Time series analysis is a powerful technique for analyzing trends, seasonality, and patterns over time. This is why today I want to explore how to handle and analyze time series data using Pandas, a versatile Python library. We’ll cover key operations like resampling, plotting, and identifying annual cycles.

Time Series and Pandas

A time series is an ordered sequence of values of a variable at equally spaced intervals. Some examples are annual temperature averages over decades or population changes of species over centuries. Despite their diversity, these datasets can be analyzed using the same tools and techniques.

Pandas is a popular Python library for working with structured data, including time series. It provides a robust and optimized suite of tools to handle time-based data efficiently, enabling powerful analyses with minimal code.

Key benefits include:

• DataFrame and Series Integration: A pandas.DataFrame can hold multiple time series, each of which can be accessed as pandas.Series. Each Series object includes methods specifically tailored for time series analysis.
• Time Series Capabilities: Pandas offer built-in functions for resampling, rolling window calculations, shifting data, and working with missing values. It also integrates seamlessly with Python’s datetime module for date-based indexing and filtering.
• Ease of Visualization:Pandas integrates with visualization libraries like Matplotlib to create time series plots easily.

If you’re not familiar with Pandas, it may be helpful to review an introductory tutorial or have a basic understanding of DataFrames. You can find a good introduction to Pandas in this following article.

Getting the Data

To start playing around, we first need to get some data. In this tutorial, I will use the following daily climate time series dataset from Kaggle, which contains daily weather observations for Delhi.

It includes the following variables:

• date: The specific day of the observation, formatted as YYYY-MM-DD.
• meantemp: The average temperature of the day, calculated from multiple 3-hour interval readings, measured in degrees Celsius.
• humidity: The amount of water vapor in the air, expressed as a percentage.
• wind_speed: The speed of the wind, measured in kilometers per hour (km/h).
• meanpressure: The average atmospheric pressure for the day, measured in hectopascals (hPa).

There are 6 main steps involved in getting the data directly from Kaggle:

1. Install the Kaggle API: Use pip install kaggle to install the Kaggle API, a tool for programmatically downloading datasets from Kaggle.
2. Locate your kaggle.json file: Download your Kaggle API token (kaggle.json) from your Kaggle account settings.
3. Setup your kaggle.json file: Move the kaggle.json file to the ~/.kaggle/ directory and set proper file permissions to ensure security.
4. Download the dataset: Use the Kaggle CLI to download the desired dataset with the following command:

kaggle datasets download -d sumanthvrao/daily-climate-time-series-data

Replace with the dataset’s ID from Kaggle.

5. Extract the dataset: Unzip the downloaded file to access the dataset:

unzip *.zip

6. Load the dataset: Load the extracted data into a Pandas DataFrame for analysis:

import pandas as pd 
df = pd.read_csv('.csv')

Pandas for Time Series Analytics

Step 1: Creating a datetime Index

Now that the data is loaded, the next step is to refine it for analysis. A key improvement is changing the current integer-based index to a more functional pandas.DatetimeIndex, which is built on Python’s datetime objects. This transformation allows us to leverage time-based operations and organize the data by its primary feature: time.

To achieve this, we use the pandas.to_datetime() function to ensure the date column is correctly formatted as datetime objects. Then, we assign it directly to the .index property of the DataFrame:

df.index = pd.to_datetime(df["date"])

Step 2: Plotting DataFrame Content

We can easily use the plot method of Pandas to generate a chart for all variables within the dataframe.

df.plot(subplots=True, figsize=(15,6))

Now we can perform multiple analyses to understand the seasonality of our data.

Resampling & Rolling Calculations

To reduce the noise in the data, we can smooth it. There are various ways to do this and so there is a choice to be made about the method to use and the degree of smoothing required.

Resampling involves changing the frequency of your time-series data. This can be done either to a higher frequency (upsampling) or a lower frequency (downsampling).

What we are going to do now is to extract a DataFrame with the columns we are interested in with meantemp, and produce a week-based resampler object, which is then reduced to the new weekly time series by taking medians over each year interval.

df[["meantemp"]].resample("1w").median().plot(figsize=(15,4))

Rolling involves applying a calculation over a moving window of a fixed size. It’s often used for smoothing or calculating running statistics. So now we can generate a 31-day moving window and roll our data to smooth the noise and see the time pattern.

df[["meantemp"]].rolling(31).median().plot(figsize=(15,4))

Identifying periodicity and correlation

We can see that there is an approximately 1-year (12 months) cycle in our data. However… how to make sure of this? A handy high-level tool to identify this periodicity is the autocorrelation plot, which visualizes the correlation of a time series with itself at a range of lag times.

An autocorrelation plot shows how similar a time series is to its past values. It compares the current data to previous ones at different time steps (lags). If the bars on the plot are tall and outside the dashed lines, it means there’s a pattern or trend.

Interpreting the values range between:

• 1: Perfect positive correlation (high similarity)
• 0: No correlation (random relationship)
• -1: Perfect negative correlation (opposite relationship)

Peaks repeating at regular intervals suggest seasonality. If the bars quickly get small, the data is random or stationary. We can perform an autocorrelation plot with our data:

pd.plotting.autocorrelation_plot(df["meantemp"].resample("1m").median())

Are obtain the following chart.

Autocorrelation plot | Image by Author

Looking at the plot, we observe two distinct peaks:

• A negative peak around lag 6, which indicates an inverse relationship at this interval
• A positive peak around lag 12, which points to a repeating pattern

This pattern strongly suggests a 12-month seasonality, meaning the data cycles annually.
 

Splitting and Stacking Cycles

Since the data exhibits a clear annual cycle, we can stack the data by aligning the same days across different years. This allows us to visualize and compare year-to-year patterns more effectively.

By plotting the behavior of a selected variable against the day of the year for each year, we can uncover variations in seasonal trends, anomalies, or long-term changes.

import pandas as pd
import matplotlib.pyplot as plt

# Assuming your DataFrame is named df
# If the 'date' column is not already a datetime object, convert it
df['date'] = pd.to_datetime(df['date'])

# Extract the day of the year and year for grouping
df['day_of_year'] = df['date'].dt.dayofyear
df['year'] = df['date'].dt.year

# Plot for a selected variable, e.g., 'meantemp'
plt.figure(figsize=(12, 8))

for year in df['year'].unique():
   year_data = df[df['year'] == year]
   year_data = year_data.resample("1m").median()
   plt.plot(year_data['day_of_year'], year_data['meantemp'], label=f'{year}')

plt.title("Mean Temperature Behavior by Day of Year")
plt.xlabel("Day of Year")
plt.ylabel("Mean Temperature")
plt.legend()
plt.show()

Thus we obtain a final plot as follows: 

Mean Temperature Behavior by Day of Year | Image by Author

Final Thoughts

This tutorial has demonstrated the power and simplicity of using Pandas for time series analysis. It has highlighted how Pandas streamlines tasks such as data preprocessing, analysis, and visualization, allowing users to effectively handle time-based data.

By leveraging tools like DatetimeIndex, resampling, and rolling calculations, we explored how to uncover trends and reduce noise in the data. Additionally, methods like autocorrelation analysis and year-over-year comparisons helped identify patterns and confirm seasonality, such as the observed 12-month cycle in the climate data.

You can check out the comprehensive code in this GitHub Repo.