Image by Author | Piktochart

Anomalies in time series data are unusual patterns or deviations from expected behavior, such as sudden spikes or drops. Detecting them is important to identify faults, predict system failures, detect fraud, or understand complex trends or shifts in data.

In this article, let’s uncover how to identify anomalies in time series data in Python, using a popular resource for this end: the z-scores method.

Detecting Anomalies in Time Series Using Z-Scores

To learn how to apply this method, we first load the Chicago rides dataset used in this previous article about calculating rolling averages for time series.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/refs/heads/main/CTA_-_Ridership_-_Daily_Boarding_Totals.csv"
df = pd.read_csv(url)

df['service_date'] = pd.to_datetime(df['service_date'])
df.set_index('service_date', inplace=True)
df.sort_index(inplace=True)

filtered_data = df.loc['2019-01-01':'2020-03-31', 'bus']

plt.figure(figsize=(14, 7))
plt.plot(filtered_data, label='Bus Ridership', alpha=0.7)
plt.title('Bus Ridership (Jan 2019 - Mar 2020)')
plt.xlabel('Date')
plt.ylabel('Bus Ridership')
plt.legend()
plt.grid()
plt.show()

After loading the dataset, we set the date-time variable as the index and sorted it to avoid later inconsistencies. We also filter the data to obtain daily bus recordings between January 2019 and March 2020, and visualize the filtered dataset:

The global z-scores method for identifying anomalies uses the global mean and global standard deviation of our time series data to perform the necessary calculations. Let’s get them:

filtered_data = df.loc['2019-01-01':'2020-03-31', 'bus'].to_frame()
global_mean = filtered_data['bus'].mean()
global_std = filtered_data['bus'].std()

Calculating the z-scores of daily observations is as simple as standardizing them, assuming they follow a normal distribution. Once standardized into z-scores, and by fixing a threshold like z = |3| anomalies will be filtered as those standardized observations outside this threshold.

filtered_data['z_score_global'] = (filtered_data['bus'] - global_mean) / global_std

z_threshold = 3
global_anomalies = filtered_data[np.abs(filtered_data['z_score_global']) > z_threshold]

print("\nAnomalies detected using Global Z-Score Method:")
print(global_anomalies)

The result: no anomalies found! This is not that surprising after all, since we are considering global average and standard deviation, i.e. a “global z-scores method“, and there is a periodical fluctuation between weekdays and weekends, even on March 2020 (when the Covid-19 pandemic broke out) data points are still within the set boundaries of “normality”.

Adopting a stubborn tone, what if calculating global z-scores wasn’t the best approach? One way to add more sensitivity to the detection method is by using rolling statistics like rolling averages and standard deviations over a 30-day window, rather than the global counterparts. We could also reduce the threshold to z = |2| to be more restrictive about what should be considered normal.

The following example shows how to perform this rolling z-score detection approach:

rolling_mean = filtered_data['bus'].rolling(window=30).mean()
rolling_std = filtered_data['bus'].rolling(window=30).std()

filtered_data['z_score_rolling'] = (filtered_data['bus'] - rolling_mean) / rolling_std

# We now set |z| > 2 for higher sensitivity
rolling_anomalies = filtered_data[np.abs(filtered_data['z_score_rolling']) > 2]

plt.figure(figsize=(14, 7))
plt.plot(filtered_data.index, filtered_data['bus'], label='Bus Ridership', alpha=0.6)
plt.scatter(rolling_anomalies.index, rolling_anomalies['bus'], color='red', label='Rolling Z-Score Anomalies')
plt.plot(rolling_mean, color='green', label='30-Day Rolling Mean')
plt.fill_between(filtered_data.index,
                 rolling_mean - 2 * rolling_std,
                 rolling_mean + 2 * rolling_std,
                 color='orange', alpha=0.2, label='Threshold')
plt.title('Rolling Z-Score Anomaly Detection')
plt.xlabel('Date')
plt.ylabel('Bus Ridership')
plt.legend()
plt.grid()
plt.show()

Output:

This time, three anomalous points are detected, all of them for unusually low numbers of bus rides. Interestingly, none of them fall under the unconventional month of March 2020. Why is that? Remember, we are relying on rolling statistics for our analysis, and despite the drastic change the data showed in this period, rolling averages evolved more smoothly as the sliding window gradually entered into this month, and the normality threshold also gradually changed accordingly. That explains why the z-scores for those points at the end of the time series aren’t deemed as anomalies.

Wrapping Up

We have covered in this article how to use the z-scores method to identify anomalies in time series data under different settings, from using global statistics to rolling statistics and different tolerance thresholds.