Image by Author | Piktochart

Cross-validation is a very established methodology for training and evaluating machine learning and predictive analytics solutions more realistically and reliably, analyzing important properties like the model’s ability to generalize to future or unseen data beyond just making accurate predictions on the data it is exposed to during training.

A key aspect in cross validation processes entails partitioning the data into multiple training and validation splits, normally based on sampling and shuffling data. This leads to a question when attempting to validate predictive models for time series forecasting. Since time series data have a inherent temporal order, and that order is key to any analysis, how to apply cross-validationwithout altering the chronological order in the data?

This article explains how and illustrates the process in Python.

Step-by-Step Process

The key to performing cross-validation in time series is ensuring that the training sets are always chronologically before the corresponding validation sets.

In Python, scikit-learn’s TimeSeriesSplit class is designed to implement the aforesaid cross validation mechanism in time series, which is visually illustrated as follows.

Not quite familiar or needing a refreshing on concepts like k-fold cross validation for a better understanding? Check this article.

Cross validation in time series | Image by Author

We start the process in Python by importing the necessary modules and classes.

import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

We then load and preprocess the Chicago bus rides time series used in previous articles of the series.

url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/refs/heads/main/CTA_-_Ridership_-_Daily_Boarding_Totals.csv"
df = pd.read_csv(url)

df['service_date'] = pd.to_datetime(df['service_date'])
df.set_index('service_date', inplace=True)
df.sort_index(inplace=True)

data = df['bus'].values

Now that we preprocessed the data and fetched the bus rides univariate time series, we initialize the class for applying time series splits on the data, considering three different splits, one for each subsequent cross validation iteration (once a forecasting model has been trained; this later step is left outside the scope of the article).

n_splits = 3
tscv = TimeSeriesSplit(n_splits=n_splits)

The split() method is responsible for producing the multiple (in our case, three) split configurations. This example shows how to iterate over them to print their indexes and values (number of daily bus rides):

for fold, (train_index, test_index) in enumerate(tscv.split(data)):
    train_data = data[train_index]
    test_data = data[test_index]
    print(f"Fold {fold + 1}")
    print(f"Train indices: {train_index}")
    print(f"Test indices: {test_index}")
    print(f"Train data: {train_data}")
    print(f"Test data: {test_data}")
    print()

Take a look at the indices obtained for each fold (iteration), and how their configuration matches the one on the right-hand side of the previous image:

Fold 1
Train indices: [   0    1    2 ... 2006 2007 2008]
Test indices: [2009 2010 2011 ... 4015 4016 4017]
Train data: [297192 780827 824923 ... 950077 631294 440041]
Test data: [ 802169  450356  901332 ...  991030 1043354 1089263]

Fold 2
Train indices: [   0    1    2 ... 4015 4016 4017]
Test indices: [4018 4019 4020 ... 6024 6025 6026]
Train data: [ 297192  780827  824923 ...  991030 1043354 1089263]
Test data: [1095431  662075  454576 ...  405068  307608  847371]

Fold 3
Train indices: [   0    1    2 ... 6024 6025 6026]
Test indices: [6027 6028 6029 ... 8033 8034 8035]
Train data: [297192 780827 824923 ... 405068 307608 847371]
Test data: [846717 890581 836436 ... 311437 217204 454862]

Another method that also ensures chronological order is sliding window cross validation. By specifying a training window size (for instance 60 days), and a test window size (e.g. the 10 days that follow the previous 60 for training), this method runs a “double sliding window” throughout the time series and builds plenty of sliding folds of size 60 + 10.

train_window = 60
test_window = 10

for start in range(len(data) - train_window - test_window + 1):
    train_data = data[start : start + train_window]
    test_data = data[start + train_window : start + train_window + test_window]
    print(f"Train data (from index {start} to {start + train_window - 1}): {train_data}")
    print(f"Test data (from index {start + train_window} to {start + train_window + test_window - 1}): {test_data}")
    print()

Wrapping Up

We have outlined two possible approaches that help preparing your time series data for cross-validation processes in Python. Whilst the choice of the best method often depends on your specific requirements, using the TimeSeriesSplit is a more automated and simplified approach, while manual sliding windows might be best when you need more fine-grained control for certain data that may drift over time.