Image by Editor | Midjourney

Predictive models are inherently statistical, with many being based on machine learning techniques. One question that often comes to mind after training a predictive model is: how can we validate its effectiveness in making accurate and generalizable predictions, and which validation method(s) should we use?

This article provides a succinct and illustrative answer to both questions by outlining three common validation methods for predictive models.

Train-Test-Validation Splits

This validation approach splits the data not into two parts (train and test sets), but into three: train, test, and validation sets. One for training the model, one for tuning and/or validating its effectiveness, and one for final evaluation after validation, as a previous step to deploying for real use.

In Python, you can use the scikit-learn library to do this split of your data. All it takes is using the train_test_split method twice: one for splitting the whole dataset into training and validation splits, and another one for further splitting the “raw” validation split into the final validation and test sets. The test_size argument is used to set the percentage of the data to be split that goes to the latter of the two resulting splits: do not be misled by its name, it is purely a convention of the scikit-learn library, not necessarily exclusive for test sets.

X_train, X_val_test, y_train, y_val_test = train_test_split(X, y, test_size=0.4, random_state=0)
X_val, X_test, y_val, y_test = train_test_split(X_val_test, y_val_test, test_size=0.5, random_state=0)

Once your data is split, you can train a predictive model like this:

model.fit(X_train, y_train)
val_score = model.score(X_val, y_val)

K-fold Cross-Validation

A more methodological approach, particularly useful to ensure that your predictive model generalizes well to diverse data it is exposed to. This method splits the data into k equal parts (folds), training a version of the model on k-1 of them and validating it on the remaining one. This process is repeated k times for all possible training-validation combinations, and performance results like predictive accuracy are averaged.

Python greatly simplifies this process:

scores = cross_val_score(model, X, y, cv=5)
mean_score = scores.mean()

This complete guide explains in detail how k-fold cross-validation works.

Bootstrapping

The previous two methods work well with larger datasets, but what if your predictive model has been built on a smaller dataset? Bootstrapping is particularly useful in these situations, enabling simulations with multiple samples from the original data distribution. Bootstrapping consists of randomly resampling the dataset with a replacement strategy several times. How many? As many as needed to ensure the stability of your model’s performance.

Again, Python libraries make this process straightforward. This example applies bootstrapping throughout 100 iterations, and finally averages the results of all iterations:

boot_scores = [model.fit(*resample(X, y)).score(X, y) for _ in range(100)]
avg_boot_score = np.mean(boot_scores)

Wrapping Up

This article illustrates through Python examples three methods to validate the effectiveness of predictive models like regressors, classifiers, or time series forecasting models. Depending on the nature of your problem, model, and data, you may want to opt for one of these methods over the others.