Image by Author | Ideogram

Validating machine learning models is an essential step to ensure that your trained model not only performs well on the training data but also generalizes properly to unseen data. While there are well-known methods for this in practice, such as train-test splitting and cross-validation, there are additional nuances and specific situations where these methods might not always be the best strategy to follow. For instance, consider an imbalanced classification problem, such as detecting fraud in bank transactions, where a very small percentage of transactions are classified as fraudulent. In situations like this, strategies like stratified k-fold cross-validation would be a more reliable approach to measure the performance of a classification model trained to perform this fraud detection task.

In this article, we uncover several additional strategies for validating a machine learning model, concretely supervised learning models for tasks like classification, regression, and time series forecasting. The focus of the article is not to explain in detail how each strategy works, but rather to help you decide when to choose one or another, and why. To this end, a decision tree-based guide has been designed to navigate towards the most suitable approach depending on the scenario, goal, or nature of the data being used to build the model.

Decision Tree to Choose the Right Model Validation Strategy

Based on the following visual depiction of a decision tree, below we describe some guidelines to choose the best strategies to adopt when evaluating a machine learning model. The choice of a validation method is, in many cases, not necessarily limited to using only one: it is often possible (and recommended) to use several compatible validation techniques as long as it is feasible to do so.

Choosing the right model validation strategy (click to enlarge)
Each strategy is labeled depending on its usual computational cost: (\$) low, (\$\$) moderate, (\$\$\$) high

Each yellow-labeled method listed in the decision tree has an associated computational cost indicator — see image caption — so that it can be considered as an additional decision factor when multiple options are possible.

Because the majority of methods considered in this guide assume that data instances are i.i.d. (independent and identically distributed), time-ordered datasets like time series are only compatible with validation mechanisms that do not abide by this condition, such as time series split (or more advanced variants like Rolling Window CV): these specialized method do not randomly suffle the data, unlike the rest of methods do, which would completely alter the time ordering inherent to these datasets.

That said, are all the not time-ordered data i.i.d.? Not necessarily. Think of data that are clustered a priori, like multiple medical measurements from the same patient, hierarchical instances like students nested within schools, or spatial data instances like geolocated weather observations. In all these cases, the i.i.d. assumption cannot be made, but fortunately, there are specific methods to handle validation procedures on models trained on these datasets. One important aspect: while in most i.i.d. situations it is fine to combine specialized validation methods with standard ones like train-test splitting and k-fold CV, for non-i.i.d. datasets this is highly dangerous, as it may lead to data leakage and overly optimistic performance estimates.

Let’s move on to specific situations, namely goals or scenarios, in which the dataset instances are assumed to be i.i.d:

• Train/test split is the simplest approach, consisting of training a machine learning model on a training subset of the entire dataset, and testing it on the remaining part. This is a commonly used and acceptable approach to get a quick and simple baseline test, but not as reliable as other methods in certain cases. Where there are no recommendations against its use, it can be combined with other more sophisticated strategies.
• K-fold cross-validation (CV) is a very popular and balanced way to perform a general-purpose validation to measure not only the model’s accuracy, but also its ability to generalize to future unseen instances. It is usually a recommended option to try in most i.i.d. scenarios.
• For imbalanced classification scenarios with at least one class being rarer than the others (remember the fraud detection scenario described earlier), stratified K-fold CV helps ensure proper class representation in the data partitions or folds made as part of the validation process.
• In certain use cases like clinical trials or risk-sensitive applications, we may be interested in reducing the variance in performance estimates, because high variance can lead to unreliable conclusions. Repeated K-fold CV is designed to this end, but its repeated training on multiple fold combinations means it normally incurs a higher computational cost when undertaken.
• If dealing with a very small dataset, one thing to be wary of is bias in model validation. If reducing bias is important, LOOCV (standing for Leave-One-Out Cross-Validation) is the right approach to apply, whereas bootstrapping is a valid strategy otherwise.
• On a last scenario or goal, if we were interested in estimating uncertainty in the form of confidence intervals, rather than precise performance estimates, we can again resort to bootstrapping when sampling with replacement is allowed, or Monte Carlo CV otherwise, i.e., when we need to avoid sampling instances repeatedly during the validation process.

If still in doubt, a final general advice for i.i.d. scenarios is to default to:

• Stratified K-fold CV for classification models.
• K-fold CV for regression models.

Wrapping Up

This article uncovered a decision tree-driven approach to help you select the most appropriate strategy to validate a machine learning model, depending on factors like the specific scenario being addressed, the format or structure of the data (e.g. is it time-ordered?), the intended goal of the validation and use of the model, and the computational cost of each validation strategy.