Machine learning models are powerful tools for businesses and researchers, as their outputs provide tremendous value. However, a model only becomes valuable if its predictions closely match reality. That is why we only use models that have already undergone rigorous evaluation.
In supervised machine learning, a model is trained using labeled data as guidance. The model then learns patterns from the data and develops a generalized version that can be used to make predictions on new, incoming data. But how can we be sure that the model will accurately predict unseen data? This is where the concept of train-test splits comes into play.
Train-test splits are simply defined as dividing our dataset into training data and testing data, as shown in the image below.
However, why is it essential to perform train-test splits to evaluate our models? That is what we will explore now.
Train-Test Splits
Train-test splits are a technique used to divide a dataset for evaluation purposes.
In practice, we often split the available dataset into a training set, which is used to train our model, and a test set to evaluate the model’s performance on unseen data. This technique evaluates our model’s ability to generalize and ensures that it is not simply memorizing patterns from the training set. It is crucial to employ this technique properly to prevent overfitting.
By using the test set, we can obtain a realistic measure of how our model will perform when deployed, and identify any problems that might arise in production.
Commonly, we split our dataset into different ratios, such as 80% training data and 20% test data, or 70% training data and 30% test data, with the larger portion allocated for training. The training set requires more data to ensure that the model has sufficient information to learn, while still leaving enough test data for an unbiased evaluation.
There are two common methods for splitting: random or stratified. As the name implies, a random split divides the data without considering the distribution of the target labels, which can lead to imbalanced subsets if the overall dataset is imbalanced. In contrast, a stratified split ensures that the distribution of the target labels is maintained in both the training and test sets.
For time-series data, a unique approach is required because the properties of time-series data differ from those of other datasets. With sequential splits, the training set contains earlier data points, while the test set includes later ones.
A train-test split can be performed using the common code below, leveraging Scikit-learn’s model_selection module and train_test_split() function.
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
There are some pitfalls to avoid when performing train-test splits, such as data leakage — where the training set inadvertently contains information about the test set — or a lack of reproducibility if a fixed random seed is not used during splitting. These pitfalls must be guarded against, but a well-designed model-training process will include such considerations for dataset preparation.
That’s what you need to know about train-test splits to get started.
