Image by Editor
In machine learning, the coefficient of determination, also known as R-squared and typically denoted by R2, is a useful metric for quantifying how much of a target variable’s variance is explained by a model, and for comparing multiple models trained on the same dataset for the same predictive goal. However, it can be misleading if interpreted incorrectly or used in isolation — without proper evaluation on unseen data or additional validation metrics to get a holistic view of model performance. This is particularly true in the event of overfitting.
Overfitting in machine learning is a common problem in which trained models fit the training data too tightly and fail to generalize well to new data once deployed. It’s often diagnosed when there is a large gap between a high training performance score — based on a metric like R-squared — and a much lower validation or testing score.
To better understand when an R-squared value might be too high, or simply misleading, consider two example scenarios:
Example 1: High Training R², Poor Generalization
This is a common overfitting situation. Suppose we trained a regression model to predict house prices using a dataset with a few thousand instances. If we first build a linear regression model with a reasonable number of features and the resulting R² is 0.72 for the training data and 0.70 for a held-out validation set, that suggests reasonable predictive power and stable generalization (notice the small difference between both scores).
Now say we increase the model’s complexity, turning it into a polynomial regressor by adding polynomial features and interaction terms — the latter referring to new features created by multiplying two or more of the original features. The training R² rises to an astounding 0.99, meaning the model nearly perfectly fits the training data. The problem appears when we evaluate the model on the validation set: the validation R² drops to 0.55.
This is a classic example of a misleading training R², likely due to learning noise along with signal. Increasing R² by making the model more complex did not yield a better model; it produced one that overfits. (In such cases, tools like regularization, learning curves, and comparing adjusted R² — which penalizes unnecessary predictors — can be more informative.)
Example 2: R² “Inflated” By A Flexible Model
Consider a dataset with high dimensionality — say, a few hundred instances with many predictor features. Gene expression data is a common example. Flexible models like deep neural networks or high-degree polynomial regressors may also yield an R² close to 1 on the training set.
Here’s the catch: R² has a monotonic non-decreasing property as more input features are added. In models with many trainable parameters (weights), fitting the training data nearly perfectly is possible even when the model learns little true signal. This almost-perfect R² can be a mirage, hiding the fact that a seemingly trustworthy model may perform much worse once deployed on new data. (Again, adjusted R² and proper out-of-sample evaluation help guard against this.)
Summary
In sum, we may suspect an unduly high R² when:
- It is high only on training data
- It increases with model complexity or the number of selected predictors
- Its value is not corroborated by other metrics, validation sets, or cross-validation
