Mallows’ Cp is a metric that is used to pick the best regression model among several different models.
It is calculated as:
Cp = RSSp/S2 – N + 2(P+1)
where:
- RSSp: The residual sum of squares for a model with p predictor variables
- S2: The residual mean square for the model (estimated by MSE)
- N: The sample size
- P: The number of predictor variables
Mallows’ Cp is used when we have several potential predictor variables that we’d like to use in a regression model and we’d like to identify the best model that uses a subset of these predictor variables.
We can identify the “best” regression model by identifying the model with the lowest Cp value that is less than P+1, where P is the number of predictor variables in the model.
The following example shows how to use Mallows’ Cp to pick the best regression model among several potential models.
Example: Using Mallows’ Cp to Pick the Best Model
Suppose a professor would like to use hours studied, prep exams taken, and current GPA as predictor variables in a regression model to predict the score that a student will receive on the final exam.
He fits seven different regression models and calculates the value for Mallows’ Cp for each model:
If the value of Mallows’ Cp is less than the number of coefficients in the model (P+1) then the model is said to be unbiased.
We can see that there are two models that are unbiased:
- The model with Hours and GPA as the predictor variables (Mallows’ Cp = 2.9, P+1 = 3)
- The model with Prep Exams and GPA as the predictor variables (Mallows’ Cp = 2.7, P+1 = 3)
Among these two models, the model that uses Prep Exams and GPA as the predictor variables has the lowest value for Mallows’ Cp, which tells us that it’s the best model that leads to the least amount of bias.
Notes on Mallows’ Cp
Here are few things to keep in mind with regards to Mallows’ Cp:
- Models that have a Mallows’ Cp value near P+1 are said to have low bias.
- If every potential model has a high value for Mallows’ Cp, this is an indication that some important predictor variables are likely missing from each model.
- If several potential models have low values for Mallow’s Cp, choose the model with the lowest value as the best model to use.
Also keep in mind that Mallows’ Cp is only one way to measure the quality of fit of a regression model.
Another commonly used metric is adjusted R-squared, which tells us the proportion of variance in the response variable that can be explained by the predictor variables in the model, adjusted for the number of predictor variables used.
When deciding which regression model is best among a list of several different models, it’s a good idea to look at both Mallows’ Cp and adjusted R-squared.
Additional Resources
How to Calculate Mallows’ Cp in R
How to Calculate Adjusted R-Squared in R
Hello Zach:
Could you help me with the following question?
I want to know how you can get from the formula Cp = RSSp/S2 – N + 2(P+1) to the one calculated as Cp < (p + 1)
Hi! Sure, I can help clarify this step. The formula \( C_p = \frac{\text{RSS}_p}{\hat{S}^2} – N + 2(P+1) \) leads to the interpretation that \( C_p \) should be approximately equal to \( p+1 \), and I’ll break down how this interpretation arises step by step.
—
### 1. **Understanding the Components in the Formula**
The given formula is:
\[
C_p = \frac{\text{RSS}_p}{\hat{S}^2} – N + 2(P+1)
\]
Where:
– \( \text{RSS}_p \): Residual Sum of Squares for the model with \( p \) predictors.
– \( \hat{S}^2 \): Estimate of the variance of the residuals (error variance), usually computed from the full model (or a saturated model).
– \( N \): Number of observations.
– \( P \): Number of predictors (not including the intercept).
—
### 2. **Ideal Case: What Happens with a Good Model?**
A “good” model achieves the following:
– **RSS approximates the true error variance:** For an ideal model, \( \text{RSS}_p \approx \hat{S}^2 (N – P – 1) \). This comes from the fact that RSS measures the remaining variance, which scales with \( N – P – 1 \) (degrees of freedom for residuals).
So, if \( \text{RSS}_p \approx \hat{S}^2 (N – P – 1) \), substituting this back into the formula:
\[
C_p = \frac{\hat{S}^2 (N – P – 1)}{\hat{S}^2} – N + 2(P+1)
\]
—
### 3. **Simplify the Formula**
Simplify \( \frac{\hat{S}^2 (N – P – 1)}{\hat{S}^2} \):
\[
C_p = (N – P – 1) – N + 2(P+1)
\]
Combine terms:
\[
C_p = -P – 1 + 2(P + 1)
\]
Expand \( 2(P + 1) \):
\[
C_p = -P – 1 + 2P + 2
\]
Simplify:
\[
C_p = P + 1
\]
—
### 4. **Why \( C_p \approx P+1 \)?**
This result shows that, under ideal circumstances (i.e., the model fits the data well without underfitting or overfitting), \( C_p \) will approximately equal \( P+1 \), where \( P+1 \) includes the number of predictors plus the intercept.
### 5. **When is \( C_p < P+1 \)?** If \( C_p \) is less than \( P+1 \), it suggests the model is underfitting, meaning it has excluded important predictors. If \( C_p \) is much greater than \( P+1 \), it suggests overfitting, meaning the model has included too many predictors that do not significantly improve the fit. --- ### 6. **Summary of the Connection** The formula simplifies to \( C_p = P+1 \) under ideal conditions because: - \( \text{RSS}_p \approx \hat{S}^2 (N - P - 1) \) holds for a well-fitted model. - This equality reflects a balance between model complexity and goodness-of-fit. The heuristic that "a good model has \( C_p \approx P+1 \)" comes directly from this derivation. This helps us interpret \( C_p \) when comparing multiple models: the closer \( C_p \) is to \( P+1 \), the better the balance between bias and variance.