In regression analysis, Mallows’ Cp is a metric that is used to pick the best regression model among several potential models.
We can identify the “best” regression model by identifying the model with the lowest Cp value that is close to p+1, where p is the number of predictor variables in the model.
The easiest way to calculate Mallows’ Cp in R is to use the ols_mallows_cp() function from the olsrr package.
The following example shows how to use this function to calculate Mallows’ Cp to pick the best regression model among several potential models in R.
Example: Calculating Mallows’ Cp in R
Suppose we would like to fit three different multiple linear regression models using variables from the mtcars dataset.
The following code shows how to fit the following regression models:
- Predictor variables in Full Model: All 10 variables
- Predictor variables in Model 1: disp, hp, wt, qsec
- Predictor variables in Model 2: disp, qsec
- Predictor variables in Model 3: disp, wt
The following code shows how to fit each of these regression models and use the ols_mallows_cp() function to calculate the Mallows’ Cp of each model:
library(olsrr) #fit full model full_model <- lm(mpg ~ ., data = mtcars) #fit three smaller models model1 <- lm(mpg ~ disp + hp + wt + qsec, data = mtcars) model2 <- lm(mpg ~ disp + qsec, data = mtcars) model3 <- lm(mpg ~ disp + wt, data = mtcars) #calculate Mallows' Cp for each model ols_mallows_cp(model1, full_model) [1] 4.430434 ols_mallows_cp(model2, full_model) [1] 18.64082 ols_mallows_cp(model3, full_model) [1] 9.122225
Here’s how to interpret the output:
- Model 1: p + 1 = 5, Mallows’ Cp = 4.43
- Model 2: p + 1 = 3, Mallows’ Cp = 18.64
- Model 3: p + 1 = 30, Mallows’ Cp = 9.12
We can see that model 1 has a value for Mallows’ Cp that is closest to p + 1, which indicates that it’s the best model that leads to the least amount of bias among the three potential models.
Notes on Mallows’ Cp
Here are few things to keep in mind with regards to Mallows’ Cp:
- If every potential model has a high value for Mallows’ Cp, this is an indication that some important predictor variables are likely missing from each model.
- If several potential models have low values for Mallow’s Cp, choose the model with the lowest value as the best model to use.
Keep in mind that Mallows’ Cp is only one way to identify the “best” regression model among several potential models.
Another commonly used metric is adjusted R-squared, which tells us the proportion of variance in the response variable that can be explained by the predictor variables in the model, adjusted for the number of predictor variables used.
When deciding which regression model is best among a list of several different models, it’s recommended to look at both Mallows’ Cp and adjusted R-squared.
Additional Resources
How to Calculate Adjusted R-Squared in R
How to Calculate AIC in R
Generally speaking, what is the ideal value of Cp. Is it 1.33 or what?
Please elucidate me with this question.
Thanks
Dr. Hector A. Quevedo
Mallows’ \( C_p \) is a statistical metric used in regression analysis to assess the relative quality of different regression models. It helps to compare models by balancing the trade-off between goodness-of-fit and model complexity. Here’s a detailed breakdown:
### Formula for Mallows’ \( C_p \)
\[ C_p = \frac{\text{Residual Sum of Squares}}{\hat{\sigma}^2} – (n – 2p) \]
Where:
– \( \text{Residual Sum of Squares (RSS)} \): Measures the total deviation of the predicted values from the observed data.
– \( \hat{\sigma}^2 \): Estimate of the variance of the errors (usually derived from the full model).
– \( n \): Number of observations.
– \( p \): Number of predictors in the model (including the intercept).
### Interpretation of \( C_p \):
The goal is to select a model where \( C_p \) is close to the number of predictors (\( p \)).
#### Ideal Value:
– Ideally, \( C_p \approx p \), which suggests the model is neither underfitting nor overfitting.
– A **small \( C_p \)** (close to \( p \)) indicates that the model provides a good fit without unnecessary complexity.
– A **large \( C_p \)** suggests overfitting, where too many predictors are included relative to their explanatory power.
### Why not 1.33 specifically?
The value 1.33 may come from rules of thumb in specific contexts (e.g., in engineering or economics), but it is not a universal or ideal value for \( C_p \). The ideal \( C_p \) depends on:
1. The number of predictors \( p \).
2. The quality of the fit of the regression model.
3. The application and tolerance for complexity in your field.
In general:
– If \( C_p \) is significantly greater than \( p \), the model is likely overfitted.
– If \( C_p \) is much less than \( p \), the model might be underfitting and missing important predictors.
### Calculating \( C_p \) in R
To calculate Mallows’ \( C_p \) in R, you typically follow these steps:
#### Example
Suppose you have a dataset `data` and a full model `full_model`:
1. **Fit the full model:**
“`R
full_model <- lm(y ~ ., data = data) sigma2_hat <- summary(full_model)$sigma^2 # Variance of residuals ``` 2. **Fit candidate models and compute \( C_p \):** ```R models <- list( model1 = lm(y ~ x1, data = data), model2 = lm(y ~ x1 + x2, data = data) ) cp_values <- sapply(models, function(mod) { p <- length(coef(mod)) # Number of predictors including intercept rss <- sum(residuals(mod)^2) # Residual Sum of Squares cp <- rss / sigma2_hat - (nrow(data) - 2 * p) return(cp) }) names(cp_values) <- names(models) cp_values ``` This calculates the \( C_p \) for each candidate model and helps you decide which one has the best balance of fit and complexity. ### Key Takeaway The "ideal" \( C_p \) is not a fixed number like 1.33—it is context-dependent. The main criterion is to find a model where \( C_p \) is approximately equal to \( p \), indicating a good balance between fit and simplicity.
Dear Zach:
I believe the U. S. economic system is biassed because it is being modeled as a stand-alone system. This is because economic modelers are not including the contingent participation of all other world systems as the social, health, environmental systems, etc., that interact with it. In other words, by using Cp < (p + 1), p is being put as equal to 0, thus, if Cp is taken as the consensus 1.3, hence, such implication proves, at least, theoretically, that this economic system is biassed. What do you think about this? Please elucidate me in this question.
Thanks
Dr. Hector A. Quevedo
Hi Hector…We appreciate your question! We are better able to assist you with questions regarding the post topics and cannot speak to broader questions regarding politics, ethics and economic theory and practice.
Dear Zach:
I am in the process of publishing a scientific article in which I will be applying the Mallow’s Cp to a problem related to the health system of the commonly use of different kinds of diseases. Of course, I will be applying the Cp-Mallows mathematical function quoted by you and other researchers. However, do I still got to have your permission to use your version of the C-Mallows, even though I will be quoting your name and the article: What is Mallow’s Cp? (Definition & Example)
Hi Hector…Sounds great! Do you still have questions regarding the post?