You can use the regsubsets() function from the leaps package in R to find the subset of predictor variables that produces the best regression model.
The following example shows how to use this function in practice.
Example: Using regsubsets() for Model Selection in R
For this example we’ll use the built-in mtcars dataset in R, which contains measurements on 11 different attributes for 32 different cars.
#view first six rows of mtcars dataset
head(mtcars)
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
Suppose we would like to fit a regression model using hp as the response variable and the following potential predictor variables:
- mpg
- wt
- drat
- qsec
We can use the regsubsets() function from the leaps package to perform an exhaustive search to find the best regression model:
library(leaps)
#find best regression model
bestSubsets <- regsubsets(hp ~ mpg + wt + drat + qsec, data=mtcars)
#view results
summary(bestSubsets)
Subset selection object
Call: regsubsets.formula(hp ~ mpg + wt + drat + qsec, data = mtcars)
4 Variables (and intercept)
Forced in Forced out
mpg FALSE FALSE
wt FALSE FALSE
drat FALSE FALSE
qsec FALSE FALSE
1 subsets of each size up to 4
Selection Algorithm: exhaustive
mpg wt drat qsec
1 ( 1 ) "*" " " " " " "
2 ( 1 ) " " "*" " " "*"
3 ( 1 ) "*" "*" " " "*"
4 ( 1 ) "*" "*" "*" "*"
The stars ( * ) at the bottom of the output indicate which predictor variables belong in the best regression model for each possible model with a different number of predictor variables.
Here is how to interpret the output:
For a model with only one predictor variable, the best regression model is produced by using mpg as the predictor variable.
For a model with two predictor variables, the best regression model is produced by using wt and qsec as the predictor variables.
For a model with three predictor variables, the best regression model is produced by using mpg, wt and qsec as the predictor variables.
For a model with four predictor variables, the best regression model is produced by using mpg, wt, drat and qsec as the predictor variables.
Note that you can also extract the following metrics for each model:
- rsq :The r-squared value for each model
- RSS: The residual sum of squares for each model
- adjr2: The adjusted r-squared value for each model
- cp: Mallows’ cp for each model
- bic: The BIC value for each model
For example, we can use the following syntax to extract the adjusted R-squared value for each of the four best models:
#view adjusted R-squared value of each model
summary(bestSubsets)$adjr2
[1] 0.5891853 0.7828169 0.7858829 0.7787005
From the output we can see:
- The adjusted R-squared value for the model with mpg as the predictor variable is 0.589.
- The adjusted R-squared value for the model with wt and qsec as the predictor variables is 0.783.
- The adjusted R-squared value for the model with mpg, wt and qsec as the predictor variables is 0.786.
- The adjusted R-squared value for the model with mpg, wt, drat and qsec as the predictor variables is 0.779.
These values give us an idea of how well the set of predictor variables are able to predict the value of the response variable, adjusted for the number of predictor variables in the model.
Additional Resources
The following tutorials explain how to perform other common tasks in R:
How to Perform Multiple Linear Regression in R
How to Perform Piecewise Regression in R
How to Perform Spline Regression in R