Multicollinearity in regression analysis occurs when two or more predictor variables are highly correlated to each other, such that they do not provide unique or independent information in the regression model.
If the degree of correlation is high enough between variables, it can cause problems when fitting and interpreting the regression model.
For example, suppose you run a regression analysis using the response variable max vertical jump and the following predictor variables:
- height
- shoe size
- hours spent practicing per day
In this case, height and shoe size are likely to be highly correlated with each other since taller people tend to have larger shoe sizes. This means that multicollinearity is likely to be a problem in this regression.
This tutorial explains why multicollinearity is a problem, how to detect it, and how to resolve it.
Why Multicollinearity is a Problem
One of the main goals of regression analysis is to isolate the relationship between each predictor variable and the response variable.
In particular, when we run a regression analysis, we interpret each regression coefficient as the mean change in the response variable, assuming all of the other predictor variables in the model are held constant.
This means we assume that we’re able to change the values of a given predictor variable without changing the values of the other predictor variables.
However, when two or more predictor variables are highly correlated, it becomes difficult to change one variable without changing another.
This makes it difficult for the regression model to estimate the relationship between each predictor variable and the response variable independently because the predictor variables tend to change in unison.
In general, multicollinearity causes two types of problems:
- The coefficient estimates of the model (and even the signs of the coefficients) can fluctuate significantly based on which other predictor variables are included in the model.
- The precision of the coefficient estimates are reduced, which makes the p-values unreliable. This makes it difficult to determine which predictor variables are actually statistically significant.
How to Detect Multicollinearity
The most common way to detect multicollinearity is by using the variance inflation factor (VIF), which measures the correlation and strength of correlation between the predictor variables in a regression model.
Utilizing the Variance Inflation Factor (VIF)
Most statistical software has the ability to compute VIF for a regression model. The value for VIF starts at 1 and has no upper limit. A general rule of thumb for interpreting VIFs is as follows:
- A value of 1 indicates there is no correlation between a given predictor variable and any other predictor variables in the model.
- A value between 1 and 5 indicates moderate correlation between a given predictor variable and other predictor variables in the model, but this is often not severe enough to require attention.
- A value greater than 5 indicates potentially severe correlation between a given predictor variable and other predictor variables in the model. In this case, the coefficient estimates and p-values in the regression output are likely unreliable.
For example, suppose we run a regression analysis using predictor variables height, shoe size, and hours spent practicing per day to predict max vertical jump for basketball players and receive the following output:
From the last column, we can see that the VIF values for height and shoe size are both greater than 5. This indicates that they’re likely suffering from multicollinearity and that their coefficient estimates and p-values are likely unreliable.
If we look at the coefficient estimate for shoe size, the model is telling us that for each additional one unit increase in shoe size, the average increase in max vertical jump is -0.67498 inches, assuming height and practice hours are held constant.
This doesn’t seem to make sense, considering we would expect players with larger shoe sizes to be taller and thus have a higher max vertical jump.
This is a classic example of multicollinearity causing the coefficient estimates to appear a bit whacky and unintuitive.
How to Resolve Multicollinearity
If you detect multicollinearity, the next step is to decide if you need to resolve it in some way. Depending on the goal of your regression analysis, you might not actually need to resolve the multicollinearity.
Namely:
1. If there is only moderate multicollinearity, you likely don’t need to resolve it in any way.
2. Multicollinearity only affects the predictor variables that are correlated with one another. If you are interested in a predictor variable in the model that doesn’t suffer from multicollinearity, then multicollinearity isn’t a concern.
3. Multicollinearity impacts the coefficient estimates and the p-values, but it doesn’t impact predictions or goodness-of-fit statistics. This means if your main goal with the regression is to make predictions and you’re not concerned with understanding the exact relationship between the predictor variables and response variable, then multicollinearity doesn’t need to be resolved.
If you determine that you do need to fix multicollinearity, then some common solutions include:
1. Remove one or more of the highly correlated variables. This is the quickest fix in most cases and is often an acceptable solution because the variables you’re removing are redundant anyway and add little unique or independent information the model.
2. Linearly combine the predictor variables in some way, such as adding or subtracting them from one way. By doing so, you can create one new variables that encompasses the information from both variables and you no longer have an issue of multicollinearity.
3. Perform an analysis that is designed to account for highly correlated variables such as principal component analysis or partial least squares (PLS) regression. These techniques are specifically designed to handle highly correlated predictor variables.
Love your article. So easy to follow and understand. Thank you.
How would I know which items are related? If I have 20 variables and 5 of them have high VIF values…how do I know which ones are related to each other? And which one do you remove when you know?
This is well-simplified and highly informative.
Sorry to bother you Zack, if you got a classification problem, but you use, for example, a logistic regression, do you still need to account for the Multicollinearity ? I would say yes, but im not sure.
Thanks a lot. I have been enlightened on the whole issue of multicollinearity
Many thanks, great explanation, could you please share the refrences for that, I really need them?
How can we use VIF in logistic regression?
Hi Rajendra…Variance Inflation Factor (VIF) is a measure of multicollinearity among the predictor variables in a regression model. High multicollinearity can make the estimates of the regression coefficients unstable and difficult to interpret. While VIF is traditionally used in linear regression, it can also be applied to logistic regression to assess multicollinearity among the predictor variables.
Here’s how you can use VIF in the context of logistic regression using Python:
### Step-by-Step Guide
1. **Import Necessary Libraries:**
You need `statsmodels` for logistic regression and `statsmodels` or `scikit-learn` for calculating VIF.
“`python
import pandas as pd
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.linear_model import LogisticRegression
“`
2. **Load and Prepare Data:**
Ensure your data is in the form of a Pandas DataFrame.
“`python
# Example: Load data
data = pd.read_csv(‘your_dataset.csv’)
# Select the features and the target variable
X = data[[‘feature1’, ‘feature2’, ‘feature3’, ‘feature4’]]
y = data[‘target’]
“`
3. **Add a Constant to the Features:**
Adding a constant is necessary for calculating VIF.
“`python
X_with_constant = sm.add_constant(X)
“`
4. **Calculate VIF:**
Calculate VIF for each feature.
“`python
vif_data = pd.DataFrame()
vif_data[“feature”] = X_with_constant.columns
vif_data[“VIF”] = [variance_inflation_factor(X_with_constant.values, i) for i in range(X_with_constant.shape[1])]
print(vif_data)
“`
5. **Interpret VIF Results:**
– A VIF value of 1 indicates no correlation among the kth predictor and the remaining predictor variables.
– A VIF between 1 and 5 indicates moderate correlation.
– A VIF greater than 5 indicates high correlation, suggesting potential multicollinearity.
6. **Fit the Logistic Regression Model:**
After checking the VIF, fit your logistic regression model using features with acceptable VIF values.
“`python
# Remove features with high VIF values if necessary
X_selected = X[[‘feature1’, ‘feature3’]] # Example of selecting features with acceptable VIF
# Fit the logistic regression model
log_reg = LogisticRegression()
log_reg.fit(X_selected, y)
# Print model summary if using statsmodels for detailed output
logit_model = sm.Logit(y, sm.add_constant(X_selected))
result = logit_model.fit()
print(result.summary())
“`
### Example with a Synthetic Dataset
Here is a complete example with a synthetic dataset to illustrate the process:
“`python
import pandas as pd
import numpy as np
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.linear_model import LogisticRegression
# Generate a synthetic dataset
np.random.seed(0)
data = pd.DataFrame({
‘feature1’: np.random.rand(100),
‘feature2’: np.random.rand(100) * 10,
‘feature3’: np.random.rand(100) * 100,
‘feature4’: np.random.rand(100) * 1000,
‘target’: np.random.choice([0, 1], 100)
})
# Select features and target
X = data[[‘feature1’, ‘feature2’, ‘feature3’, ‘feature4’]]
y = data[‘target’]
# Add a constant for VIF calculation
X_with_constant = sm.add_constant(X)
# Calculate VIF
vif_data = pd.DataFrame()
vif_data[“feature”] = X_with_constant.columns
vif_data[“VIF”] = [variance_inflation_factor(X_with_constant.values, i) for i in range(X_with_constant.shape[1])]
print(“VIF Data:”)
print(vif_data)
# Fit the logistic regression model with features having acceptable VIF values
X_selected = X[[‘feature1’, ‘feature2’, ‘feature3’]] # Example of selecting features with acceptable VIF
log_reg = LogisticRegression()
log_reg.fit(X_selected, y)
# Model summary using statsmodels for detailed output
logit_model = sm.Logit(y, sm.add_constant(X_selected))
result = logit_model.fit()
print(result.summary())
“`
By following these steps, you can assess and handle multicollinearity in your logistic regression models using VIF, leading to more stable and interpretable models.