Linear regression is a statistical method we can use to understand the relationship between two variables, x and y.
When interpreting the results of a regression model, we must first make sure that four assumptions are met:
1. Linear relationship: There exists a linear relationship between the independent variable, x, and the dependent variable, y.
2. Independence: The residuals are independent. In particular, there is no correlation between consecutive residuals in time series data.
3. Homoscedasticity: The residuals have constant variance at every level of x.
4. Normality: The residuals of the model are normally distributed.
If one or more of these assumptions are violated, then the results of the regression model could be unreliable.
In this tutorial we explain how to check each of these assumptions in R, using the built-in mtcars dataset as an example with the mpg variable as the response variable and the disp variable as the predictor variable.
Assumption 1: Linear Relationship
The first assumption of linear regression is that there is a linear relationship between the independent variable, x, and the independent variable, y.
The easiest way to detect if this assumption is met is to create a scatter plot of x vs. y.
We can use the following syntax to do so:
#create scatter plot of disp vs. mpg
plot(mtcars$disp, mtcars$mpg)
This produces the following scatter plot:
Since the points in the scatter plot fall roughly along a straight line, the linear relationship assumption is met.
Assumption 2: Independence
The next assumption of linear regression is that the residuals are independent.
The simplest way to test if this assumption is met is to look at a residual time series plot, which is a plot of residuals vs. time.
Ideally, most of the residual autocorrelations should fall within the 95% confidence bands around zero, which are located at about +/- 2-over the square root of n, where n is the sample size.
You can also use the Durbin-Watson test to check this assumption.
Assumption 3: Homoscedasticity
The next assumption of linear regression is that the residuals have constant variance at every level of x.
When this is not the case, the residuals are said to suffer from heteroscedasticity.
The simplest way to detect heteroscedasticity is by creating a fitted value vs. residual plot.
You can use the following syntax to do so in R:
#fit regression model model <- lm(mpg ~ disp, data=mtcars) #create fitted values vs residuals plot plot(model, 1)
This produces the following plot:
The x-axis shows the fitted values and the y-axis shows the residuals.
We can see that the residuals do seem to get larger over time, but not enough for us to be concerned that heteroscedasticity is present.
Assumption 4: Normality
The next assumption of linear regression is that the residuals are normally distributed.
The most common way to check this assumption is to create a Q-Q plot.
If the points on the plot roughly form a straight diagonal line, then the normality assumption is met.
We can use the following syntax to create this plot in R:
#fit regression model model <- lm(mpg ~ disp, data=mtcars) #create Q-Q plot plot(model, 2)
This produces the following Q-Q plot:
From the plot we can see that the points fall roughly along a straight line, with minor deviations along each tail.
Based on this plot, we would assume that the normality assumption is not violated.
Additional Resources
The following tutorials explain how to perform other common tasks in R:
How to Perform Spline Regression in R
How to Perform OLS Regression in R
How to Perform Power Regression in R
To test independence, is there anything else besides Durbin Watson that can be used?
Hi Phuc…Yes, there are several other tests and methods besides the Durbin-Watson test that can be used to test for independence (or autocorrelation) in a time series or regression residuals. Here are a few alternatives:
### 1. **Ljung-Box Test**
The Ljung-Box test is a statistical test used to determine if there are significant autocorrelations in a time series. It tests the null hypothesis that the data are independently distributed.
– **Null Hypothesis (H₀)**: The data are independently distributed (no serial correlation).
– **Alternative Hypothesis (H₁)**: The data are not independently distributed (there is serial correlation).
In R, you can use the `Box.test` function:
“`R
Box.test(residuals, lag = k, type = “Ljung-Box”)
“`
In Python, you can use the `acorr_ljungbox` function from the `statsmodels` library:
“`python
from statsmodels.stats.diagnostic import acorr_ljungbox
acorr_ljungbox(residuals, lags=[k], return_df=True)
“`
### 2. **Breusch-Godfrey Test**
The Breusch-Godfrey test (also known as the LM test) is a test for higher-order autocorrelation in the residuals from a regression model.
– **Null Hypothesis (H₀)**: No autocorrelation at lag k.
– **Alternative Hypothesis (H₁)**: Autocorrelation at lag k.
In R, you can use the `bgtest` function from the `lmtest` package:
“`R
library(lmtest)
bgtest(model, order = k)
“`
In Python, you can use the `acorr_breusch_godfrey` function from the `statsmodels` library:
“`python
from statsmodels.stats.diagnostic import acorr_breusch_godfrey
acorr_breusch_godfrey(model, nlags=k)
“`
### 3. **Runs Test**
The Runs Test is a non-parametric test that examines the randomness of data. It tests whether the order of data points is random or if there is some form of trend or autocorrelation.
– **Null Hypothesis (H₀)**: The sequence is random.
– **Alternative Hypothesis (H₁)**: The sequence is not random.
In Python, you can use the `runs_test` function from the `statsmodels` library:
“`python
from statsmodels.sandbox.stats.runs import runstest_1samp
runstest_1samp(residuals)
“`
### 4. **Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) Plots**
ACF and PACF plots are graphical tools used to identify the presence of autocorrelation at different lags.
– **ACF Plot**: Shows the autocorrelation of the series at different lags.
– **PACF Plot**: Shows the partial autocorrelation of the series at different lags, removing the influence of intermediate lags.
In R, you can use the `acf` and `pacf` functions:
“`R
acf(residuals)
pacf(residuals)
“`
In Python, you can use the `plot_acf` and `plot_pacf` functions from the `statsmodels` library:
“`python
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
plot_acf(residuals)
plot_pacf(residuals)
“`
### 5. **Augmented Dickey-Fuller (ADF) Test**
While primarily used to test for stationarity, the ADF test can also be useful in identifying the presence of autocorrelation since non-stationary data often exhibit autocorrelation.
– **Null Hypothesis (H₀)**: The series has a unit root (is non-stationary).
– **Alternative Hypothesis (H₁)**: The series does not have a unit root (is stationary).
In R, you can use the `adf.test` function from the `tseries` package:
“`R
library(tseries)
adf.test(residuals)
“`
In Python, you can use the `adfuller` function from the `statsmodels` library:
“`python
from statsmodels.tsa.stattools import adfuller
adfuller(residuals)
“`
### Summary
These tests and methods provide various ways to check for independence or autocorrelation in your data or residuals. Depending on the context and specific requirements of your analysis, you may choose one or more of these techniques to complement or replace the Durbin-Watson test.