Step-by-Step Guide to Linear Regression in Python

Linear regression is one of the first algorithms you’ll add to your statistics and data science toolbox.

It helps model the relationship between one more independent variables and a dependent variable. In this tutorial, we’ll review how linear regression works and build a linear regression model in Python. You can follow along with this Google Colab notebook if you like.

Let’s begin.

Linear Regression: An Overview

Linear regression aims to fit a linear equation to observed data given by:

Where:

y and x are the dependent and independent variables, respectively.
β₁ is the slope of the line or the regression coefficient.
β₀ is the y-intercept.

As you might already be familiar, linear regression finds the best-fitting line through the data points by estimating the optimal values of β₁ and β₀ that minimize the sum of the squared residuals—the differences between the observed and predicted values—the Ordinary Least Squares (OLS) method.

When there are multiple independent variables, the multiple linear regression model is given by:

Here, y is predicted by multiple x variables, each with its coefficient β.

Building a Linear Regression Model on the California Housing Dataset

Let’s use the California Housing dataset from sklearn.datasets for linear regression. We’ll load the dataset, read it into a dataframe, and perform linear regression one step at a time.

Step 1: Import Libraries

First, let’s import the necessary libraries, modules, and classes to use.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

Step 2: Load the Dataset

Next, load the California Housing dataset that’s built into scikit-learn.

# Load the dataset
housing = fetch_california_housing()

# Create a DataFrame
df = pd.DataFrame(housing.data, columns=housing.feature_names)
df['MedHouseVal'] = housing.target

df.head()

Step 3: Explore the Dataset

Let’s understand the dataset a bit.

# Check for missing values
print(df.isnull().sum())

This outputs:

MedInc         0
HouseAge       0
AveRooms       0
AveBedrms      0
Population     0
AveOccup       0
Latitude       0
Longitude      0
MedHouseVal    0
dtype: int64

It also helps to get the summary statistics for the features in the dataset.

# Summary statistics
print(df.describe())

Step 4: Split the Dataset into Training and Test Sets

The next step is to separate the data into training and test sets.

# Define features and target variable
X = df.drop('MedHouseVal', axis=1)
y = df['MedHouseVal']

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=25)

Let’s also check the dimensions of the train and test sets—features and the labels.

print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)

You should get a similar output:

(14448, 8) (6192, 8) (14448,) (6192,)

Step 5: Create and Fit the Model

Instantiate and fit the LinearRegression model on the training data like so.

# Create the model
model = LinearRegression()

# Fit the model
model.fit(X_train, y_train)

# Display the coefficients
print('Intercept:', model.intercept_)
print('Coefficients:', model.coef_)

These are the coefficients and the intercept for the model:

Intercept: -37.552166316589286
Coefficients: [ 4.41181942e-01  9.70730794e-03 -1.19993946e-01  7.84709051e-01
 -3.39466724e-07 -3.28239095e-03 -4.23679731e-01 -4.39311822e-01]

Step 6: Make Predictions

Use the linear regression model to make predictions on the test set.

# Make predictions
y_pred = model.predict(X_test)

Step 7: Visualize the Predictions

Now that we have the predictions, let’s plot the true vs. predicted values.

plt.scatter(y_test, y_pred, color='blue')
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], color='red', linewidth=2)
plt.xlabel('Actual MedHouseVal')
plt.ylabel('Predicted MedHouseVal')
plt.title('Actual vs Predicted MedHouseVal')
plt.show()

Step 8: Evaluate the Model

Calculate and print the mean squared error and R-squared value:

The Mean Squared Error (MSE) is the average of the squared differences between the observed and predicted values.
R-squared measures the proportion of the variance in the dependent variable that is predictable from the independent variable(s).

# Calculate Mean Squared Error
mse = mean_squared_error(y_test, y_pred)

# Calculate R-squared
r2 = r2_score(y_test, y_pred)

print('Mean Squared Error:', mse)
print('R-squared:', r2)