To evaluate the performance of a model on a dataset, we need to measure how well the predictions made by the model match the observed data.

One commonly used method for doing this is known as leave-one-out cross-validation (LOOCV), which uses the following approach:

**1.** Split a dataset into a training set and a testing set, using all but one observation as part of the training set.

**2.** Build a model using only data from the training set.

**3.** Use the model to predict the response value of the one observation left out of the model and calculate the mean squared error (MSE).

**4.** Repeat this process *n* times. Calculate the test MSE to be the average of all of the test MSE’s.

This tutorial provides a step-by-step example of how to perform LOOCV for a given model in Python.

**Step 1: Load Necessary Libraries**

First, we’ll load the necessary functions and libraries for this example:

**from sklearn.model_selection import train_test_split
from sklearn.model_selection import LeaveOneOut
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression
from numpy import mean
from numpy import absolute
from numpy import sqrt
import pandas as pd
**

**Step 2: Create the Data**

Next, we’ll create a pandas DataFrame that contains two predictor variables, x_{1} and x_{2}, and a single response variable y.

**df = pd.DataFrame({'y': [6, 8, 12, 14, 14, 15, 17, 22, 24, 23],
'x1': [2, 5, 4, 3, 4, 6, 7, 5, 8, 9],
'x2': [14, 12, 12, 13, 7, 8, 7, 4, 6, 5]})
**

**Step 3: Perform Leave-One-Out Cross-Validation**

Next, we’ll then fit a multiple linear regression model to the dataset and perform LOOCV to evaluate the model performance.

**#define predictor and response variables
X = df[['x1', 'x2']]
y = df['y']
#define cross-validation method to use
cv = LeaveOneOut()
#build multiple linear regression model
model = LinearRegression()
#use LOOCV to evaluate model
scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error',
cv=cv, n_jobs=-1)
#view mean absolute error
mean(absolute(scores))
3.1461548083469726
**

From the output we can see that the mean absolute error (MAE) was **3.146**. That is, the average absolute error between the model prediction and the actual observed data is 3.146.

In general, the lower the MAE, the more closely a model is able to predict the actual observations.

Another commonly used metric to evaluate model performance is the root mean squared error (RMSE). The following code shows how to calculate this metric using LOOCV:

**#define predictor and response variables
X = df[['x1', 'x2']]
y = df['y']
#define cross-validation method to use
cv = LeaveOneOut()
#build multiple linear regression model
model = LinearRegression()
#use LOOCV to evaluate model
scores = cross_val_score(model, X, y, scoring='neg_mean_squared_error',
cv=cv, n_jobs=-1)
#view RMSE
sqrt(mean(absolute(scores)))
3.619456476385567**

From the output we can see that the root mean squared error (RMSE) was **3.619**. The lower the RMSE, the more closely a model is able to predict the actual observations.

In practice we typically fit several different models and compare the RMSE or MAE of each model to decide which model produces the lowest test error rates and is therefore the best model to use.

**Additional Resources**

A Quick Intro to Leave-One-Out Cross-Validation (LOOCV)

A Complete Guide to Linear Regression in Python

Hello,

Thank you for the tutorial.

I’m inexperienced with these procedures, so I have a question.

How can I print the model that was built?

I’m applying this to my data and I get the RMSE, but I’d like to know the model that was built during the LinearRegression step.

Thanks in advance.

Shyam

Thanks for the example Zach. Helpful.

There may be an error in your last comment about the RMSE. Shouldn’t that be the square root of the mean *squared* error, and not the mean absolute error? The ‘S’ in RMSE is for ‘squared’, but the differences between predicted and measured values are not being squared, if they’re just getting their absolute values taken.

RMSE should be the square root of the mean of the squared errors, not the absolute errors.