Choosing the Best Regression Diagnostic Tool: A Decision Tree Approach

Image by Author| Ideogram

Introduction

Assessing the validity and predictive quality of regression models in machine learning is essential to ensure a model is reliable, interpretable, and generalizes well to new data. To this end, there exist several diagnostic tools like residual plots, VIF, Cook’s distance, and more that specialize in evaluating model validity, multicollinearity, robustness to influential points like outliers, and violations of key assumptions. To help you decide on the right diagnostic tool to use depending on your problem and needs, this article conveys a decision tree-style guide that navigates you towards choosing the right regression diagnostic method (or set of methods) to apply.

Decision Tree to Choose the Best Regression Diagnostic Tool

The diagram below depicts a visual guide to help you choose the right method to diagnose your trained regression model.

Choosing the right regression diagnosis tool (click to enlarge)

The decision tree broadly considers two main scenarios:

Diagnosing a newly built regression model and unsure where to start.
Having a specific concern about an already existing (and usually evaluated) model.

Initial Diagnosis of New Models

A common procedure to perform an initial diagnosis of a newly built regression model is to apply three methods sequentially: (1) residual vs. fitted plot, (2) variance inflation factor (VIF), and (3) Cook’s distance. If results for all three of them are favorable — see green ticks in diagram above — that hints at your model looking good. Otherwise, there are several workarounds to consider depending on the specific diagnosis outcomes:

The residual vs. fitted plot tool can yield several kinds of patterns as outcomes. A curved pattern normally indicates that data are not linear and the (presumably linear) regression model is not a good fit; hence, you may want to try extracting polynomial features or interaction terms between features to better fit the input-output variable relationships.
A funnel pattern is a sign of heteroscedasticity; in other words, the variability of the residuals (prediction errors made) is not constant across different predicted values. Transforming the target variable, e.g., by using logarithmic transformations, sometimes helps alleviate this issue.
A systematic pattern other than random scatter near zero generally means the model is not properly capturing the relationships between predictors and the target variable. Try getting new variables that might help capture this relationship.
If none of the aforesaid patterns appear, and the residual vs. fitted plot yields random scatter near zero, there are still two more trials to pass: VIF and Cook’s distance. VIF is “passed” is all VIFs are lower than 5, meaning there is no high correlation between variables. Otherwise, it will be necessary to remove highly correlated predictor variables and restart the diagnosis process with the updated model.
Both residual vs. fitted plot and VIF led to favorable outcomes? Great news! Now it only remains to try Cook’s distance test, aimed at detecting highly influential observations (instances) in a regression model, typically outliers or unusual data points that might alter the model parameters and subsequent predictions if they were removed. In essence, highly influential points are problematic, and Cook’s distance is the tool to be utilized for detecting and handling them. Once possible influential points are removed and the three tools lead to positive outcomes, your model is finally in good shape!

Diagnosing Specific Concerns

Chances are, you have already performed an initial diagnosis on your model, and you are at a point where you have specific concerns and would like to know which diagnostic tool to use for analyzing it. The right-hand side part of the above diagram shows six possible concerns to look at. For four of them, you may be directed to (re)apply either one of the already outlined tools, or adjusted R² if your goal is to compare different models. If having multiple regression models, each with their own parameters, adjusted R² helps determine which one provides a better fit to the data by highlighting the proportion of variance in the target variable explained by the predictor variables.

Additionally, if the goal or concern is to perform statistical tests or determine confidence intervals, the QQ plot (Quantile-Quantile plot) is intended to analyze the normality of residuals, that is, whether the errors between predictions and actual target values for observations follow a normal distribution. QQ plot can help detect deviations from normality, like heavy tails or skewness, which means the assumptions underlying the regression model may not be met.

Finally, when the model seems to behave arbitrarily, so that some predictions are good whereas others are bad, try using the scale-location plot, which is another approach to measure the homoscedasticity assumption. Just like in residuals vs. fitted plot, if a funnel pattern shows up, it’s time to act: use weighted least squares or transform the target, dependent variable.

Wrapping Up

This article introduced a guide to help you how to proceed for diagnosing your regression model. Depending on whether you are looking for an initial diagnosis or have a specific concern to analyze, the decision tree is designed to point you in the right direction.