Image by Editor | Midjourney
Regression modeling is a type of predictive data problem present in many sectors: finance (to assess risk), healthcare (for disease progression modeling), economics (for income prediction based on multiple socio-economic factors), marketing (to estimate customers’ lifetime value), and engineering (for failure rate estimation), to name a few. These example problems share one thing in common: we have a target variable to be predicted based on one or multiple different variables or predictors.
From classical methods inherited from statistics like linear regression to more advanced solutions involving ensemble machine learning models or complex deep learning architectures, there is a wealth of approaches with varying levels of complexity, interpretability, and data requirements. Therefore, a question that often arises is: which type of regression model should I choose based on my problem, data, and needs?
This article provides a decision tree-based taxonomy of regression models to guide you in identifying the most suitable method to apply among the diverse landscape of choices available, depending on the problem you are tackling and the nature of your data.
A Decision Tree Approach to Selecting the Ideal Regression Model
The following diagram illustrates our proposed decision tree approach for navigating you towards choosing the right regression modelling approach.
A decision tree approach to selecting the ideal regression model (click to enlarge)
Let’s further analyze it along the following key points and aspects to consider:
1. Since regression models are built by modeling a relationship between predictor features and a target variable, so that the latter can be accurately predicted based on the former, the most important question arguably is: is there a linear or nonlinear relationship between predictor features and the target variable? Linearity in this context means that the target variable can be expressed as a weighted sum of the predictor features, plus a constant rate of change when any of them varies. For example, in a linear relationship, doubling an input feature like years of experience would result in a proportional change in the target variable, say annual salary.
2. For linear data, i.e. data exhibiting a linear relationship between predictors and target, next thing to look at is the number of independent features or predictors: the simplest scenario occurs when we have only one predictor feature, e.g. predicting annual salary based on years of experience solely. Simple linear regression models, given by a straight line, are then sufficient in this scenario. Otherwise, when there are multiple predictor features, we may want to check whether there is multicollinearity, that is, a strong correlation between independent variables, which can distort coefficient estimates and reduce model reliability. If no multicollinearity exists, a multiple linear regression model can do the job; otherwise, slightly tweaked approaches like ridge or lasso regression, or Elastic Nets, would be best.
If our data are nonlinear, we may have a few additional questions to ask to find the right regression approach. We further elaborate on them below.
1. Could feature transformations like log or polynomial transformations turn the data linear? If yes, we can resort to polynomial regression or generalized linear models (GLMs). Otherwise, we may continue asking questions about the data and problem at hand: keep reading.
2. For data that cannot be made linear, let’s look at the size of the dataset in terms of number of observations. If your dataset has less than 5K observations, it can be considered as small or middle-sized, in which case a good approach to follow is using tree-based regressors: decision trees, random forests, XGBoost, and similar.
3. For large, nonlinear datasets spanning over 5K instances, things get slightly trickier, and we need to shift towards more complex models. The next question to ask if we reached this point is: should interpretability prevail over accuracy, or is a highly accurate model our top priority? If interpretability truly matters, let’s stick to explainable models like XGBoost and Gradient Boosting, or consider the added use of Explainable AI mechanisms together with other model types.
4. If accuracy is more important than interpretability, it’s definitely time to venture into sophisticated regression techniques. Let’s look at the dataset size in terms of number of features, i.e. its dimensionality. If your dataset has low dimensionality (say up to 15-25 features), consider using support vector machines for regression (SVR) or Kernel Methods. For datasets of larger dimensionality, the final question to ask is: does the model (and the underlying problem) require automatic extraction of meaningful features?. This is an instance of the case of unstructured or visual data, where the original features (e.g. pixels, words) are not meaningful enough and higher-level features should be extracted. In this case, deep learning models for regression are the solution: from feed-forward regressors to convolutional networks or transformers for very complex tabular data. Otherwise, it is still possible to use intermediate complexity models like ensembles with chances of success.
Wrapping Up
This article provided a structured taxonomy, shaped as a decision tree, to help you identify the right regression modelling method depending on the properties and size of your data, in other words, based on the complexity of the problem to address. Choosing the most adequate approach is a crucial step to developing an effective and successful predictive solution that yields high impact.
