Step-by-Step Guide to Logistic Regression in R

Logistic regression is a powerful statistical technique that is used to model binary outcomes, allowing you to predict the probability of an event occurring based on one or more predictors. This guide will walk you through the process of implementing a logistic regression in R, covering everything from data preparation to model evaluation and refinement.

Introduction to Logistic Regression

Logistic regression is used for predicting a binary outcome, such as true and false or yes and no. It models the probability of the outcome class, such as true, as a function of the predictors. The relationship between the predictors and outcome is expressed as the log odds (logit) of the probability with the coefficients indicating the change in the log odds of the outcome for a one unit increase in that predictor.

Logistic regression is used in a wide range of fields, from medicine and social sciences to finance and marketing, due to its ability to handle binary classification problems effectively. The method’s simplicity and interpretability make it a popular choice for modeling and predicting outcomes where the response variable is categorical. However, to achieve a good model fit, certain assumptions must be met, and proper model evaluation techniques should be applied.

Preparation

The first step in any modeling process is data preparation. This involves cleaning the data, handling missing values, encoding categorical variables, and splitting the data into training and testing sets. This ensures that any models are built on a solid foundation and helps avoid potential issues during analysis and interpretation.

This guide will utilize the mtcars dataset that is available in R by default to predict whether a car has an automatic or manual transmission based on other variables. In the data, 0 represents an automatic transmission while 1 is a manual transmission. In order to use the data for a logistic regression, the outcome variable of transmission needs to be converted to a factor variable. The dataset is already clean of data errors, but be sure to fully vet any dataset you use for analysis.

# load the necessary libraries for data manipulation
library(tidyverse)

# load the mtcars dataset
data(“mtcars”)

# convert am (transmission) to a factor variable
mtcars$am <- as.factor(mtcars$am)

Data Exploration

Understanding the data is a crucial step in any analysis project. Before diving into a full logistic regression, you should have a basic grasp of the dataset, including the distribution of key variables. For example, you can run summary to get basic statistics for the numeric variables and table to obtain the counts for the outcome variable.

summary(mtcars)

table(mtcars$am)

Additional checks such as correlations, missingness, and exploration of the data structure can be helpful during this phase. Your findings from the data exploration process will help identify if there are additional data preparation needs, such as handling missing values, inspecting outliers, or transforming variables.

Fitting a Logistic Regression Model

To fit a logistic regression model in R, use the glm function with the family argument set to binomial. To build a logistic regression model that predicts transmission using horsepower and miles per gallon, you can run the following code.

model <- glm(am ~ hp + mpg, data = mtcars, family = binomial)
summary(model)

Running summary on your model outputs standard model metrics, including coefficients, standard errors, and the Z and p-values for each predictor. The coefficients represent the effect of each predictor on the log-odds of the outcome. In the output, the coefficient for MPG is positive. This indicates that a higher MPG value is associated with a higher likelihood of transmission equalling 1, indicating a manual transmission. The errors, Z, and p-values are used to indicate precision and statistical significance of each predictor.

Model Assumptions

In order to properly utilize the model and ensure its validity, you must ensure key assumptions have been met. First is that each observation should be independent, which is validated during the study design or data collection phase. There should also be no multicollinearity among the predictors, which occurs when predictors are highly correlated with each other. This inflates the standard errors fo the coefficients, making it difficult to isolate the individual contribution of each. Multicollinearity can be checked using the Variance Inflation Factor, which reports how much of the variance in a coefficient is inflated due to multicollinearity. Any value above 5-10 is considered problematic.

library(car)
vif(model)

Another assumption is that the relationship between a continuous predictor and the log-odds of the outcome is linear. This means that a unit change in the predictor is associated with a consistent change in the log-odds of the outcome. If this assumption is violated, the model’s predictions may be inaccurate. Linearity can be checked with the Box-Tidwell test, which adds a logarithmic transformation of the continuous predictor to the model. If this new term is significant, the linearity assumption is violated.

library(car)

# create the transformed variables
mtcars$log_hp <- log(mtcars$hp + 1) * mtcars$hp
mtcars$log_mpg <- log(mtcars$mpg + 1) * mtcars$mpg

# run a new model with the transformed variables
box_tidwell <- glm(am ~ hp + log_hp + mpg + log_mpg, data = mtcars, family = binomial)

summary(box_tidwell)

Model Evaluation

To assess the model, you can consider various metrics:

Null and Residual Deviance: These are provided in the basic model summary. The null deviance represents a model with no predictor variables included while the residual deviance is for the model with all predictors included. A large reduction in deviance suggests a good model fit.
Akaike Information Criterion (AIC): Also part of the basic model summary, the AIC is a measure of the relative quality of the model. A lower value indicates a better fit, but there is no accepted threshold. Instead, this value is used to compare between multiple models.
Confusion Matrix: If you have the actual values for a training dataset, you can build a confusion matrix to compare the predicted versus the actual values to assess model accuracy.

# get the model’s predictions
predicted_probabilities <- predict(model, type = “response”)

# convert the probabilities predicted into 0’s and 1’s based on a 0.5 cutoff value
predicted_values <- ifelse(predicted_probabilities > 0.5, 1, 0)

# confusion matrix comparing predicted to the dataset
table(predicted = predicted_values, actual = mtcars$am)

Predicting New Values

Your fitted model can be used to predict the probability of the outcome for new data. For example, you can pass in a new datapoint and output the probability that a car with these metrics has a manual transmission. You can then convert this probability to a 0 or 1 output by using 50% as a threshold.

new_data <- data.frame(hp = 120, mpg = 20)
prediction <- predict(model, newdata = new_data, type = “response”)
prediction

prediction_final <- ifelse(prediction > 0.5, 1, 0)

In this model, the prediction probability was 0.141, so the value for prediction_final was 0. Therefore, the model predicts that a car with 120 horsepower and 20 MPG will have automatic transmission.

Model Improvements

There are many strategies you can use to improve the predictive power of your logistic regression model. A few of these include:

Feature selection: Often, your dataset will have irrelevant variables or variables that do not improve predictive power. Careful feature selection or utilization of methods like stepwise selection can result in an improved final model.
Class imbalance: If the target classes are too imbalanced in your dataset where one is overrepresented over the other, consider resampling where examples from the minority class are duplicated or using weighted logistic regression to improve predictive power.
Interaction terms: Sometimes, two variables can have an interaction where the combined effect is more impactful than simply adding the individual effects together. Interaction terms can be added by multiplying the two predictors together in the glm statement. If this term becomes significant, the combined effect of the two variables is different from their individual effects.

Common Pitfalls and Troubleshooting

One of the most common pitfalls in logistic regression is the misinterpretation of the model coefficients. In linear regression, coefficients represent the change in the outcome variable for a one-unit change in the predictor. However, in logistic regression, coefficients represent the change in the log-odds of the outcome. One solution is to report the odds ratios, calculated as exponentials of the coefficients, to make them easier to interpret.

An important issue to keep in mind in all types of modeling applications is overfitting. This occurs when the model fits the training data too closely to be helpful in generating predictions of new, unseen data. Common solutions for overfitting include limiting the number of predictors and performing cross-validation on the model to test it on different subsets of the data.

Finally, convergence problems can also occur if the model fails to find a solution due to small sample sizes, missing data, quasi-separation, or multicollinearity. Each of these causes has different solutions that make the analysis more complicated, but an easy first pass solution is to reduce the number of predictors.

Summary

Logistic regression is a powerful and widely used tool for binary classification problems. There are many steps and considerations to keep in mind when using this type of analysis. This guide provided a brief overview of how to implement logistic regression in R. With these techniques, you can confidently apply logistic regression to a variety of datasets and scenarios, while keeping more advanced techniques in mind to further refine your analysis.