Logistic regression is one of the common algorithms you can use for classification. Just the way linear regression predicts a continuous output, logistic regression predicts the probability of a binary outcome.
In this step-by-step guide, we’ll look at how logistic regression works and how to build a logistic regression model using Python. We’ll use the Breast Cancer Wisconsin dataset to build a logistic regression model that predicts whether a tumor is malignant or benign based on certain features.
Let’s get started.
How Logistic Regression Works
Logistic regression works by modeling the probability of a binary outcome based on one or more predictor variables.
Let’s take a linear combination of input features or predictor variables. If x represents the input features and β represents the coefficients or parameters of the model:
Where β0 is the intercept term and the βs are model coefficients.
The logistic or sigmoid function maps inputs to the output, a probability between 0 and 1. We plug in the linear combination z into the sigmoid function:
The output σ(z) represents the probability that the input belongs to class 1. Then 1 – σ(z) is the probability that the input belongs to class 0.
The class label is decided based on a chosen threshold. A commonly used threshold is 0.5:
The model parameters are estimated by maximizing the likelihood of observing the given data—maximum likelihood estimation (MLE). The cost function used for optimization is the logistic loss:
where yi is the actual class label, ŷi is the predicted probability, and N is the number of samples.
The trained model is evaluated using metrics such as accuracy, precision, recall, and F1 score as needed.
By iteratively adjusting the coefficients to minimize the cost function, logistic regression finds the best-fitting model to predict the probability of the binary outcome based on the input features.
Building a Logistic Regression Model
Let’s now get to building a logistic regression model. As mentioned, we’ll use the Breast Cancer Wisconsin dataset. The goal is to predict the target label—1 for malignant and 0 for benign tumors, respectively.
Step 1: Import the Necessary Libraries
First, we need to import the necessary libraries, modules, and various classes.
import pandas as pd import matplotlib.pyplot as plt import seaborn as sns from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
Step 2: Load the Dataset
Next, we load the Breast Cancer Wisconsin dataset—contains features of the breast cancer data and the corresponding target labels—and read it into a pandas dataframe for easier analysis.
# Load the dataset data = load_breast_cancer() # Create a DataFrame df = pd.DataFrame(data.data, columns=data.feature_names) df['target'] = data.target df.head()
Here’s the head of the dataframe (truncated as there are 30 features and a target label):
Step 3: Explore the Dataset
Exploring the dataset helps us understand its structure, check for missing values, and get summary statistics of the features.
- df.isnull().sum() checks for missing values in the dataset, which is important for data preprocessing.
- df.describe() provides summary statistics of the features, helping us understand the distribution and scale of the data.
# Check for missing values print(df.isnull().sum()) # Summary statistics print(df.describe())
The code snippet above returns the number of missing values and the summary statistics for all the numeric features in the dataset. Please refer to the Colab notebook for the output.
Step 4: Visualize the Data
Visualizing the data helps identify patterns and relationships between features and the target variable.
We use sns.pairplot to create pair plots of a subset of selected features colored by the target variable, allowing us to see how these features relate to each other and the target.
# Pairplot of the dataset sns.pairplot(df, vars=['mean radius', 'mean texture', 'mean area'], hue='target') plt.show()
Step 5: Split the Dataset into Train and Test Sets
The next step is to split the dataset into training and test sets.
# Define features and target variable
X = df.drop('target', axis=1)
y = df['target']
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=25)
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
Output:
(455, 30) (114, 30) (455,) (114,)
Step 6: Scale the Features
Feature scaling ensures that all features contribute equally to the model, preventing features with larger scales from skewing the model.
# Create the scaler scaler = StandardScaler() # Fit and transform the training data X_train_scaled = scaler.fit_transform(X_train) # Transform the test data X_test_scaled = scaler.transform(X_test)
Step 7: Create and Fit the Logistic Regression Model
We instantiate and fit the logistic regression model on the scaled training data.
# Create the model
model = LogisticRegression()
# Fit the model
model.fit(X_train_scaled, y_train)
# Display the coefficients
print('Intercept:', model.intercept_)
print('Coefficients:', model.coef_)
Output:
Intercept: [0.41929956] Coefficients: [[-0.45929248 -0.30161599 -0.45511443 -0.52552987 -0.16507015 0.11832179 -0.96241461 -0.80821553 -0.35213898 0.32945034 -0.74022291 0.62535611 -0.38453488 -0.74842793 -0.39347969 0.66808284 -0.43221848 0.03793236 0.21596437 0.81085901 -1.26270626 -1.51681407 -1.10589228 -1.18791003 -0.66945572 0.08468715 -0.91710651 -0.90888628 -0.65189289 -0.1779736 ]]
Step 8: Make Predictions
We use the trained model to make predictions on the scaled test data.
# Make predictions y_pred = model.predict(X_test_scaled)
Step 9: Evaluate the Model
Let’s evaluate the model’s performance:
- accuracy_score calculates the accuracy of the model.
- confusion_matrix is a matrix showing true positives, true negatives, false positives, and false negatives.
- classification_report is a more comprehensive view of model performance: precision, recall, and F1-score.
# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
# Calculate confusion matrix
conf_matrix = confusion_matrix(y_test, y_pred)
# Calculate classification report
class_report = classification_report(y_test, y_pred)
print('Accuracy:', accuracy)
print('Confusion Matrix:\n', conf_matrix)
print('Classification Report:\n', class_report)
Output:
Accuracy: 0.9649122807017544
Confusion Matrix:
[[36 3]
[ 1 74]]
Classification Report:
precision recall f1-score support
0 0.97 0.92 0.95 39
1 0.96 0.99 0.97 75
accuracy 0.96 114
macro avg 0.97 0.95 0.96 114
weighted avg 0.97 0.96 0.96 114
Step 10: Visualize the Confusion Matrix
We visualize the confusion matrix using sns.heatmap creates a heatmap of the confusion matrix to better understand the model’s performance.
# Plot the confusion matrix
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=data.target_names, yticklabels=data.target_names)
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()
Wrapping Up
And that’s a wrap! In this tutorial, we reviewed how logistic regression works and built a logistic regression model in Python. We imported the necessary libraries, loaded and preprocessed the data, trained the model, made predictions, and evaluated the model’s performance.
Logistic regression is a simple and an easy-to-interpret algorithm for binary classification tasks. You can also use it for multiclass classification. With proper data preprocessing and model evaluation, you can use it in applications like medical diagnosis, spam detection, and the like.
