Image by Editor (Kanwal Mehreen) | Canva

When using statistical models in Python, preparing your data correctly is very important. It helps you get accurate and reliable results. The statsmodels library has powerful tools for analysis, but your data must be in the right format.

This article will show simple steps to clean, change, and organize your data. This will make sure it works well with statsmodels.

Understanding Data Requirements for statsmodels

statsmodels is a Python library for statistical modeling and hypothesis testing. To ensure accurate results, data must meet certain requirements:

1. Data Format: Supports Pandas DataFrame, NumPy arrays, and dictionary-like objects.
2. Handling Missing Data: Remove or impute missing values using mean or median.
3. Data Types and Encoding: Encode categorical variables and ensure numerical data is in the correct format.
4. Scaling and Normalization: Standardize or normalize data if required by the model.
5. Outliers: Detect and handle outliers using Z-scores, IQR, or visualization methods.

Importing Required Libraries

Before preparing your data, ensure you have the necessary libraries installed. You can install them using:

pip install pandas numpy statsmodels

Then, import the required libraries:

import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf

Loading the Data

The dataset should be loaded into a Pandas DataFrame. You can read data from a CSV file or other formats such as Excel, SQL databases, or JSON.

import pandas as pd

# Load dataset
file_path = "data.csv"
df = pd.read_csv(file_path)

# Display basic info
df.info()

Handling Missing Data

Missing data can affect analysis accuracy. It is important to handle it properly before using statsmodels. Common methods include:

• Removing missing values
• Filling gaps with the mean or median
• Using advanced techniques like interpolation or regression

df = df.dropna()
print(df.head())

Encoding Categorical Variables

The dataset contains categorical variables such as country and gender. Convert them into numerical representations:

# One-hot encoding for categorical variables
df = pd.get_dummies(df, columns=['country', 'gender'], drop_first=True)
print(df.head())

Handling Outliers

Outliers can distort statistical analysis. It is important to detect and handle them properly. Common methods include identifying outliers using Z-scores or IQR.

from scipy import stats

# Using Z-score
df = df[(np.abs(stats.zscore(df.select_dtypes(include=[np.number]))) < 3).all(axis=1)]
print(df.head())

Normalization and Standardization

Scaling data helps improve model performance. It ensures all features have a similar range. Normalization scales values between 0 and 1. Standardization gives data a mean of 0 and a standard deviation of 1. You can either normalize (scale to [0,1]) or standardize (mean=0, std=1):

from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Standardization
scaler = StandardScaler()
df[['credit_score', 'age', 'tenure', 'balance', 'products_number', 'estimated_salary']] = scaler.fit_transform(df[['credit_score', 'age', 'tenure', 'balance', 'products_number', 'estimated_salary']])

# Normalization
scaler = MinMaxScaler()
df[['credit_score', 'age', 'tenure', 'balance', 'products_number', 'estimated_salary']] = scaler.fit_transform(df[['credit_score', 'age', 'tenure', 'balance', 'products_number', 'estimated_salary']])

print(df.head())

Defining Independent and Dependent Variables

For regression models, separate the dependent variable (churn) and independent variables:

X = df[['credit_score', 'age', 'tenure', 'balance', 'products_number', 'credit_card', 'active_member', 'estimated_salary']]
Y = df['churn']

Add a constant term to the independent variables to include an intercept in the model:

X = sm.add_constant(X)

Performing Logistic Regression Analysis

Since churn is a binary variable (0 or 1), we use logistic regression:

# Fit an OLS (Ordinary Least Squares) regression model
logit_model = sm.Logit(Y, X).fit()
print(logit_model.summary())

Conclusion

Proper data preparation is essential for accurate analysis with statsmodels. Without well-structured data, the results of your models may be misleading or inaccurate.

Key steps include:

• Handling missing values and outliers
• Encoding categorical variables
• Normalizing or standardizing data
• Defining independent and dependent variables
• Performing logistic regression

Following these steps ensures reliable results and meaningful insights.