Image by Editor

Answering “why” in data analysis typically involves seeking relationships among variables in your dataset. For example, which features influence customer churn? Or which operational metrics correlate with production costs?

When the dataset has only two or three variables, conducting a t-test or calculating Pearson correlations is a simple process, but things change as the number of variables grows, and real-world datasets usually have many variables. In such situations, manually implementing statistical tests for every pair of variables can become tedious, to say the least.

To alleviate this challenge, this article shows how to use open-source Python libraries to build a reusable hypothesis testing engine that ingests a dataset stored in a Pandas DataFrame and automatically iterates through it. It applies the appropriate statistical tests and outputs a clean list of statistically significant findings identified throughout the process.

Building Your Reusable Hypothesis Testing Engine Step-by-Step

We will consider an HR scenario related to employee attrition, that is, workforce reduction due to causes like resignations, retirements, and so on. Suppose we want to analyze a dataset that contains employee information arranged across a few dozen continuous variables, including age, commute distance, training hours, tenure, satisfaction rating, and engagement score. There may also be categorical, binary variables like whether an employee quit or stayed.

Two primary analysis questions are formulated:

• Considering a categorical target like “quit vs. stay”, which continuous variables significantly differ between employees who quit compared to employees who stayed? (two groups, t-test required)
• Considering the satisfaction rating as a target variable, which of the other numerical variables strongly correlate with it? (Pearson correlations required)

Now, we can write a script capable of answering both questions automatically, given an input dataset of employees. Both Pandas and scipy.stats will play a central role in manipulating the data and running the statistical tests, respectively. First, let’s import them:

import pandas as pd
import numpy as np
from scipy import stats

To emulate the scenario described earlier, we create a dataset of 100 fictional employees with the variables listed above. We can do this using NumPy’s random number generators based on probability distributions, which we also imported in the previous step. Importantly, we will intentionally add some relationships mixed with noise in the random generation process for the sake of illustrating the process of testing and finding significant results:

np.random.seed(42)
n_employees = 100

# Target variables
attrition = np.random.choice(['Yes', 'No'], n_employees)
satisfaction = np.random.uniform(1, 10, n_employees)

# The generation is partly random, but it also intentionally adds
# some relationships to the targets for later testing
engagement = np.where(attrition == 'Yes', 
                      np.random.normal(50, 10, n_employees), 
                      np.random.normal(75, 10, n_employees))
tenure = (satisfaction * 1.5) + np.random.normal(0, 2, n_employees)
tenure = np.clip(tenure, 0.5, 20)

# Other variables will be generated as purely random
data = {
    'Age': np.random.randint(22, 60, n_employees), 
    'Tenure_Years': tenure,                        
    'Commute_Distance_Miles': np.random.uniform(1, 40, n_employees),
    'Training_Hours': np.random.randint(5, 50, n_employees),         
    'Engagement_Score': engagement,                
    'Satisfaction_Rating': satisfaction,           
    'Attrition': attrition                         
}
df = pd.DataFrame(data)

Next, we establish our analysis targets: one categorical variable and one continuous numerical variable. The threshold for statistical significance in the tests will be 0.05, a commonly used significance level:

categorical_target = 'Attrition'
continuous_target = 'Satisfaction_Rating'
alpha = 0.05

We want to print only results that are statistically significant, meaning results with p-values below 0.05. Thus, we create an empty list that will store them:

significant_results = []

Here comes the main part. Through an overarching loop, we iterate through all non-target variables in the dataset. For each variable, we apply the two specified tests: one against the categorical variable (t-test) and one against the numerical variable (Pearson correlation). If either test returns a statistically significant result, the information is stored in significant_results to be shown later. It’s simpler than it looks:

for column in df.columns:
    
    # Skipping the targets themselves
    if column in [categorical_target, continuous_target]:
        continue
        
    # --- TEST A: T-Test against the Categorical Target (Attrition) ---
    # Split the continuous variable into two groups based on Attrition
    group_yes = df[df[categorical_target] == 'Yes'][column]
    group_no = df[df[categorical_target] == 'No'][column]
    
    # Running independent t-tests
    t_stat, t_p_value = stats.ttest_ind(group_yes, group_no, equal_var=False)
    
    if t_p_value < alpha:
        significant_results.append({
            'Variable': column,
            'Test_Type': 'T-Test (Attrition)',
            'Statistic': round(t_stat, 3),
            'P-Value': round(t_p_value, 4),
            'Flag': 'Significant Difference'
        })
        
    # --- TEST B: Pearson Correlation against the Continuous Target (Satisfaction) ---
    # Running Pearson correlation
    corr_stat, corr_p_value = stats.pearsonr(df[column], df[continuous_target])
    
    if corr_p_value < alpha:
        significant_results.append({
            'Variable': column,
            'Test_Type': 'Correlation (Satisfaction)',
            'Statistic': round(corr_stat, 3),
            'P-Value': round(corr_p_value, 4),
            'Flag': 'Significant Correlation'
        })

Finally, we place the captured results into a DataFrame for easier printing and display the findings, if any, sorted by p-value:

results_df = pd.DataFrame(significant_results)

# Displaying the findings, sorted by p-value
if not results_df.empty:
    print(results_df.sort_values(by='P-Value').to_string(index=False))
else:
    print("No statistically significant results found.")

Here is an example of the results obtained. These may change for you if you remove the random seed, as the initial dataset has been generated at random:

        Variable                  Test_Type  Statistic  P-Value                    Flag
    Tenure_Years Correlation (Satisfaction)      0.915   0.0000 Significant Correlation
Engagement_Score         T-Test (Attrition)    -12.304   0.0000  Significant Difference
  Training_Hours Correlation (Satisfaction)     -0.268   0.0069 Significant Correlation

Is this script reusable? Leaving the synthetic data generation part aside, yes. This code is highly reusable and can be applied to future versions of an HR employee dataset like this or to a different dataset or analysis question with minimal changes.

Analysis of Results and Closing Remarks

This article showed how to implement a simple but effective Python script that iterates through a dataset containing multiple variables and applies statistical significance and correlation tests, identifying statistically significant results for any variables on the fly.