Image by Editor

Introduction

In my years of experience working with statistical testing, one of the most critical aspects of any experiment is the sample size. The size of your sample directly impacts the reliability and accuracy of your results. A larger sample size generally provides more accurate estimates, while a smaller sample size can lead to greater variability and potential errors.

This article explores the impact of sample size on hypothesis testing. Specifically, we will simulate the same statistical effect (e.g. comparing the means of two groups) with different sample sizes. The goal is to observe how the results change and interpret the findings across varying sample sizes. This type of analysis is fundamental in fields like clinical trials, A/B testing, and social science experiments, where resource limitations often require a trade-off between sample size and statistical power.

Why Sample Size Matters

When conducting hypothesis testing, the null hypothesis typically assumes that there is no significant difference between groups or treatments. However, due to random sampling variability, even if there is a true effect, small sample sizes may fail to detect it, a phenomenon known as a Type II error (false negative). Conversely, very large samples can detect even the smallest differences. This can lead to results that are statistically significant but not practically meaningful, and it underscores the importance of considering effect size alongside the p-value.

In this experiment, we will use multiple sample sizes (from small to large) to test the same statistical effect and evaluate how the p-value and confidence intervals change as the sample size increases. This experiment will also give us insights into the statistical power of hypothesis testing and why choosing an appropriate sample size is crucial when designing experiments.

The Experiment Setup

In this experiment, the effect we are testing is the difference between the means of two groups. Specifically, we’ll simulate two groups where we assume one group has a treatment (e.g. a new drug or intervention) and the other group has a placebo or control. We will then check if the mean of the treatment group differs significantly from the control group. This difference in means is the “effect” we’re trying to detect.

In real-world applications, this could involve comparing the average recovery time between patients treated with a new drug and those given a placebo or comparing test scores between two groups using different teaching methods.

Hypothesis Testing Framework

Before running the experiment, let’s quickly review the concept of hypothesis testing.

• Null Hypothesis (H₀): This is the hypothesis that there is no effect or no difference between the groups. In our case, the null hypothesis would state that the mean difference between the groups is zero.
• Alternative Hypothesis (H₁): This is the hypothesis we want to test. It asserts that there is a significant difference between the groups. For example, it could state that the mean difference is not zero.

We use a t-test (or Z-test) to compare the means and calculate the p-value, which tells us the probability of observing the data if the null hypothesis were true.

• If the p-value is below a pre-set significance level (typically 0.05), we reject the null hypothesis and conclude that there is a statistically significant difference between the groups.
• If the p-value is greater than 0.05, we fail to reject the null hypothesis and conclude that there is no significant difference.

Choosing the Sample Sizes

To see how sample size affects the experiment’s results, we will simulate tests using a range of sample sizes. Here’s the breakdown:

• Small sample size: 10
• Medium sample size: 50
• Large sample size: 100
• Very large sample size: 1000

By running the experiment with different sample sizes, we will observe how the Type I (false positive) and Type II (false negative) errors are impacted. A small sample size might lead to higher Type II errors, while very large sample sizes might result in significant but trivial differences (i.e. detecting an effect that is too small to matter).

Tools Used

• Python: For data generation, simulation, and statistical testing
• SciPy: To perform statistical tests (e.g. t-test and Z-test) and calculate p-values
• NumPy: To generate random data and perform mathematical operations
• Matplotlib / Seaborn: For visualizing results

We’ll simulate the data using NumPy’s normal distribution to create random values for the two groups. Then, we’ll use SciPy’s t-test to compare the means.

Running the Experiment

We will simulate the data for two groups with synthetic data generated from a normal distribution. The treatment group will have a mean of 10, and the control group will have a mean of 8. We will assume that the standard deviation for both groups is 2, so the data will be drawn from normal distributions with different means.

Here’s the code to generate the data:

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Define parameters
population_size = 100000
mean_treatment = 10
mean_control = 8
std_dev = 2

# Simulate data for treatment and control groups
treatment_group = np.random.normal(mean_treatment, std_dev, population_size)
control_group = np.random.normal(mean_control, std_dev, population_size)

# Visualize the generated data (optional)
import matplotlib.pyplot as plt
plt.hist(treatment_group, bins=30, alpha=0.5, label="Treatment Group")
plt.hist(control_group, bins=30, alpha=0.5, label="Control Group")
plt.legend()
plt.show()

Image by Author

This code generates 100,000 random data points for each group, assuming normal distributions with different means and the same standard deviation. The resulting histogram will show how the two groups overlap or differ.

Code Walkthrough:

• Generating Data: We used np.random.normal() to generate data for the treatment and control groups. The mean and standard deviation are specified, and the size of each population is set to 100,000.
• Visualization: We use Matplotlib to generate histograms for both groups. The alpha parameter controls transparency so that we can see the overlap of the two distributions.

Simulating Multiple Trials

To get a reliable measure of statistical power, we will run the experiment multiple times for each sample size. By repeating the simulation, we can average the p-values and assess the statistical significance across different sample sizes. We will also calculate the statistical power: the probability that the test correctly rejects the null hypothesis when it is false.

Here’s the code to run multiple trials and calculate the p-value for each sample size:

from scipy.stats import ttest_ind

# Function to simulate the test and return p-value
def run_experiment(sample_size, num_trials=100):
    p_values = []
    for _ in range(num_trials):
        # Sample random data for each group
        treatment_sample = np.random.choice(treatment_group, sample_size, replace=False)
        control_sample = np.random.choice(control_group, sample_size, replace=False)
        
        # Perform t-test
        t_stat, p_value = ttest_ind(treatment_sample, control_sample)
        p_values.append(p_value)
    
    # Calculate the average p-value
    avg_p_value = np.mean(p_values)
    return avg_p_value

# Simulate experiments with different sample sizes
sample_sizes = [10, 50, 100, 1000]
results = {}

for size in sample_sizes:
    avg_p_value = run_experiment(size)
    results[size] = avg_p_value

print("Average p-values for each sample size:", results)

Code Explanation:

• Simulating the Test: For each sample size, we randomly sample from the treatment and control populations and perform a t-test using ttest_ind() from SciPy.
• Multiple Trials: We run the test 100 times for each sample size to get an average p-value, which helps smooth out any randomness in the results.
• Expected Outcome: As the sample size increases, the p-value is expected to become smaller and more consistently indicate a significant difference, reflecting the true effect in the data.

Analyzing the Results

We can now analyze the results of our experiment based on the p-values we calculated for each sample size. Below are the average p-values for different sample sizes:
Average p-values for each sample size:

Image by Author

• Sample size 10: 0.1078
• Sample size 50: 0.0004679
• Sample size 100: 3.7181e-08
• Sample size 1000: 1.0915e-84

Here’s what we can infer from these results:

• For sample size 10, the p-value is 0.1078, which is above the typical significance threshold of 0.05. This means we fail to reject the null hypothesis and cannot claim a significant difference between the treatment and control groups. This is a Type II error (false negative), as the sample size is too small to detect the true effect.
• As we increase the sample size to 50, the p-value drops to 0.0004679, which is well below 0.05. This means we reject the null hypothesis and find a statistically significant difference between the two groups. While this result is highly significant, with smaller sample sizes, there is greater variability in the p-value across repeated experiments compared to larger samples.
• With sample size 100, the p-value is even smaller (3.7181e-08), and with sample size 1000, the p-value drops drastically to 1.0915e-84. These extremely small p-values indicate that the larger the sample size, the more likely we are to detect a true effect. Larger sample sizes help reduce the variance and bias in our estimates, leading to more stable and reliable results.

Visualization

To better visualize the impact of sample size on statistical significance, we can plot p-values for each sample size. This helps us understand how the p-value decreases as the sample size increases and how the experiment stabilizes over time.

Here’s the code to create a line plot of p-values across different sample sizes:

import matplotlib.pyplot as plt

# Sample sizes and corresponding average p-values
sample_sizes = [10, 50, 100, 1000]
p_values = [0.1078, 0.0004679, 3.7181e-08, 1.0915e-84]

# Plotting the p-values as a line plot
plt.figure(figsize=(8, 6))
plt.plot(sample_sizes, p_values, marker='o', color='b', linestyle='-', linewidth=2, markersize=8)
plt.yscale('log')  # Log scale for better visualization of small p-values
plt.title('Effect of Sample Size on P-Value')
plt.xlabel('Sample Size')
plt.ylabel('P-Value (Log Scale)')
plt.grid(True)

plt.show()

Image by Author

This plot shows how the p-value sharply decreases as the sample size increases, particularly from 10 to 1000. Notice the log scale on the y-axis to better visualize the drastic drop in p-values for larger samples.

Statistical Significance

From the results and the plot, we see a clear trend: smaller sample sizes lead to higher variability in the results, while larger sample sizes stabilize the results around the true effect. Smaller samples are more prone to Type II errors, where we fail to detect a significant difference even when one exists.

With larger samples, the test becomes more sensitive and precise. This is why statistical power increases with sample size; the test becomes more likely to detect a true effect and less likely to produce a false negative.

Visualizing the Data: Sample Size vs. P-Value

A scatter plot or line plot showing how p-values or test statistics change as sample size increases clearly visualizes the impact of sample size on statistical significance and test power.

Below is the code to create a scatter plot that shows the relationship between sample size and the p-value for each sample size:

# Plotting the p-values with sample sizes
plt.figure(figsize=(8, 6))
plt.scatter(sample_sizes, p_values, color='green', s=100, edgecolor='black', alpha=0.7)
plt.yscale('log')
plt.title('Sample Size vs. P-Value')
plt.xlabel('Sample Size')
plt.ylabel('P-Value (Log Scale)')
plt.grid(True)
plt.show()

Image by Author

This scatter plot shows the data points for each sample size, highlighting how larger sample sizes lead to smaller p-values, indicating more reliable results. The logarithmic scale will help in visualizing the dramatic changes in p-value as the sample size grows.

Power Analysis

In statistical hypothesis testing, power analysis is used to determine the probability that a test will correctly reject the null hypothesis when it is false. Power increases with sample size, meaning that larger samples are more likely to detect a true effect when one exists.

In our case, the statistical power of the test increases dramatically as we move from a sample size of 10 to 1000. Larger samples give us more precise estimates, making it easier to detect small effects that might be missed with small samples. Power analysis is important when designing experiments to ensure that your sample size is large enough to detect meaningful differences without wasting resources.

Wrap Up

From our experiment, we can conclude several key insights:

• Sample size is critical: As expected, larger sample sizes produce more reliable results. In the case of our hypothesis test, the p-value decreased dramatically as the sample size increased from 10 to 1000, indicating that larger samples lead to more stable and accurate results.
• Small sample sizes tend to increase variability and are more likely to lead to Type II errors, where we fail to detect a true effect. For instance, with a sample size of 10, the p-value was 0.1078, which is above the typical significance threshold of 0.05.
• Statistical power improves with sample size: Larger samples allow the test to correctly reject the null hypothesis when it is false and to detect even small effects. In our case, with a sample size of 1000, the p-value was so small (1.0915e-84) that we were confident in detecting a statistically significant difference.

When designing experiments, choosing the right sample size is a balance between statistical power and resource constraints. While larger samples give you more reliable results, they also require more time and resources to collect. Here’s a general approach:

• Small-scale experiments (e.g. pilot studies) can often start with smaller sample sizes (e.g. 10 or 50), but be mindful that statistical power may be limited.
• For larger, more robust experiments, especially when detecting subtle effects or working in fields like clinical trials or A/B testing, consider increasing the sample size to at least 100 or more.
• Power analysis tools can help determine the minimum sample size required to achieve a certain power level (usually 80% or higher). Tools like G*Power are helpful in planning experiments.