Image by Editor | ChatGPT

In this article, we simulate the law of large numbers to observe how sample averages converge toward the true mean of a population. By running repeated trials and tracking the average, we will see randomness dominate early on but diminish steadily as more samples accumulate.

By the end of the experiment, you will:

• See how sample averages stabilize as the number of trials grows
• Understand why variability decreases with larger sample sizes
• Gain a practical sense of why the LLN underpins so many statistical and machine learning techniques

The Law of Large Numbers Explained

At its core, the law of large numbers states that as the number of independent, identical trials increases, the sample mean approaches the population mean. For example, flipping a fair coin a few times may give you a lopsided ratio of heads to tails, but as the number of flips grows, the ratio gets closer to 50%.

There are two main versions of the LLN:

• Weak Law of Large Numbers: As the sample size grows, the probability that the sample mean is close to the population mean approaches 1.
• Strong Law of Large Numbers: With probability 1, the sample mean converges exactly to the population mean as the number of trials goes to infinity.

The difference lies in how strictly convergence is defined, but both capture the same essential idea: averages get more reliable with scale.

This principle shows up everywhere. Casinos rely on it to guarantee profits despite short-term fluctuations at the tables. Pollsters use it to estimate public opinion with limited samples. In machine learning, the LLN justifies why large training datasets produce more stable estimates of model parameters.

Simulating the LLN makes what looks abstract look concrete, showing the convergence in action rather than relying only on theory.

Designing the Experiment

To demonstrate the law of large numbers, we need a source of randomness. A fair coin is the most intuitive choice, but to make the experiment richer, we will also use continuous distributions such as the normal and exponential. Each provides a slightly different perspective:

• Coin flips (Bernoulli distribution): The true mean is 0.5, representing the probability of heads.
• Normal distribution: We can generate values with a mean of 50 and a standard deviation of 10. The true mean is 50.
• Exponential distribution: This distribution is skewed, which slows convergence but still guarantees it. The true mean equals the reciprocal of the rate parameter.

The procedure is straightforward:

• Generate a long sequence of random values from the chosen distribution
• Compute the running average at each step, from the first observation up to the final one
• Compare the running averages with the true population mean
• Repeat the process across different distributions to see whether convergence behaves consistently

Expectations:

• With small sample sizes (10 or 100), the running mean will fluctuate heavily
• As the number of trials grows (1,000 or 10,000), the running mean will settle near the true mean
• The fluctuations will shrink as the sample size increases, illustrating the stability that the law of large numbers predicts

Running the Simulation

Before starting, ensure the necessary Python libraries—NumPy, Matplotlib, and Seaborn—are installed:

pip install numpy matplotlib seaborn

Now let’s implement the simulation.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Optional: set a seed for reproducibility
np.random.seed(42)

# Set style
sns.set(style="whitegrid")

# Function to simulate and plot LLN
def simulate_lln(distribution, params, n_trials=10000, true_mean=None, title=""):
    """
    Simulate the Law of Large Numbers for a given distribution.
    
    distribution: np.random function (e.g., np.random.normal)
    params: dictionary of parameters for the distribution
    n_trials: number of random draws
    true_mean: the population mean (if known; otherwise estimated from the sample)
    title: plot title
    """
    # Step 1: Draw samples
    samples = distribution(size=n_trials, **params)
    
    # Step 2: Compute running means
    running_means = np.cumsum(samples) / np.arange(1, n_trials + 1)
    
    # Step 3: Set/estimate true mean
    if true_mean is None:
        true_mean = np.mean(samples)
    
    # Step 4: Plot results
    plt.figure(figsize=(10, 6))
    plt.plot(running_means, label="Running Mean", color="blue", alpha=0.7)
    plt.axhline(true_mean, color="red", linestyle="--", label=f"True Mean = {true_mean:.2f}")
    plt.title(f"Law of Large Numbers Simulation: {title}")
    plt.xlabel("Number of Trials")
    plt.ylabel("Sample Mean")
    plt.legend()
    plt.show()

# Coin flips (Bernoulli with p=0.5)
simulate_lln(
    distribution=np.random.binomial,
    params={"n": 1, "p": 0.5},
    n_trials=10000,
    true_mean=0.5,
    title="Fair Coin Flips"
)

# Normal distribution (mean=50, std=10)
simulate_lln(
    distribution=np.random.normal,
    params={"loc": 50, "scale": 10},
    n_trials=10000,
    true_mean=50,
    title="Normal Distribution (μ=50, σ=10)"
)

# Exponential distribution (λ=1 ⇒ mean=1; in NumPy, use scale=1/λ)
simulate_lln(
    distribution=np.random.exponential,
    params={"scale": 1.0},
    n_trials=10000,
    true_mean=1.0,
    title="Exponential Distribution (λ=1)"
)

Explanation of the code

• simulate_lln function: Takes a distribution, generates random values, computes the cumulative average, and plots the result
• Coin flips: Modeled with a binomial distribution (n=1, p=0.5). The running mean converges to 0.5
• Normal distribution: The sample means converge to 50, with smaller fluctuations at larger sample sizes
• Exponential distribution: Shows convergence but with more noise in early stages because of skewness

Computational Considerations

Running 10,000 trials is fast on most machines. Scaling to millions of trials is possible, but memory-intensive if you store every running mean. To handle larger scales, you can compute averages in streaming mode without saving the entire sequence or relying on parallel processing.

Law of Large Numbers Simulation-Fair-Coin-Flips

Law of Large Numbers Simulation-Normal-Distribution

Law of Large Numbers Simulation-Exponential-Distribution

Visualization Explanation

After running the simulation across the three distributions, we can examine the plots and interpret what they reveal about the law of large numbers in practice.

• Fair coin flips: At the beginning, the running mean swings widely: with only a few flips, results such as 0 heads out of 2 flips (mean = 0) or 3 heads out of 4 flips (mean = 0.75) are common. However, as the number of flips grows into the thousands, the average steadily narrows toward 0.5. The red reference line at 0.5 highlights how close the running mean becomes once enough trials accumulate.
• Normal distribution (mean = 50): Here the convergence is smoother because the distribution is symmetric and less prone to extreme values. The running mean begins with fluctuations around 50 but stabilizes quickly. By the time we reach 10,000 draws, the sample mean sits almost exactly on the true population mean.
• Exponential distribution (mean = 1): The exponential case highlights the role of skewness. The early running means fluctuate more because exponential draws occasionally produce large outliers. Despite this, the convergence is still clear: with enough samples, the average levels out near 1, showing that LLN holds even with skewed data.

Variability and Convergence

• Variance of running means: Across all three cases, variability is high for small sample sizes but shrinks as the number of trials increases
• Speed of convergence: Symmetric distributions (like normal) stabilize faster than skewed ones (like exponential)
• Bias: None of the simulations showed systematic bias. The sample means approached the true means as predicted

The visualizations make the abstract idea of convergence tangible. You can watch randomness lose influence as more samples are added.

Key Insights

The simulation confirms several important points about the law of large numbers:

• Convergence is real but not immediate: In all distributions, small sample sizes produced unstable averages. This demonstrates why relying on just a handful of trials can be misleading. The law does not erase randomness; it ensures that randomness averages out with enough repetition.
• Distribution shape affects the journey: The normal distribution’s symmetry made convergence smooth and fast. The exponential distribution, with its long tail, converged more slowly and showed more noise in the early trials. This highlights that while LLN always holds, the path to convergence depends on the underlying distribution.
• Practical lesson: scale matters: Whether you are flipping coins, analyzing customer transactions, or estimating model accuracy, the number of observations directly controls how much you can trust the average. The simulation makes it clear that stability comes only with sufficient scale.

Takeaways for Practice

• Never assume that small samples tell the whole story; instability is the norm at low counts
• Expect convergence patterns to differ by distribution, especially with skewed data
• Use LLN as a reminder that larger datasets not only provide more information but also more reliable estimates

Limitations of the Experiment

Although the simulation illustrates the law of large numbers effectively, it comes with limitations. The results depend on specific assumptions:

• Independent and identical samples: LLN assumes each draw is independent and from the same distribution. If data are correlated or biased, convergence may fail or take much longer.
• Dataset size and computation: We simulated up to 10,000 trials without issue. Scaling to millions of trials increases memory use and runtime, requiring optimized or streaming approaches.
• Illustration vs. proof: This experiment shows patterns but does not replace formal proofs. The visuals demonstrate convergence, but the mathematical guarantees extend far beyond what we simulated.

Practical Applications

The law of large numbers is more than a theoretical curiosity. It powers many of the tools and methods used in applied statistics and machine learning:

• Monte Carlo simulations: Financial modeling, physics experiments, and risk analysis all rely on large repeated trials. LLN ensures that the simulated averages approximate real expectations.
• Polls and surveys: Opinion polls draw small samples from large populations. LLN guarantees that with sufficient sample size, the sample mean is a good approximation of the true population mean.
• Machine learning: Training models requires estimating parameters from data. Larger datasets give more stable and reliable estimates because LLN reduces the variability in parameter estimates.

Conclusion

This experiment simulated the law of large numbers across coin flips, normal distributions, and exponential distributions. In every case, the running averages started unstable but gradually converged toward the true population mean. The rate of convergence depended on the distribution, with symmetric data stabilizing faster than skewed data.

The key lesson is that randomness dominates in the short run but fades in the long run. Averages become trustworthy only with scale, which is why the law of large numbers is so central in both theory and practice.

To extend this experiment, you could try biased coins, heavy-tailed distributions, or correlated samples to see how convergence behaves in less ideal conditions. You might also compare convergence rates quantitatively, measuring how variance shrinks as sample size increases.

By simulating the LLN, we make a foundational principle of probability visible. What begins as noise eventually gives way to stability, a reminder of why large samples matter in every field that depends on data.