Image by Author 

Machine learning is often seen as pure coding and model tuning, but behind every prediction is statistics. From understanding how your data behaves to validating model performance, statistical concepts form the foundation of every intelligent system. Whether you’re training a neural network or building a simple regression model, knowing the “why” behind your numbers is what separates a good machine learning practitioner from a great one. In this article, we’ll explore seven key statistical concepts that every machine learning developer needs to understand. Let’s get started.

1. P-Values & Hypothesis Testing

A p-value is the probability of observing data as extreme (or more extreme) than what you saw, assuming the null hypothesis is true. It is not the probability that the null hypothesis is true. In machine learning settings, you might see p-values when doing feature selection (e.g., in regression), A/B testing, or model comparisons. Some common misunderstandings and pitfalls are:

• A low p-value (for example, 0.01) does not mean the effect is “important” or “useful”; it only means that it’s unlikely under the null hypothesis
• P-values shrink with big samples: when datasets are large, we can detect even tiny but unimportant effects
• Too many tests cause false hits: checking lots of features increases the chance of getting small p-values just by luck
• Assumptions matter: if the data isn’t normal or independent, the p-value may not be trustworthy

In Practice (Machine Learning Use Cases)

• Feature engineering/regression: use p-values to guide feature inclusion, but also consider effect sizes, domain knowledge, and cross-validation
• In A/B tests: p-values often determine whether you “accept” or “reject” a change, but always consider confidence intervals and practical impact
• When comparing nested models: likelihood ratio tests (LRT) yield p-values for comparing model fits

Graphical Illustration

One of the tails of the bell curve is shaded in pink. This shaded area represents the p-value, the likelihood of observing a result as extreme as, or more extreme than, your data assuming the null hypothesis (H₀) is true.

The larger the shaded area → higher the p-value → less evidence against H₀. A small shaded area (e.g., < 0.05) → unlikely under H₀ → reject H₀.

2. Correlation vs Causation (and Nonlinearity)

Correlation indicates a statistical relationship — whether two variables change together. But it doesn’t reveal causation or whether a third factor is influencing both. In machine learning, models often use correlations to make predictions, but if your aim is understanding or intervening, caution is needed. The classic Pearson correlation (r) only captures linear relationships. Two variables can be strongly related (e.g., quadratic) but yield a near-zero r.

When This Matters in Machine Learning

• Feature selection: you might drop a feature because its linear correlation looks weak, but in fact, it has a nonlinear predictive relationship
• Model interpretation: claiming “X causes Y” because of correlation is risky; if you deploy a model under that assumption, interventions may fail when data shifts
• Causal inference: when you go beyond pure prediction and into “if I change X, what happens to Y?” you need causal frameworks (e.g., do-calculus, instruments, experiment design)

Graphical Illustration

Below are two scatter plots, which show that as temperature increases, both ice cream sales and drownings rise (positive correlations for each). The arrow labeled “Hidden Variable: Season” connects them to indicate that the underlying cause of both trends is the season rather than a direct causal relationship between the two variables.

Both variables correlate because of a third factor (a confounder), not direct causation, as correlation simply measures association, not influence.

3. Bias–Variance Tradeoff

Bias is the error that arises from overly simplistic assumptions in your model — occurring when it’s too basic to capture the actual patterns in the data — which is known as underfitting. Variance is the error that occurs when a model is too complex, learning not just the patterns but also the random noise in the training data — called overfitting. The trade-off between bias and variance is important: lowering bias usually increases variance and vice versa. The aim is to strike a balance to get the “sweet spot” of model complexity, where your model generalizes well to unseen data. Mathematically, the expected prediction error can be expressed as:
\[
\text{Total Error} = \text{Bias}^2  + \text{ Variance } + \text{ Irreducible Noise}
\]
This decomposition explains why we can’t eliminate both bias and variance completely, since improving one typically worsens the other.

Why This Matters in Machine Learning

• Overfitting vs. underfitting is one of the most central challenges in machine learning
• When tuning model complexity (e.g., depth of tree, degree of polynomial, regularization strength), you’re implicitly navigating the bias–variance tradeoff
• Regularization (L1, L2, dropout, early stopping) is a tool to control variance
• Cross-validation helps estimate generalization error, not just training error

Graphical Illustration

The following graph is a curve that shows model complexity on the x-axis and error on the y-axis. Bias (error due to overly simple assumptions) decreases as model complexity grows, and variance (error due to sensitivity to training data) increases with increasing model complexity. Its total error curve, the combination of both down and up, exhibits a U-shape. At the sweet spot of this tradeoff is where we have found an optimal model complexity and can identify a point between bias and variance that gives us the lowest total error and best generalization to new, unseen data.

4. Sampling, Estimation & the Central Limit Theorem (CLT)

Population vs. Sample: The population usually refers to the full set of possible observations, which is often unobservable. A sample, on the other hand, is what you actually observe. Sampling refers to the method by which you select data points (random sampling, stratified, convenience, etc.), and good sampling is critical to avoid bias.

Point Estimation: Point estimation uses sample data to approximate an unknown population parameter — for example, estimating the population mean (μ) with the sample mean (x̄). An estimator is called unbiased if its expected value equals the true parameter. In practice, some estimators may have a slight bias but lower variance, which can be preferable since it produces more stable and reliable estimates.

Central Limit Theorem (CLT): The CLT states that, given certain conditions, the distribution of sample means approaches a normal distribution as the sample size becomes large, regardless of the original data’s distribution. Its implications include:

• Even if raw data is skewed, sample means tend to be well behaved
• It enables the use of normal-based confidence intervals, p-values, etc., even when the data is not normal, as long as the sample size N (number of observations) is large enough. Generally, when N ≥ 30, the CLT approximation becomes quite reliable in many practical cases

In Machine Learning Settings

• Model evaluation metrics (e.g., average error) often rely on the CLT to approximate the sampling distribution of the metric
• In bootstrapping, repeated sampling leverages similar ideas to approximate uncertainty
• When splitting data into train/validation/test, you hope your sample represents the full population; poorly sampled splits lead to biased generalization estimates

Graphical Illustration

The graph below shows three histograms side by side. The first histogram represents skewed raw data, such as an income distribution, which is not symmetric. The second histogram shows the distribution of sample means when the sample size is small, which is still somewhat irregular. The third histogram shows the distribution of sample means when the sample size is large; it now appears smooth and bell-shaped, resembling a normal distribution. This helps us understand why averaging predictions, as done in ensemble methods such as bagging, tends to reduce variance and produce more stable models.

5. Probability Distributions & Likelihood

Role of distributions in machine learning: many models assume (or exploit) particular probability distributions. For instance:

• Gaussian (normal) distribution in linear regression or Gaussian noise models
• Bernoulli/Binomial for binary classification or logistic regression
• Poisson distribution for count data
• Exponential or Weibull for survival models

It’s important to understand a distribution’s shape, tails, variance, and skew when doing modeling.

Likelihood & maximum likelihood estimation (MLE): the likelihood is the data’s probability model viewed as a function of the parameters. Maximum likelihood estimation picks the parameter values that maximize this likelihood. Many machine learning methods (such as logistic regression, Naive Bayes, and Gaussian mixture models) are built on MLE or its variants, including regularized MLE and penalized likelihood.

Bayesian view (posterior, prior, etc.): in this approach, model parameters are treated as random variables rather than fixed values. Priors are assigned to these parameters, and observations are used to update them, producing a posterior distribution. The posterior is proportional to likelihood × prior. In large data regimes, the posterior often concentrates near the MLE, as the likelihood dominates the prior.

Where This Shows Up in Machine Learning

• Model fitting: e.g., logistic regression maximizes log-likelihood
• Probabilistic models: e.g., Gaussian mixture models, hidden Markov models, Bayesian networks
• Regularization: you can interpret L2 regularization as imposing a Gaussian prior on weights (in a Bayesian view)
• Generative vs. discriminative modeling: generative models define likelihoods over data + label; discriminative models focus on conditional likelihood

Graphical Illustration

Below, there is a curve showing how likely the observed data is for different parameter values θ. The peak of the curve is at θ̂ (theta-hat), the maximum likelihood estimate (MLE). Likelihood quantifies how compatible your data is with a parameter setting, and MLE chooses the parameter where this likelihood is highest.

6. Confidence Intervals & Uncertainty Quantification

A confidence interval gives a range of likely values for a parameter, such as a mean or difference between means. For example, a 95% CI means that if we repeated the experiment many times, about 95% of those intervals would contain the true value. It does not mean there’s a 95% chance that your single interval includes the truth.

Why This Matters in Machine Learning

• A single number (like accuracy) is not enough; you need to know how uncertain that value is
• In deployment, confidence helps you judge how trustworthy a model’s prediction is, especially in high-risk fields like medicine
• It’s key in A/B testing and model calibration
• It reminds you that high accuracy doesn’t always mean high certainty
• In active learning, models often focus on data points where they are least confident

Graphical Illustration

This graph shows multiple confidence intervals estimated from different samples. The dashed black line marks the true mean. Most intervals (green) include the true mean, while a few (red) miss it. This illustrates that with a 95% confidence level, about 95% of intervals from repeated samples will capture the true value, while roughly 5% will not.

7. VC Theory / Generalization Bounds

VC (Vapnik–Chervonenkis) theory explains how a learning algorithm performs on unseen data. The VC dimension measures how complex a model class is. A very complex model may memorize the training data, leading to overfitting, while a very simple model may miss important patterns, resulting in underfitting. With enough samples, the training error approaches the true error for many model classes. The generalization bounds in VC theory link three key aspects: the amount of data, the model’s complexity (VC dimension), and how well the model generalizes.

Why This Matters in Machine Learning

• Helps decide how complex a model should be
• Explains why regularization is needed
• Shows that too much flexibility leads to overfitting
• Clarifies why simpler models sometimes work better

Graphical Illustration

The graph below shows two curves. As model complexity increases, the training error keeps decreasing, while the test (generalization) error first drops and then rises again. The lowest point on the test error curve represents the optimal capacity zone. At this point, the model achieves the best balance between underfitting and overfitting.

Wrapping Up

The whole article focuses on how statistics lay the groundwork for machine learning. You cannot learn machine learning without learning statistics. Let’s take a look at the summary of what we discussed above:

Concept	Graph Shape / Feature	What It Teaches / Description	Key Machine Learning Connection
P-Value	Bell curve tail	Significance of results	A/B testing, regression
Correlation	Scatter plots + hidden factor	Correlation ≠ causation	Feature interpretation
Bias–Variance	U-shaped	Tradeoff between simplicity & flexibility	Model tuning
CLT	Histograms → bell shape	Sampling averages become normal	Evaluation, bagging
Likelihood	Peak curve	How parameters fit data	Model training
Confidence Interval	Error bars	Quantifying uncertainty	Metric reliability
VC Theory	Train/test error curves	Optimal model complexity	Generalization