Image by Editor

Outliers are a common issue in real-world data. Whether due to manual errors, external factors, or rare events, it’s not unusual for some observations to be unreasonably distant from the vast majority of other data points. Balancing statistical intuition with a light touch of data science, this article uses small experiments to show how outliers can impact the results of several statistical methods.

What Outliers Are In A Nutshell

An outlier is an observation that deviates markedly from the rest of your data. Although they’re often dismissed as “noise,” outliers can disproportionately influence statistical analyses or even the training of machine learning models rooted in statistical principles. That said, they can also reveal important structural facts about a dataset. It all depends on the specific problem and the data at hand.

Let’s examine the potential impact of outliers through several common statistical methods, aided by Python code examples.

Mean vs. Median: Noticing Outlier Sensitivity Made Simple

Did you know the widely used arithmetic mean is particularly sensitive to extreme values (i.e., outliers), whereas the median is more robust? The following example illustrates this phenomenon. We’ll use NumPy for simple array math:

import numpy as np

data = np.array([10, 11, 12, 13, 14])
data_with_outlier = np.append(data, 100)

print("Mean:", np.mean(data), "Median:", np.median(data))
print("Mean w/ outlier:", np.mean(data_with_outlier),
      "Median w/ outlier:", np.median(data_with_outlier))

Mean: 12.0 Median: 12.0
Mean w/ outlier: 26.666666666666668 Median w/ outlier: 12.5

The second dataset is identical to the first except for one added value—an outlier of 100. Note how it severely shifts the mean, whereas the median remains largely unchanged. Takeaway: if a single observation drastically alters your summary statistic, the estimator you’re using may not be robust enough for that dataset.

Linear Regression: How A Single Point May Tilt The Whole Model

Traditional linear regression methods typically minimize squared errors (e.g., Ordinary Least Squares), which gives extreme points outsized leverage. Below, we fit models with and without an outlier using scikit-learn:

from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[1],[2],[3],[4],[5]])
y = np.array([2, 4, 6, 8, 10])

X_out = np.vstack([X, [[6]]])
y_out = np.append(y, 50)

model = LinearRegression().fit(X, y)
model_out = LinearRegression().fit(X_out, y_out)

print(model.coef_, model.intercept_)
print(model_out.coef_, model_out.intercept_)

Here’s a visualization of the two models:

 
How an outlier affects linear regression analysis
 

Notice how one point like (6, 50) noticeably affects both parameters of the simple linear regression model: the slope and the intercept. If outliers cannot be removed, consider robust alternatives such as Huber regression or RANSAC.

Correlation: Strong vs. Weak … (vs. Utterly Artificial)

Pearson correlation is often the go-to measure in statistical analyses (including pre-modeling checks), but it is highly sensitive to outliers—it can be inflated or deflated by a single extreme value. In such cases, rank-based measures like Spearman correlation are preferable. We’ll compute both using SciPy:

import numpy as np
from scipy.stats import pearsonr, spearmanr

x = np.array([1,2,3,4,5])
y = np.array([2,4,6,8,10])

x_out = np.append(x, 6)
y_out = np.append(y, 50)

print("Pearson:", pearsonr(x, y)[0])
print("Spearman:", spearmanr(x, y)[0])
print("Pearson w/ outlier:", pearsonr(x_out, y_out)[0])
print("Spearman w/ outlier:", spearmanr(x_out, y_out)[0])

Result:

Pearson: 1.0
Spearman: 1.0
Pearson w/ outlier: 0.7642652566278798
Spearman w/ outlier: 1.0

As you can see, Spearman remains robust in this toy example, while Pearson drops substantially once a single outlier is introduced.

Final Thoughts

Through these small, humble experiments, the big lesson is clear: many commonly used statistical tools and measures are fragile when data contain outliers. Modern machine learning and data science workflows demand robustness as much as accuracy. If the data in your next project contain outliers, choosing the right strategies to navigate them is an essential prerequisite for a trustworthy analysis.