Most real-world datasets contain outliers—data points that are drastically different from the rest—due to measurement errors, incorrect data entry, or super rare events. But they can skew statistical analyses and lead to inaccurate and unreliable results. Identifying and handling these outliers is, therefore, important for effective data analysis.

This comprehensive tutorial goes over tips for detecting and managing outliers in real-world datasets. We’ll cover the following:

• Understanding the context and domain of the data
• Visualizing data distributions
• Applying statistical methods
• Considering data transformations
• Handling outliers

By the end of this tutorial, you’ll have learned enough to apply these techniques in your data analysis workflow. You can follow along with the Google Colab notebook. Let’s get started.

1. Understand the Context and Domain

Outlier detection is not a one-size-fits-all process. The effectiveness of identifying outliers largely depends on the context and domain of the dataset you’re working with. Here’s how domain knowledge and contextual factors can influence outlier detection.

Domain Knowledge

When you’re working with a dataset, you should first have the domain knowledge—understanding and expertise in the specific area or industry related to the data. This knowledge helps in defining what constitutes an outlier. Because anomalies in one field might be regular data points in another.

For example, a financial analyst might consider a 5% change in stock prices as normal, whereas in weather data, a 5-degree temperature change could be significant. With sufficient domain knowledge, you can set more accurate thresholds for outlier detection based on historical data and experience.

Let’s consider a sample dataset from a hospital containing patients’ blood glucose levels. A domain expert (such as a doctor) knows that blood glucose levels typically range between 70 and 140 mg/dL for a healthy individual:

import pandas as pd

# Sample data
data = {'patient_id': [1, 2, 3, 4, 5],
    	'glucose_level': [85, 92, 130, 200, 75]}
df = pd.DataFrame(data)

# Use domain knowledge to define outlier threshold 
lower_threshold = 70
upper_threshold = 140

# Identify outliers
df['is_outlier'] = df['glucose_level'].apply(lambda x: x  upper_threshold)
print(df)

Here, the knowledge of permissible glucose levels identify patients with levels outside the normal range:

   patient_id  glucose_level  is_outlier
0           1             85       False
1           2             92       False
2           3            130       False
3           4            200        True
4           5             75       False

Contextual Factors

Contextual factors are external influences that can affect the data. These include seasonal trends, economic conditions, special events, and the like.

• Seasonality: Data might have seasonal patterns. For instance, retail sales typically spike during holidays.
• Economic factors: Economic downturns can lead to unusual patterns in financial data.
• Special events: Events like natural disasters can cause significant deviations in normally stable data.

Consider a sample sales dataset. Sales typically increase in December due to holiday shopping. But these are valid and aren’t necessarily outliers. So we mark December as a month with expected high sales:

import pandas as pd

# Sample data
data = {'month': ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'],
    	'sales': [200, 220, 230, 250, 260, 270, 300, 320, 310, 330, 350, 600]}
df = pd.DataFrame(data)

# Contextual knowledge: Define December as a month with expected high sales
expected_high_sales_month = 'Dec'

# Identify outliers considering the context
df['is_outlier'] = df.apply(lambda row: row['sales'] > 500 and row['month'] != expected_high_sales_month, axis=1)
print(df)

Here’s the output where the December sales are not marked as outliers given the context:

   month  sales  is_outlier
0    Jan    200       False
1    Feb    220       False
2    Mar    230       False
3    Apr    250       False
4    May    260       False
5    Jun    270       False
6    Jul    300       False
7    Aug    320       False
8    Sep    310       False
9    Oct    330       False
10   Nov    350       False
11   Dec    600       False

2. Visualize the Data

Visualization is a powerful tool for identifying outliers in data. By visualizing the data, you can spot unusual values and patterns that might indicate the presence of outliers. Here are some common plots to visualize the data.

Box Plots

Box plots, or box-and-whisker plots, are another way to understand the distribution of data. They highlight the interquartile range (IQR), which helps in identifying data points that lie outside 1.5 times the IQR from the quartiles, typically considered outliers (coming up next!).

Box plots are also useful for comparing distributions across different categories. Let’s take an example:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Sample data
data = {'category': ['A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B'],
    	'values': [10, 12, 14, 15, 30, 10, 11, 13, 13, 50]}
df = pd.DataFrame(data)

# Create a box plot
sns.boxplot(x='category', y='values', data=df)
plt.title('Box Plot of Values by Category')
plt.show()

In this example, the box plot helps identify outliers in each category. The data points lying outside the “whiskers” are considered potential outliers:

Scatter Plots

Scatter plots are great for visualizing values for two variables for a set of data. They are useful for examining relationships between variables and identifying outliers in multivariate data.

Here’s an example:

import pandas as pd
import matplotlib.pyplot as plt

# Sample data
data = {'x': [2, 4, 6, 8, 10, 12, 14, 16, 18, 20],
    	'y': [4, 8, 12, 16, 20, 24, 28, 32, 36, 63]}
df = pd.DataFrame(data)

# Create a scatter plot
plt.scatter(df['x'], df['y'])
plt.title('Scatter Plot of X vs Y')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()

The scatter plot shows a linear relationship between x and y, except for the last data point, which stands out as an outlier:

Histograms

Histograms help understand the distribution of a variable by dividing the data into bins—showing the frequency of data points in each bin.

They help in understanding the distribution of data, including skewness, modality, and the presence of outliers. Histograms are especially helpful for visualizing the frequency of different ranges of values in the dataset.

The histogram shows the frequency distribution of the values, highlighting the outlier at the far right end of the distribution:

import pandas as pd
import matplotlib.pyplot as plt

# Sample data
data = {'values': [1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5, 20]}
df = pd.DataFrame(data)

# Create a histogram
plt.hist(df['values'], bins=10, edgecolor='black')
plt.title('Histogram of Values')
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.show()

The histogram shows the frequency distribution of the values with a possible outlier at the far right end of the distribution:

Visualizations such as box plots, scatter plots, and histograms, therefore, provide intuitive and immediate insights into the data, allowing you to identify unusual values and patterns that might indicate outliers.

3. Use Statistical Methods

Statistical methods provide a quantitative approach to identifying outliers. These methods are based on the statistical properties of the data and help in setting objective criteria for detecting anomalies. Here are two commonly used statistical methods for outlier detection.

Z-Score

The Z-score measures of how far away from the mean—in terms of the standard deviations—a data point is.

Mathematically, it’s given by:

where X is the data point, μ is the mean of the data, and σ is the standard deviation.

A Z-score outside the range [-3,3] for a point indicates that it’s likely an outlier, as it’s more than three standard deviations away from the mean.

Let’s take an example:

import numpy as np
import pandas as pd

# Sample data
data = {'values': [10, 12, 13, 15, 18, 19, 20, 22, 25, 100]}
df = pd.DataFrame(data)

# Calculate Z-scores
df['z_score'] = (df['values'] - df['values'].mean()) / df['values'].std()

# Identify outliers
threshold = 3
df['is_outlier'] = df['z_score'].abs() > threshold
print(df)

In this example, the data point 100 has a Z-score greater than 3, indicating it is likely an outlier:

   values   z_score  is_outlier
0      10 -0.578386       False
1      12 -0.503271       False
2      13 -0.465714       False
3      15 -0.390599       False
4      18 -0.277926       False
5      19 -0.240368       False
6      20 -0.202811       False
7      22 -0.127696       False
8      25 -0.015023       False
9     100  2.801794       False

Interquartile Range (IQR)

The Interquartile Range (IQR) also measures dispersion and is the difference between the third quartile (Q3) and the first quartile (Q1).

Mathematically, the interquartile range (IQR) is given by:

IQR = Q3 – Q1

Where Q1 and Q3 are the first and third quartiles, respectively.

Outliers generally are data points below Q1 – 1.5 * IQR or above Q3 + 1.5 * IQR. So the IQR helps identify outliers based on the spread of the data.

Here’s an example:

import pandas as pd

# Sample data
data = {'values': [10, 12, 13, 15, 18, 19, 20, 22, 25, 100]}
df = pd.DataFrame(data)

# Calculate Q1, Q3, and IQR
Q1 = df['values'].quantile(0.25)
Q3 = df['values'].quantile(0.75)
IQR = Q3 - Q1

# Identify outliers
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df['is_outlier'] = (df['values']  upper_bound)
print(df)

In this particular example, 100 lies outside the bounds defined by the IQR, marking it as an outlier:

   values  is_outlier
0  	10   	False
1  	12   	False
2  	13   	False
3  	15   	False
4  	18   	False
5  	19   	False
6  	20   	False
7  	22   	False
8  	25   	False
9 	100    	True

As seen, statistical methods like Z-score and IQR provide a systematic way to detect outliers.

4. Transform Your Data

You can use transformations such as log and Box-Cox transformations to help stabilize variance and normalize data distributions, especially when the data points are spread out over a wide range.

While these transformations are not primarily designed for outlier detection, they can indirectly assist in identifying and handling outliers by making the data more suitable to standard statistical techniques.

Log Transformation

Log transformation is applying logarithm to each data point. This transformation is particularly useful for skewed data, as it compresses the range of the data and reduces the impact of large values.

This helps make skewed data more symmetric, which can improve the performance of outlier detection methods that assume normally distributed data.

Here’s an example:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Sample data
data = {'values': [1, 2, 5, 10, 50, 100, 200, 500, 7000, 15000]}
df = pd.DataFrame(data)

# Apply log transformation
df['log_values'] = np.log(df['values'])

# Plot original vs. log-transformed data
plt.figure(figsize=(10, 5))

plt.subplot(1, 2, 1)
plt.hist(df['values'], bins=10, edgecolor='black')
plt.title('Original Data')

plt.subplot(1, 2, 2)
plt.hist(df['log_values'], bins=10, edgecolor='black')
plt.title('Log-Transformed Data')

plt.show()

This is the histogram of the log-transformed data:

Box-Cox Transformation

Box-Cox Transformation is a family of power transformations that can make non-normal data more normal. The transformation is defined as:

where X is the original data, Y is the transformed data, and λ is the transformation parameter.

The parameter λ is estimated to achieve the best transformation, making the data as close to normal as possible. which is beneficial for outlier detection methods that rely on normality assumptions.

While log and Box-Cox transformations are not direct methods for outlier detection, they help in making data more suitable for analysis by stabilizing variance and normalizing distributions.

5. Handle Outliers Appropriately

Once outliers are identified, the next step is to handle them appropriately based on their nature and impact on the analysis. Here are three common strategies for handling outliers: removal, capping, and imputation.

Removing Outliers

Removing outliers involves deleting data points that are errors or irrelevant to the analysis. This method is suitable when outliers are due to data entry errors or measurement anomalies. And it ensures that the remaining data is representative and accurate for analysis.

Consider a dataset of students’ test scores, where a few scores are incorrect due to known data entry errors:

import pandas as pd

# Sample data
data = {'student_id': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    	'test_score': [85, 90, 95, 100, 75, 89, 100, 105, 110, 300]}
df = pd.DataFrame(data)

# Define a threshold to identify outliers (test scores above 150 are considered errors)
threshold = 150

# Remove outliers
df_cleaned = df[df['test_score'] <= threshold]
print(df_cleaned)

This outputs:

   student_id  test_score
0           1          85
1           2          90
2           3          95
3           4         100
4           5          75
5           6          80
6           7         100
7           8         105
8           9         110

Winsorization

Winsorization involves replacing the extreme values beyond a certain percentile with the values at that percentile. It sets the extreme values to the closest value within the specified limits.

For example, if you Winsorize data at the 5th and 95th percentiles, values below the 5th percentile are set to the 5th percentile value, and values above the 95th percentile are set to the 95th percentile value. The code snippet below shows how to do it:

import pandas as pd
import numpy as np

# Sample data
data = {'values': [10, 12, 13, 15, 18, 19, 20, 22, 25, 100]}
df = pd.DataFrame(data)

# Define the Winsorization percentiles (e.g., 5th and 95th percentiles)
lower_percentile = 5
upper_percentile = 95

# Calculate the percentile values
lower_value = df['values'].quantile(lower_percentile / 100)
upper_value = df['values'].quantile(upper_percentile / 100)

print(f"lower value: {lower_value:.2f}")
print(f"upper value: {upper_value:.2f}")

# Winsorize the extreme values
df['winsorized_values'] = df['values'].apply(lambda x: max(lower_value, min(x, upper_value)))
print(df)

This is the output:

lower value: 10.90
upper value: 66.25
   values  winsorized_values
0      10              10.90
1      12              12.00
2      13              13.00
3      15              15.00
4      18              18.00
5      19              19.00
6      20              20.00
7      22              22.00
8      25              25.00
9     100              66.25

Imputing Outliers

You can also impute outliers with more plausible values, such as the mean or median of the data. This reduces the impact of outliers by replacing them with values that are more representative of the dataset.

Consider a dataset of ages where a few values are extremely high due to errors:

import pandas as pd

# Sample data
data = {'person_id': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    	'age': [25, 30, 35, 40, 28, 32, 36, 38, 150, 200]}
df = pd.DataFrame(data)

# Define a threshold to identify outliers (ages above 100 are considered errors)
threshold = 100

# Calculate the median age based on valid values
median_age = df[df['age'] <= threshold]['age'].median()

# Impute outliers with the median age
df['age_imputed'] = df['age'].apply(lambda x: x if x <= threshold else median_age)
print(df)

This should give you the following output:

   person_id  age  age_imputed
0          1   25         25.0
1          2   30         30.0
2          3   35         35.0
3          4   40         40.0
4          5   28         28.0
5          6   32         32.0
6          7   36         36.0
7          8   38         38.0
8          9  150         33.5
9         10  200         33.5

Summary

For accurate data analysis, detecting and handling outliers is necessary. You can use domain knowledge, context, and visualizations to help identify outliers.

Once you detect outliers, you can remove erroneous entries or irrelevant outliers, apply Winsorization to mitigate the impact of extreme values without losing data points, and impute outliers with more plausible values to preserve data completeness.

By applying these techniques and strategies, you can effectively manage outliers in real-world datasets. Happy outlier detection!