Working with data and extracting value from it has become one of the most important professionals in the last years. However, to get worthy insights from data, we need to make sure the data is clear and represents a general pattern.
This is why when having data points that behave differently from the rest is usually considered a problem, as it can bias the overall behavior and lead to wrong insights. These abnormal points are referred to as outliers.
Outliers are datapoints within a dataset that present a different behavior.
To make this idea easier to understand, picture a room full of balloons. Each balloon stands for a piece of data in a dataset. Most of the balloons are floating at the same level, but two of them are much higher than the others.
What can be the main reason for this difference?
If we dig a little bit, we might discover that these two balloons that float higher are filled with helium. This is what we call an outlier.
In this article, I aim to easily explain several methods to efficiently identify and remove outliers from your data.
What Are Outliers?
Outliers are data points that stand out because they are very different from the rest of the dataset, making them unrepresentative of the overall data. While this definition sounds straightforward, spotting these abnormal data points is challenging. They can occur for various reasons, such as errors in data collection, inaccuracies in measurement, or even rare events.
Let’s first address a fundamental question:
Why do we need to detect outliers in a dataset?
There are two main reasons.
- Outliers’ distortion effect: Outliers can obscure meaningful patterns in data and distort the learning process of machine learning algorithms. For instance, in a dataset representing house prices based on various features, an unusually high price for a small, poorly located apartment could be an outlier, leading to biased predictions. Our main goal is to remove these outliers.
- Outliers detection as an objective: Outlier detection is the primary objective in many data science applications. In these situations, finding anomalies isn’t just part of getting the data ready—it’s the main goal of the analysis. Take fraud detection in the finance industry, for instance. Here, the aim is to spot unusual transaction patterns that suggest fraudulent activities. Our main goal is to determine these outliers.
Image by Author
In this introductory guide, I will focus on an algorithmic approach to detecting outliers using sci-kit learn. However, it’s important to note that there are three primary methods for identifying outliers:
- Graphical Approach: Detect outliers through data visualizations.
- Statistical Approach: Detect outliers through statistical analysis and probability distributions.
- Algorithmic Approach: Detect outliers using machine learning models.
Applying ML algorithms to detect outliers using Scikit-Learn
The Algorithmic Approach takes advantage the power of ML algorithms to detect outliers, addressing the limitations of simpler statistical methods.
There are many robust models for outlier detection. Today I will be discussing 4 of them:
- Isolation Forest
- Local Outlier Factor
- One-Class Support Vector Machine
- Elliptic Envelope
These models offer reliable techniques for identifying anomalies in complex and multidimensional datasets, as they excel at understanding intricate patterns in the data and defining accurate decision boundaries.
1. Isolation Forest
The first outlier detection algorithm is the Isolation Forest, a model built on the principles of the well-known Random Forest algorithm and Ensemble Learning techniques.
The core idea of the Isolation Forest is that anomalies are rare and often require fewer steps to isolate within a tree structure compared to normal data.
The Isolation Forest constructs an ensemble of decision trees, which isolate outliers more quickly due to their sparse nature. By measuring the average path length of data points in these trees, it effectively quantifies an anomaly score for each data point.
Implementing the Isolation Forest in Python is straightforward, when using the Scikit-Learn library.
# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.ensemble import IsolationForest
# Load the Iris dataset and select features
iris = load_iris(as_frame=True)
X = iris.data[['sepal length (cm)', 'sepal width (cm)']]
# Initialize the Isolation Forest model
model = IsolationForest(contamination=0.05, random_state=42)
# Fit the model to the data
model.fit(X)
# Compute the anomaly scores for each point
scores = model.decision_function(X)
# Identify outliers based on the anomaly scores
outliers = np.where(scores < np.percentile(scores, 5))[0]
# Plot the results
plt.figure(figsize=(10, 6))
plt.scatter(X.iloc[:, 0], X.iloc[:, 1], color='green', label='Normal Data')
plt.scatter(X.iloc[outliers, 0], X.iloc[outliers, 1], color='red', label='Outliers')
plt.xlabel('Sepal Length (cm)', fontsize=13)
plt.ylabel('Sepal Width (cm)', fontsize=13)
plt.title('Anomaly Detection using Isolation Forest', fontsize=16)
plt.legend()
plt.show()
2. Local Outlier Factor (LOF)
It is an algorithm based on the concept that an anomaly (or outlier) is more isolated in the feature space compared to its k nearest neighbors. This is why, LOF evaluates the local neighborhood of each data point, calculating its density relative to its neighbors.
Outliers typically exhibit significantly lower local densities than their neighbors, making them detectable using the LOF algorithm.
Scikit-Learn offers a straightforward implementation of LOF in Python.
# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.neighbors import LocalOutlierFactor
# Load the Iris dataset and select features
iris = load_iris(as_frame=True)
X = iris.data[['sepal length (cm)', 'sepal width (cm)']]
# Initialize the Local Outlier Factor model
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.05)
# Fit the model and calculate the outlier scores
outlier_labels = lof.fit_predict(X)
scores = lof.negative_outlier_factor_
# Identify the indices of the outliers
outliers = np.where(outlier_labels == -1)[0]
# Plot the results
plt.figure(figsize=(10, 6))
plt.scatter(X.iloc[:, 0], X.iloc[:, 1], color='green', label='Normal Data')
plt.scatter(X.iloc[outliers, 0], X.iloc[outliers, 1], color='red', label='Outliers')
plt.xlabel('Sepal Length (cm)', fontsize=13)
plt.ylabel('Sepal Width (cm)', fontsize=13)
plt.title('Anomaly Detection using Local Outlier Factor', fontsize=16)
plt.legend()
plt.show()
When using this model, there are two important variables to adjust:
- Number of Neighbors: This parameter sets how many neighbors to check when estimating density. It affects strongly on how sensitive the algorithm is to changes in local density.
- Contamination Coefficient: This variable determines the expected proportion of outliers in the dataset. It helps to set a threshold for identifying a data point as an outlier.
Getting these parameters right is essential for accurately detecting outliers.
3. One-class Support Vector Machines (SVMs):
It is another robust ML model that can be applied for detecting outliers. Its main goal is to classify these outliers by learning the properties of the normal class and distinguishing them from the abnormal one.
To do so, it determines the boundary that best separates normal data points from outliers by maximizing the margin between support vector machines.
Scikit-Learn provides an easy implementation of One-Class SVM in Python.
# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.svm import OneClassSVM
# Load the Iris dataset and select features
iris = load_iris(as_frame=True)
X = iris.data[['sepal length (cm)', 'sepal width (cm)']]
# Initialize the One-Class SVM model
model = OneClassSVM(nu=0.05, kernel='rbf', gamma='scale')
# Fit the model to the data
model.fit(X)
# Compute the anomaly scores for each point
scores = model.decision_function(X)
# Identify outliers based on the anomaly scores
outliers = np.where(scores < np.percentile(scores, 5))[0]
# Plot the results
plt.figure(figsize=(10, 6))
plt.scatter(X.iloc[:, 0], X.iloc[:, 1], color='green', label='Normal Data')
plt.scatter(X.iloc[outliers, 0], X.iloc[outliers, 1], color='red', label='Outliers')
plt.xlabel('Sepal Length (cm)', fontsize=13)
plt.ylabel('Sepal Width (cm)', fontsize=13)
plt.title('Anomaly Detection using One-Class SVM', fontsize=16)
plt.legend()
plt.show()
4. Elliptic Envelope
The Elliptic Envelope algorithm fits a Gaussian distribution to our set of data points and predicts the probability of each point to be under this distribution. Data points with the lowest probability are the ones considered outliers.
This method can only be applied if the data is assumed to follow a Gaussian distribution. Scikit-Learn makes it easy to implement it in Python.
# Import necessary librariesimport numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.covariance import EllipticEnvelope
# Load the Iris dataset and select features
iris = load_iris(as_frame=True)
X = iris.data[['sepal length (cm)', 'sepal width (cm)']]
# Initialize the Elliptic Envelope model
model = EllipticEnvelope(contamination=0.05, support_fraction=0.8, random_state=42)
# Fit the model to the data
model.fit(X)
# Compute the anomaly scores for each point
scores = model.decision_function(X)
# Identify outliers based on the anomaly scores
outliers = np.where(scores < np.percentile(scores, 5))[0]
# Plot the results
plt.figure(figsize=(10, 6))
plt.scatter(X.iloc[:, 0], X.iloc[:, 1], color='green', label='Normal Data')
plt.scatter(X.iloc[outliers, 0], X.iloc[outliers, 1], color='red', label='Outliers')
plt.xlabel('Sepal Length (cm)', fontsize=13)
plt.ylabel('Sepal Width (cm)', fontsize=13)
plt.title('Anomaly Detection using Elliptic Envelope', fontsize=16)
plt.legend()
plt.show()
And this is all!
In Brief
To further understand this concept, we can finsih by summarizing three main bullet points:
- Outliers: Outliers are anomalies in any set of data that can impact the general pattern and decrease any ML model result, making their removal crucial.
- Detection Models: Outliers can be identified using visual techniques, statistical methods or ML algorithms.
- Algorithms: Effective ML models for outlier detection are Isolation Forest, LOF, SVM, and Elliptic Envelope.
