Anomaly detection identifies unusual data patterns or outliers. In R, different techniques are available to help detect such anomalies. Methods include statistical tests, clustering, and machine learning models. This article will show how to use these methods in R.
What is Anomaly Detection
Anomaly detection is finding unusual patterns in data, which can be outliers or rare events. Anomaly detection helps to spot problems like fraud or system errors, and is useful in areas like finance, healthcare, and security. It helps keep data accurate and improve decisions.
Anomalies can be classified into three types:
- Point anomalies: A single data point is anomalous
- Contextual anomalies: A data point is anomalous in a specific context
- Collective anomalies: A collection of data points is anomalous
Prerequisites and Setting Up R
First, we need to install the required libraries.
install.packages(c("outliers", "dbscan", "bnlearn"))
Anomaly Detection Methods in R
Now, let’s explore the different methods for anomaly detection:
Z-Score
Z-Score is a method to detect outliers. It measures how far a value is from the average. A high Z-Score indicates an outlier. It helps identify values that are unusual in a dataset.
# Sample dataset values <- c(75, 82, 90, 68, 88, 94, 78, 60, 72, 85) # Calculate the average and standard deviation avg_value <- mean(values) std_dev_value <- sd(values) # Calculate Z-scores for each value z_scores_values <- (values - avg_value) / std_dev_value # Display the Z-scores print(z_scores_values)
Output:
[1] -0.3945987 0.2630658 1.0146823 -1.0522632 0.8267782 1.3904906 -0.1127425 [8] -1.8038797 -0.6764549 0.5449220
Grubbs’ Test
Grubbs’ Test is used to find one outlier in a dataset. It looks for the data point farthest from the mean. The test compares this point to others using standard deviations. If the p-value is low, it means the point is an outlier.
# Load the required package library(outliers) # Sample exam data data <- c(75, 82, 90, 68, 88, 94, 78, 60, 72, 85) # Perform Grubbs' Test to detect a single outlier grubbs_result <- grubbs.test(data) # Display the result of Grubbs' Test print(grubbs_result)
Output:
Grubbs test for one outlier data: data G = 1.80388, U = 0.59827, p-value = 0.2455 alternative hypothesis: lowest value 60 is an outlier
K-Means Clustering
K-Means Clustering groups data into clusters. It finds the center of each cluster, called the centroid, with the number of clusters is set beforehand. Points far from their centroid may be considered outliers. K-Means helps to identify patterns and anomalies based on distance from centroids.
library(tidyverse)
# Sample exam data
data <- c(75, 82, 90, 68, 88, 94, 78, 60, 72, 85)
# Reshape the data to a dataframe
data_df <- data.frame(value = data)
# Apply K-Means clustering with 2 clusters
kmeans_result <- kmeans(data_df, centers = 2)
# Display the K-Means result
print(kmeans_result)
# Calculate the distance of each point from the nearest cluster center
distances <- apply(data_df, 1, function(x) {
min(sqrt(rowSums((t(kmeans_result$centers) - x)^2)))
})
# Set a threshold to identify anomalies
threshold <- quantile(distances, 0.9)
# Identify anomalies based on whether their distance exceeds the threshold
anomalies threshold)
# Display the anomalous data points
print(data[anomalies])
Output:
K-means clustering with 2 clusters of sizes 5, 5 Cluster means: value 1 87.8 2 70.6 Clustering vector: [1] 2 1 1 2 1 1 2 2 2 1 Within cluster sum of squares by cluster: [1] 84.8 195.2 (between_SS / total_SS = 72.5 %) Available components: [1] "cluster" "centers" "totss" "withinss" "tot.withinss" [6] "betweenss" "size" "iter" "ifault" [1] 60
DBSCAN
DBSCAN is a density-based clustering method, grouping points into clusters based on their density. Points in low-density areas are marked as outliers. It works well with clusters of different shapes. Points labeled as cluster 0 are considered anomalies.
# Load required package library(dbscan) # Sample exam data (reshaped as matrix for DBSCAN) data <- c(75, 82, 90, 68, 88, 94, 78, 60, 72, 85) # Reshape the data to a matrix data_matrix <- matrix(data, ncol = 1) # Apply DBSCAN with epsilon value of 5 and minPts as 3 dbscan_result <- dbscan(data_matrix, eps = 5, minPts = 3) # Print DBSCAN result print(dbscan_result) # Identifying outliers outliers_dbscan <- which(dbscan_result$cluster == 0) # Print the indices of outliers print(outliers_dbscan)
Output:
DBSCAN clustering for 10 objects. Parameters: eps = 5, minPts = 3 Using euclidean distances and borderpoints = TRUE The clustering contains 1 cluster(s) and 1 noise points. 0 1 1 9 Available fields: cluster, eps, minPts, metric, borderPoints [1] 8
Bayesian Networks
Bayesian Networks are graphical models used for probabilistic reasoning. They represent relationships between variables using nodes and edges. Each node shows a variable, and edges indicate dependencies. They can be used for anomaly detection by analyzing conditional dependencies.
# Load the required package
library(bnlearn)
# Generate a new example dataset
set.seed(42)
data Y -> Z)
network_structure <- model2network("[X][Y|X][Z|Y]")
# Build the Bayesian network using the structure and data
bn_model <- bn.fit(network_structure, data)
# Print the fitted Bayesian network model
print(bn_model)
Output:
Bayesian network parameters Parameters of node X (Gaussian distribution) Conditional density: X Coefficients: (Intercept) -0.02874049 Standard deviation of the residuals: 1.005498 Parameters of node Y (Gaussian distribution) Conditional density: Y | X Coefficients: (Intercept) X -0.01265089 0.07538724 Standard deviation of the residuals: 0.9723072 Parameters of node Z (Gaussian distribution) Conditional density: Z | Y Coefficients: (Intercept) Y -0.01916620 -0.08886731 Standard deviation of the residuals: 0.9164108
Conclusion
Anomaly detection is useful for finding unusual data points. It helps in detecting problems like fraud or system errors. In R, we can use methods like Z-Score, Grubbs’ Test, K-Means, DBSCAN, and Bayesian Networks. Using these methods can improve the quality of data and the accuracy of models. Anomaly detection helps maintain data integrity and supports better decision-making.
