Statistical validation of data quality often requires more sophisticated approaches than simple range checks or null value detection. When working with datasets where the acceptable range of values might vary across different segments or time periods, traditional threshold-based validations may not provide sufficient insight. This is where Z-score based validation becomes particularly useful.

Z-scores help us understand how far a data point deviates from the mean in terms of standard deviations, making them especially valuable for detecting outliers in normally distributed data. Great Expectations, through its ExpectColumnValueZScoresToBeLessThan validation method, provides a robust way to implement these statistical checks in your data quality pipeline.

The Great Expectations Gallery

The Great Expectations Gallery is a comprehensive resource that showcases various data validation approaches, from basic checks to advanced statistical methods. It is a well-organized library where each validation method comes with detailed documentation, implementation guidelines, and practical examples.

The Gallery helps data practitioners in several important ways:

• It provides documented validation methods that address common data quality challenges
• It explains the technical parameters and configurations for each validation approach
• It includes concrete examples showing how to implement these validations in practice
• It guides users in selecting the most appropriate validation method for their specific needs

ExpectColumnValueZScoresToBeLessThan represents a sophisticated statistical validation method from this gallery. By leveraging Z-scores, it enables us to identify unusual patterns and potential anomalies in our data through a standardized statistical approach. This becomes particularly valuable when dealing with datasets where the definition of “normal” or “acceptable” values might change based on the underlying data distribution.

Basic Outlier Detection with Single-Sided Z-Scores

Our first implementation focuses on identifying extreme outliers in BMI values using a single-sided Z-score validation approach. This means we’re looking for values that deviate significantly above our threshold, which is particularly useful when we’re more concerned about upper bounds than lower bounds in our data.

import great_expectations as gx
import pandas as pd
from sklearn.datasets import load_diabetes

# Load diabetes dataset
diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)

# Setup Great Expectations
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="diabetes_data")
data_asset = data_source.add_dataframe_asset(name="diabetes_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})

# Create and validate Z-score expectation
expectation = gx.expectations.ExpectColumnValueZScoresToBeLessThan(
    column="bmi",
    threshold=3,
    double_sided=False
)

# Validate and show results
results = batch.validate(expectation)

print("\nValidation Results:")
print(f"Success: {results.success}")
if not results.success:
    print(f"Number of outliers: {results.result['unexpected_count']}")
    print("Sample outliers:", results.result['partial_unexpected_list'][:5])

When we run this validation, we get these results:

Validation Results:
Success: False
Number of outliers: 2
Sample outliers: [0.16085491731571683, 0.17055522598064407]

To better understand what these results mean in the context of our data distribution, let’s visualize it:

# After the first validation, let's visualize the distribution
import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))
plt.hist(df['bmi'], bins=30, edgecolor='black')
plt.axvline(df['bmi'].mean(), color='red', linestyle='dashed', label='Mean')
plt.axvline(df['bmi'].mean() + 3*df['bmi'].std(), color='green', linestyle='dashed', label='3σ threshold')
plt.axvline(df['bmi'].mean() - 3*df['bmi'].std(), color='green', linestyle='dashed')
plt.title('Distribution of BMI Values')
plt.xlabel('BMI (standardized)')
plt.ylabel('Frequency')
plt.legend()
plt.grid(True, alpha=0.3)

Image By Author

Looking at our validation results and the visualization, we discovered exactly two values exceeding the 3σ upper threshold. The histogram shows why: these points fall beyond the right-most green dashed line, representing our +3σ threshold. While we’ve plotted both the upper and lower boundaries for completeness, our validation is configured to only flag values above the upper threshold (double_sided=False in our configuration).

What makes this visualization particularly insightful is how it demonstrates the relationship between our chosen threshold and the actual data distribution. Most values cluster around the mean (red dashed line), with the frequency gradually decreasing as we move away from the center. By setting our validation to only monitor the upper boundary at +3σ, we’ve identified two statistically significant high outliers while ignoring any potential low outliers.

Advanced Validation: Double-Sided Z-Score Approach

While our initial one-sided validation was useful for detecting high outliers, many scenarios require us to identify both unusually high and low values. This is where double-sided Z-score validation becomes valuable. Let’s modify our approach to detect outliers on both sides of the distribution using a more sensitive threshold of 2σ instead of 3σ.

import great_expectations as gx
import pandas as pd
import numpy as np
from sklearn.datasets import load_diabetes
import matplotlib.pyplot as plt

# Load and prepare data
diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)

# Setup Great Expectations (same as before)
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="diabetes_data")
data_asset = data_source.add_dataframe_asset(name="diabetes_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})

# Create double-sided Z-score expectation
expectation = gx.expectations.ExpectColumnValueZScoresToBeLessThan(
    column="bmi",
    threshold=2,
    double_sided=True
)

# Validate
results = batch.validate(expectation)

# Print results with more detail
print("\nValidation Results:")
print(f"Success: {results.success}")
print(f"Total records: {results.result['element_count']}")
print(f"Records outside ±2 std dev: {results.result['unexpected_count']}")

Our validation now uses two key modifications: double_sided=True to check both tails of the distribution, and a threshold of 2 instead of 3. The results show a significant change in our findings:

Validation Results:
Success: False
Total records: 442
Records outside ±2 std dev: 15

To better understand these results, let’s visualize the distribution with our new thresholds:

# After the second validation, let's visualize the distribution

plt.figure(figsize=(10, 6))
plt.hist(df['bmi'], bins=30, edgecolor='black')
plt.axvline(df['bmi'].mean(), color='red', linestyle='dashed', label='Mean')
plt.axvline(df['bmi'].mean() + 2*df['bmi'].std(), color='green', linestyle='dashed', label='2σ threshold')
plt.axvline(df['bmi'].mean() - 2*df['bmi'].std(), color='green', linestyle='dashed')
plt.title('Distribution of BMI Values')
plt.xlabel('BMI (standardized)')
plt.ylabel('Frequency')
plt.legend()
plt.grid(True, alpha=0.3)

Image By Author

The visualization reveals the impact of our modified approach. The green dashed lines now represent the ±2σ boundaries, and we’ve identified 15 records (approximately 3.4% of our 442 total records) that fall outside these thresholds. This aligns closely with statistical theory – in a normal distribution, approximately 95% of values fall within two standard deviations of the mean, leaving about 5% as potential outliers.

By lowering our threshold from 3σ to 2σ and implementing double-sided validation, we’ve adopted a more sensitive approach to outlier detection. This might be particularly useful in scenarios where early detection of anomalies is important, such as monitoring patient health metrics or quality control in manufacturing processes where both unusually high and low values could indicate potential issues.

Conclusion: Implementing Z-Score Validation

Great Expectations’ ExpectColumnValueZScoresToBeLessThan validation method provides a sophisticated approach to data quality monitoring that goes beyond simple threshold checks. By offering both single-sided and double-sided validation options along with configurable thresholds, it enables data teams to implement statistically rigorous outlier detection that adapts to their specific needs. The method’s integration into Great Expectations’ broader validation framework means these statistical checks can be seamlessly incorporated into existing data pipelines.

When implementing Z-score validation with Great Expectations, the choice between 3σ and 2σ thresholds becomes a powerful tool for fine-tuning sensitivity. The 3σ threshold captures approximately 99.7% of normally distributed data, making it ideal for identifying extreme outliers, while the 2σ threshold encompasses about 95%, providing earlier detection of potential anomalies. This flexibility, combined with the double-sided parameter, allows teams to balance between sensitivity to outliers and tolerance for natural variation in their data.

The real power of Great Expectations lies in how it transforms statistical concepts into practical, production-ready validations. The framework’s standardized approach to implementing and maintaining statistical checks makes it valuable across domains that require robust outlier detection. By integrating statistical validation into your data quality suite through Great Expectations, you can ensure that your outlier detection strategy is both theoretically sound and operationally robust.