Image by Editor
What are Statology Sprints?
Statology Sprints are carefully curated learning paths that distill complex concepts into practical tools you can apply immediately. We fuse foundational statistical material with expert insights to enhance your skills while skipping unnecessary filler. Each Sprint includes:
- Core concepts explained clearly
- Practical code examples you can adapt
- Connections between related ideas
- Links to deeper coverage when you need it
This Sprint focuses on Great Expectations, a library that transforms how data teams approach quality control and validation, making data validation less challenging and more accessible.
Getting Started with Great Expectations
Great Expectations provides a framework for expressing what you expect from your data, then validating those expectations. Before diving into advanced techniques, let’s establish the foundations:
# Install Great Expectations
pip install great-expectations
# Import libraries
import great_expectations as gx
import pandas as pd
# Create sample data
data = {'age': [25, 30, -5, 45, 150]}
df = pd.DataFrame(data)
# Set up Great Expectations with this data
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="my_datasource")
data_asset = data_source.add_dataframe_asset(name="my_dataframe")
batch_definition = data_asset.add_batch_definition_whole_dataframe("my_batch_definition")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
# Add a simple expectation
expectation = gx.expectations.ExpectColumnValuesToBeBetween(
column="age", min_value=0, max_value=120
)
# Validate data against expectation
results = batch.validate(expectation)
When we run this validation, we see that not all of our age values meet our criteria. Let’s examine which values passed and which ones failed:
# Extract indices of failed rows from the validation result
failed_indexes = results.result["partial_unexpected_index_list"]
# Retrieve rows with failed values using their indices
failed_df = df.iloc[failed_indexes]
# Retrieve rows that passed the expectation by dropping the failed rows
passed_df = df.drop(index=failed_indexes)
print("Passed values:")
print(passed_df)
print("\nFailed values:")
print(failed_df)
Passed values: age 0 25 1 30 3 45 Failed values: age 2 -5 4 150
This output reveals exactly which values violated our expectations – a negative age (-5) and an implausibly high age (150). Great Expectations doesn’t just tell us that validation failed; it provides detailed information about which specific values caused the failure, making troubleshooting more efficient.
This simple example demonstrates the three core pillars of Great Expectations:
- Data Assets: Containers for your data (our pandas DataFrame)
- Expectations: Rules your data should follow (age between 0-120)
- Validation: The process of checking if expectations are met
➡️ Learn more about Great Expectations fundamentals
Organizing Validations with Expectation Suites
As your validation needs grow, managing individual checks becomes cumbersome. Expectation Suites let you group related validations into a cohesive unit that can be run together and maintained as a single entity. Let’s see how this works in practice:
import great_expectations as gx
import pandas as pd
# Create sample data for product inventory
data = {
'product_id': ['A001', 'B002', 'C003', 'D004', 'E005'],
'price': [24.99, 0, 149.99, 999.99, -10.00],
'quantity': [100, 25, 0, -5, 75]
}
df = pd.DataFrame(data)
print("Sample inventory data:")
print(df)
# Set up Great Expectations environment
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="inventory_data")
data_asset = data_source.add_dataframe_asset(name="product_inventory")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
# Create an Expectation Suite for inventory validation
suite = gx.core.expectation_suite.ExpectationSuite(name="inventory_validation")
suite = context.suites.add(suite)
# Add multiple expectations to the suite
# 1. Product IDs must follow the pattern of a letter followed by 3 digits
suite.add_expectation(
gx.expectations.ExpectColumnValuesToMatchRegex(
column="product_id",
regex="^[A-Z][0-9]{3}$"
)
)
# 2. Prices must be positive
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="price",
min_value=0.01
)
)
# 3. Quantities must be non-negative
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="quantity",
min_value=0
)
)
# Validate the batch against the suite
validation_results = batch.validate(suite)
# Print readable validation results
print("\nValidation Results:")
print(f"Overall Success: {validation_results.success}")
if not validation_results.success:
print("\nFailed Expectations:")
for result in validation_results.results:
if not result.success:
column = result.expectation_config.kwargs['column']
failed_values = result.result.get('partial_unexpected_list', [])
print(f"- {column}: {failed_values}")
When we run this code, we get these results:
Sample inventory data: product_id price quantity 0 A001 24.99 100 1 B002 0.00 25 2 C003 149.99 0 3 D004 999.99 -5 4 E005 -10.00 75 Validation Results: Overall Success: False Failed Expectations: - price: [0.0, -10.0] - quantity: [-5]
This output reveals three validation issues with our inventory data:
- Two products have invalid prices: 0.00 (item B002) and -10.00 (item E005)
- One product has an impossible negative quantity: -5 (item D004)
The suite approach offers several advantages:
- Organization: Related validations are grouped logically
- Maintenance: Update multiple validations in one place
- Documentation: The suite serves as self-documenting code for your data requirements
- Reusability: Apply the same suite to different datasets or pipeline stages
You can extend this approach by creating different suites for different validation needs – perhaps a basic suite for data structure checks and a more complex suite for business logic validations. This modular approach scales well as your data quality requirements grow.
➡️ Learn more about organizing validations with suites
Advanced Validation Techniques
Once you’ve mastered the basics, Great Expectations offers sophisticated validation approaches that address more complex data quality scenarios. Let’s explore three powerful techniques that extend your validation capabilities.
Conditional Expectations
Sometimes you need to apply different validation rules to different segments of your data. Conditional expectations let you define which rows should be evaluated based on specific criteria:
import great_expectations as gx
import pandas as pd
# Create sample Titanic data
data = {
'PClass': ['1st', '2nd', '1st', '3rd', '1st'],
'Survived': [1, 0, 1, 0, 1],
'Age': [24, 35, 28, 19, 42]
}
df = pd.DataFrame(data)
# Set up the environment
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="titanic_data")
data_asset = data_source.add_dataframe_asset(name="passenger_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
# Create conditional expectation that checks first-class passengers over 30
expectation = gx.expectations.ExpectColumnValuesToBeInSet(
column="Survived",
value_set=[1], # Expecting Survived=1 (they survived)
condition_parser="pandas",
row_condition='PClass=="1st" and Age > 30'
)
# Validate and show results
results = batch.validate(expectation)
print("Conditional Validation Results:")
print(f"Success: {results.success}")
print(f"Number of rows evaluated: {results.result['element_count']}")
if not results.success:
print("Unexpected values:", results.result['partial_unexpected_list'])
Conditional Validation Results: Success: True Number of rows evaluated: 1
This validation specifically checks if first-class passengers over 30 years old survived. The condition PClass==”1st” and Age > 30 filters the data before applying the expectation. In our sample, only one passenger meets both criteria (first-class AND over 30), and they did indeed survive as expected, resulting in a successful validation. This technique is valuable when different segments of your data should adhere to different rules – for instance, validating shipping addresses only for international orders or ensuring premium customers have complete profile information.
➡️ Learn more about conditional expectations
Statistical Validation with Z-Scores
Fixed thresholds aren’t always appropriate for detecting outliers. Z-score based validation identifies values that deviate significantly from the statistical distribution of your data:
import great_expectations as gx
import pandas as pd
from sklearn.datasets import load_diabetes
# Load diabetes dataset
diabetes = load_diabetes()
df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)
# Setup Great Expectations
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="diabetes_data")
data_asset = data_source.add_dataframe_asset(name="diabetes_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
# Create Z-score expectation to find BMI values > 2 standard deviations from mean
expectation = gx.expectations.ExpectColumnValueZScoresToBeLessThan(
column="bmi",
threshold=2,
double_sided=True # Check both high and low outliers
)
# Validate
results = batch.validate(expectation)
print("\nZ-Score Validation Results:")
print(f"Success: {results.success}")
print(f"Total records: {results.result['element_count']}")
print(f"Records outside ±2 std dev: {results.result['unexpected_count']}")
Z-Score Validation Results: Success: False Total records: 442 Records outside ±2 std dev: 15
The results identify 15 BMI values out of 442 total records that fall outside two standard deviations from the mean. This finding aligns with statistical theory – in a normal distribution, approximately 95% of values should fall within ±2 standard deviations, meaning we would expect about 5% (roughly 22 records) to be outside this range. Finding 15 records (about 3.4%) is reasonably close to the theoretical expectation. This approach excels in scenarios where you can’t define fixed thresholds – such as with medical measurements that vary by population, financial indicators affected by market conditions, or sensor readings influenced by environmental factors.
➡️ Learn more about Z-score validation
Custom Expectation Classes
When built-in expectations don’t quite fit your needs, you can create custom classes that encode your specific business logic:
import great_expectations as gx
import pandas as pd
class ExpectOrderValueWithinRange(gx.expectations.ExpectColumnValuesToBeBetween):
"""An expectation that validates if order values fall within an acceptable range based on business rules."""
# Default values that reflect business logic
column: str = "order_value"
min_value: float = 10.0 # Minimum order value
max_value: float = 10000.0 # Maximum order value
# Custom description for Data Docs
description: str = """
Order values must be between **$10** and **$10,000**.
- Orders below $10 may indicate pricing errors
- Orders above $10,000 require special approval
"""
# Create sample data
data = {
'order_value': [5.0, 25.0, 150.0, 12000.0, 50.0],
'order_date': ['2025-01-01'] * 5
}
df = pd.DataFrame(data)
# Set up Great Expectations context
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="order_data")
data_asset = data_source.add_dataframe_asset(name="orders")
batch_definition = data_asset.add_batch_definition_whole_dataframe("order_batch")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
# Create and validate expectation
expectation = ExpectOrderValueWithinRange()
results = batch.validate(expectation)
print("\nCustom Expectation Results:")
print(f"Success: {results.success}")
if not results.success:
print(f"Invalid values: {results.result['partial_unexpected_list']}")
print(f"Total failures: {results.result['unexpected_count']}")
Custom Expectation Results: Success: False Invalid values: [5.0, 12000.0] Total failures: 2
Our custom class immediately identifies two problematic orders: one below the minimum threshold (5.00) that likely indicates a pricing error, and one above the maximum threshold (12,000.00) that requires special approval according to our business rules. The beauty of this approach is that it encapsulates both the validation logic and the business reasoning in a reusable component. Rather than repeatedly writing the same validation code with inline comments, team members can simply use this well-documented class that clearly communicates why these thresholds matter. This self-documenting aspect becomes even more valuable when validation logic is complex or subject to change over time.
➡️ Learn more about custom expectation classes
Bringing It All Together: Weather Station Monitoring
Let’s apply what we’ve learned by building a complete data validation solution for a weather monitoring network. Our system will check station identifiers, validate sensor readings, and ensure the reliability of data collection equipment.
Step 1: Setting Up Basic Validations
First, we’ll create sample weather station data and set up basic validations for station IDs, temperature readings, and battery levels:
import great_expectations as gx
import pandas as pd
import numpy as np
# Create sample weather station data
np.random.seed(42)
data = {
'station_id': ['S001', 'S002', 'X003', 'S004', 'S005', 'S006', '007', 'S008'],
'temperature': [32.5, 28.1, -50.0, 31.2, 27.8, 110.5, 29.3, 30.4],
'humidity': [45, 60, 55, 52, 48, 20, 53, 62],
'battery_level': [95, 80, 30, 85, 15, 90, 75, 60]
}
df = pd.DataFrame(data)
print("Weather Station Data:")
print(df)
# Set up Great Expectations
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="weather_data")
data_asset = data_source.add_dataframe_asset(name="stations")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
# Create validation suite
suite = gx.core.expectation_suite.ExpectationSuite(name="weather_station_checks")
suite = context.suites.add(suite)
# Add expectations to validate data format and ranges
suite.add_expectation(
gx.expectations.ExpectColumnValuesToMatchRegex(
column="station_id",
regex="^S[0-9]{3}$" # Station IDs should be S followed by 3 digits
)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="temperature",
min_value=-40, # Valid temperature range
max_value=50
)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="battery_level",
min_value=20, # Minimum battery level for reliable operation
)
)
# Run validation
results = batch.validate(suite)
# Display results
print("\nValidation Results:")
print(f"Success: {results.success}")
if not results.success:
print("\nFailed Expectations:")
for result in results.results:
if not result.success:
column = result.expectation_config.kwargs.get('column')
failed_values = result.result.get('partial_unexpected_list', [])
print(f"- {column}: {failed_values}")
Weather Station Data: station_id temperature humidity battery_level 0 S001 32.5 45 95 1 S002 28.1 60 80 2 X003 -50.0 55 30 3 S004 31.2 52 85 4 S005 27.8 48 15 5 S006 110.5 20 90 6 007 29.3 53 75 7 S008 30.4 62 60 Validation Results: Success: False Failed Expectations: - station_id: ['X003', '007'] - temperature: [-50.0, 110.5] - battery_level: [15]
Our basic validation immediately identifies several issues:
- Two stations (X003 and 007) have incorrectly formatted IDs
- Two stations report implausible temperatures: -50.0°C (too cold) and 110.5°C (too hot)
- One station (S005) has a critically low battery level (15%)
This simple validation suite already provides actionable information for maintaining our weather station network. By combining pattern matching (for station IDs) with range checks (for temperature and battery levels), we quickly identify stations that need attention.
Step 2: Adding Conditional Validations
Next, we’ll add a conditional validation to check humidity readings specifically from stations with low battery levels. This is important because low battery can affect sensor accuracy:
# Add a conditional expectation to check humidity readings from stations with low battery
# Low-battery stations (< 25%) should have humidity values within a narrower range
# since battery issues can cause sensor inaccuracy
conditional_expectation = gx.expectations.ExpectColumnValuesToBeBetween(
column="humidity",
min_value=45,
max_value=55,
condition_parser="pandas",
row_condition="battery_level < 25" # Only apply to stations with low battery
)
# Run the conditional validation
conditional_results = batch.validate(conditional_expectation)
print("\nLow-Battery Station Validation:")
print(f"Success: {conditional_results.success}")
print(f"Stations with low battery evaluated: {conditional_results.result['element_count']}")
if not conditional_results.success:
failed_values = conditional_results.result.get('partial_unexpected_list', [])
failed_indices = conditional_results.result.get('partial_unexpected_index_list', [])
print("\nStations requiring maintenance:")
for idx in failed_indices:
station = df.iloc[idx]
print(f"Station {station['station_id']}: Battery at {station['battery_level']}%, " +
f"Humidity reading {station['humidity']}% (outside reliable range)")
Low-Battery Station Validation: Success: True Stations with low battery evaluated: 1
Our conditional validation evaluates only stations with battery levels below 25% (just one station in our dataset). Fortunately, this station’s humidity reading of 48% falls within our acceptable range of 45-55%, so no additional issues are detected.
This is a great example of how conditional expectations allow us to apply specialized validation rules to different segments of data. In this case, we’re using tighter validation thresholds for stations that may have compromised accuracy due to low power. This approach is particularly valuable when different data quality standards apply to different subsets of your data.
Step 3: Creating a Comprehensive Monitoring System
Finally, let’s combine our validations into a comprehensive monitoring suite that includes all our checks plus statistical outlier detection:
# Create a comprehensive monitoring suite that combines format, range,
# and conditional checks - using a unique name
monitoring_suite = gx.core.expectation_suite.ExpectationSuite(name="weather_monitoring_comprehensive")
monitoring_suite = context.suites.add(monitoring_suite)
# Add all our previous expectations
monitoring_suite.add_expectation(
gx.expectations.ExpectColumnValuesToMatchRegex(
column="station_id",
regex="^S[0-9]{3}$"
)
)
monitoring_suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="temperature",
min_value=-40,
max_value=50
)
)
monitoring_suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="battery_level",
min_value=20
)
)
monitoring_suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="humidity",
min_value=45,
max_value=55,
condition_parser="pandas",
row_condition="battery_level < 25"
)
)
# Add a statistical validation for temperature
monitoring_suite.add_expectation(
gx.expectations.ExpectColumnValueZScoresToBeLessThan(
column="temperature",
threshold=3,
double_sided=True
)
)
# Run the complete suite
final_results = batch.validate(monitoring_suite)
# Generate human-readable report
print("\n===== WEATHER STATION MONITORING REPORT =====")
print(f"Total stations checked: {df.shape[0]}")
print(f"Overall validation success: {final_results.success}")
# Count stations by issue type
format_issues = 0
temperature_issues = 0
battery_issues = 0
humidity_issues = 0
outlier_issues = 0
for result in final_results.results:
if not result.success:
# Fixed: more accurately identify the expectation type
expectation_type = result.expectation_config.type.lower()
column = result.expectation_config.kwargs.get('column')
if 'match_regex' in expectation_type and column == 'station_id':
format_issues = len(result.result.get('partial_unexpected_list', []))
elif column == 'temperature' and 'z_score' not in expectation_type:
temperature_issues = len(result.result.get('partial_unexpected_list', []))
elif column == 'battery_level':
battery_issues = len(result.result.get('partial_unexpected_list', []))
elif column == 'humidity':
humidity_issues = len(result.result.get('partial_unexpected_list', []))
elif 'z_score' in expectation_type:
outlier_issues = len(result.result.get('partial_unexpected_list', []))
# Display summary
print("\nIssues Detected:")
print(f"- Station ID format issues: {format_issues}")
print(f"- Temperature range issues: {temperature_issues}")
print(f"- Low battery warnings: {battery_issues}")
print(f"- Humidity sensor issues: {humidity_issues}")
print(f"- Statistical outliers: {outlier_issues}")
print("\nRecommended Actions:")
if format_issues > 0:
print("- Update station ID naming to follow standard format (S followed by 3 digits)")
if temperature_issues > 0:
print("- Investigate stations reporting temperatures outside normal range")
if battery_issues > 0:
print("- Schedule battery replacement for stations below 20% capacity")
if humidity_issues > 0:
print("- Calibrate humidity sensors on low-battery stations")
if outlier_issues > 0:
print("- Check stations with statistically unusual temperature readings")
===== WEATHER STATION MONITORING REPORT ===== Total stations checked: 8 Overall validation success: False Issues Detected: - Station ID format issues: 2 - Temperature range issues: 2 - Low battery warnings: 1 - Humidity sensor issues: 0 - Statistical outliers: 0 Recommended Actions: - Update station ID naming to follow standard format (S followed by 3 digits) - Investigate stations reporting temperatures outside normal range - Schedule battery replacement for stations below 20% capacity
Our comprehensive monitoring system now provides a complete picture of the weather station network’s health, categorizing issues by type and providing specific maintenance recommendations. The report accurately identifies:
- Two station ID format issues: Stations ‘X003’ and ‘007’ don’t follow the required naming pattern
- Two temperature range violations: The extreme readings of -50.0°C and 110.5°C fall outside our acceptable range
- One battery warning: Station S005’s battery level (15%) is below our 20% threshold
- No humidity sensor issues: All stations with low battery have humidity readings within acceptable ranges
- No statistical outliers: After removing the extreme temperature readings caught by the range check, the remaining values don’t show unusual statistical patterns
The report automatically generates maintenance recommendations based on the validation results. This is particularly valuable for operations teams who need to prioritize maintenance activities but may not understand the technical details of the data quality checks.
This weather station monitoring example demonstrates how Great Expectations can transform data validation from a technical exercise into a practical operational tool:
- Data Quality as a Business Tool: Validation results directly translate into actionable maintenance tasks, bridging the gap between data quality and business operations
- Multiple Validation Approaches: Combining pattern matching (for IDs), range checks (for sensors), conditional rules (for low-battery stations), and statistical methods (for outlier detection) creates a comprehensive validation system
- Automated Reporting: Categorizing issues and providing specific recommendations makes validation results accessible to both technical and non-technical stakeholders
The power of Great Expectations lies not just in identifying data problems, but in providing structured, actionable information about those problems. This transforms data validation from a technical gatekeeping function into a business enablement tool that improves data quality while supporting operational decision-making.
Next Steps
After completing this Sprint, consider these advanced applications of Great Expectations:
- Integrating validation into automated data pipelines
- Building domain-specific expectation libraries for your organization
- Setting up continuous monitoring of data quality metrics
- Generating data quality documentation with Great Expectations’ Data Docs feature
With the foundation you’ve built in this Sprint, you’re well-equipped to implement robust data validation practices that ensure the reliability of your data-driven applications.
