How to Organize Data Validations Using Great Expectations Suites

Data validation rarely involves just one rule. While it’s straightforward to write simple checks like “ensure all ages are positive” or “verify email addresses contain @ symbols,” most projects require multiple validations working in concert. Take a taxi ride dataset, for example — you need to verify passenger counts, fare amounts, pickup locations, and timestamps simultaneously. Managing these checks individually becomes cumbersome as your validation requirements expand.

Great Expectations addresses this challenge through its suite capabilities. Let’s explore how to organize multiple data quality checks effectively and create reusable validation rules that can grow with your needs.

Why Move Beyond Individual Checks?

When validating data, you might start with simple pandas checks on individual columns – verifying passenger counts are reasonable or ensuring fare amounts are positive. While these basic checks work for quick analyses, they fall short as your data validation needs mature.

Consider what happens when your project grows:

You need to validate multiple datasets with the same rules
Other team members need to understand and use your validation logic
Stakeholders ask which specific values failed validation and why
New team members need to understand the reasoning behind each validation rule
You want to track validation results over time

This is where Great Expectations offers significant advantages. Instead of scattered validation code, it provides a structured way to:

Create reusable validation rules
Generate detailed reports on failed validations
Document the purpose of each check
Share validation logic across teams
Track validation history

Let’s explore how to create validation rules that achieve these goals.

Building Flexible Validation Rules

We begin with parameterized expectations – validation rules that can adapt based on the values you provide. Think of them as templates where you fill in the specifics later. Instead of hardcoding values like “passenger count must be between 1 and 6,” you create a flexible rule where the minimum and maximum values can change depending on your needs.

For example, a parameterized passenger count validation could be used to:

Check standard taxi rides (1-6 passengers)
Validate minivan services (1-8 passengers)
Verify bus services (1-50 passengers)

All without changing the underlying validation code. You simply adjust the parameters when running the validation. Below is a sample code to demonstrate how this works in practice:

import great_expectations as gx
import pandas as pd

# Create context and sample data
context = gx.get_context()
data = {
    'passenger_count': [1, 2, 3, 4, 5, 6, 7],
    'fare_amount': [-5, 0, 10, 20, 30, 40, 500]
}
df = pd.DataFrame(data)

# Set up data source and batch
data_source = context.data_sources.add_pandas(name="my_taxi_data")
data_asset = data_source.add_dataframe_asset(name="taxi_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})

# Create parameterized expectations
passenger_expectation = gx.expectations.ExpectColumnValuesToBeBetween(
    column="passenger_count",
    min_value={"$PARAMETER": "passenger_min"},
    max_value={"$PARAMETER": "passenger_max"}
)

# Test with different parameters
validation_results = batch.validate(
    passenger_expectation,
    expectation_parameters={
        "passenger_min": 1,
        "passenger_max": 6
    }
)

print("\nValidation Results:")
print(f"Success: {validation_results.success}")
if not validation_results.success:
    print("Failed values:", validation_results.result['partial_unexpected_list'])

When we run this validation, we get detailed results:

Validation Results:
Success: False
Failed values: [7]

Unlike a simple boolean check, this output tells us exactly which values violated our rules. Plus, we can easily adjust our validation parameters without changing the code.

Creating Validation Suites

While parameterized expectations are powerful, most data quality checks often require multiple validations working together. This is where Expectation Suites come in. A suite lets you group related expectations and run them as a single unit.

Let’s create a suite that validates both passenger counts and fare amounts:

import great_expectations as gx
import pandas as pd

# Create context and sample data
context = gx.get_context()
data = {
    'passenger_count': [1, 2, 3, 4, 5, 6, 7],
    'fare_amount': [-5, 0, 10, 20, 30, 40, 500]
}
df = pd.DataFrame(data)

# Set up data source and batch
data_source = context.data_sources.add_pandas(name="taxi_data")
data_asset = data_source.add_dataframe_asset(name="taxi_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})

# Create an Expectation Suite
suite = gx.core.expectation_suite.ExpectationSuite(name="taxi_data_quality")
suite = context.suites.add(suite)

# Add multiple expectations to the suite
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(
        column="passenger_count",
        min_value=1,
        max_value=6
    )
)

suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(
        column="fare_amount",
        min_value=0
    )
)

# Validate the batch against the suite
validation_results = batch.validate(suite)

print("\nValidation Results:")
print(f"Success: {validation_results.success}")
if not validation_results.success:
    print("\nFailed Expectations:")
    for result in validation_results.results:
        if not result.success:
            print(f"- {result.expectation_config.kwargs['column']}: {result.result['partial_unexpected_list']}")

When we run this suite, we get comprehensive results that show us all validation failures:

Validation Results:
Success: False

Failed Expectations:
- passenger_count: [7]
- fare_amount: [-5]

This output immediately shows us that we have two issues: a passenger count that’s too high (7) and a negative fare amount (-5). The suite approach offers several advantages:

1. Organization: Related validations are grouped logically
2. Maintenance: Update multiple validations in one place
3. Documentation: The suite serves as self-documenting code for your data requirements
4. Reusability: Apply the same suite to different datasets or pipeline stages

Best Practices for Suite Organization

When organizing your validations into suites, consider these patterns:

1. Group by Purpose: Create separate suites for different validation goals

Data quality suites for basic cleanliness checks
Business logic suites for domain-specific rules
Compliance suites for regulatory requirements

2. Layer Your Validations: Structure suites in tiers

Core suites for must-pass validations
Extended suites for warning-level checks
Custom suites for specific use cases

3. Version Your Suites: Maintain suite stability

Use clear naming conventions
Document changes to validation rules
Track suite versions in your code repository

Conclusion

Organizing data validations into suites transforms scattered checks into a maintainable quality control system. By grouping related validations, documenting their purpose, and making them reusable, you create a scalable foundation for data quality. As your validation needs grow, these organized suites become invaluable – they help teams understand data requirements, track quality issues, and maintain consistent standards across datasets.

The next time you find yourself writing individual validation checks, consider how organizing them into suites could make your data quality process more robust and manageable. Start small with a few related checks, document your validation rules, and gradually build a comprehensive suite that grows with your project’s needs.