Data validation isn’t always straightforward. Sometimes we need to apply different rules to different parts of our dataset. For example, we might want to validate shipping addresses only for international orders, or verify account balances only for active accounts. Great Expectations handles these scenarios through conditional expectations.
In this article, we’ll explore how to create and use conditional expectations effectively. We’ll use the classic Titanic dataset as our example, as it provides an intuitive way to demonstrate these concepts.
Setting Up Our Environment
First, let’s create a sample dataset and set up our Great Expectations environment:
import great_expectations as gx
import pandas as pd
# Sample Titanic data
data = {
'PClass': ['1st', '2nd', '1st', '3rd', '1st'],
'Survived': [1, 0, 1, 0, 1],
'Age': [24, 35, 28, 19, 42]
}
df = pd.DataFrame(data)
print("Initial DataFrame:")
print(df)
When we run this code, we see our sample dataset:
Initial DataFrame: PClass Survived Age 0 1st 1 24 1 2nd 0 35 2 1st 1 28 3 3rd 0 19 4 1st 1 42
This dataset contains passenger information from the historic Titanic voyage, including their passenger class (PClass), survival status (Survived), and age. The Titanic dataset is widely used in data analysis as it provides information on demographic and survival patterns. In this sample, we focus on three key variables: passenger class (1st, 2nd, or 3rd), survival status (1 for survived, 0 for did not survive), and the passenger’s age. With this meaningful data in hand, we can proceed to set up the Great Expectations environment:
# Set up the environment
context = gx.get_context()
data_source = context.data_sources.add_pandas(name="titanic_data")
data_asset = data_source.add_dataframe_asset(name="passenger_metrics")
batch_definition = data_asset.add_batch_definition_whole_dataframe("batch_def")
batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
print("Batch created successfully")
After running this setup code, we receive confirmation that our batch was created successfully. This setup establishes a data context, registers our DataFrame as a data source, and creates a batch definition – essential components for running validations. As covered in our previous article on Great Expectations fundamentals, this infrastructure enables us to systematically validate our data against defined expectations.
Batch created successfully
Creating Basic Conditional Expectations
Now that our environment is ready, let’s create our first conditional expectation. We’ll start with a simple case: checking if first-class passengers survived. To make this interesting, we’ll expect them to have not survived (Survived=0), which we know will fail based on our data:
# Create and validate basic conditional expectation
expectation_with_condition = gx.expectations.ExpectColumnValuesToBeInSet(
column="Survived",
value_set=[0], # Expecting Survived=0
condition_parser="pandas",
row_condition='PClass=="1st"'
)
# Validate and show results
results = batch.validate(expectation_with_condition)
print("\nValidation Results (Expecting first-class passengers to have Survived=0):")
print(f"Success: {results.success}")
if not results.success:
print("Unexpected values:", results.result['partial_unexpected_list'])
print("Total unexpected values:", results.result['unexpected_count'])
Running this validation gives us:
Validation Results (Expecting first-class passengers to have Survived=0): Success: False Unexpected values: [1, 1, 1] Total unexpected values: 3
Let’s break down what happened:
- The row_condition=’PClass==”1st”‘ filtered our validation to only look at first-class passengers
- We expected these passengers to have Survived=0 (which means we expect them to not have survived)
- The validation failed because all three first-class passengers actually survived (Survived=1)
- The results show us exactly how many values failed (3) and what those values were ([1, 1, 1])
Advanced Conditional Expectations
We can make our conditions more complex by combining multiple criteria using standard logical operators. In many situations, data validation often requires checking multiple conditions simultaneously – for instance, validating transaction amounts only for high-risk customers in specific regions. For our example, we’ll create an expectation that checks survival status for first-class passengers over 30, combining both class and age conditions:
# Create and validate complex conditional expectation
expectation_complex = gx.expectations.ExpectColumnValuesToBeInSet(
column="Survived",
value_set=[0], # Expecting Survived=0
condition_parser="pandas",
row_condition='PClass=="1st" and Age > 30'
)
# Validate and show results
results = batch.validate(expectation_complex)
print("\nComplex Validation Results (Expecting first-class passengers over 30 to have Survived=0):")
print(f"Success: {results.success}")
if not results.success:
print("Unexpected values:", results.result['partial_unexpected_list'])
print("Number of rows evaluated:", results.result['element_count'])
print("Total unexpected values:", results.result['unexpected_count'])
The results show:
Complex Validation Results (Expecting first-class passengers over 30 to have Survived=0): Success: False Unexpected values: [1] Number of rows evaluated: 1 Total unexpected values: 1
This more complex validation reveals the power of conditional expectations in filtering and validating specific subsets of data:
- Only one passenger met our conditions (first-class AND over 30), demonstrating how multiple conditions effectively narrow down our focus to specific cases
- That passenger survived (value=1), failing our expectation of Survived=0, which highlights how we can identify specific cases that don’t meet our criteria
- The results tell us both how many rows were evaluated (1) and how many failed (1), providing complete transparency about the validation’s scope and impact
This granular level of insight helps us understand not just what failed, but exactly which subset of our data was involved in the validation process.
Best Practices for Conditional Expectations
When working with conditional expectations, keep these guidelines in mind:
- Use Clear Conditions: Write row conditions that are easy to understand and maintain. Complex conditions can be hard to debug.
- Follow Syntax Rules:
- Use double quotes in condition strings
- Avoid single quotes and newlines
- Use the correct condition_parser for your data source
- Test Incrementally:
- Verify your conditions independently before adding them to expectations
- Start with simple conditions before building complex ones
- Check the number of rows being evaluated to ensure your conditions are filtering as expected
- Document Your Logic:
- Include comments explaining why certain conditions are needed
- Document any assumptions about the data
- Keep track of which conditions apply to which business rules
Next Steps
Conditional expectations are powerful tools for implementing complex validation rules. As you work with them, consider:
- Combining multiple conditions for more sophisticated validations
- Creating reusable condition patterns for common scenarios
- Integrating conditional expectations into your validation suites
- Setting up automated testing to ensure conditions remain valid as data changes
By experimenting with conditional expectations, you can create more precise and meaningful data validations that better reflect your business rules and data quality requirements.
This article is part of our series on Great Expectations. Check out our previous articles on basic expectations and validation suites for more background.
