Image by Editor | Midjourney

Hypothesis testing is a powerful statistical tool that helps us make data-driven decisions. It allows us to test assumptions about our dataset and determine whether observed patterns are due to chance or genuine effects.

In this article, we’ll use DuckDB to perform hypothesis testing on a sample cab ride dataset. Specifically, we’ll explore:

• Testing if tipping behavior is different for different payment methods.
• Checking if longer trips are associated with higher tip percentages.

So you’ll learn how to:

• Formulate hypotheses
• Use statistical tests like the t-test, chi-square test, and correlation test
• Interpret the results to make data-backed conclusions

Let’s get started!

🔗 Link to the Google Colab notebook

Setting Up DuckDB

First, install DuckDB if you haven’t already:

pip install duckdb

Now, import it in Python:

import duckdb

▶️ Get the sample cab ride dataset (CSV file)

You can query the CSV file like so:

duckdb.query("SELECT * FROM read_csv_auto('/content/cab_ride_data.csv') LIMIT 5").df()

Hypothesis 1: Does Payment Type Affect Tipping Behavior?

Question: Do customers tip differently depending on their payment method?

Hypothesis:

• Null Hypothesis (H₀): The distribution of tipping is the same across all payment methods.
• Alternative Hypothesis (H₁): Tipping behavior varies based on the payment method.

This is a chi-square test problem since we are comparing categorical distributions. And you need to compute expected counts for each cell in the contingency table. 

First, we’ll count the number of rides where customers tipped or did not tip, grouped by payment type:

SELECT 
    payment_type, 
    SUM(CASE WHEN tip > 0 THEN 1 ELSE 0 END) AS tipped,
    SUM(CASE WHEN tip = 0 THEN 1 ELSE 0 END) AS no_tip
FROM read_csv('/content/cab_ride_data.csv')
GROUP BY payment_type;

Since DuckDB does not have a built-in chi_squared_test function, we will use Python’s scipy.stats.chi2_contingency to conduct the test:

import pandas as pd
from scipy.stats import chi2_contingency

# Query data
query = """
    SELECT 
        payment_type, 
        SUM(CASE WHEN tip > 0 THEN 1 ELSE 0 END) AS tipped,
        SUM(CASE WHEN tip = 0 THEN 1 ELSE 0 END) AS no_tip
    FROM read_csv('/content/cab_ride_data.csv')
    GROUP BY payment_type;
"""
df = duckdb.query(query).df()

# Create contingency table
contingency_table = df[['tipped', 'no_tip']].values

# Perform chi-square test
chi2_stat, p_value, dof, expected = chi2_contingency(contingency_table)

# Display results
print(f"Chi-Square Statistic: {chi2_stat:.4f}")
print(f"P-Value: {p_value:.4f}")

Interpreting the Results

• If p-value < 0.05, we reject the null hypothesis and conclude that tipping behavior significantly differs across payment methods.
• If p-value > 0.05, we fail to reject H₀, meaning there is no strong evidence that tipping behavior varies based on payment method.

Here we get:

Chi-Square Statistic: 1.8509
P-Value: 0.3964

So we conclude that there’s no strong evidence to support that tipping behavior depends on the payment method.

Hypothesis 2: Does Trip Length Correlate with the Tip Amount?

Question: Do longer trips lead to higher tip percentages?

Hypothesis:

• H₀: There is no correlation between trip duration and tip percentage.
• H₁: There is a significant correlation between trip duration and tip percentage.

We use Pearson’s correlation test to measure the relationship.

duckdb.query("""
    SELECT corr(duration_min, tip_percentage) AS correlation
    FROM (
        SELECT duration_min,
               (tip / fare) * 100 AS tip_percentage
        FROM read_csv('/content/cab_ride_data.csv')
        WHERE tip IS NOT NULL AND fare > 0
    )
""").df()

Interpreting the Results

• A correlation close to 1 suggests longer trips strongly correlate with higher tips.
• A correlation close to 0 means no relationship.
• A negative correlation suggests longer trips lead to lower tip percentages (unfortunately!).

Here we get a small negative correlation indicating that longer trips may lead to lower tips.

 	correlation
0 	-0.016022

Wrapping Up

Hypothesis testing is an essential tool for drawing meaningful insights from data. With DuckDB, we can efficiently perform statistical tests directly on large datasets without extensive data loading.

We’ve used the following:

1. Chi-square test evaluates if payment type affects tipping behavior.
2. Correlation test examines whether longer trips lead to higher tip percentages.

Try modifying these queries to test other hypotheses on the dataset. The more questions you ask, the more insights you’ll discover.