Image by rawpixel.com on Magnific
Data dredging is the practice of running many statistical tests on the same dataset and reporting only the ones that reach statistical significance. It also goes by p-hacking, data snooping, data fishing, and significance chasing.
The problem it creates is not a matter of intuition. It is arithmetic: every additional test raises the probability that at least one result clears the significance threshold on chance alone, even when no real effect exists anywhere in the data.
Data Dredging: Why It Inflates False Positives
Every hypothesis test carries a fixed risk of a false positive.
At a significance level of α = 0.05, a test of a true null hypothesis returns a significant result 5% of the time.
Suppose an analyst runs 20 such tests. Does this mean the analysis carries twenty separate 5% risks, each isolated from the others?
Not quite. The quantity that matters is the probability that at least one test in the whole set produces a false positive.
That probability climbs with every test added, and it climbs faster than a simple sum of the individual risks would suggest.
Each test still operates at its stated α. What changes is the size of the family.
The more chances a researcher gives noise to clear the threshold, the more often noise clears it.
Data Dredging: The Family-Wise Error Rate
The family-wise error rate is the probability that a set of tests produces at least one false positive.
For m independent tests, each conducted at significance level α, it is calculated as:
FWER = 1 − (1 − α)m
where:
- α: the significance level of each individual test
- m: the number of tests performed
- FWER: the probability of at least one false positive across all m tests
The term (1 − α) is the probability that a single test avoids a false positive, and (1 − α)m is the probability that all m tests avoid one. Subtracting from 1 gives the probability that at least one test does not.
This formula assumes the m tests are independent of one another.
Data Dredging: An Example
Suppose a marketing analyst wants to know which visitor characteristics are associated with conversion rate on a company website.
She has 20 unrelated visitor metrics available, including session duration and referral source, and she tests each one against conversion rate at a significance level of α = 0.05.
Step 1: State the setup
- α: significance level of each test = 0.05
- m: number of tests performed = 20
Assume that none of the 20 metrics has any real association with conversion rate, so every null hypothesis is true.
Step 2: Apply the family-wise error rate formula
FWER = 1 − (1 − α)m
Step 3: Calculate the result
FWER = 1 − (1 − 0.05)20 = 1 − (0.95)20 = 0.6415
Step 4: Interpret the result
There is a 64% chance that at least one of the 20 metrics reads as statistically significant, even though none of them is associated with conversion rate at all.
A single test at the same significance level carries only a 5% risk.
Step 5: Draw a conclusion
An analyst who runs all 20 tests and reports only the significant ones is more likely than not to be reporting noise.
Data Dredging: The Bonferroni Correction
The Bonferroni correction controls the family-wise error rate by lowering the threshold each individual test must clear.
The corrected significance level is calculated as:
αcorrected = α / m
where:
- α: the desired family-wise significance level
- m: the number of tests performed
- αcorrected: the significance level applied to each individual test
For the marketing analyst’s 20 tests at a family-wise level of α = 0.05, the corrected threshold is:
αcorrected = 0.05 / 20 = 0.0025
Each of her 20 tests must now return a p-value below 0.0025 to be declared significant, rather than 0.05.
The family-wise error rate across her full set falls to 1 − (1 − 0.0025)20 = 0.0488.
The correction isn’t without trade-off; raising the bar for each test reduces the power to detect effects that are real, which increases the Type II error rate.
