Image by Author

More data is almost always assumed to be better. In statistics, larger samples reduce sampling error, increase statistical power, and lead to more reliable conclusions. That assumption is usually correct. But not always.

There are real cases where collecting more data made results harder to interpret, more misleading, or outright wrong. These are not theoretical curiosities. They happened to real researchers, real companies, and real polling organizations. Here are five of the most instructive examples.

1. The Literary Digest Poll That Predicted the Wrong Winner (1936)

In 1936, Literary Digest magazine conducted one of the largest opinion polls in American history. They mailed out 10 million ballots and received 2.4 million responses, predicting that Republican Alf Landon would defeat Franklin D. Roosevelt in a landslide. Roosevelt won by one of the largest margins in U.S. presidential election history.

The problem was not the sample size. The problem was where the ballots were sent. Literary Digest used automobile registration records and telephone directories, sources that were heavily skewed toward wealthier, more Republican-leaning households. The massive sample amplified a biased data collection process, giving the illusion of precision while encoding a systematic error into every single response.

This is the clearest demonstration of a principle that still catches researchers off guard: a larger biased sample does not converge on the truth. It converges on a wrong answer, with growing confidence.

2. Facebook’s Emotional Contagion Study Found Something Real but Meaningless (2014)

In 2014, Facebook researchers published a study based on 689,003 users. They manipulated news feeds to show more positive or negative content and measured whether emotional tone spread between users. The effect was statistically significant across all tests.

The effect size, however, was tiny. The researchers themselves reported that the change in emotional expression amounted to roughly one extra emotional word per thousand words written. At 689,000 people, even a difference that small will register as statistically significant. The study was real, the methodology was sound, and the result was essentially nothing in any practical sense.

Large samples detect everything, including effects too small to matter. When a sample is large enough, a null hypothesis will almost always be rejected, even when the underlying phenomenon would never influence any real decision. Statistical significance tells you whether an effect exists. It does not tell you whether the effect is worth knowing about.

3. Genomic Studies Discovered Thousands of Genes That Explained Nothing (Early 2000s)

When genome-wide association studies became feasible in the early 2000s, researchers scanned hundreds of thousands of genetic variants across large populations, looking for genes linked to conditions like schizophrenia, depression, and diabetes. The larger the study, the more associations they found.

Many of those associations were false positives generated by the multiple comparisons problem. When you test 500,000 genetic variants simultaneously, even a standard significance threshold of p < 0.05 will produce roughly 25,000 false positives by chance alone. Larger datasets meant more variants tested, which meant more spurious findings published as breakthroughs.

The field eventually responded by adopting much stricter significance thresholds for genomic research. But the early wave of literature was partly a consequence of applying conventional standards to an unconventional scale. More data created more opportunities for statistical noise to masquerade as signal.

4. Simpson’s Paradox Reversed the UC Berkeley Admissions Findings (1973)

In 1973, a study of UC Berkeley graduate admissions found what appeared to be clear evidence of gender bias. Across the university as a whole, men were admitted at a meaningfully higher rate than women. The dataset covered over 12,000 applicants, large enough to look conclusive.

When researchers broke the data down by department, the pattern reversed. Women were admitted at equal or higher rates than men in most individual departments. The aggregate result had masked a compositional effect: women tended to apply to more competitive departments with lower overall acceptance rates, which pulled down their university-wide average.

The larger dataset did not clarify the picture. It created a misleading one. Pooling more data across heterogeneous groups can produce aggregate trends that do not exist within any single group, and the fix requires disaggregation, not more data collection.

5. Overfitted Election Forecasting Models Got More Confident as They Got More Wrong (2016)

In the lead-up to the 2016 U.S. presidential election, several forecasting models trained on large volumes of polling and historical data assigned very high probabilities to one outcome. The models had been trained to identify patterns across decades of elections and thousands of polling results.

Some of those models had learned noise rather than signal. They fit the historical data with high precision, capturing patterns that were artifacts of specific past elections rather than generalizable political dynamics. More training data gave the models more structure to fit, which increased their apparent confidence without improving their predictive accuracy.

This is overfitting at scale. A model that memorizes its training data will perform well in-sample and poorly out-of-sample. Adding more data to a model with the wrong structure does not improve it. It often makes it more confidently wrong.

The Pattern Across All Five

Each of these cases shares a common thread. The researchers involved were not careless. In most cases, they were doing exactly what statisticians are trained to do: collect large samples to reduce uncertainty. The problem was that sample size addressed only one source of error, while leaving others untouched or, in some cases, amplifying them.

Bias in data collection does not shrink with sample size. False discovery rates rise when more tests are run. Practical significance and statistical significance diverge as samples grow. Aggregate data can conceal group-level patterns. And complex models can overfit when given more material to memorize.

Sample size is one variable in a much larger equation. It matters enormously, but it is not a substitute for careful study design, clearly defined hypotheses, and honest interpretation of effect sizes. More data gives you more information. What you do with that information is a separate question entirely.