Image by Editor

Running an experiment, analyzing a dataset, and getting a “no significant difference” result — i.e. failing to reject a null hypothesis, which is also known as “Type II Error” — might often sound disappointing, to the point of abandoning an idea you might have had in mind. But there’s a catch: sometimes that idea might actually be brilliant despite those results, just your data didn’t “speak sufficiently loud to be listened to”.

This is what occurs when using a small sample size: it can remarkably affect how much of the “data truth” calculations are able to detect. Let’s break down three reasons why such small samples can diminish the meaningfulness of results:

Triggering a “False Negative”

These are the so-called Type II Errors at their maximum expression. In statistics, these errors signify a dramatic connection that went missing. Put another way, there is a meaningful effect in the real-world scenario the data comes from, but your test over the gathered sample failed to spot it.

Suppose you invented a miraculous remedy to cure headaches. Testing it on five people and (coincidentally) having two of them naturally resistant to it would statistically yield a group result equivalent to failure, even though the drug truly works.

Suffering from “Low Power”

Statistical power is comparable to a digital camera’s resolution. Trying to immortalize a small bird sitting in a tree branch far away with a 2-megapixel camera will likely result in a blurry spot. The bird was there, but the equipment did not have enough resolution to properly photograph it. Small sample sizes are just like these low-resolution cameras, and in such situations, the low statistical power conveyed by a tiny sample is a scientific guarantee of failure even before starting calculations.

Mistaking “Not Proven” for “Not Real”

Most people mistakenly interpret a “statistically non-significant” result yielded by a small sample as proof that their idea or hypothesis was wrong. It is a tempting error to think the math says “your idea was false”, but in fact, what it says is “you didn’t provide me enough information to make a reliable, convincing decision”.

In the end of the day, statistical formulae — including hypothesis testing ones — have by default a cautious behavior. When not enough data is provided, they default to saying nothing happens — even though there might be a massive discovery underneath.