In 2011, three psychologists ran an experiment that had nothing to do with the topic they claimed to be studying. They wanted to know how easy it is to make a false finding look statistically real using nothing but ordinary, common flexibility in how a study gets analyzed, checking results a bit early, trying a second outcome measure, and dropping an awkward condition. Their paper found that stacking a handful of these everyday choices together could push a test’s false-positive rate as high as 60%, in a test that’s supposed to be wrong only 5% of the time by design.

That’s the real scale of the problem this article is about. P-hacking isn’t usually a scientist inventing data. It’s a hundred small, defensible-sounding decisions made after the results are already in view, each one nudging a null finding a little closer to “significant” until it crosses the line. Most of the time, nobody involved would call what they did fraud. That’s exactly what makes it worth knowing how to spot from the outside, since the paper itself will never announce that it happened.

Here are five concrete signs worth checking before trusting a “statistically significant” claim, plus a short simulation further down that shows exactly how much these practices distort the numbers.

A Ten-Second Refresher on What a P-Value Actually Claims

A p-value answers one narrow question: if there were truly no effect at all, how likely is it that you’d see data this extreme purely by chance? A p-value under the conventional 0.05 threshold means that, assuming no real effect exists, a result this strong would show up by chance less than 5% of the time. It does not mean the hypothesis is 95% likely to be true, and it says nothing about how large or important the effect actually is. Keep that narrow definition in mind, because every red flag below is really a different way of asking the same question: was that 5% threshold earned honestly, or was it reached by trying enough things until something crossed the line?

Red Flag 1: The P-Value Lands Suspiciously Just Under the Cutoff

Watch for results reported as p = .048, or p = .049, sitting right under the 0.05 line rather than comfortably below it. One result like that means nothing on its own. A pattern of it across a field is a different story.

In 2015, a team of researchers text-mined the p-values reported across more than 100,000 papers and found a real, measurable cluster of results bunched just under the 0.05 threshold, more than would be expected if researchers were reporting whatever their analysis produced with no further adjustment. That’s consistent with a specific, common behavior: running an analysis, checking the number, and stopping the moment it crosses the line rather than continuing to a cleaner, pre-planned result. A single borderline p-value isn’t proof of anything. A field’s worth of them clustering suspiciously close to the pass mark is a real, documented pattern worth remembering the next time a headline leans on a p-value of exactly .049.

Red Flag 2: Significance Without a Reported Effect Size

A p-value tells you whether an effect is probably not zero. It tells you nothing about whether that effect is big enough to care about. A large enough sample can turn a genuinely trivial difference — a fraction of a point on a scale, a few seconds on a task — into a “statistically significant” result, because statistical significance is partly just a function of how much data you collected.

The tell is a paper that reports a p-value prominently and never states the actual size of the effect: no percentage difference, no standardized measure, nothing a reader could use to judge whether the finding matters practically. A study that found a real, honest effect usually wants to tell you how big it is, because a bigger effect is more impressive. A study leaning on significance alone, with the actual magnitude left vague or absent, is a reasonable place to slow down and ask what number is missing and why.

Red Flag 3: The Sample Size or Stopping Rule Was Never Locked In

This is the practice known as optional stopping, and it’s one of the most common forms of p-hacking precisely because it feels methodologically responsible rather than dishonest. A researcher checks their results partway through data collection. If the result is already significant, they stop, since collecting more data “just to be thorough” feels unnecessary. If it isn’t significant yet, they collect a bit more and check again.

The problem is that this single habit invalidates the 5% error rate on which the whole test depends. Every time you peek at the data and get to decide whether to stop, you’re giving chance another opportunity to produce a fluke result, and stopping only when it looks good means you’re selectively keeping the flukes and discarding the honest non-results. The fix researchers are supposed to use — a pre-registered sample size decided before data collection starts — exists specifically to close this loophole, and its absence, a paper that never states its stopping rule or planned sample size in advance, is a legitimate thing to notice.

Red Flag 4: The Hypothesis Conveniently Matches Whatever Came Out Significant

This one has a name: HARKing, short for Hypothesizing After the Results are Known. It happens when a researcher runs an exploratory analysis, notices an interesting pattern in the data that they weren’t specifically looking for, and then writes the paper’s introduction as though that pattern had been the hypothesis all along.

The result reads as clean, confirmatory science: a clear prediction, a clear test, a clear confirmation. What’s hidden is that the prediction was written after the answer was already known, which means it was never actually at risk of being wrong. A useful, if imperfect, reader-level check is comparing how specific and confident a paper’s stated hypothesis sounds against how exploratory or broad its actual methods section seems. A tightly specific hypothesis paired with an analysis that clearly explored several possible outcomes before landing on the one reported is worth a second look.

Red Flag 5: Nobody Else Can Reproduce It

This is the strongest signal on this list, because it’s not circumstantial; it’s a direct test. If a finding is real, an independent team following the same method on new data should get a similar result most of the time. If a finding only existed because of the flexible choices covered in flags 1 through 4, it usually won’t survive that test.

In 2015, the Open Science Collaboration attempted to replicate 100 published psychology studies using the original methods and materials wherever possible. Fewer than half produced a statistically significant result the second time around. That’s not a fringe result from one skeptical team; it’s one of the most cited findings in the broader replication crisis conversation, and it’s a direct, measured consequence of exactly the practices described above being common enough to shape published literature at scale. A related, quieter version of this same red flag is a paper whose methods section is too vague to actually replicate from — missing exact measures, unclear exclusion criteria, and no shared data or code. A study that can’t be checked is a study whose “significant” result you’re being asked to take on faith.

P-Hacking Simulation

The five flags above describe patterns. Here’s what they actually do to the numbers, demonstrated directly rather than just asserted. This simulation runs two groups of data drawn from the exact same distribution, meaning there is, by construction, no real effect to find, and tests how often each analysis style still produces a “significant” result anyway.

Prerequisites: Python 3.9 or newer, pip install numpy scipy matplotlib.

"""
p_hacking_simulation.py
Simulates the false-positive rate under three conditions, all of which
have NO real effect (both groups drawn from the identical distribution),
to show how p-hacking practices inflate the rate of false "significant"
findings well above the nominal 5% alpha level.
"""
import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=42)
N_SIMULATIONS = 10000
ALPHA = 0.05

def honest_test(n_per_group=20):
    """A single, pre-registered test at a fixed sample size. No peeking."""
    group_a = rng.normal(0, 1, n_per_group)
    group_b = rng.normal(0, 1, n_per_group)  # identical distribution, no real effect
    _, p = stats.ttest_ind(group_a, group_b)
    return p < ALPHA

def optional_stopping_test(start_n=10, max_n=50, check_every=5):
    """Peek at the p-value every few participants and stop as soon as
    it crosses significance, a classic p-hack."""
    group_a = list(rng.normal(0, 1, start_n))
    group_b = list(rng.normal(0, 1, start_n))
    for n in range(start_n, max_n, check_every):
        _, p = stats.ttest_ind(group_a, group_b)
        if p < ALPHA:
            return True  # stopped early because it "worked"
        group_a.extend(rng.normal(0, 1, check_every))
        group_b.extend(rng.normal(0, 1, check_every))
    _, p = stats.ttest_ind(group_a, group_b)
    return p < ALPHA

def multiple_dv_test(n_per_group=20, n_outcomes=5):
    """Measure five outcome variables, all with no real effect, and
    report a 'finding' if ANY single one crosses significance."""
    for _ in range(n_outcomes):
        group_a = rng.normal(0, 1, n_per_group)
        group_b = rng.normal(0, 1, n_per_group)
        _, p = stats.ttest_ind(group_a, group_b)
        if p < ALPHA:
            return True
    return False

def combined_p_hack(start_n=10, max_n=50, check_every=5, n_outcomes=3):
    """Stack both practices together: peek early on several outcome
    variables at once, and stop as soon as anything crosses significance."""
    groups_a = [list(rng.normal(0, 1, start_n)) for _ in range(n_outcomes)]
    groups_b = [list(rng.normal(0, 1, start_n)) for _ in range(n_outcomes)]
    for n in range(start_n, max_n, check_every):
        for i in range(n_outcomes):
            _, p = stats.ttest_ind(groups_a[i], groups_b[i])
            if p < ALPHA:
                return True
        for i in range(n_outcomes):
            groups_a[i].extend(rng.normal(0, 1, check_every))
            groups_b[i].extend(rng.normal(0, 1, check_every))
    for i in range(n_outcomes):
        _, p = stats.ttest_ind(groups_a[i], groups_b[i])
        if p < ALPHA:
            return True
    return False

results = {}
for label, fn in [
    ("Honest, single pre-registered test", honest_test),
    ("Optional stopping (peek and stop early)", optional_stopping_test),
    ("Multiple outcome variables (report any hit)", multiple_dv_test),
    ("Combined: optional stopping + multiple DVs", combined_p_hack),
]:
    false_positives = sum(fn() for _ in range(N_SIMULATIONS))
    rate = false_positives / N_SIMULATIONS
    results[label] = rate
    print(f"{label}: {rate:.1%} false-positive rate (nominal alpha = {ALPHA:.0%})")

# --- Chart: false-positive rate by practice ---
import matplotlib.pyplot as plt

labels = list(results.keys())
rates = [results[l] * 100 for l in labels]
short_labels = ["Honest test", "Optional\nstopping", "Multiple\noutcomes", "Combined\np-hack"]

fig, ax = plt.subplots(figsize=(8, 5))
bars = ax.bar(short_labels, rates, color=["#4C9F70", "#E8A33D", "#E8A33D", "#D64545"])
ax.axhline(5, color="black", linestyle="--", linewidth=1, label="Nominal 5% alpha")
ax.set_ylabel("False-positive rate (%)")
ax.set_title("How p-hacking practices inflate false positives\n(10,000 simulations per condition, no real effect exists)")
ax.legend()
for bar, rate in zip(bars, rates):
    ax.text(bar.get_x() + bar.get_width() / 2, rate + 1, f"{rate:.1f}%", ha="center")
plt.tight_layout()
plt.savefig("p_hacking_false_positive_rates.png", dpi=150)
print("\nChart saved to p_hacking_false_positive_rates.png")

Output:

p_hacking_false_positive_rates

Code explanation:

• Every group in every condition is drawn from the exact same distribution (rng.normal(0, 1, ...) for both groups every time), so there is, by construction, no real effect anywhere in this script. Anything that comes back "significant" is a false positive.
• honest_test runs one t-test at a fixed sample size with no peeking — the baseline case that should land at roughly the nominal 5% alpha.
• optional_stopping_test and multiple_dv_test each simulate one common p-hacking habit on its own: checking the p-value every few participants and stopping the moment it crosses 0.05, or testing five outcome variables and reporting a "finding" if any single one crosses the line.
• combined_p_hack stacks both habits together, peeking early across several outcome variables at once, which is closer to how p-hacking usually happens in practice — more than one flexible choice stacked at the same time rather than just one.
• The final loop runs each condition 10,000 times, tallies how often each one falsely crossed p < .05, then plots all four false-positive rates as a bar chart against a dashed line marking the honest 5% baseline, so the inflation is visible at a glance.

How to run it: save the file as p_hacking_simulation.py, then from a terminal with Python 3.9+ installed, run:

python p_hacking_simulation.py

What These Red Flags Don't Mean

A borderline p-value, a modest sample size, or a paper that doesn't headline its effect size isn't proof of misconduct on its own, and treating any single flag as a verdict is its own kind of overreach. Most p-hacking isn't a deliberate act of deception; it's what happens when a researcher trained under real-time pressure and funding constraints makes a series of individually reasonable-feeling decisions without realizing how much they compound. The five signs above are worth using as reasons to look closer at the methods section, at whether the study was pre-registered, at whether anyone has tried to replicate it — not as a checklist for declaring a study fraudulent from the abstract alone.

Wrapping Up

The practical version of everything above fits in one habit: the next time a headline leans on the phrase "statistically significant," look past the p-value itself and ask three questions. Was the sample size and analysis plan decided before the data came in? Is the actual size of the effect reported anywhere, not just its significance? And has anyone besides the original team gotten the same result? A study that answers all three cleanly has earned the claim it's making. One that dodges all three might still be right, but you're the one taking it on faith — and now you know exactly what to check before you do.