Understanding p-Hacking and Researcher Degrees of Freedom

Image by Author | Canva

Introduction

In this article, we’re talking about p-hacking, a way of analyzing data until you get results that look statistically significant. It’s more common than you’d think and often leads to findings that can’t be trusted or reproduced. The issue usually revolves around something called the p-value.

A p-value is a number that helps researchers determine whether their results might have happened by random chance. A common threshold is 0.05, which means that if your p-value is below that, your result is considered statistically significant. If you run enough different analyses, eventually, something will fall below 0.05 just by luck, even if there’s no real effect.

This is where p-hacking is introduced. It happens when researchers try different combinations of tests, variables, or subsets of data until they get that magic p < 0.05, and then only report those results.

How p-Hacking Actually Happens in Practice

p-Hacking isn’t always done on purpose. A lot of the time, it’s unintentional; researchers want interesting results, so they keep exploring until they find something. But here’s what it can look like:

Running lots of tests and only reporting the one that “works”: Maybe someone tests 20 different variables, and just one shows a p-value below 0.05. Instead of saying, I tested 20 things, they just present that one as if it’s meaningful.
Collecting more data after seeing your results: Sometimes, when researchers run an analysis and they don’t get the result they hoped for, they decide to add more data until the numbers look better. This is risky because it increases the chance of getting a false positive.
Trying different ways to define your variables: For example, if a developer is studying stress, they might define it as sleep hours, then switch to heart rate, then to blood pressure, until something gives them a significant result.
Dropping or keeping certain data points: Removing a few outliers or choosing to analyze only a specific group can also shift results.

Why This Is Statistically Misleading

p-Hacking creates a false sense of confidence. If you’re running 20 tests, the chance of at least one showing significance just by random chance is pretty high. But if you don’t tell people you ran 20 tests, they assume that one result is meaningful on its own.

This inflates false positives and leads to findings that can’t be repeated, which is a big reason why some research doesn’t hold up when other researchers try to replicate it.

But on the other hand, it’s not just about bad intentions.

Most p-hacking doesn’t come from people trying to cheat, it comes from curiosity, pressure to publish, or just not being aware of the risks. But good intentions don’t change the fact that the results can be misleading.

So, why does this matter? Science relies on trust and reproducibility. If people can’t reproduce results, then the whole point of doing research starts to fall apart. That’s why understanding p-hacking matters. It helps both researchers and readers become more critical about what results really mean.

How Researcher Choices Can Quietly Shape Results

The term “researcher degrees of freedom” was popularized in Simmons, Nelson, and Simonsohn’s 2011 paper False-Positive Psychology. It refers to all the decisions researchers make while analyzing data and how those choices, even if they seem harmless, can quietly increase the chances of getting a false positive (a.k.a. saying something is real when it’s not).

These decisions include things like:

When to stop collecting data
Which variables to include or exclude
How to handle outliers
Which statistical tests to run
How to define and group the sample

So here is the catch: these decisions aren’t always made ahead of time. They often happen after the researcher has seen the data. That kind of flexibility can easily lead to tweaking the results, sometimes even without noticing it’s happening.

In their study, Simmons et al. showed that just a few of these flexible decisions could inflate the false positive rate from the standard 5% to over 60%. That means you could be publishing results that look statistically solid, but are actually random noise dressed up as insight.

This isn’t just theory, it’s something that’s been seen across fields like psychology, economics, medicine, and beyond. The problem often isn’t bad intentions; it’s that the system rewards significant results, and researchers (like anyone else) tend to gravitate toward what gets published.

That kind of flexibility can easily lead to tweaking the results, sometimes even without noticing it’s happening.

In the next section, we will take a look at the idea of p-hacking and researcher degrees of freedom using a simple Python simulation. This will show how running multiple tests without a clear hypothesis can lead to false positives, even when the data shows no real effect.

p-Hacking Python Simulation

The Python script we will write will generate two groups of random data from the same distribution (i.e., there is no real difference between them). Then we will run multiple t-tests by trying different groupings or slicing the data in various ways. Finally, we will track how often we get a statistically significant p-value (i.e., p < 0.05), even though nothing is actually different.

import numpy as np
import scipy.stats as stats
import random

# Set random seed for reproducibility
np.random.seed(42)

# Step 1: Simulate two groups from the same distribution
group_a = np.random.normal(loc=50, scale=10, size=100)
group_b = np.random.normal(loc=50, scale=10, size=100)

# Step 2: Simulate researcher degrees of freedom
# Try multiple slicing conditions and record how often we get a false positive
significant_results = 0
total_tests = 0

# Simulate 1000 different random test combinations
for i in range(1000):
    # Simulate choosing random slices from the data
    slice_a = random.sample(list(group_a), k=random.randint(20, 100))
    slice_b = random.sample(list(group_b), k=random.randint(20, 100))

    # Run an independent t-test
    t_stat, p_value = stats.ttest_ind(slice_a, slice_b)

    total_tests += 1
    if p_value < 0.05:
        significant_results += 1

# Step 3: Report how many "significant" results we got by chance
false_positive_rate = (significant_results / total_tests) * 100
print(f"Out of {total_tests} random tests, {significant_results} were 'significant' (p < 0.05)")
print(f"False positive rate: {false_positive_rate:.2f}%")

In the above code, we created two statistically similar groups. However, by randomly slicing the data and running 1000 t-tests, which simulates a researcher trying many combinations, we ended up getting several results that were statistically significant.

This is simulated p-hacking in action. The more tests you run, the more likely you are to find something significant purely by chance, and that something can make it into a published paper if you're not careful.

This shows why it’s important to:

Pre-register your hypotheses
Limit flexibility in your analysis
Correct for multiple comparisons (e.g., Bonferroni correction)
Be transparent about your methodology

Wrapping Up: What We’ve Learned About p-Hacking

In this article, we walked through what p-hacking really looks like, and simulated it in Python code. We explored how running lots of tests or tweaking analysis decisions (even with no real effect on the data) can lead to false positives. It’s not always done with bad intent, but the result is the same: findings that can’t be trusted or reproduced.

We also introduced the idea of researcher degrees of freedom, which refers to how much wiggle room researchers have in making decisions about data and how that flexibility, if unchecked, can compromise a study's integrity.

If you’re doing research or working with data, the big takeaway here is this: be mindful of how you analyze your data. Try to plan ahead, document your process, and don’t just chase p-values. It’s tempting, but it hurts good science.

There’s so much more to explore, like how to correct for multiple comparisons, the role of pre-registration, or how tools like Bayesian analysis offer alternatives. So keep digging. And if you ever find yourself getting that magical p < 0.05 result after trying a bunch of different things... pause, and ask yourself why.