Image by Author

P-value confusion is the single most common statistical mistake I see when mentoring data science students. Students who can calculate p-values perfectly still stumble when interpreting what these numbers actually mean. The consequences go beyond grades: I’ve seen professionals make critical business decisions based on fundamental p-value misinterpretations, leading to wasted resources and flawed conclusions.

The problem isn’t that p-values are inherently difficult. Statistical education often focuses on computation while glossing over interpretation. In this guide, I’ll share the five most common p-value mistakes I see students make, drawn directly from patterns I’ve observed in hundreds of hypothesis testing sessions.

Quick Context: What Is a P-Value?

A p-value measures the probability of obtaining results at least as extreme as those you observed (or more extreme) if the null hypothesis were true. It’s not the probability that your hypothesis is correct, nor is it the probability that your results occurred by chance. Think of it as a measure of surprise: how surprising would your data be if there were truly no effect?

P-values appear throughout statistical testing to help researchers make decisions about null hypotheses. You’ll see them in t-tests, ANOVA, regression analysis, and countless other statistical procedures.

Common Mistake #1: Thinking p < 0.05 “Proves” Your Hypothesis

Students often get this wrong by treating p-values as proof. I hear statements like “My p-value is 0.03, so my hypothesis is true!” at least once per week. Just last month, a student concluded that caffeine “definitely increases” productivity because their t-test returned p = 0.04.

This happens because we’re taught to “reject the null hypothesis” when p < 0.05, and students equate rejection of one hypothesis with confirmation of another. But p-values only tell you about the null hypothesis. A small p-value means your data would be surprising if the null hypothesis were true, nothing more.

The fix: Change your language entirely. Instead of “proving” your hypothesis, say “the data are inconsistent with the null hypothesis,” or “we found evidence against the null hypothesis.” When that student reanalyzed their caffeine study, we reframed the conclusion as “the data suggest an association between caffeine and productivity, inconsistent with no effect.”

Common Mistake #2: Using P-Values to Compare Effect Sizes

Students routinely compare p-values to judge which effect is larger. I’ve reviewed dozens of analyses where students concluded “Variable A has a stronger effect than Variable B because p = 0.001 versus p = 0.04.” This represents a fundamental confusion between statistical significance and practical magnitude.

P-values are functions of both effect size and sample size. A tiny, practically meaningless effect can have a very small p-value if you have enough data. A large, important effect might have p = 0.08 with a small sample.

Recently, a student was analyzing survey data comparing two teaching methods. Method A showed a 2-point improvement (p = 0.001, n = 500) while Method B showed a 15-point improvement (p = 0.06, n = 30). They initially concluded Method A was “more effective” based solely on p-values.

The solution: Always calculate and report actual effect sizes. Use Cohen’s d for t-tests, R-squared for regression, or raw mean differences. These tell you how large the effect is, independent of sample size. When the student added effect sizes to their analysis, they recognized that Method B’s larger improvement deserved serious consideration despite the higher p-value.

Common Mistake #3: Running Multiple Tests Without Correction

This is the mistake I see most often in student projects involving exploratory analysis. Students run 15 or 20 t-tests comparing different groups, then report only the “significant” findings. The problem: when you run 20 tests at α = 0.05, you expect about one false positive purely by chance.

Students fall into this trap because each individual test seems legitimate. They’re not intentionally p-hacking; they genuinely don’t understand the multiple comparisons problem. I recently worked with a student analyzing gene expression data who ran 50 comparisons and excitedly reported the five “significant” findings. They hadn’t considered that some of their “discoveries” might be statistical noise.

The correction is straightforward but requires discipline. If you’re running multiple tests, apply a correction method like Bonferroni (divide your α by the number of tests) or control the False Discovery Rate with the Benjamini–Hochberg procedure. For that gene expression student, after applying FDR correction, only two of their five “discoveries” remained significant.

Common Mistake #4: Interpreting Non-Significant Results as “No Effect”

When students get p = 0.08 or p = 0.12, I consistently hear “there is no relationship” or “there is no effect.” This binary thinking about the 0.05 threshold leads to completely dismissing potentially important findings.

The mistake happens because statistical education emphasizes the significance threshold without discussing what happens near the boundary. Students learn the rule “p < 0.05 is significant” and conclude that anything above this line means nothing happened.

Three weeks ago, a student was analyzing the relationship between study hours and exam scores. Their correlation test returned p = 0.09, and they concluded “study hours don’t affect exam scores.” When we examined the data together, there was a clear positive relationship; the study was simply underpowered with only 25 students.

The fix requires looking beyond the p-value. Examine confidence intervals, consider statistical power, and look at the actual data. For continuous relationships, report the effect size and its uncertainty rather than making binary significant/not significant declarations. That exam-scores analysis became much more informative when we reported “each additional study hour was associated with a 3.2-point increase in exam scores (95% CI: -0.5 to 6.9, p = 0.09), suggesting a positive relationship worth investigating with a larger sample.”

Common Mistake #5: Ignoring Test Assumptions Before Computing P-Values

Students rush to get a p-value without checking whether their chosen test is appropriate for their data. I’ve seen t-tests applied to highly skewed data, parametric tests used on ordinal scales, and independence assumptions violated in repeated-measures designs. The resulting p-values are meaningless.

This happens because assumption checking feels like extra work that delays the “real” analysis. Students want to answer their research question, and checking assumptions seems like a bureaucratic hurdle. Last week, a student proudly showed me their analysis comparing customer satisfaction scores (on a 1–5 scale) between three stores with ANOVA, getting p = 0.03. The data were severely non-normal with clear ceiling effects.

The solution: Always check assumptions before interpreting p-values. For t-tests and ANOVA, check normality and equal variances. For regression, examine residuals for linearity, independence, normality, and equal variance. If assumptions are violated, either transform your data, use a nonparametric alternative, or use a more appropriate model. When that student switched to the Kruskal–Wallis test (appropriate for ordinal data), their p-value changed to 0.18, completely reversing their conclusion.

How to Avoid These Mistakes: Quick Checklist

Before you interpret any p-value, run through this checklist:

• Remember that p-values test the null hypothesis, not your alternative hypothesis
• Never use p-values to compare effect sizes; calculate actual effect sizes instead
• Apply multiple comparison corrections when running more than one test
• Treat p-values as continuous measures of evidence, not binary declarations
• Check all test assumptions before computing p-values
• Report confidence intervals alongside p-values
• Consider your statistical power, especially for non-significant results
• Use precise language: “inconsistent with the null” rather than “proves my hypothesis”

Conclusion

These five mistakes represent the most common p-value interpretation errors I’ve observed in student work. Once students understand these pitfalls, their statistical reasoning improves dramatically. Recognizing these patterns in your own work is the first step toward more rigorous analysis.