A single value for years has determined the relevance of an observation: the p-value. An observation that has a p-value less than 0.05 is said to be statistically significant, while an observation that does not cross the threshold of significance is conveniently disregarded. It is easy to use, but it loses data that the confidence interval retains.
What A P-Value Actually Measures
The p-value addresses only one question, and a very limited one at that. Given that the null hypothesis is true, how likely is it for the data collected to be as extreme as it actually was? As important as what it does address is what it does not.
The p-value gives no indication of the size of the effect or how well it was estimated, nor of the probability that the hypothesis is true. This is why two experiments can yield the exact same p-value yet produce vastly different estimates.
What A Confidence Interval Adds
A confidence interval conveys information that a p-value omits. A 95% interval reports the range of parameter values compatible with the data. Formally, of all intervals built this way across repeated samples, 95% contain the true parameter.
The width of the interval reflects the precision of the estimate: a narrow interval far from zero points to a well-estimated, sizeable effect, while a wide interval that spans zero points to uncertainty, regardless of whether the matching p-value falls below a threshold.
The interval also contains the significance test. If a 95% interval for a mean difference excludes zero, the two-sided test rejects the null at alpha = 0.05. Reporting the interval loses nothing and adds the magnitude and precision that a bare p-value hides.
The Mechanics And The Choice Of Method
When determining the confidence interval for the mean, the procedure involves finding the estimate ± critical value × standard error, using either the t-distribution or the normal approximation, depending on whether the sample size is small or large.
The confidence intervals for proportions, mean differences, and regression coefficients follow the same formula, estimate ± critical value × standard error, though with different standard error formulas. The difficult part is selecting the appropriate technique, since a one-sample t-interval, a Wilson interval, and a bootstrap interval all rest on different assumptions.
A Short Worked Example
Consider a study where the difference between two groups is 4.2 units and the standard error is 1.6. The 95% confidence interval is 4.2 ± 1.96 × 1.6 = [1.1, 7.3]. Zero is not included in this interval, so this effect is statistically significant at the 5% level.
But the interval says more than that: it shows the true effect could plausibly be as small as roughly 1 unit or as large as roughly 7 units, a distinction “p = 0.02” alone would never have revealed.
How Reporting Norms Are Changing
Methodologists and editors have moved in this direction for years. The American Statistical Association’s 2016 statement warned against treating statistical significance as a stand-in for scientific importance, and reporting guidelines in psychology and medicine now ask for effect sizes and intervals alongside, or in place of, p-values.
This is not a rejection of hypothesis testing, but a correction to how its output is read.
Tools For Computing And Checking Intervals
Once the method is fixed, computing an interval is a mechanical process, and a verified calculator is faster and less error-prone than working out standard errors by hand. Omni Calculator maintains a large, freely accessible library of statistical tools covering confidence intervals, standard errors, t-tests, effect sizes, and related procedures.
Each tool is paired with an explanation of the formula it applies and the assumptions it relies on, making quantitative methods usable without hiding the reasoning behind them.
Putting It Into Practice
Adopting intervals costs little. Report the point estimate, the interval, and the method; keep the p-value if a journal requires it; and let the interval do the interpretive work. A line such as “the mean difference was 4.2 points (95% CI: 1.1 to 7.3)” tells a reader the direction, the size, and the precision of the effect at once, which is everything “p = 0.02” leaves unsaid.
The habit is easy to build and hard to regret, and it moves a results section from a verdict toward an estimate, which is what most analyses were trying to produce in the first place.
