Hypothesis Testing & Statistical Errors
When we are working with data, we want to come to some conclusion based on what is available. To infer the actual population from the data sample, we rely on a statistical hypothesis test to support our conclusion. Hypothesis testing is a particular statistical method to make inferences about the population based on the sample data. The way it works is often by setting the null hypothesis and the alternative hypothesis.
For example, in an independent t-test where we compare means from two different groups, the null hypothesis (often denoted as H0) states there are no significant differences between the means. In contrast, the alternative hypothesis (often denoted as H1) states that there is a difference between the means.
It is important to note that hypothesis tests rely on underlying assumptions — such as the normality of the data and the equality of variances — that must be met to ensure the validity of the conclusions.
However, there is a chance that the hypothesis testing result needs to be corrected and leads to incorrect conclusions. This is what we call a statistical error.
In general, there are two types of statistical errors: type I and type II errors. A type I error is where we have a false positive conclusion, while a type II error is when we have a false negative conclusion.
An easy to understand visualization of type I and type II errors
Statistical Power
Statistical power is the probability of detecting an effect if there is one (i.e., correctly rejecting a false null hypothesis). If the likelihood of type II error is signified as β (beta), then power can be denoted as 1 – β. Power, then, can be said to be the inverse of the type II error rate probability.
The closer the power is to 1, the more negligible risk of committing the false negative and vice versa. Commonly, the researcher set the power around 0.8 or 80%, as lower than this would be considered too high a risk for a false negative to occur.
Four things affect the statistical power, including:
- Significance level α (alpha)
- Sample size
- Variance
- Effect size (quantified magnitude calculated with specific statistical measures such as Pearson’s correlation)
Researchers often adjust these 4 factors — especially sample size and significance level — to design studies with adequate power while controlling for errors. In general, statistical power is increased when we increase sample size, effect size, and significance levels. While raising the significance level (α) increases power, it also increases the risk of committing a type I error (false positive). This nuance is an important trade-off to understand in experimental design.
Given the parameters above, we can calculate the statistical power. However, it’s often too complex to calculate power by hand. That’s why we can instead rely on computational tools such as SPSS or Python execution.
from statsmodels.stats.power import TTestIndPower effect_size = 0.5 alpha = 0.05 sample_size = 100 power_analysis = TTestIndPower() calculated_power = power_analysis.power(effect_size=effect_size, nobs1=sample_size, alpha=alpha, ratio=1.0, alternative='two-sided') print(calculated_power)
0.9404271933839894
In the code above you can see that effect_size is set to 0.5. In the context of the independent t-test, this effect size is typically interpreted as Cohen’s d, which you can find out more about here.
Bear in mind that power analysis is commonly performed during the planning phase of a study to determine the necessary sample size, and is not a post-hoc consideration or calculation.
