In a hypothesis test, there is always a type I error rate that tells us the probability of rejecting a null hypothesis that is actually true. In other words, it’s the probability of getting a “false positive”, i.e. when we claim there is a statistically significant effect, but there actually isn’t.
When we perform one hypothesis test, the type I error rate is equal to the significance level (α), which is commonly chosen to be 0.01, 0.05, or 0.10. However, when we conduct multiple hypothesis tests at once, the probability of getting a false positive increases.
For example, imagine that we roll a 20-sided dice. The probability that the dice lands on a “1” is just 5%. But if we roll two of these dice at once, the probability that one of the dice will land on a “1” increases to 9.75%. If we roll five dice at once, the probability increases to 22.6%.
The more dice we roll, the higher the probability that one of the dice will land on a 1. Similarly, if we conduct several hypothesis tests at once using a significance level of .05, the probability that we get a false positive increases to beyond 0.05.
How to Estimate the Family-wise Error Rate
The formula to estimate the family-wise error rate is as follows:
Family-wise error rate = 1 – (1-α)n
where:
- α: The significance level for a single hypothesis test
- n: The total number of tests
For example, suppose we conduct 5 different comparisons using an alpha level of α = .05. The family-wise error rate would be calculated as:
Family-wise error rate = 1 – (1-α)c = 1 – (1-.05)5 = 0.2262.
In other words, the probability of getting a type I error on at least one of the hypothesis tests is over 22%!
How to Control the Family-wise Error Rate
There are several methods that can be used to control the family-wise error rate, including:
1. The Bonferroni Correction.
Adjust the α value used to assess significance such that:
αnew = αold / n
For example, if we conduct 5 different comparisons using an alpha level of α = .05, then using the Bonferroni Correction our new alpha level would be:
αnew = αold / n = .05 / 5 = .01.
2. The Sidak Correction.
Adjust the α value used to assess significance such that:
αnew = 1 – (1-αold)1/n
For example, if we conduct 5 different comparisons using an alpha level of α = .05, then using the Sidak Correction our new alpha level would be:
αnew = 1 – (1-αold)1/n = 1 – (1-.05)1/5 = .010206.
3. The Bonferroni-Holm Correction.
This procedure works as follows:
- Use the Bonferroni Correction to calculate αnew = αold / n.
- Perform each hypothesis test and order the p-values from all tests from smallest to largest.
- If the first p-value is greater than or equal to αnew, stop the procedure. No p-values are significant.
- If the first p-value is less than αnew, then it is significant. Now compare the second p-value to αnew. If it’s greater than or equal to αnew, stop the procedure. No further p-values are significant.
By using one of these corrections to the significance level, we can dramatically reduce the probability of committing a type I error among a family of hypothesis tests.
“Dice” is the plural of “die.” So, “imagine that we roll a 20-sided die,” not “… a 20-sided dice.”
I couldn’t stoping thinking the same thing
ok. I got it. actually I hope so. for example I have 3 groups for comparing volumes of 142 brain region for each group. Should I do FWER tests? why or not?
Hi halil…When comparing volumes of 142 brain regions across three groups, you are dealing with a large number of statistical tests. In such situations, controlling the Family Wise Error Rate (FWER) becomes important to reduce the probability of making false discoveries. Here’s why and how you might want to consider using FWER tests:
### Why Control FWER?
1. **Multiple Comparisons Problem**:
– When performing multiple statistical tests, the chance of committing at least one Type I error (false positive) increases. For example, if you perform 142 tests at a significance level of 0.05, the probability of making at least one Type I error is much higher than 0.05.
2. **Interpretation of Results**:
– Controlling FWER helps ensure that the overall probability of making one or more false discoveries among all the tests remains at the desired significance level (e.g., 0.05). This makes the results more reliable and interpretable.
### Methods to Control FWER
There are several methods to control the FWER, each with its own advantages and limitations:
1. **Bonferroni Correction**:
– **Method**: Divide the desired alpha level (e.g., 0.05) by the number of tests (e.g., 142).
– **Example**: If you have 142 tests and want to maintain an overall alpha of 0.05, the corrected alpha for each test would be 0.05/142 ≈ 0.00035.
– **Pros**: Simple and conservative.
– **Cons**: Can be overly conservative, leading to a higher likelihood of Type II errors (false negatives).
2. **Holm-Bonferroni Method**:
– **Method**: A stepwise approach that adjusts the p-values in a less conservative manner than the Bonferroni correction.
– **Pros**: Less conservative than Bonferroni, reducing the risk of Type II errors.
– **Cons**: Still conservative and might not be the best for highly correlated tests.
3. **Hochberg’s Step-Up Procedure**:
– **Method**: Similar to Holm-Bonferroni but more powerful, adjusting p-values in a step-up manner.
– **Pros**: More powerful than Holm-Bonferroni.
– **Cons**: Assumes independent tests, which might not always be the case.
4. **Westfall-Young Permutation Procedure**:
– **Method**: A resampling-based method that adjusts p-values based on the distribution of the data.
– **Pros**: Can be more powerful and appropriate for correlated tests.
– **Cons**: Computationally intensive.
### Should You Use FWER Tests?
**Yes, you should consider using FWER tests when comparing volumes of 142 brain regions for each group because:**
1. **High Number of Comparisons**: With 142 tests, the risk of false positives is substantial if you do not control for multiple comparisons.
2. **Reliability of Results**: Controlling FWER ensures that the results are statistically reliable, enhancing the credibility of your findings.
3. **Research Standards**: In many fields, including neuroimaging, it is standard practice to control for multiple comparisons to ensure that the results are not due to random chance.
### Practical Steps
1. **Choose an Appropriate Method**: Depending on the correlation between brain regions, you might choose a more or less conservative method. If the tests are highly correlated, consider methods like the Westfall-Young permutation procedure.
2. **Implement the Correction**: Apply the chosen correction method to your p-values.
3. **Report the Adjusted Results**: Ensure that you report the adjusted p-values and the method used for correction in your findings.
### Example Using Bonferroni Correction in Python
Here’s an example of how you might apply the Bonferroni correction using Python:
“`python
import numpy as np
from statsmodels.stats.multitest import multipletests
# Assume p_values is an array of p-values from your 142 tests
p_values = np.random.rand(142) # Example p-values
# Apply Bonferroni correction
alpha = 0.05
corrected_p_values = multipletests(p_values, alpha=alpha, method=’bonferroni’)
# corrected_p_values contains a tuple with:
# (boolean array of hypotheses accepted, corrected p-values, …, …)
print(corrected_p_values)
“`
In summary, controlling the FWER is important in your scenario to ensure the reliability and validity of your results, especially given the large number of comparisons.
First of all, thank you very much for your answer. Finally, I would like to ask the following. Normally, we normally separate the anatomical structures of the brain into subgroups and obtain volumes in the form of total volumes and right and left volumes of these groups as a result of semi-automated software analyzing human brain mr imaging according to a specific algorithm. After that, we compare the volumes between the 3 groups with a software package such as SPSS. In fact, we are comparing both total and right and left half volumes of the same anatomical region. Since we are comparing both total and right and left volumes, don’t we minimize the p value too much according to each test?
First of all, thank you very much for your answer. Finally, I would like to ask the following. Normally, we normally separate the anatomical structures of the brain into subgroups and obtain volumes in the form of total volumes and right and left volumes of these groups as a result of semi-automated software analyzing human brain mr imaging according to a specific algorithm. After that, we compare the volumes between the 3 groups with a software package such as SPSS. In fact, we are comparing both total and right and left half volumes of the same anatomical region. Since we are comparing both total and right and left volumes, don’t we reduce the p value more than it should be according to each test?
Hi Halil…You are very welcome! When you perform multiple comparisons, such as comparing the volumes of the brain’s anatomical structures across different groups and considering both total and side-specific (right and left) volumes, there is indeed a risk of inflating the Type I error rate. This inflation occurs because each statistical test carries a probability of yielding a false positive (rejecting the null hypothesis when it is true), and conducting multiple tests increases the overall likelihood of making at least one Type I error.
### Addressing the Problem of Multiple Comparisons
To mitigate the inflation of the Type I error rate when conducting multiple comparisons, several methods can be employed. Here are some commonly used techniques:
1. **Bonferroni Correction**:
– This method involves dividing the desired significance level (e.g., α = 0.05) by the number of tests being conducted. For instance, if you are performing 10 tests, you would use a significance level of 0.05/10 = 0.005 for each test.
– This approach is conservative and can reduce the power of the tests (the probability of correctly rejecting a false null hypothesis).
2. **Holm-Bonferroni Method**:
– A stepwise method that is less conservative than the Bonferroni correction. It adjusts the p-values sequentially, starting with the smallest p-value.
3. **False Discovery Rate (FDR) Control**:
– The Benjamini-Hochberg procedure controls the expected proportion of false discoveries (false positives) among the rejected hypotheses. This method is less conservative than the Bonferroni correction and is particularly useful when dealing with many comparisons.
4. **Multivariate Analysis of Variance (MANOVA)**:
– Instead of conducting multiple separate ANOVAs, you can use MANOVA to test for differences in multiple dependent variables simultaneously. This can help control for Type I error inflation and account for correlations between dependent variables.
5. **Permutation Tests**:
– Non-parametric methods that involve shuffling the data to generate a distribution of the test statistic under the null hypothesis. These tests can be useful when the assumptions of traditional parametric tests are not met.
### Practical Example
If you are comparing the total, right, and left volumes of an anatomical structure between three groups, you would normally perform several comparisons:
– Total volume between groups.
– Right volume between groups.
– Left volume between groups.
Let’s say you are performing these comparisons for 10 different anatomical structures. This would lead to a total of 30 comparisons (10 structures × 3 volume types).
#### Applying Bonferroni Correction
“`python
# Assuming alpha is 0.05 and there are 30 comparisons
alpha_corrected = 0.05 / 30
print(f”Corrected alpha level: {alpha_corrected}”)
“`
#### Using False Discovery Rate (FDR) Control
In SPSS, you can use the FDR control through the `Analyze` > `General Linear Model` > `Multivariate…` menu. In Python, you can use `statsmodels` for FDR control:
“`python
import statsmodels.stats.multitest as smm
# Assuming p_values is a list of p-values from your multiple comparisons
p_values_corrected = smm.multipletests(p_values, alpha=0.05, method=’fdr_bh’)[1]
“`
### Summary
When comparing both total and side-specific volumes of anatomical regions, it is essential to adjust for multiple comparisons to avoid inflating the Type I error rate. Methods such as Bonferroni correction, Holm-Bonferroni method, FDR control, and MANOVA can help manage this issue. The choice of method depends on the context and the balance you wish to strike between controlling Type I errors and maintaining statistical power.