A two sample t-test is used to test whether or not the means of two populations are equal.
This type of test makes the following assumptions about the data:
1. Independence: The observations in one sample are independent of the observations in the other sample.
2. Normality: Both samples are approximately normally distributed.
3. Homogeneity of Variances: Both samples have approximately the same variance.
4. Random Sampling: Both samples were obtained using a random sampling method.
If one or more of these assumptions are violated, then the results of the two sample t-test may be unreliable or even misleading.
In this tutorial we provide an explanation of each assumption, how to determine if the assumption is met, and what to do if the assumption is violated.
Assumption 1: Independence
A two sample t-test makes the assumption that the observations in one sample are independent of the observations in the other sample.
This is a crucial assumption because if the same individuals appear in both samples then it isn’t valid to draw conclusions about the differences between the samples.
How to Check this Assumption
The easiest way to check this assumption is to verify that each observation only appears in each sample once and that the observations in each sample were collected using random sampling.
What to Do if this Assumption is Violated
If this assumption is violated, the results of the two sample t-test are completely invalid. In this scenario, it’s best to collect two new samples using a random sampling method and ensure that each individual in one sample does not belong to the other sample.
Assumption 2: Normality
A two sample t-test makes the assumption that both samples are approximately normally distributed.
This is a crucial assumption because if the samples are not normally distributed then it isn’t valid to use the p-values from the test to draw conclusions about the differences between the samples.
How to Check this Assumption
If the sample sizes are small (n < 50), then we can use a Shapiro-Wilk test to determine if each sample size is normally distributed. If the p-value of the test is less than a certain significance level, then the data is likely not normally distributed.
If the sample sizes are large, then it’s better to use a Q-Q plot to visually check if the data is normally distributed.
If the data points roughly fall along a straight diagonal line in a Q-Q plot, then the dataset likely follows a normal distribution.
What to Do if this Assumption is Violated
If this assumption is violated then we can perform a Mann-Whitney U test, which is considered the non-parametric equivalent to the two sample t-test and does not make the assumption that the two samples are normally distributed.
Assumption 3: Homogeneity of Variances
A two sample t-test makes the assumption that the two samples have roughly equal variances.
How to Check this Assumption
We use the following rule of thumb to determine if the variances between the two samples are equal: If the ratio of the larger variance to the smaller variance is less than 4, then we can assume the variances are approximately equal and use the two sample t-test.
For example, suppose sample 1 has a variance of 24.5 and sample 2 has a variance of 15.2. The ratio of the larger sample variance to the smaller sample variance would be calculated as:
Ratio: 24.5 / 15.2 = 1.61
Since this ratio is less than 4, we could assume that the variances between the two groups are approximately equal.
What to Do if this Assumption is Violated
If this assumption is violated then we can perform Welch’s t-test, which is a non-parametric version of the two sample t-test and does not make the assumption that the two samples have equal variances.
Assumption 4: Random Sampling
A two sample t-test makes the assumption that both samples were obtained using a random sampling method.
How to Check this Assumption
There is no formal statistical test we can use to test this assumption. Instead, we just need to make sure that both samples were obtained use a random sampling method such that each individual in the population of interest had an equal probability of being included in either sample.
What to Do if this Assumption is Violated
If this assumption is violated, then it’s unlikely that our two samples are representative of the population of interest. In this case, we can’t generalize the findings from the two sample t-test to the overall population with reliability.
In this scenario, it’s best to collect two new samples using a random sampling method.
Very useful and easy to understand notes on t-test
Thank you Ato for your feedback!
In a situation a researcher is collecting data from an intact classes which should not be randomized in a quasi-experimental study, what should I do?
In a quasi-experimental study involving intact classes (i.e., pre-existing groups that cannot be randomized), performing a t-test can still be valid, but you must be especially cautious about the assumptions the test makes and how your study design affects them. Here’s what you should consider for each of the **four assumptions of a t-test** in this context, and what actions you can take:
—
### **1. Independence of Observations**
* **Challenge:** Intact classes typically violate this assumption because students within a class may influence each other (shared teacher, environment, peer effects).
* **Action:**
* **Acknowledge** this limitation in your study write-up.
* Consider using **multilevel modeling (hierarchical linear modeling)** if you have multiple intact groups (e.g., multiple classes per condition), which accounts for intra-class correlations.
* If you only have one class per condition, treat your analysis as **group-level** rather than individual-level to avoid pseudo-replication (e.g., use class averages).
—
### **2. Normality**
* **Challenge:** With small sample sizes or non-random samples, data may not follow a normal distribution.
* **Action:**
* Use **Shapiro-Wilk** or **Kolmogorov-Smirnov** tests to assess normality.
* If data are non-normal and sample size is small, consider using a **non-parametric test** (e.g., **Mann-Whitney U** for independent samples).
—
### **3. Homogeneity of Variance (Equal Variances)**
* **Challenge:** Intact classes may differ substantially in size or variability due to uncontrolled classroom factors.
* **Action:**
* Test with **Levene’s test** or **F-test**.
* If variances are unequal, use **Welch’s t-test**, which adjusts for variance heterogeneity.
—
### **4. Scale of Measurement (Interval or Ratio Level)**
* **Challenge:** This is usually less problematic, as long as you are using appropriate numerical data.
* **Action:** Ensure that your outcome variable is measured on an **interval or ratio** scale (e.g., test scores, performance metrics), not ordinal or categorical.
—
### **Summary of What to Do**
* **Design Acknowledgment:** Clearly state the **quasi-experimental nature** and limitations regarding group assignment.
* **Assumption Checking:** Conduct diagnostics for normality and equal variances.
* **Alternative Analysis:** If assumptions are seriously violated, consider:
* **Non-parametric tests**
* **ANCOVA** (if you can control for pre-existing differences)
* **Multilevel modeling** (if you have more than one class per condition)
Random sampling is not an assumption of the independent samples t‑test. The test itself requires only that observations are independent, that each group is approximately normally distributed, and that the variances are roughly equal. In social science research, the independent t‑test is often applied in quasi‑experimental designs where random sampling or random assignment is not possible or ethical. Randomization is a requirement of true experimental design because it supports generalizability, but its absence does not invalidate the t‑test — it simply limits the scope of conclusions beyond the studied groups. (Statistics by Jim; Laerd Statistics) Please correct this as your incorrect comments are causing my doc students to make mistakes on their papers in my statistics classes.
and causation. I forgot to add that to my post. Random group assignment is needed to determine causation.
Thank you for your contribution to our discussion!