Image by Editor | Midjourney

In statistics, effect size is a way to quantify the magnitude of a relationship, difference, or association between data variables or groups. Unlike p-values, which are used to test whether an effect, such as a statistical difference, exists, effect sizes are designed to quantify the magnitude of that effect, thereby providing essential context for interpreting statistical results.

There are several types of effect size measures — including Pearson’s r, Cohen’s d, eta-squared, and odds ratios — and their computation is generally straightforward. However, choosing the appropriate effect size measure is not always a straightforward task and depends on several factors, including the type of data, the research question being addressed, the statistical test utilized, and the hypothesis being tested.

This article is intended to help you decide on the right effect size measure for your problem or goals, using a decision tree approach to guide you in the right direction.

A Decision Tree Approach

Below is a decision tree-shaped diagram that accounts for about ten different well-known effect size measures for different types of statistical analyses.

A decision tree approach to choosing the right effect size measure (click to enlarge)

Some of the key decisions or selection criteria to consider when navigating this tree are:

• The type of relationship to assess, primarily determined by most of the existing measures for determining either an association or a difference (comparison) between groups. On top of these two large categories of effect size measures lies a third one, related to prediction modeling or variance explained, i.e., research questions like “how well is a continuous outcome predicted upon a set of predictor variables?”. The R² measure fits in this third scenario.
• While most effect size measures are designed to study the relationships between two groups, there are also specific measures for comparisons involving three or more groups, applicable in ANOVA (analysis of variance) contexts or multiple regression models. Eta squared (η²) stands out among these measures suitable for more than two groups.
• Having just numerical data or having some categorical data in at least one group is another common selection criterion to consider when selecting the right measure, and within categorical data variables, it also matters ascertaining whether the categories are ordinal (like e.g. t-shirt sizes like ‘S’, ‘M’, ‘L’, and so on) or not ordinal (like different animal species or t-shirt colors).
• Some effect size measures are also particularly suitable for smaller (resp. larger) samples, such as Cohen’s d, which works well with large samples, or Hedge’s d for smaller samples.
• Last, the type of analysis, e.g., correlation, ANOVA, regression, etc., is another key selection criterion for choosing the right effect size measure.

With this information in mind, all that remains is briefly outlining each of the effect size measures considered in this guide:

1. Cohen’s d is utilized to measure the difference between two groups’ means when the sample sizes are large and normality assumptions are made.
2. Hedge’s g is a better approach than Cohen’s d when the sample sizes are small, with the rest of the conditions holding the same.
3. Glass’s Δ can be used when the two groups have unequal variances and, as a result, neither of the two above-listed measures is suitable, even though the data are still assumed to follow normal distributions.
4. Cliff’s Delta is a somewhat versatile measure, suitable for comparing differences between two samples that are not normal or ordinal. It can also be utilized for measuring associations between ordinal variables.
5. Moving to measures for quantifying associations, Pearson’s r is the way to go for measuring correlation between two continuous variables.
6. For comparing binary outcomes belonging to two binary categorical variables, e.g., having vs. not having lung cancer, and smoking vs. not smoking, the odds ratio builds a 2×2 contingency table to study the interaction between both binary variables.
7. Eta-squared (η²), as mentioned earlier, is useful for variance explained in ANOVA, i.e., when comparing three or more groups.
8. Cramer’s V is used to study associations in purely nominal data (categorical data without ordinal semantics) with more than two possible categories in at least one of the two variables, hence, a larger contingency table than 2×2 is needed.
9. Meanwhile, for nominal categorical data variables suitable for a 2×2 contingency table, the Phi coefficient is used.
10. R² and Adjusted R² are utilized to measure how much of the variation in a dependent variable or outcome in a regression problem can stem from the attribution of independent variables or predictors.

Wrapping Up

In this article, we presented a decision tree-based guide to select the ideal effect size measure depending on the type of relationship or aspect to study among two or more data groups or variables, their data types, and other statistical properties.