Image by Editor | Midjourney

Nonparametric tests are widely used in many applications, but frequently, users without a strong statistical background may struggle to decide on the most appropriate test to conduct, depending on the data they have and the specific real-world problem (typically a data analysis problem) they want to address.

This guide is here to help, aimed at guiding users towards the right nonparametric test to use.

As a quick recap, nonparametric tests in data analysis are useful when our data is not assumed to follow a specific distribution, oftentimes the normal distribution. They can be utilized to answer several questions, from comparing differences among data groups to assessing possible associations (correlations) between variables, to testing randomness or certain assumptions about the data distribution, to name a few.

A Decision Tree Approach

First, let’s bear in mind that nonparametric tests are generally less powerful than parametric tests in terms of detecting true effects when they exist (i.e., they generally have lower statistical power). Therefore, they should be used only when your data doesn’t meet the requirements to perform parametric tests — for instance, because your sample size is small, the data is ordinal or ranked, or it doesn’t meet assumptions like normality or variance homogeneity.

This said, once you are sure nonparametric tests are the only way to go, the starting point is to determine which analysis question you want to address. These are the most common data analysis questions that nonparametric tests are designed to address:

1. Do you need to compare several data groups?
2. Are you investigating the correlation among variables or groups?
3. Do you want to compare a data sample to a certain theoretical distribution?
4. Are you handling categorical data?
5. Are you analyzing whether there exists randomness in a sequence of data?

A decision tree approach to choosing the right nonparametric test (click to enlarge)

Much of the job is done by picking the right question from the ones above, indeed!

Let’s discuss the target tasks on the above image from the second one onwards, leaving the first one (comparing several data groups) for later discussion.

Investigating the correlation between several variables or groups when we are forced to use nonparametric tests entails using Spearman’s rank correlation or Kendall’s tau. The main difference between these two tests lies in how they measure association: Spearman’s deals with ranked values and is more sensitive to larger data samples, whereas Kendall’s focuses on analyzing concordant vs. discordant pairs, and it is a robust solution against smaller samples or samples with many tied ranks.

Comparing a dataset to a theoretical distribution boils down to the Kolmogorov–Smirnov test, as this test determines whether the samples stem from the same distribution, or a specific sample comes from a specific, hypothesized distribution. It does so by comparing cumulative distribution functions (CDFs) across samples.

Meanwhile, when we are presented with categorical data and want to check possible relationships between variables, the Chi-square test is the way to go for testing independence or goodness-of-fit. For instance, to test whether there exists an association between gender and preferred subject of study among college students.

And finally, for datasets containing binary or sequential data where we want to analyze possible randomness in the sequence, the Runs test helps detect non-random trends or patterns.

Wrapping Up

Going back to the analysis scenario of comparing several data groups without normality assumptions — i.e. being obliged to use nonparametric tests — we may use one test or another depending on: (i) the number of groups to compare, and (ii) whether the samples are paired or related or not. This article further discusses the characteristics and situations in which different tests can be used for this target group comparison problem.

In a nutshell, you may want to use:

• The Wilcoxon signed-rank test for comparing two related groups.
• The Mann Whitney U test for comparing two independent groups.
• The Friedman test is used for comparing three or more related groups.
• The Kruskal-Wallis test is used for comparing three or more unrelated groups.