How to Calculate Intraclass Correlation Coefficient in Excel

An intraclass correlation coefficient (ICC) is used to determine if items (or subjects) can be rated reliably by different raters.

The value of an ICC can range from 0 to 1, with 0 indicating no reliability among raters and 1 indicating perfect reliability.

This tutorial provides a step-by-step example of how to calculate ICC in Excel.

Step 1: Create the Data

Suppose four different judges were asked to rate the quality of 10 different college entrance exams. The results are shown below:

Step 2: Fit an ANOVA

In order to calculate the ICC for these ratings, we first need to fit an Anova: Two-Factor Without Replication.

To do so, highlight cells A1:E11 as follows:

To do so, click the Data tab along the top ribbon and then click the Data Analysis option under the Analysis group:

If you don’t see this option available, you need to first load the Analysis ToolPak.

In the dropdown menu that appears, click Anova: Two-Factor Without Replication and then click OK. In the new window that appears, fill in the following information and then click OK:

The following results will appear:

Step 3: Calculate the Intraclass Correlation Coefficient

We can use the following formula to calculate the ICC among the raters:

Intraclass correlation coefficient in Excel

The intraclass correlation coefficient (ICC) turns out to be 0.782.

Here is how to interpret the value of an intraclass correlation coefficient, according to Koo & Li:

  • Less than 0.50: Poor reliability
  • Between 0.5 and 0.75: Moderate reliability
  • Between 0.75 and 0.9: Good reliability
  • Greater than 0.9: Excellent reliability

Thus, we would conclude that an ICC of 0.782 indicates that the exams can be rated with “good” reliability by different raters.

A Note on Calculating ICC

There are several different versions of an ICC that can be calculated, depending on the following three factors:

  • Model: One-Way Random Effects, Two-Way Random Effects, or Two-Way Mixed Effects
  • Type of Relationship: Consistency or Absolute Agreement
  • Unit: Single rater or the mean of raters

In the previous example, the ICC that we calculated used the following assumptions:

  • Model: Two-Way Random Effects
  • Type of Relationship: Absolute Agreement
  • Unit: Single rater

For a detailed explanation of these assumptions, please refer to this article.

Leave a Reply

Your email address will not be published. Required fields are marked *