6 Replies to “Cohen’s Kappa Statistic: Definition & Example”

1. hi, I try to calculate the example manually,
   However, when I caculated (0.6429 – 0.5) / (1 – 0.5), I got 0.2858, not 0.2857 in the example result.
   Could you check that for me?
   Thanks,
   Yuxin

   Reply

2. please send me Kappa formula via my email

   Reply

3. Hi i have question
   i try to correlate result between two different analyzer
   the report either normal or deficient
   which one i shall refer as Yes and which one i shall refer as NO

   Reply

   1. Hi Roihan…When using **Cohen’s Kappa statistic** to measure agreement between two analyzers (or raters) reporting results as either **Normal** or **Deficient**, you can assign “Yes” and “No” in a way that aligns logically with your context. Here’s how you might decide:

      ### General Guidelines:
      1. **”Yes” for the condition of interest**:
      Assign “Yes” to the outcome that represents the presence of the condition of interest. For example:
      – If “Deficient” indicates a specific condition or abnormality, assign **Deficient = Yes** and **Normal = No**.
      – This aligns with the common practice of “Yes” indicating the presence of a condition.

      2. **”No” for the absence of the condition**:
      – Assign “Normal = No” to indicate the absence of deficiency.

      ### Why this assignment works well:
      – “Yes” typically means something is **positive** or present, making it easier to interpret metrics like agreement or disagreement.
      – It keeps the analysis intuitive, especially if you’re looking at the kappa statistic alongside other metrics.

      ### Practical Example:
      If you are comparing two analyzers on their ability to detect deficiency:
      – **Analyzer 1**: Normal, Deficient
      – **Analyzer 2**: Normal, Deficient

      Assign:
      – Deficient = Yes (1)
      – Normal = No (0)

      This way, Cohen’s Kappa will measure how consistently the analyzers agree on detecting deficiencies (the condition of interest).

      Reply

4. Hello Zach
   Just wanted formula for Fleiss’ Kappa and its application with example

   Reply

   1. Hi Nitu…### **Fleiss’ Kappa Formula**

      Fleiss’ Kappa is a statistical measure for assessing the reliability of agreement between multiple raters when assigning categorical ratings to a fixed number of items. Unlike Cohen’s Kappa, which is for two raters, Fleiss’ Kappa handles multiple raters.

      #### **Formula**
      Fleiss’ Kappa is calculated as:

      \[
      \kappa = \frac{{P_o – P_e}}{{1 – P_e}}
      \]

      Where:
      – \(P_o\): The observed agreement.
      – \(P_e\): The expected agreement by chance.

      #### **Steps to Calculate Fleiss’ Kappa**

      1. **Determine Proportions of Rater Agreements for Each Category (\(p_j\))**:
      For each category \(j\), calculate:
      \[
      p_j = \frac{1}{{N \cdot n}} \sum_{i=1}^N n_{ij}
      \]
      Where:
      – \(N\): Number of items.
      – \(n\): Number of ratings per item.
      – \(n_{ij}\): Number of raters who assigned category \(j\) to item \(i\).

      2. **Calculate the Observed Agreement (\(P_o\))**:
      For each item \(i\), compute:
      \[
      P_i = \frac{1}{{n(n-1)}} \sum_{j=1}^k n_{ij}(n_{ij}-1)
      \]
      Average over all items:
      \[
      P_o = \frac{1}{N} \sum_{i=1}^N P_i
      \]

      3. **Calculate the Expected Agreement (\(P_e\))**:
      \[
      P_e = \sum_{j=1}^k p_j^2
      \]

      4. **Compute Fleiss’ Kappa (\(\kappa\))**:
      \[
      \kappa = \frac{{P_o – P_e}}{{1 – P_e}}
      \]

      —

      ### **Application with Example**

      #### Scenario:
      Suppose you have 3 raters assessing 5 items into 3 categories (A, B, C). The ratings for each item are:

      | Item | A | B | C |
      |——|—-|—-|—-|
      | 1 | 2 | 1 | 0 |
      | 2 | 0 | 3 | 0 |
      | 3 | 1 | 1 | 1 |
      | 4 | 3 | 0 | 0 |
      | 5 | 0 | 1 | 2 |

      – \(N = 5\) (items)
      – \(n = 3\) (raters)
      – \(k = 3\) (categories)

      #### Step 1: Compute \(p_j\)
      For each category \(j\), sum ratings across all items and divide by total ratings (\(N \cdot n\)):

      \[
      p_j = \frac{\text{Total ratings for category } j}{N \cdot n}
      \]

      – \(p_A = \frac{2+0+1+3+0}{5 \cdot 3} = 0.4\)
      – \(p_B = \frac{1+3+1+0+1}{5 \cdot 3} = 0.3333\)
      – \(p_C = \frac{0+0+1+0+2}{5 \cdot 3} = 0.2667\)

      #### Step 2: Compute \(P_o\)
      For each item, calculate \(P_i\):
      \[
      P_i = \frac{1}{{n(n-1)}} \sum_{j=1}^k n_{ij}(n_{ij}-1)
      \]

      – Item 1: \(P_1 = \frac{1}{{3(3-1)}} [2(2-1) + 1(1-1) + 0(0-1)] = \frac{2}{6} = 0.3333\)
      – Item 2: \(P_2 = \frac{1}{{6}} [0(0-1) + 3(3-1) + 0(0-1)] = 1.0\)
      – Item 3: \(P_3 = \frac{1}{{6}} [1(1-1) + 1(1-1) + 1(1-1)] = 0.0\)
      – Item 4: \(P_4 = \frac{1}{{6}} [3(3-1) + 0(0-1) + 0(0-1)] = 1.0\)
      – Item 5: \(P_5 = \frac{1}{{6}} [0(0-1) + 1(1-1) + 2(2-1)] = 0.3333\)

      Average over all items:
      \[
      P_o = \frac{1}{5} (0.3333 + 1.0 + 0.0 + 1.0 + 0.3333) = 0.5333
      \]

      #### Step 3: Compute \(P_e\)
      \[
      P_e = \sum_{j=1}^k p_j^2 = 0.4^2 + 0.3333^2 + 0.2667^2 = 0.16 + 0.1111 + 0.0711 = 0.3422
      \]

      #### Step 4: Compute \(\kappa\)
      \[
      \kappa = \frac{P_o – P_e}{1 – P_e} = \frac{0.5333 – 0.3422}{1 – 0.3422} = \frac{0.1911}{0.6578} = 0.2905
      \]

      —

      ### **Interpretation**
      – \(\kappa = 0.29\): Indicates **fair agreement** among raters.
      – Fleiss’ Kappa ranges:
      – \(< 0.00\): Poor agreement - \(0.00 - 0.20\): Slight agreement - \(0.21 - 0.40\): Fair agreement - \(0.41 - 0.60\): Moderate agreement - \(0.61 - 0.80\): Substantial agreement - \(0.81 - 1.00\): Almost perfect agreement --- This example demonstrates the calculation of Fleiss’ Kappa and its interpretation. Let me know if you'd like further clarification!

      Reply