The Kuder-Richardson Formula 20, often abbreviated KR-20, is used to measure the internal consistency reliability of a test in which each question only has two answers: right or wrong.
The Kuder-Richardson Formula 20 is as follows:
KR-20 = (k / (k-1)) * (1 – Σpjqj / σ2)
where:
- k: Total number of questions
- pj: Proportion of individuals who answered question j correctly
- qj: Proportion of individuals who answered question j incorrectly
- σ2: Variance of scores for all individuals who took the test
The value for KR-20 ranges from 0 to 1, with higher values indicating higher reliability.
The following example shows how to calculate the value for KR-20 in practice.
Example: Calculating Kuder-Richardson Formula 20
Suppose we administer a test with 7 questions to 10 students.
The results of the test are listed below in Excel, with 1 indicating a correct answer and 0 indicating an incorrect answer:
The following screenshot shows how to calculate the KR-20 value for this test:
Here are the formulas used in various cells:
- B13: =SUM(B2:B11) / 10
- B14: =1-B13
- B15: =B13*B14
- B17: =COUNTA(B1:H1)
- B18: =SUM(B15:H15)
- B19: =VAR.S(I2:I11)
- B20: =(B17/(B17-1))*(1-B18/B19)
The KR-20 value turns out to be 0.0603.
Since this value is extremely low, this indicates that the test has low reliability.
This means the questions may need to be re-written or re-phrased in such a way that the reliability of the test can be increased.
Additional Resources
The following tutorials provide explanations of terms commonly used when assessing the validity of tests and questionnaires:
thank you for providing this source. i am chemistry teacher without statistic background. this page help me a lot!
You are very welcome AMAL!
What is the most acceptable number of question items for K-R20 reliability test?
It is observed that the more the number of question items the less the K-R20 result. For example, using 50 question items compared to using 20 question items for a dichotomously scored instrument. What is the most acceptable number of question items for a pilot test?
Hi Ephraim…The Kuder-Richardson Formula 20 (K-R20) is used to measure the internal consistency reliability of a test with dichotomous items. A few key points about the relationship between the number of question items and the K-R20 result are:
1. **Length of the Test**: Generally, longer tests (more items) tend to have higher reliability coefficients because they sample more extensively from the domain of interest, thus providing a more accurate measure of the underlying trait or ability.
2. **Item Quality**: The quality of the items is crucial. Adding more items can sometimes lower the K-R20 if the new items are not well-aligned with the construct being measured or are poorly written.
3. **Pilot Testing**: For pilot testing, the goal is to balance between having enough items to ensure reliability and not overwhelming the participants. Too many items can lead to fatigue, which may affect the quality of responses.
**Recommended Number of Items for Pilot Test:**
– **Standard Practice**: A common practice is to start with a medium-sized test, typically around 20 to 30 items. This is usually sufficient to provide an initial estimate of reliability and to identify problematic items.
– **Iterative Process**: Pilot testing should be an iterative process. Start with 20-30 items, analyze the data, and refine the items. Based on the initial results, you can add or remove items and conduct further pilot tests.
– **Contextual Considerations**: The acceptable number of items can also depend on the context of the test (e.g., the age of participants, the testing environment) and the specific requirements of the research or assessment.
### Practical Steps for Pilot Testing:
1. **Initial Pilot Test**: Start with around 20-30 items.
2. **Data Analysis**: Calculate the K-R20 coefficient and examine item statistics (e.g., item difficulty, discrimination indices).
3. **Refinement**: Modify, add, or remove items based on the analysis.
4. **Further Testing**: Conduct additional pilot tests with the revised set of items.
5. **Final Decision**: Decide on the final number of items based on the reliability, validity, and practicality of the test.
By starting with a moderate number of items and refining through iterative testing, you can achieve a reliable and valid instrument for your main study.
Firstly, could you tell me about tools that can identify questions that are correctly attempted by high-mark achievers but have low reliability? Secondly, I have noticed that, through the refinement process, weak and low achievers tend to benefit more as their grades improve due to deletions. My concern is how to address this issue.
Hi Saqib…The Kuder-Richardson Formula 20 (KR-20) is a statistical measure used to assess the reliability of dichotomous (right/wrong) test items. Here’s how you can address your concerns and the tools that can help:
—
### **1. Tools to Identify Questions with Low Reliability**
To identify items that are correctly attempted by high achievers but have low reliability, you can use **item analysis** techniques. Here are the tools and approaches that can help:
#### **a. Item Discrimination Index**
– Measures how well an item distinguishes between high and low performers.
– Tools:
– **SPSS**: Use “Reliability Analysis” to compute item statistics.
– **JMP or Minitab**: Both offer item analysis features.
– **Excel**: Perform manual calculations for the discrimination index.
#### **b. Point-Biserial Correlation**
– Correlates an item score (dichotomous) with the total test score. Items with low or negative correlations may need refinement.
– Tools:
– **R**: `psych` package for psychometric analysis.
– **Python**: Libraries like `pingouin` or `scipy` can compute point-biserial correlations.
#### **c. Classical Test Theory (CTT)**
– Tools like **Winsteps** or **TAP** (Test Analysis Program) provide item difficulty and discrimination indices.
#### **d. Item Response Theory (IRT)**
– Provides a detailed analysis of each item’s performance.
– Tools:
– **IRT software** like R’s `ltm` or `TAM` packages.
– **jMetrik**: Free software for IRT-based analyses.
#### **e. Visualization Tools**
– Tools like **Tableau** or **Power BI** can create scatterplots to compare item difficulty and discrimination visually.
—
### **2. Addressing Grade Improvements Among Weak and Low Achievers**
Your observation that weak and low achievers tend to benefit more from the refinement process is common, as the deletion of poorly performing items reduces randomness and unfair penalization. Here’s how to address this:
#### **a. Balance Item Removal**
– Avoid over-refining by ensuring that deleted items are balanced across difficulty levels. This prevents skewing the test toward favoring weaker students.
#### **b. Adaptive Testing**
– Consider using **adaptive testing**, where item difficulty adjusts to the test taker’s ability in real time. This approach personalizes assessments, ensuring high achievers remain challenged while weak achievers are tested at appropriate levels.
#### **c. Retain High-Discrimination Items**
– Retain items that strongly differentiate between high and low achievers, even if they seem difficult for weaker students.
#### **d. Evaluate Item Difficulty Index**
– Retain moderately difficult items (p-values between 0.4 and 0.7) to ensure balanced assessment for all ability levels.
#### **e. Analyze Post-Test Impact**
– After refinement, analyze post-test results to see if high achievers are underperforming. Tools like **Excel** or statistical software (SPSS, R) can help compare pre- and post-refinement scores across groups.
#### **f. Weighting Scores**
– Introduce weighting to adjust the impact of easier items on the overall score. This ensures weak students don’t gain disproportionately.
—
### **Practical Suggestions**
1. **Use Technology**:
– Automated tools like **Edulastic**, **ExamSoft**, or **ClassMarker** often have in-built item analysis features.
2. **Diversify Assessments**:
– Use a mix of multiple-choice, short answers, and higher-order thinking questions to assess different levels of cognition.
3. **Continuous Monitoring**:
– Regularly review item performance with each test cycle to ensure fairness and reliability.
4. **Expert Review**:
– Engage subject matter experts to vet items flagged by statistical tools for potential retention or revision.
—
By focusing on these tools and strategies, you can ensure that test refinement benefits all students equitably while maintaining the validity and reliability of your assessments.