Image by Author | ChatGPT

Elo ratings and Arena scores provide a dynamic, tournament-style way to rank large language models (LLMs) based on head-to-head comparisons, similar to how chess players are ranked through competitive matches. Unlike static benchmarks that test specific skills, these systems measure which models users actually prefer in conversation through millions of pairwise votes.

What Makes Elo Different from Traditional AI Benchmarks

Most AI evaluation methods like MMLU test models on standardized multiple-choice questions with predetermined correct answers. Elo ratings take a completely different approach by treating model evaluation like a sports tournament where models compete directly against each other.

The Elo system originated from chess master Arpad Elo’s work in the 1960s to create fairer player rankings. Instead of relying on win-loss records alone, Elo considers the strength of opponents faced. When a lower-rated player defeats a higher-rated opponent, the rating change is more dramatic than when the favorite wins as expected.

This concept translates perfectly to LLM evaluation. When users compare responses from two models side-by-side, they vote for which answer they prefer. These votes feed into the Elo calculation, creating rankings that reflect user preferences rather than performance on academic tests.

How Elo Scoring Works for LLMs

The Elo system uses a mathematical formula to update ratings after each comparison:

New Rating = Old Rating + K × (Actual Result – Expected Result)

The K-factor determines how much ratings change after each match. Higher K values make ratings more responsive to recent results but also more volatile. The expected result comes from the rating difference between models—a 200-point gap suggests the higher-rated model should win about 76% of the time.

Here’s how to interpret common Elo score ranges for LLMs:

1000-1100: Basic conversational ability but frequent errors and limitations. Early or smaller models typically score in this range.

1100-1200: Competent performance with noticeable improvements in reasoning and factual accuracy. Many current open-source models fall here.

1200-1300: Strong performance approaching the quality of leading commercial models. Shows good reasoning across diverse topics.

1300+: Exceptional performance that consistently impresses users. Only the most advanced models like GPT-4, Claude, and Gemini reach these levels.

Where to Find Current Elo Rankings

Several platforms maintain up-to-date Elo rankings, each with different strengths and focuses:

Chatbot Arena (LMSYS) provides the most comprehensive and widely-cited Elo rankings through a blind voting system. LMSYS, an academic research consortium at UC Berkeley collaborating with Stanford, UC San Diego, and Carnegie Mellon University, collects over 3.2 million user votes by having people compare model responses without knowing which model wrote what. The platform shows both current standings and historical trends, allowing you to track how models improve over time. This creates the most authoritative Elo ratings in the LLM community.

Elo and Arena Score Visualization | Image credit: LMSYS Chatbot Arena

OpenLM Arena aggregates scores from multiple sources and combines Elo ratings with traditional benchmarks like MMLU. This gives a more complete picture of model capabilities across different evaluation methods.

Arena Leaderboard Table | Image credit: OpenLM Arena

The OpenLM Arena page displays two distinct leaderboards: the top “OpenLM” table focuses exclusively on open-source models with their Arena Elo, MMLU scores, and licenses, while the “Full Leaderboard” below includes both proprietary and open-source models with more detailed breakdowns across categories like coding, vision, and Arena Hard tasks. For the most comprehensive Elo comparison including top-performing models like GPT-4, Gemini, and Claude, refer to the Full Leaderboard. The OpenLM-specific table is ideal when you need to evaluate only open-source alternatives with permissive licensing.

Reading Elo Results and Choosing Models for Your Projects

When examining Elo leaderboards, models with similar scores (within 20-30 points) should be considered roughly equivalent in performance. Pay attention to category-specific breakdowns when available—a model might excel at creative writing but struggle with mathematical reasoning. Models with thousands of battles provide more reliable ratings than those with only hundreds of comparisons.

Elo ratings excel at predicting user satisfaction with conversational AI applications, making them valuable for chatbots, writing assistants, and customer service. For technical applications requiring specific domain knowledge, combine Elo scores with relevant benchmarks since a model with high Arena scores might perform poorly on specialized tasks like code generation. Consider your specific use case when interpreting scores—a model with slightly lower overall Elo might actually perform better for your particular application.

Limitations and Considerations

User Bias and Platform Effects: Elo ratings reflect the preferences of the specific user base voting in each platform. Different communities might prefer different response styles, potentially creating bias in the rankings. The scoring also depends on the types of prompts and tasks included in evaluations.

Preference vs. Accuracy: The system measures preference rather than accuracy. Users might prefer confident-sounding but incorrect responses over cautious but accurate ones. This means high Elo scores don’t guarantee factual reliability, especially for specialized domains.

Rating Drift and Relative Scores: Elo ratings represent relative model performance within a specific competitor set rather than absolute measures of capability. As new, more capable models enter the arena, previously high-performing models might experience rating drops despite no actual performance degradation. This phenomenon, known as rating drift, underscores the dynamic nature of Elo scores and cautions against direct comparisons between ratings from different platforms or different time periods.

Rating Volatility: Early scores can be unreliable for new models or those with limited comparison data. Ratings might not reflect true performance until sufficient battles have been completed. Additionally, the pool of competing models affects individual scores—introducing stronger competitors can lower existing ratings even without performance degradation.

Data Contamination: If models have seen similar prompts during training, their Arena performance might not reflect genuine generalization ability. This has led to the development of contamination-resistant evaluation methods.

Using Elo Scores Effectively

Treat Elo ratings as one indicator among many rather than the sole decision factor. While high scores suggest strong user satisfaction, they don’t guarantee performance on your specific tasks or accuracy in specialized domains.

Track model performance trends over time instead of relying on single snapshots. Models showing consistent Elo improvements often signal active development and optimization, which can be valuable for long-term projects requiring ongoing model support.

Elo ratings represent a shift toward user-centric AI evaluation that complements traditional academic benchmarks. As conversational AI becomes more prevalent, these preference-based metrics will likely become increasingly important for real-world model selection decisions.