Study Finds Chatbot Arena Voters Grow Harsher Over Time, Even on Unchanged Models
Researchers analyzed 1,260,967 blind votes cast on Chatbot Arena between May 2023 and August 2024, restricting the analysis to pairs of models whose versions were fixed. For the same unchanged pair, the share of votes saying "both are bad" rose 1.71 percentage points per month against a mean of 17.1%, and 1.91 points per month in 2024. In fourteen pairs observed often in both years, the rate rose in every one, from 15.5% to 23.3%, while the stronger model's share of decisive votes stayed flat in 2024. The authors argue this reflects standards that track the category's progress, so absolute verdicts drift even as relative comparisons hold steady.
It gives concrete evidence that human preference data and LLM leaderboards drift over time, which matters for anyone interpreting Arena rankings or training on human feedback.
Comments & Questions (0)
No comments yet
Be the first to comment!