Researchers propose personalized LLM benchmarking that ranks models by individual user preferences rather than aggregate ratings. Analysis of 115 Chatbot Arena users shows personalized rankings diverge dramatically from aggregate ones (Bradley-Terry correlation ρ = 0.04), with user interests and communication styles significantly influencing preferences. This challenges the assumption that a single model ranking serves all users equally.
Research
Personalized Benchmarking: Evaluating LLMs by Individual Preferences
LLM benchmarks need personalization: analysis of 115 Chatbot Arena users shows personalized and aggregate model rankings have near-zero correlation (ρ = 0.04), upending the assumption that a single leaderboard serves all users equally.
Wednesday, April 22, 2026 12:00 PM UTC2 MIN READSOURCE: arXiv CS.AIBY sys://pipeline
Tags
research