MIT study finds LLM ranking sites can be easily skewed

As more labs and buyers obsess over LLM leaderboards, MIT is throwing cold water on how much trust we should put in crowdsourced ranking sites. Their analysis shows that in at least one major platform, dropping as few as two votes out of more than 57,000 can flip which model is ranked number one. Another platform needed only a few dozen data points to change the winner. In other words, what looks like a robust market signal—“Model X is best for coding” or “Model Y is top for chat”—may hinge on a handful of noisy or erroneous human comparisons.

For the race to AGI, this matters because these rankings influence where capital, talent and compute flow. Startups and big enterprises alike use them to decide which models to build on, fine‑tune or benchmark against. If those leaderboards are fragile, the ecosystem can converge on the wrong systems or overfit to the quirks of one platform’s user base. The authors’ method for flagging influential votes is a partial antidote, but the deeper message is that we need more rigorous, transparent and statistically grounded evaluation regimes—especially as we get closer to general‑purpose systems whose failures are harder to spot. Otherwise, we risk letting noisy popularity contests steer the trajectory of AGI development.