A November 10 preprint tests how votes, prompts and model submissions can alter AI rankings. Its experiments used public data, not live ranking manipulation.

The big change

  • An arena ranking combines people's choices between anonymous answers. It measures preferences in those comparisons, which may differ from the tasks a particular customer needs.
  • The new experiments moved rankings without improving the target model. Their results depend on specified voting, prompt and model-copy assumptions.
  • Earlier voting research shows why safeguards need testing too. Detecting unusual votes can reduce manipulation, but the researchers found that some strategies survived filtering.

How comparisons become a ranking

The 2024 Chatbot Arena paper describes users asking questions and comparing two anonymous model answers. They choose the response they prefer before learning the models' identities. A statistical method called Bradley–Terry combines those pairwise choices into estimates of relative strength.

This permits open-ended questions for which there may be no single correct answer. It also makes the participants and their questions part of what is measured. The original paper identifies a selection bias: its users lean toward people interested in language models. A broad preference score therefore needs context before a buyer treats it as evidence for a particular workload.

What the November experiment changed

The dataset contained over 100,000 preferences. Adding up to 10,000 strategic votes moved a model as many as 20 places. Other tests repeated comparisons in a language favoring the target, or inserted up to ten copies of a selected model, assuming identical outcomes. This changes the comparison pool without changing the target’s answers. Combining prompting and copies produced a maximum 45-place shift.

Those maxima are conditional simulations, not typical improvements. The main voting results assume model identification; larger leaderboards may require more votes. No provider misconduct is established. Proposed defenses include limiting similar submissions and balancing prompt groups.

Why other models' votes matter

The January 2025 vote-rigging paper explains an indirect route: a vote can change a target's rating even when its answer is absent. The ranking depends on comparisons throughout the pool, so changing opponents' estimated strength can change the target's relative position. Its experiments used approximately 1.7 million historical votes, a different dataset from the November study.

That earlier work also examined imperfect model identification and ordinary users voting at the same time. These conditions matter because an attacker may misidentify an answer or have strategic votes diluted by other participants. Its defense tests found that filtering unlikely outcomes reduced ranking gains but did not eliminate them. This supports a narrower conclusion than declaring either rankings or safeguards useless: the collection process and its protections need evaluation alongside the resulting score.

For a team selecting a model, the practical next step is to compare candidates on its own representative tasks. BIG CHANGE did not reproduce either study or test a live leaderboard.