# Study tests manipulation of AI rankings
> Offline experiments examine votes, prompts and model submissions.
By BIGCHANGE.AI Editorial
Published: 2026-11-14T18:41:38.137Z
Updated: 2026-11-14T18:41:38.137Z
Canonical: https://bigchange.ai/blog/preference-ranking-manipulation-study

AI-generated illustration for BIG CHANGE.
A [November 10 preprint](https://arxiv.org/abs/2611.14769) tests how votes, prompts and model submissions can alter AI rankings. Its experiments used public data, not live ranking manipulation.
## The big change
- An arena ranking combines people's choices between anonymous answers. It measures preferences in those comparisons, which may differ from the tasks a particular customer needs.
- The new experiments moved rankings without improving the target model. Their results depend on specified voting, prompt and model-copy assumptions.
- Earlier voting research shows why safeguards need testing too. Detecting unusual votes can reduce manipulation, but the researchers found that some strategies survived filtering.
## How comparisons become a ranking
The [2024 Chatbot Arena paper](https://arxiv.org/abs/2403.04132) describes users asking questions and comparing two anonymous model answers. They choose the response they prefer before learning the models' identities. A statistical method called Bradley–Terry combines those pairwise choices into estimates of relative strength.
This permits open-ended questions for which there may be no single correct answer. It also makes the participants and their questions part of what is measured. The original paper identifies a selection bias: its users lean toward people interested in language models. A broad preference score therefore needs context before a buyer treats it as evidence for a particular workload.
## What the November experiment changed
The dataset contained over 100,000 preferences. Adding up to 10,000 strategic votes moved a model as many as 20 places. Other tests repeated comparisons in a language favoring the target, or inserted up to ten copies of a selected model, assuming identical outcomes. This changes the comparison pool without changing the target’s answers. Combining prompting and copies produced a maximum 45-place shift.
Those maxima are conditional simulations, not typical improvements. The main voting results assume model identification; larger leaderboards may require more votes. No provider misconduct is established. Proposed defenses include limiting similar submissions and balancing prompt groups.
## Why other models' votes matter
The [January 2025 vote-rigging paper](https://arxiv.org/abs/2501.17858) explains an indirect route: a vote can change a target's rating even when its answer is absent. The ranking depends on comparisons throughout the pool, so changing opponents' estimated strength can change the target's relative position. Its experiments used approximately 1.7 million historical votes, a different dataset from the November study.
That earlier work also examined imperfect model identification and ordinary users voting at the same time. These conditions matter because an attacker may misidentify an answer or have strategic votes diluted by other participants. Its defense tests found that filtering unlikely outcomes reduced ranking gains but did not eliminate them. This supports a narrower conclusion than declaring either rankings or safeguards useless: the collection process and its protections need evaluation alongside the resulting score.
For a team selecting a model, the practical next step is to compare candidates on its own representative tasks. BIG CHANGE did not reproduce either study or test a live leaderboard.
## Sources
- [Leaderboard Hacking](https://arxiv.org/abs/2611.14769) — November 10 preprint and experiments.
- [Chatbot Arena paper](https://arxiv.org/abs/2403.04132) — 2024 comparison framework.
- [Vote rigging study](https://arxiv.org/abs/2501.17858) — Earlier 2025 research.
The BIG CHANGE newsletter
The big picture. At your pace.
Recent stories on AI and robotics, the shifts worth watching and practical ideas to use. Choose a daily briefing, weekly digest or monthly perspective.
Next scheduled send (UTC): . Your first edition arrives at the next scheduled send after you confirm.
Your privacy, your choice.
Necessary storage supports site security and remembers your choices. Optional Google Analytics stays off until you allow it. You can read every story with necessary storage only. Privacy details