THE WORLD IS NOT STANDING STILL.RSS
BIG CHANGE.
BIG CHANGE, one tap away
BIG CHANGE, one tap away

Make room for the big picture.

Add BIG CHANGE to your home screen or desktop. It opens in its own window, with the latest stories when you’re online.

iPhone & iPad
Open bigchange.ai in Safari. Tap Share, then Add to Home Screen. Turn on Open as Web App if shown, then tap Add.
Android
Open bigchange.ai in Chrome. In the browser menu, choose Install app or Add to Home screen.
Desktop
In Chrome or Edge, use the install icon in the address bar or the browser’s app menu. In Safari on Mac, choose File → Add to Dock.

Markdown edition

# Study tests manipulation of AI rankings

> Offline experiments examine votes, prompts and model submissions.

By BIGCHANGE.AI Editorial

Published: 2026-11-14T18:41:38.137Z
Updated: 2026-11-14T18:41:38.137Z
Canonical: https://bigchange.ai/blog/preference-ranking-manipulation-study

![Illustration of a person comparing two responses on a screen.](https://bigchange.ai/api/media/file/ranking-v1.png)
AI-generated illustration for BIG CHANGE.

A [November 10 preprint](https://arxiv.org/abs/2611.14769) tests how votes, prompts and model submissions can alter AI rankings. Its experiments used public data, not live ranking manipulation.

## The big change

- An arena ranking combines people's choices between anonymous answers. It measures preferences in those comparisons, which may differ from the tasks a particular customer needs.
- The new experiments moved rankings without improving the target model. Their results depend on specified voting, prompt and model-copy assumptions.
- Earlier voting research shows why safeguards need testing too. Detecting unusual votes can reduce manipulation, but the researchers found that some strategies survived filtering.

## How comparisons become a ranking

The [2024 Chatbot Arena paper](https://arxiv.org/abs/2403.04132) describes users asking questions and comparing two anonymous model answers. They choose the response they prefer before learning the models' identities. A statistical method called Bradley–Terry combines those pairwise choices into estimates of relative strength.

This permits open-ended questions for which there may be no single correct answer. It also makes the participants and their questions part of what is measured. The original paper identifies a selection bias: its users lean toward people interested in language models. A broad preference score therefore needs context before a buyer treats it as evidence for a particular workload.

## What the November experiment changed

The dataset contained over 100,000 preferences. Adding up to 10,000 strategic votes moved a model as many as 20 places. Other tests repeated comparisons in a language favoring the target, or inserted up to ten copies of a selected model, assuming identical outcomes. This changes the comparison pool without changing the target’s answers. Combining prompting and copies produced a maximum 45-place shift.

Those maxima are conditional simulations, not typical improvements. The main voting results assume model identification; larger leaderboards may require more votes. No provider misconduct is established. Proposed defenses include limiting similar submissions and balancing prompt groups.

## Why other models' votes matter

The [January 2025 vote-rigging paper](https://arxiv.org/abs/2501.17858) explains an indirect route: a vote can change a target's rating even when its answer is absent. The ranking depends on comparisons throughout the pool, so changing opponents' estimated strength can change the target's relative position. Its experiments used approximately 1.7 million historical votes, a different dataset from the November study.

That earlier work also examined imperfect model identification and ordinary users voting at the same time. These conditions matter because an attacker may misidentify an answer or have strategic votes diluted by other participants. Its defense tests found that filtering unlikely outcomes reduced ranking gains but did not eliminate them. This supports a narrower conclusion than declaring either rankings or safeguards useless: the collection process and its protections need evaluation alongside the resulting score.

For a team selecting a model, the practical next step is to compare candidates on its own representative tasks. BIG CHANGE did not reproduce either study or test a live leaderboard.

## Sources

- [Leaderboard Hacking](https://arxiv.org/abs/2611.14769) — November 10 preprint and experiments.
- [Chatbot Arena paper](https://arxiv.org/abs/2403.04132) — 2024 comparison framework.
- [Vote rigging study](https://arxiv.org/abs/2501.17858) — Earlier 2025 research.