UserSWE is an October 21 preprint testing 11 coding agents on 72 repository tasks with simulated users.

The big change

  • What changed: The user’s clarity and proposed cause can change across turns.
  • How it works: Each turn triggers tests and cross-family verdict review.
  • Why it matters: The leader changes with the exchange budget.

What the score captures

A fixed pool constrains disclosure. Messages react to the agent and tests; a persona chain calibrated on 20,859 private sessions shapes their wording. The users were simulated.

Claude Fable 5.1 leads at one turn (31.9%); GPT-6-Astra at eight (81.9%), one task above Claude Opus 5. Opus leads on the score that rewards faster completion. Different scaffolds complicate model-only comparisons. The authors promise a benchmark and framework release; v1 links none. Tasks cover one or two issues.