Giving a coding agent teammates helped in one of four matched comparisons in Vals AI’s October 9 Vibe Code Bench study. A GPT 6 Sol team at medium reasoning effort passed 84.9% of the benchmark’s app tests, against 77.6% for a single Sol agent. The 7.3 percentage-point gain cost $3.07 rather than $1.22 per app on average. Vals reported a 95% interval of 2.3 to 12.4 points for that gain and a paired-test p-value of 0.005. None of the other three team-versus-single comparisons showed a statistically significant score difference.
The result gives developers a more precise budgeting question than whether teams are “better.” On this set of web-app builds, Sol at medium effort bought a measurable quality gain for roughly $1.85 more per app, with almost the same median elapsed time. At higher effort, and with Claude Opus 5.5 at either effort, extra team spending did not establish a score improvement in this test.
The big change
- What changed: Vals measured single coding agents against teams on the same 50 web-app tasks at two reasoning settings. Only Sol at medium effort had a statistically significant team score gain.
- Why it matters: The winning team setup cost 2.5 times as much per app; the other teams cost 1.8 to 5.1 times as much without a significant score gain. Team design and model settings belong in the same purchasing decision.
- What to watch: Repeated runs and tests on other software tasks could show whether that gain is dependable beyond these particular apps. This study ran each setup once.
What the eight runs measured
Vals asked GPT 6 Sol and Claude Opus 5.5 to build 50 complete web apps from product specifications. The apps were deployed with Docker Compose and scored by browser agents working through user-interface tests. The reported score is the average share of each app’s tests that passed. The benchmark method says each app specification has 20 to 60 automated tests and an individual test counts as passing when at least 90% of its steps succeed.
For each model, Vals compared a single agent with a lead that could delegate to as many as five subagents concurrently, at both medium and max reasoning effort. The leads got an added instruction to divide, delegate, integrate and verify the work. The task prompt, sandbox and grader were otherwise held constant within each model comparison. Each of the eight configurations built each of the 50 apps once.
Model and effort | Single score | Team score | Mean cost per app: single → team | Median runtime: single → team |
|---|---|---|---|---|
GPT 6 Sol, medium | 77.6% | 84.9% | $1.22 → $3.07 | 12.4 → 11.9 min |
GPT 6 Sol, max | 89.0% | 90.4% | $4.82 → $8.54 | 38.4 → 27.6 min |
Claude Opus 5.5, medium | 91.5% | 91.2% | $4.08 → $9.10 | 20.3 → 26.6 min |
Claude Opus 5.5, max | 89.8% | 93.2% | $23.77 → $122 | 74.7 → 174.1 min |
These are Vals’s mean model costs for one app build, not subscription prices or costs per successful UI test. The runtimes are medians of wall-clock time, so a row’s cost and time describe different summaries of the 50 runs. The max-effort Opus team’s mean includes one timed-out run, whose roughly $189 cost Vals estimated from token counts.
Why the result depends on the setup
Vals compared each team with its matching single agent on the same apps using a paired t-test. The Sol max-effort team gained 1.4 points, with a 95% interval from −1.7 to 4.5 points. Opus’s medium-effort team scored 0.3 points lower, with an interval from −3.2 to 2.5; its max-effort team gained 3.4 points, with an interval from −1.4 to 8.3. Those intervals include no gain. They do not prove that teams never help those configurations, but this experiment did not establish a gain.
For Sol, spending on reasoning effort was another tested choice. Raising a single Sol agent from medium to max effort increased its score from 77.6% to 89.0%, which Vals reports as statistically significant. That max-effort single agent cost $4.82 per app and had a 38.4-minute median runtime, compared with $3.07 and 11.9 minutes for the medium-effort team. The 4.1-point score difference between those two configurations was not statistically significant (p = 0.08). Those figures let a developer see the measured price and time trade-off without assuming one configuration will win on a new project.
The teams also spent their time differently. Vals says Sol’s lead delegated work early and tested the assembled app itself. Its max-effort team finished in a median 27.6 minutes, down from 38.4 for the single agent. Opus’s leads tended to write a shared contract and delegate in waves. At max effort the team took a median 174.1 minutes, versus 74.7 for the single agent, and cost about five times as much per app. Vals attributes much of the extra bill to cached input tokens held in subagents’ separate contexts.
How far to take this finding
The significance intervals reflect differences among the 50 apps, not variation between repeated attempts at the same app. A different run could change the comparison. The team runs also had the added delegation instruction, so the experiment measured the combined team-and-instruction setup. Sol used the OpenAI Agents API and Opus used Claude Code; cross-model score comparisons also mix different harnesses. Vals’s Sol cost estimates assume its reported token usage includes subagents and do not account for any higher price on requests crossing the long-context threshold, because request-level usage was unavailable.
The study supports a bounded decision: when building similar full-stack apps with these configurations, a Sol medium-effort team showed a measurable gain at higher cost, while the other teams did not show a reliable score gain. It cannot settle whether teams pay off for another codebase, a different model, or a workflow with a different way to split and check the work. BIG CHANGE reviewed the published Vals post and method; it did not run or replicate the benchmark.
