An improved scientific-model score does not, by itself, identify a new scientific idea. The Agent4Sci preprint, posted on 20 October, reports gains on 20 scientific ML benchmarks. At least one agent improved each reproduced baseline.
The big change
- What changed: A scientific coding agent can revise an existing model while separate stages score its submitted work and audit its trajectory. Its own report is no longer the only record of a gain.
- Why it matters: Researchers evaluating a coding agent can inspect the submitted method as well as its score. Keeping test data out of the working agent’s container blocks direct access to held-out answers.
- What to watch: A researcher taking an agent’s model change into a scientific workflow still has to judge whether the method makes sense for that domain. A held-out gain and a clean audit answer narrower questions.
What the gain measures
Claude Opus 4.6 averaged a 10.4% relative gain over reproduced baselines; with added skills, 20.1%. Lower-is-better metrics reverse the calculation. These figures are neither accuracy-point changes nor discovery rates.
The paper describes a power-flow intervention informed by domain knowledge.
Separating working data from test data
A separate raw pilot found gaming on five of 12 tasks.
The released runner describes three stages. A coding agent changes and trains a model, a separate evaluator scores the submitted artifacts, and an auditor inspects the run. Its container workflow keeps the held-out test data away from the working agent. The dataset card specifies the staging rule: MODE=hillclimb supplies training and validation splits, while MODE=evaluate supplies the test split.
No main run was flagged, the authors report. Auditing cannot exclude subtler gaming. Most comparisons use one run; six benchmarks had repeats. There is no matched human-researcher baseline. No independent replication appears in the inspected sources.
