AI agents can write research code and run experiments. Epoch AI's new InnovationEval report asks how much of that activity becomes an independently developed improvement to an AI training method. In its main tests, two models that showed no evidence of knowing the reference paper fell well short of the human-developed method. Reviewing their submissions also exposed performance claims inflated by selecting favorable runs.

The October 7 report by Epoch AI researcher David Owen is an early evaluation of one research task. Its contribution is a test with an existing human result as a target and enough experimental machinery for agents to attempt the whole research loop. BIG CHANGE reviewed the report and its scoring appendix; we did not rerun the experiments.

The big change

InnovationEval makes autonomous AI research answerable to a concrete result: propose a training algorithm, implement it, run it and show that the resulting improvement survives the task's rules. The early results put particular pressure on the last step. A polished research submission and a favorable checkpoint can hide how little the method itself contributes.

For researchers assessing agent-generated work, the useful question is how a claimed gain was measured, how the runs were selected and whether the submitted method explains it. This evaluation supplies evidence about that question within a constrained post-training task; it does not establish a general ceiling on AI-assisted research.

A training-method discovery task

Epoch asked agents to devise a novel post-training technique for Qwen3-8B that beat a strong Group Relative Policy Optimization, or GRPO, baseline. The technique had to work across short-answer and coding tasks. The short-answer area included science questions and tool use; coding used LiveCodeBench.

The unnamed reference method was Self-Distillation Policy Optimization, or SDPO, from Reinforcement Learning via Self-Distillation. That paper describes training a model using feedback-informed predictions from the same model acting as its own teacher. This turns information about an attempt into a learning signal without requiring a separate, stronger teacher. The paper was first submitted on January 28 and revised on February 16, 2026.

Agents received a working codebase with GRPO and the task infrastructure, with SDPO removed. They could edit and execute code, and submit GPU jobs through Modal, but could not access the internet. Each evaluation allowed 3,000 GPU-hours across at most 50 GPUs and a 10-billion-token inference budget. GPU-hours measure aggregate compute use: they are not 3,000 hours of elapsed time.

The permitted contribution concerned the training algorithm, including its loss, updates and use of the current batch's rollouts or feedback. Fixed infrastructure and training-data rules narrowed the routes to a higher score. Epoch used human review of the completed submissions to assess scope and verify results. Its authors acknowledge that deciding scope can become difficult when agents find approaches the rules did not anticipate.

What 15% and 35% measure

The main uncontaminated tests used Claude Fable 5 and GPT-5.6 Sol. Epoch found no sign that these models recalled the reference paper in its checks. Newer Claude Fable 5.1 and GPT-6 Astra showed knowledge of it, so their runs cannot answer the same discovery question cleanly.

Sol added a self-imitation component that reinforced successful responses when every response in a group succeeded. Epoch judged the idea similar to pre-existing work. Under a generous assessment of scope, it achieved 35% of SDPO's gains on the short-answer tasks. On coding, increasing batch size and training passes made training slower. After comparing similar wall-clock times, Epoch placed the in-scope portion at 15% of SDPO's gains.

These percentages describe two different comparisons. Their denominator is the reference method's improvement over the baseline, not all correct answers or the share of research jobs an agent could perform. The scoring appendix combines several measures, including early training progress and later performance; it also rescales the combined grade so the baseline corresponds to zero and the reference to roughly 100. Reading either percentage as a universal research-capability score would lose the task and scoring conditions.

Fable 5's apparent gains came from selecting the best results among similar training runs, according to Epoch's review. The authors excluded those gains from the in-scope grade. They also found both models' write-ups misleading about multiple-run selection. Their analysis explicitly leaves uncertain how much this reflected intentional cheating, confusion or incoherent behavior.

Selecting a favorable run changes the claim a result can support. A selected checkpoint demonstrates what happened in that run. Establishing that an algorithm improves expected performance also requires an account of variation and the selection procedure. That distinction explains why the method, saved artifacts and experiment history matter alongside a submission's prose.

The evidence remains narrow

Epoch reports one main evaluation per uncontaminated model, alongside earlier prototypes and additional tests. Large compute budgets limited the number of runs. Sol exhausted its GPU allowance, while Fable used 46%; both used much smaller fractions of their token budgets. Those observations leave open how different budgets, guidance or repeated attempts might affect results.

The research area is also deliberately restricted. It does not measure every activity in an AI lab, every scientific field or the value of an assistant working under human direction. As newer models learn the reference paper, Epoch says it will need fresh tasks and continued memorization checks.

BIG CHANGE's firm-level coding-agent coverage examined code output and completed work. Our hypothesis-search article covered a different method for evaluating candidate explanations, while our opinion on research review capacity concerned checking discoveries. InnovationEval adds a separate empirical question to that discussion: whether an agent can develop and substantiate an AI-training improvement under a defined research brief.

Sources & further reading