Sakana AI researchers tested a system that reads scientific papers before drafting a review. In an October 8 preprint, their Multi-Layered Review system identified 73.43% of contradictions deliberately inserted into papers’ main claims when four generated reviews were considered together. That figure describes the most central category of a synthetic benchmark. On a separate set of withdrawn papers with real reported problems, its reviews matched the retraction comments in 16.11% of cases under a strict test, or raised a similar concern in 26.07% under a looser one.

The big change

  • What changed: Sakana’s team tested AI reviews against planted, identifiable mistakes instead of relying only on how closely a generated review resembles a human one. The prominent 73.43% result covers the most central planted contradictions and four attempts per paper.
  • Why it matters: Reviewers and editors need tools that point to substantive errors, not just plausible prose. The much lower match rates on withdrawn papers show how much performance depends on the kind of error and the way detection is measured.
  • What to watch: A useful next test is whether systems can find naturally occurring problems across subjects and longer papers, then give reviewers evidence they can check. This preprint also found that hidden instructions in submitted text could influence review scores.

How the contradiction test worked

The researchers began with 257 AI conference papers from ACL, AISTATS, CVPR and ICML 2025, plus NeurIPS 2024. They used Gemini 2.5 Pro to map claims and supporting details in each paper. GPT-4.1 then produced contradictory passages at different distances from a main claim in that map. Replacing the original passages and recompiling the papers yielded 1,164 altered versions, each with one planted contradiction. Distance zero denotes a contradiction in a main claim; larger distances reach more peripheral details, according to the paper’s design.

Sakana’s reviewer, built around Claude Sonnet 4, first outlines a paper, then reads more closely for evidence, assumptions and weaknesses, and finally drafts a review. The full framework can add an appendix summary and literature search. For the contradiction test, the researchers used only the Review Agent, with neither of those optional components. An o3 model compared each review with the original and altered passages to judge whether it had caught the intended contradiction. The researchers repeated that judging step ten times and reported the average.

The 73.43% figure comes from Table 14 of the preprint: a contradiction at distance zero was counted as caught if any of four generated reviews identified it. The benchmark construction provides one distance-zero altered version for each of the 257 source papers. A single MLR review scored 60.79% on that category. For all 1,164 planted contradictions together, four reviews scored 40.95% and one scored 28.32%. The rate fell as errors moved away from main claims, though the most distant groups contained few examples.

The comparison needs its settings attached. On distance-zero errors, LLM-Review using GPT-4.1 scored 14.56%; changing that baseline to Claude Sonnet 4 and removing its text truncation raised it to 35.40%. The single-review MLR result of 60.79% was higher again. That sequence suggests both model choice and review procedure mattered, while the four-review 73.43% result also benefits from multiple chances to flag an error. The baselines used different models and review formats, so the headline number is not an isolated measurement of the three-pass procedure.

The real-paper check was harder

The authors also evaluated WithdrarXiv-Check, a dataset of withdrawn arXiv papers paired with retraction comments. They used 211 papers of at most 30 pages from an original 245-paper test set. Their o3 judge looked for a concern in the generated review that matched a retraction comment. An “exact” match produced MLR’s 16.11% result; allowing a similar concern produced 26.07%. The next best baseline in the similar setting, AgentReview, scored 18.48%. These are matches to the documented retraction reasons, not a measurement of every possible flaw in each paper.

The comparison across the two datasets has limits. The authors say many withdrawn papers are in mathematics and physics, while the systems were designed mainly around AI conference submissions. Their appendix reports declining detection as page count rose. The synthetic benchmark has its own advantage: in a manual audit of 50 planted contradictions, human raters judged 34% to interrupt the surrounding prose’s natural flow. That could make some artificial edits easier to spot than mistakes that arose in the course of research or writing.

The preprint also tests deliberate manipulation. Researchers hid text after the conclusion in tiny white type in 50 rejected papers. The paper reports that all tested review systems were influenced by the inserted instructions; MLR explicitly noticed them in eight cases. Its mean change in its own review score was small, but individual changes varied widely. This is a separate warning about treating submitted manuscript text as trustworthy instructions.

The work offers a more concrete way to ask whether an AI reviewer catches a known error. It does not establish reliable automated peer review. The results are reported by the system’s creators in a preprint, and BIG CHANGE did not run the code or independently reproduce the evaluations.

Sources & further reading