An October 21 preprint found that 327 of 335 PubMed IDs supplied by closed-book agents resolved to real records. A separate language-model judge rated only three of those 327 relevant to the biomedical claims.
Author Mohammed Saqlain compared closed-book answering, fixed-query PubMed retrieval, agent-written queries and an oracle given the correct abstracts. Each ran on 100 sampled SciFact development claims with two Gemini Flash Lite models. Agent search reached 62% verdict accuracy with both; its advantage over other realistic setups was not statistically significant. Oracle accuracy reached 79% and 86%.
The big change
A citation existence check leaves relevance untested. The relevance finding rests on one automated judge checked against 50 controls. This small, single-provider study does not establish clinical reliability.
