# Real PubMed IDs can still point to the wrong evidence

> A preprint found most cited PubMed IDs existed while few were judged relevant.

By BIGCHANGE.AI Editorial

Published: 2026-10-25T13:44:39.985Z
Updated: 2026-10-25T13:44:39.985Z
Canonical: https://bigchange.ai/blog/real-pubmed-ids-wrong-evidence-biomedical-agents

![Journal shelves with one orange-marked volume.](https://bigchange.ai/api/media/file/pubmed-real-id-evidence-check-hero-v2.png)
AI-generated conceptual illustration by BIG CHANGE.

An October 21 preprint found that 327 of 335 PubMed IDs supplied by closed-book agents resolved to real records. A separate language-model judge rated only three of those 327 relevant to the biomedical claims.

Author Mohammed Saqlain compared closed-book answering, fixed-query PubMed retrieval, agent-written queries and an oracle given the correct abstracts. Each ran on 100 sampled SciFact development claims with two Gemini Flash Lite models. Agent search reached 62% verdict accuracy with both; its advantage over other realistic setups was not statistically significant. Oracle accuracy reached 79% and 86%.

## The big change

A citation existence check leaves relevance untested. The relevance finding rests on one automated judge checked against 50 controls. This small, single-provider study does not establish clinical reliability.

## Sources

- [Saqlain's preprint](https://arxiv.org/html/2610.29812v1) — Methods, citation audit and statistical limits.