In Who Verifies the Tools?, William Jeffries reports 20 retrieved documents from 4,772 matches, with the cap omitted.

The big change

  • What changed: Paired tool wording changed how identical results were interpreted.
  • Why it matters: Tool protocols let builders expose boundaries in structured results and compare them with the text delivered to an agent or verifier. A shared text view cannot supply fields it omitted.
  • What to watch: An output schema can require an identifier type. Validating that field's presence and form still leaves the actual record reference to check against its owning system.

Where the information went missing

Four omissions involved search limits, result caps, page estimates and document-versus-scan IDs. The estimate was reconstructed from an unreleased draft.

What the experiment found

Forty cases had eight pilots and 32 holdouts. Three models, five repeats and two wordings yielded 1,200 runs including pilots. For scope, completeness and uncertainty, fixed false answers flipped all 90 verifier judgments from acceptance to rejection. Under clear wording, 113/120 identifier agent answers still overclaimed. That wording bundled disclosures and instructions, so their effects remain inseparable.

What a tool contract can check

The Model Context Protocol tool specification separates a tool result's ordinary content from optional JSON structuredContent. A tool may declare an outputSchema. When it does, the server must return conforming structured content, and the client should validate it. The specification also recommends a text rendering of structured content for compatibility. A builder therefore has two surfaces to compare: the machine-readable value and the words the model may see.

JSON Schema's object reference makes an important distinction. Listing a property under properties describes how to validate it if present; absence remains valid by default. The required keyword makes its presence mandatory, while type constrains the value's form. A proposed contract could require resultLimit, totalKnown, isEstimated and idNamespace fields. A missing field could fail validation before it is turned into prose.

Presence is only the first check. A numeric total can still be inaccurate, and a well-typed identifier can still point to the wrong record. Checking those values against the search implementation or owning data store is a separate operation. This is a contract design inference from the standards, not a tested repair in the archive study or a BIG CHANGE experiment.

One archive and a one-answer harness cannot establish general failure rates.

Sources & further reading