# Testing agent histories against reality

> Controlled experiments distinguish records from effects.

By BIGCHANGE.AI Editorial

Published: 2026-11-14T19:19:39.548Z
Updated: 2026-11-14T19:19:39.548Z
Canonical: https://bigchange.ai/blog/agent-history-world-mismatch

![Illustration of a person checking a calendar on a phone.](https://bigchange.ai/api/media/file/history-v2.png)
AI-generated illustration for BIG CHANGE.

Salesforce's [November 10 preprint](https://arxiv.org/abs/2611.14654) tests agents continuing from records that disagree with saved data. Cleaner histories sometimes hindered recovery.

## The big change

- The researchers separated what an agent was told from what the system stored. Their controlled tests do not establish production failure rates.
- AppWorld checks whether simulated applications actually reach the requested state. This makes it possible to distinguish a convincing completion message from a task completed correctly.
- A failed connection does not necessarily mean a failed operation. Stripe's retry guidance explains how supported requests can avoid repeating an effect that already happened.

## What the experiments changed

AgentCE uses planning grids. At 319 frozen states containing wrong values, two receiver models continued under original, falsely corrected, truthful-note or neutral-note histories. Truthful notes raised recovery from 51% to 66% versus neutral notes; false completed actions lowered it from 50% to 44% versus original histories. Recovery measured correct values among the next two or three outstanding slots, not whole-task completion.

In 65 AppWorld states, the researchers independently varied success/failure receipts and whether database writes took effect. Success receipts hurt whole-task success when effects were absent, but helped when present.

Wording, roles and placement also differed between histories; receipt contents differed too. These tests do not isolate truth alone or validate a general handoff policy.

## Why the destination matters

The original [2024 AppWorld paper](https://arxiv.org/abs/2407.18901) describes a controlled world of applications populated with fictitious users and activities. Agents write code and call application programming interfaces, the functions through which software requests actions. The benchmark then checks the resulting database rather than relying on a model's account of what it did.

Its tasks include everyday work across applications, such as ordering groceries using information from notes, messaging and shopping services. Evaluation allows different ways to complete the same request and checks for unintended changes as well as the requested result. That distinction is important: reaching a desired value is not sufficient if the agent also damages unrelated data.

This environment gives researchers a repeatable place to examine actions and their effects. It remains a simulation, not a customer account or a measurement of live reliability. A successful benchmark run should therefore be read with its task definition and checks, rather than treated as proof that the same agent can safely handle every application.

## An interrupted request needs a different kind of check

[Stripe's engineering account of idempotency](https://stripe.com/blog/idempotency) describes a concrete ambiguity. A server may finish an operation, but the connection can break before its reply reaches the requester. The requester then lacks confirmation even though the change happened. Simply repeating an unprotected payment request could produce an unwanted second effect.

For supported Stripe requests, a caller supplies an idempotency key: a unique identifier that lets the service recognize retries of the same operation. The documented payment example retries with the same key after a network error so that the customer is charged once. This protection comes from the service's request-handling contract; a written note saying “retry” does not create it.

These are specific engineering mechanisms, not a guarantee for every tool. Before retrying a consequential action, a team needs to know what the service can confirm and what protection it provides against duplicate effects. BIG CHANGE did not run the preprint's experiments or make payment requests.

## Sources

- [History and world preprint](https://arxiv.org/abs/2611.14654) — November 10 controlled experiments.
- [AppWorld benchmark](https://arxiv.org/abs/2407.18901) — Original simulated application environment.
- [Stripe on idempotency](https://stripe.com/blog/idempotency) — Retry behavior for supported requests.
