The WorldFailBench v1 preprint, submitted October 24, finds seven models understated success shifts on selected RoboTwin scenes.

The big change

  • What changed: On selected RoboTwin boundary pairs, 400 of 420 model responses understated the absolute reference success change.
  • Why it matters: Robot teams need reference checks before diagnosing failures or setting operating limits.
  • What to watch: Physical G1 humanoid tasks, sweeping trash and collecting clothes, measured type and stage only.

What an outcome comparison measures

A robot policy chooses actions from observations and an instruction. A world model returns predicted observations after those actions, letting the policy continue in a generated environment. WorldArena's policy-evaluator track runs policies through action-controllable models, judges task completion from their videos, and compares success rates with RoboTwin simulator runs. Its reported correlation describes how well an evaluator orders policies by completion rate under that setup.

REFLECT's failure-explanation research shows why diagnosis needs more than a final outcome: its robot records distinguish mistaken object choices from failed handling of an intended object and locate the relevant step. Two evaluations can agree that a task failed while pointing to different repairs. A task-wide success percentage erases the distinction. The team choosing what to fix needs a record of observed behavior and timing; the team ranking policies by completion rate is answering another question.

What the new benchmark adds

WorldFailBench pairs starting conditions across reference and generated runs. Its simulated boundary pairs alter object position, orientation or size.

Two experts labeled observable failures; a third resolved disagreements. BIG CHANGE did not rerun the experiments.

Seven models on 60 selected RoboTwin pairs made 420 responses, not 420 scenes. Each compares the absolute success-rate difference across pair endpoints. Simulator reruns from matched starts gauge variation, not independent replication.

Using the finding

An earlier evaluation position paper argues that world-model claims should name the decision they are meant to support, then use evidence matched to that decision. It separates policy ranking from interventional action fidelity and closed-loop rollout validity. That distinction helps frame a review: deciding which policy completes more tasks, explaining a failed action, and estimating the effect of a changed condition each require an appropriate reference measurement. The position paper proposes evaluation practice; it does not report a test of those recommendations.

WorldArena's completion comparison can help rank policies by task outcome. REFLECT's observed failure explanations address what went wrong after an execution. For a generated environment to support either use, its output needs a reference comparison for that same question. When testing a change in operating conditions, compare success at both conditions instead of relying on an average. A single ranking discards the evidence needed to distinguish an outcome difference from a diagnostic one.

Our earlier H-JEPA report examined simulated visual planning. It offers background on another world-model use.