EmbodiedCLBench finds no steady gain from extra examples

EmbodiedCLBench, a preprint posted October 9, tests whether AI agents can use experience with simple household tasks to solve new combinations of those tasks. In the authors' simulated evaluation, one prior example produced a large improvement. Adding more examples did not reliably lift success further. That makes the benchmark useful for separating an initial benefit from evidence of steadily accumulating skills.

The big change

• What changed: The researchers released a benchmark of 168 basic and 100 advanced household tasks designed to test whether agents can carry experience into unseen task combinations.
• How it works: Agents first complete related basic tasks. They then choose actions from candidates in an offline state graph built from the TongSim household simulator. A task succeeds when the agent reaches its goal within a fixed step budget.
• What the authors found: In the main comparison, access to two basic-task experiences improved advanced-task success across all eight tested harness and model pairings. In a narrower sample-count analysis, moving from zero to one experience helped substantially, while moving from one to six produced no consistent upward trend.
• Why it matters: The results give developers a way to test whether an agent uses new experience to solve unfamiliar combinations, rather than assuming that storing more examples improves performance. The study evaluates simulated decisions, not physical robot operation.

A household task becomes a sequence of choices

The benchmark uses TongSim, a simulator with indoor scenes, interactive objects and actions such as walking to, picking up, opening and switching on. The authors build basic tasks from short sequences of object interactions. Advanced tasks combine actions drawn from two to four basic-task categories. The paper gives “put the cup on the table” as a basic example and “tidy up the room” as an advanced one.

The agent does not control a running simulator during the published evaluation. The authors used a closed-source TongSim version to construct an offline state-transition graph. Each node contains a simulator image and candidate actions. An executable choice moves the agent to another state; an invalid action cannot be executed there. The agent repeats the choice until it reaches the goal or runs out of steps. The paper says task descriptions and action candidates were generated with Qwen3.7-Plus and manually checked.

This design makes the measured outcome precise: the success rate is the share of tasks completed within the budget. Advanced tasks in the main test have a 12-step shortest path, with three extra steps allowed at test time. A failure can therefore reflect a wrong or inefficient sequence under that limit. It does not measure grasp reliability, motor control or performance in a changing physical home.

Earlier tasks help, then the gains level off

The main results table compares agents attempting advanced tasks with no prior examples against two ways of carrying forward experience from two related basic tasks. In-context learning keeps the full earlier interaction trajectories. Skill learning replaces those trajectories with an agent-written Markdown skill summary. Across four harnesses, each paired with two backbone models, both approaches beat the corresponding zero-shot advanced-task result. For example, the Codex harness with Qwen3.8-Flash rose from 4% without the learning phase to 34% with trajectories and 33% with a skill summary. Those are the authors' results under this benchmark's task set and budget.

The sample-count analysis varies the number of basic-task examples from zero to six. Here the authors used only Hermes and OpenClaw, both with Qwen3.8-Flash. Across the two harnesses and two learning methods, one example moved advanced-task success from an 8%–10% zero-example range to 28%–40%. After that first example, none of the four reported curves climbed consistently as more examples were added. The finding applies to these agents and memory methods under the paper's evaluation conditions; it does not establish a general limit on how much experience can help.

The authors' action analysis offers a narrower clue about the improvement. Compared with zero-shot runs, experienced agents chose fewer invalid actions in the recorded decisions. The paper reports a 13%–20% reduction across the four harnesses it analyzed with Qwen3.8-Flash. That pattern is consistent with better use of the available action choices, although it does not establish that the agents learned a durable, general skill.

For researchers choosing an evaluation, the useful question is whether a method improves success as the number and variety of prior tasks increase, on the same held-out compositions and step budget. EmbodiedCLBench supplies a structured test of that question. Its evidence remains the authors' October 9 preprint and project release; BIG CHANGE has not run the benchmark, and the results have not been independently replicated here.

Sources & further reading

• Jie Huang and coauthors, “EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents,” arXiv v1, posted October 9, 2026. The primary paper documents TongSim task construction, the offline action graph, step budgets, eight main harness/model configurations and the narrower zero-to-six-example analysis. All results are the authors' reported simulation results; this is a preprint, not a physical robot trial or an independent replication.
• EmbodiedCLBench project page, checked October 13, 2026. The authors' project page summarizes the benchmark and links to code and dataset releases. It repeats the paper's findings and is not an independent experimental source.
• arXiv version record, checked October 13, 2026. Confirms the paper title, authors and October 9 v1 date; it supplies metadata rather than separate performance evidence.