Markdown edition

# H-JEPA improves visual planning in simulated tasks with a hierarchical world model

> H-JEPA's three-level planner raises simulated AntMaze success from 18% to 73% with less planner compute. Its real-robot-video results measure offline path fidelity, not robot task completion.

By BIG CHANGE Editorial

Published: 2026-10-12T02:56:09.290Z
Updated: 2026-10-12T02:56:09.290Z
Canonical: https://bigchange.ai/blog/h-jepa-hierarchical-world-model-visual-planning

![A small orange sphere rests inside a raised tabletop maze, with a segmented orange route visible through the passages.](https://bigchange.ai/api/media/file/hjepa-maze-concept-hero-v2.png)
AI-generated conceptual illustration of hierarchical navigation planning by BIG CHANGE. The maze is not the AntMaze environment or a physical robot test.

Researchers behind [H-JEPA](https://arxiv.org/html/2610.06805v1) report a way for a visual world model to plan at several timescales. In one simulated maze, their three-level planner completed 73% of tasks, compared with 18% for a flat model, while using less planning compute. The October 5 paper also tests recorded real-robot videos, but those tests score proposed paths offline; they do not show a robot carrying out the plans.

For robotics researchers and engineers, the useful question is whether the improvement comes from breaking a long plan into shorter steps, from learning a better measure of progress toward the goal, or both. The paper tests those explanations separately. Its results are promising for visual planning research, with important limits on what can be inferred about physical robots.

## The big change

- **What changed:** H-JEPA trains a stack of action-conditioned predictive models. Each level looks farther ahead in its own learned representation, and the planner turns the upper level's predicted states into subgoals for the level below.
- **Why it matters:** In Visual AntMaze, the authors report 73.3% success for three-level planning against 18.0% for a flat baseline. Ablations indicate that shorter planning segments and a more useful high-level goal representation both contribute. A robotics team can study those two mechanisms separately instead of treating hierarchy as one indivisible improvement.
- **What to watch:** The four task environments are simulated. The additional DROID test compares planned robot-arm paths with recorded expert paths; a closed-loop trial in varied physical settings would address a different, still-open question.

## A plan at several timescales

A world model predicts how an environment may change after an action. H-JEPA uses a joint-embedding predictive architecture, or JEPA, which predicts compact representations of future observations rather than reconstructing every image pixel. Its [paper](https://arxiv.org/html/2610.06805v1) describes a hierarchy trained jointly on observation and action trajectories. The lowest level follows the finest time steps. Each higher level receives representations from the level below, aggregates actions over a longer stride and predicts farther ahead in a separate latent space.

Planning starts with a current observation and a goal observation. The top level searches for a coarse route to the goal. Its predicted states then guide the next level, which refines the route. The lowest level produces actions. In the simulated closed-loop evaluations, the system executes an action prefix, observes the resulting state and plans again. The researchers compare it with LeWM, the same first-level JEPA trained and planned on its own, and with a temporal hierarchy whose levels share one representation.

The separate representations matter in the maze because a goal can be about reaching a location even while the ant's leg pose changes rapidly. The authors' probes found that higher H-JEPA levels retained the ant's position while making leg-pose detail less recoverable. They found a similar pattern in FourRoom, where an independently moving distractor became less prominent at higher levels. The effect was much weaker in the manipulation datasets, whose measured fast and slow motions were less separated. The paper does not establish that every task benefits from discarding fine detail.

## What the comparisons show

The authors evaluated navigation in Visual AntMaze and FourRoom Distractors, and manipulation in Push-T and OGBench Cube. These are simulated tasks with fixed cameras and backgrounds. Across planning budgets, adding levels through three improved the success-versus-compute tradeoff in AntMaze, FourRoom and Cube. Push-T was the exception: two levels matched or exceeded the flat baseline at tested budgets, while three levels performed poorly. The authors point to its short episodes, which leave fewer usable training clips for deeper models.

On a separate AntMaze task set, the flat LeWM baseline averaged **18.0% success, with a 3.5-point standard error**, while full three-level H-JEPA planning averaged **73.3%, with a 3.5-point standard error**. Both figures are author-reported means over three paired model and planner seeds. The [paper's success-versus-planner-FLOPs comparison](https://arxiv.org/html/2610.06805v1) also shows the hierarchy reaching higher success with lower measured planning compute. It does not measure the total cost of training and deploying a robot.

The ablations clarify the contribution. A temporal hierarchy with one shared representation improved over the flat baseline across all four environments, indicating value in breaking up the plan. On AntMaze, using an upper-level representation merely to score a flat planner's predicted distance to the goal also improved success. Yet that change alone did not reach the full hierarchical planner's 73.3% mean. The two effects complement each other in this evaluation. On Push-T and Cube, where the paper found little selective abstraction, changing only the goal-cost representation produced no clear benefit.

## The real-robot-video boundary

The researchers also used [DROID robot-manipulation videos](https://arxiv.org/html/2610.06805v1). Scenes, objects and lighting change between recorded episodes. In that setting, the paper says a purely predictive model can favor the stable background and lose information about the moving robot. The team added an inverse-dynamics loss, training a component to infer the action between consecutive learned states so that the representation retains the agent's motion. With that addition, the learned two-level hierarchy improved the paper's offline planning-fidelity measure over the flat baseline at lower planner compute.

That measure compares a proposed three-dimensional end-effector path with an expert path in the recording. It is 100% for the expert path and 0% for a stationary arm under the paper's calibration. It does not measure whether a robot would complete a task after executing the proposed actions. The authors evaluated these DROID plans open-loop because the dataset has no simulator for the changing scenes. BIG CHANGE did not train the model, run the [public code](https://github.com/kevinghst/H-JEPA), or test a robot.

The code repository offers training and planning scripts and instructions for reproducing selected paper figures and tables. It gives researchers a route to inspect the method, but the paper remains a preprint and its physical-world evidence remains offline. Our earlier [GPT-Policy report](https://bigchange.ai/blog/gpt-policy-robot-learning-vlm-agents) covers a separate approach based on contextual examples and robot-tool requests; its results do not validate H-JEPA's world-model planning.

## Sources & further reading

- [Zhang and colleagues, “H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning,” arXiv v1](https://arxiv.org/html/2610.06805v1): Primary preprint submitted October 5, 2026. Sections 2–4 and the planning appendix establish the method, simulator comparisons, three-seed AntMaze figures and DROID offline protocol. These are the authors' results, not independent replication or physical deployment evidence.
- [H-JEPA project page](https://h-jepa.com/): Authors' visual explanation of hierarchical prediction, top-down planning, four simulated environments and the DROID fidelity test. Its examples illustrate the reported experiments; selected videos are not trial-level performance data.
- [H-JEPA public code repository](https://github.com/kevinghst/H-JEPA): Authors' training and planning implementation, environment list and reproduction instructions for selected results. BIG CHANGE inspected the repository description and instructions but did not execute them.
- [BIG CHANGE, “GPT-Policy tests context in robot-arm and mobile-exploration tasks”](https://bigchange.ai/blog/gpt-policy-robot-learning-vlm-agents): Earlier coverage of a distinct robotics research method. It provides context, not evidence for H-JEPA's results.

## Sources

- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning, arXiv v1](https://arxiv.org/html/2610.06805v1) — Primary preprint for method, four simulated task evaluations, AntMaze figures and ablations, and DROID offline Fréchet-fidelity protocol. Author-run results; not independent replication or physical deployment.
- [H-JEPA authors' project page](https://h-jepa.com/) — Authors' explanatory page and visual examples of hierarchy and experiments. Its selected videos illustrate results but do not supply trial-level outcome data.
- [H-JEPA public code repository](https://github.com/kevinghst/H-JEPA) — Authors' training/planning implementation and instructions for selected figure/table reproductions. Inspected as documentation; code was not run by BIG CHANGE.
- [BIG CHANGE, GPT-Policy tests context in robot-arm and mobile-exploration tasks](https://bigchange.ai/blog/gpt-policy-robot-learning-vlm-agents) — Related prior coverage of a different robotics method and experiment. Context only; no followUpTo relation.