Researchers behind H-JEPA report a way for a visual world model to plan at several timescales. In one simulated maze, their three-level planner completed 73% of tasks, compared with 18% for a flat model, while using less planning compute. The October 5 paper also tests recorded real-robot videos, but those tests score proposed paths offline; they do not show a robot carrying out the plans.

For robotics researchers and engineers, the useful question is whether the improvement comes from breaking a long plan into shorter steps, from learning a better measure of progress toward the goal, or both. The paper tests those explanations separately. Its results are promising for visual planning research, with important limits on what can be inferred about physical robots.

The big change

  • What changed: H-JEPA trains a stack of action-conditioned predictive models. Each level looks farther ahead in its own learned representation, and the planner turns the upper level's predicted states into subgoals for the level below.
  • Why it matters: In Visual AntMaze, the authors report 73.3% success for three-level planning against 18.0% for a flat baseline. Ablations indicate that shorter planning segments and a more useful high-level goal representation both contribute. A robotics team can study those two mechanisms separately instead of treating hierarchy as one indivisible improvement.
  • What to watch: The four task environments are simulated. The additional DROID test compares planned robot-arm paths with recorded expert paths; a closed-loop trial in varied physical settings would address a different, still-open question.

A plan at several timescales

A world model predicts how an environment may change after an action. H-JEPA uses a joint-embedding predictive architecture, or JEPA, which predicts compact representations of future observations rather than reconstructing every image pixel. Its paper describes a hierarchy trained jointly on observation and action trajectories. The lowest level follows the finest time steps. Each higher level receives representations from the level below, aggregates actions over a longer stride and predicts farther ahead in a separate latent space.

Planning starts with a current observation and a goal observation. The top level searches for a coarse route to the goal. Its predicted states then guide the next level, which refines the route. The lowest level produces actions. In the simulated closed-loop evaluations, the system executes an action prefix, observes the resulting state and plans again. The researchers compare it with LeWM, the same first-level JEPA trained and planned on its own, and with a temporal hierarchy whose levels share one representation.

The separate representations matter in the maze because a goal can be about reaching a location even while the ant's leg pose changes rapidly. The authors' probes found that higher H-JEPA levels retained the ant's position while making leg-pose detail less recoverable. They found a similar pattern in FourRoom, where an independently moving distractor became less prominent at higher levels. The effect was much weaker in the manipulation datasets, whose measured fast and slow motions were less separated. The paper does not establish that every task benefits from discarding fine detail.

What the comparisons show

The authors evaluated navigation in Visual AntMaze and FourRoom Distractors, and manipulation in Push-T and OGBench Cube. These are simulated tasks with fixed cameras and backgrounds. Across planning budgets, adding levels through three improved the success-versus-compute tradeoff in AntMaze, FourRoom and Cube. Push-T was the exception: two levels matched or exceeded the flat baseline at tested budgets, while three levels performed poorly. The authors point to its short episodes, which leave fewer usable training clips for deeper models.

On a separate AntMaze task set, the flat LeWM baseline averaged 18.0% success, with a 3.5-point standard error, while full three-level H-JEPA planning averaged 73.3%, with a 3.5-point standard error. Both figures are author-reported means over three paired model and planner seeds. The paper's success-versus-planner-FLOPs comparison also shows the hierarchy reaching higher success with lower measured planning compute. It does not measure the total cost of training and deploying a robot.

The ablations clarify the contribution. A temporal hierarchy with one shared representation improved over the flat baseline across all four environments, indicating value in breaking up the plan. On AntMaze, using an upper-level representation merely to score a flat planner's predicted distance to the goal also improved success. Yet that change alone did not reach the full hierarchical planner's 73.3% mean. The two effects complement each other in this evaluation. On Push-T and Cube, where the paper found little selective abstraction, changing only the goal-cost representation produced no clear benefit.

The real-robot-video boundary

The researchers also used DROID robot-manipulation videos. Scenes, objects and lighting change between recorded episodes. In that setting, the paper says a purely predictive model can favor the stable background and lose information about the moving robot. The team added an inverse-dynamics loss, training a component to infer the action between consecutive learned states so that the representation retains the agent's motion. With that addition, the learned two-level hierarchy improved the paper's offline planning-fidelity measure over the flat baseline at lower planner compute.

That measure compares a proposed three-dimensional end-effector path with an expert path in the recording. It is 100% for the expert path and 0% for a stationary arm under the paper's calibration. It does not measure whether a robot would complete a task after executing the proposed actions. The authors evaluated these DROID plans open-loop because the dataset has no simulator for the changing scenes. BIG CHANGE did not train the model, run the public code, or test a robot.

The code repository offers training and planning scripts and instructions for reproducing selected paper figures and tables. It gives researchers a route to inspect the method, but the paper remains a preprint and its physical-world evidence remains offline. Our earlier GPT-Policy report covers a separate approach based on contextual examples and robot-tool requests; its results do not validate H-JEPA's world-model planning.

Sources & further reading