# Mission Control 2.5 adds Run:ai training recovery

> Mission Control recovery targets KAI-scheduled Kubeflow PyTorch jobs; continuing training requires application checkpoints.

By BIGCHANGE.AI Editorial

Published: 2026-10-27T14:11:38.596Z
Updated: 2026-10-27T14:11:38.596Z
Canonical: https://bigchange.ai/blog/nvidia-mission-control-2-5-kubernetes-training-recovery

![One operator faces a blank console beside three closed compute cabinets.](https://bigchange.ai/api/media/file/mission-control-operator-compute-room-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE.

[NVIDIA’s 26 October 2026 announcement](https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes/) extends training recovery to Kubernetes. The [2.5.0 release notes](https://docs.nvidia.com/mission-control/docs/systems-quick-start-guide/latest/nmc-release-notes.html) specify Run:ai KAI scheduling with Kubeflow PyTorch operators. This account analyzes documentation; BIG CHANGE has not deployed or tested the feature.

## The big change

- **What changed:** An eligible job can requeue with an implicated node excluded from its specification.
- **Why it matters:** Job recovery can start during hardware diagnosis.
- **What to watch:** Checkpoint and telemetry integration determine what a restart can preserve. Measured reliability remains unreported.

## Recovery paths

**Autonomous Resiliency Engine (ARE)** includes job and hardware recovery; job recovery 1.7.0 integrates with Run:ai 2.26.

Transient faults may retry without hardware quarantine. Suspected hardware faults trigger separate job and node responses. Job-level exclusion differs from hardware quarantine.

The [autonomous job recovery manual](https://docs.nvidia.com/mission-control/docs/systems-quick-start-guide/latest/ajr/ajr-overview.html) identifies application and system logs, node and network metrics, and health checks as possible evidence. An `UNKNOWN` result leaves the cause unclassified. Its enrollment examples and attribution display concern **Slurm**; they are not demonstrated Run:ai instructions or a verified Run:ai interface.

## The application carries saved progress

The manual separates manual investigation from the time still spent detecting a failure, stopping a job and loading a checkpoint. Work since the last saved checkpoint must be repeated. A scheduler can start another attempt, but the training application needs saved state and reload code to continue from earlier progress.

The manual recommends NVIDIA’s training telemetry client library for structured throughput, checkpoint and hang messages. That requires application integration. Teams can inspect their own logs and checkpoint files to determine which failure signals and saved state their current jobs actually provide; merely restarting a container does not prove that training resumed from a checkpoint.

**NVIDIA Resiliency Extensions (NVRx) 0.6.0** is supported for asynchronous checkpointing but installed separately. Its [versioned notes](https://nvidia.github.io/nvidia-resiliency-ext/release-notes.html) describe checkpoint worker changes. Training code still needs save and reload hooks; package support does not establish that a particular application has them or that its saved state is usable. BIG CHANGE has not verified a save-and-restart cycle.

## Limits

Vera Rubin NVL72 lacks Run:ai and ARE support in 2.5.0.

No AI-based controller or uptime gain is established.

## Sources

- [NVIDIA's recovery announcement](https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes/) — Vendor scenarios; unmeasured.
- [Release notes](https://docs.nvidia.com/mission-control/docs/systems-quick-start-guide/latest/nmc-release-notes.html) — Version and support matrix.
- [NVIDIA autonomous job recovery user guide](https://docs.nvidia.com/mission-control/docs/systems-quick-start-guide/latest/ajr/ajr-overview.html) — Attribution, telemetry and checkpoints; Slurm setup is not Run:ai guidance.
- [NVIDIA Resiliency Extension 0.6.0 release notes](https://nvidia.github.io/nvidia-resiliency-ext/release-notes.html) — Separate checkpoint utilities; the current page includes later versions.