NVIDIA’s 26 October 2026 announcement extends training recovery to Kubernetes. The 2.5.0 release notes specify Run:ai KAI scheduling with Kubeflow PyTorch operators. This account analyzes documentation; BIG CHANGE has not deployed or tested the feature.

The big change

  • What changed: An eligible job can requeue with an implicated node excluded from its specification.
  • Why it matters: Job recovery can start during hardware diagnosis.
  • What to watch: Checkpoint and telemetry integration determine what a restart can preserve. Measured reliability remains unreported.

Recovery paths

Autonomous Resiliency Engine (ARE) includes job and hardware recovery; job recovery 1.7.0 integrates with Run:ai 2.26.

Transient faults may retry without hardware quarantine. Suspected hardware faults trigger separate job and node responses. Job-level exclusion differs from hardware quarantine.

The autonomous job recovery manual identifies application and system logs, node and network metrics, and health checks as possible evidence. An UNKNOWN result leaves the cause unclassified. Its enrollment examples and attribution display concern Slurm; they are not demonstrated Run:ai instructions or a verified Run:ai interface.

The application carries saved progress

The manual separates manual investigation from the time still spent detecting a failure, stopping a job and loading a checkpoint. Work since the last saved checkpoint must be repeated. A scheduler can start another attempt, but the training application needs saved state and reload code to continue from earlier progress.

The manual recommends NVIDIA’s training telemetry client library for structured throughput, checkpoint and hang messages. That requires application integration. Teams can inspect their own logs and checkpoint files to determine which failure signals and saved state their current jobs actually provide; merely restarting a container does not prove that training resumed from a checkpoint.

NVIDIA Resiliency Extensions (NVRx) 0.6.0 is supported for asynchronous checkpointing but installed separately. Its versioned notes describe checkpoint worker changes. Training code still needs save and reload hooks; package support does not establish that a particular application has them or that its saved state is usable. BIG CHANGE has not verified a save-and-restart cycle.

Limits

Vera Rubin NVL72 lacks Run:ai and ARE support in 2.5.0.

No AI-based controller or uptime gain is established.