In an October 26 technical account, NVIDIA describes Minerva investigations of slow training jobs. Its internal cases do not establish a launch or customer results.

The big change

  • What changed: NVIDIA detailed a path from anomalous devices to affected jobs.
  • Why it matters: A link can pass a health check while traffic crossing it slows.
  • What to watch: Inspect the incident's underlying events before changing hardware or job placement.

A job can fail before a device looks broken

A 2017 Microsoft Research paper on gray failure describes differential observability: the requester sees degraded service while the component's failure detector sees nothing wrong. Its reduced-bandwidth link example is close to the operator's problem here. A test of connectivity asks whether the link works at all; the job depends on the performance it delivers under load. The paper establishes a diagnostic blind spot, not Minerva's effectiveness.

To investigate that gap, compare what a service experienced with what component monitors recorded over the same interval. Matching timestamps and dependencies can make an explanation plausible, but proximity alone cannot prove it. A stale relationship map or a slowdown that began before the component changed would weaken the proposed cause. This is an analytical check, not a measured result.

From a suspect component to a job

NVIDIA says Minerva compares peer devices, flags unusual logs and maps infrastructure to active jobs. Operators can investigate from a job or interface. Correlation narrows the search; the team determines cause and action.

The maintained Minerva 1.0.1 user guide describes on-premises collectors, a graph of device relationships, and metrics and events attached to those objects. Its incident view groups events by shared attributes or network topology and exposes a timeline. An operator can compare the proposed trigger with the constituent events before draining a node.

The guide's overview mentions SaaS, but the 1.0.1 release notes explicitly say Cloud/SaaS is unsupported for this version. Follow that release-specific restriction when planning deployment.

What the cases show

NVIDIA says Minerva found one degrading NVLink connection among roughly 400,000 interfaces; engineers drained nodes. It grouped 81 events into one incident and estimates 54 hours of total cluster uptime saved. An NVMe case has an estimated 5,530 hours of job runtime saved. No comparison or fleet sample supports generalizing those estimates.

The guide's Sherlock section describes live-data questions after LLM configuration. Its beta MCP interface provides authenticated, read-only queries and cannot remediate faults. MCP is not tenant-aware: authorized users can read all deployment data; namespace filters are not user access controls. The guide gives no complete customer price.

BIG CHANGE did not deploy or test Minerva.