NVIDIA released Nemotron 3 Diarization on September 23. The open-weight, roughly 100-million-parameter model marks when each of up to eight speakers talks in recorded or streaming audio. It gives developers speaker activity and timestamps for a conversation; producing a transcript with words also requires speech recognition.
The big change
- What changed: NVIDIA's previous streaming Sortformer checkpoint supports four speakers. This release offers eight arrival-ordered speaker channels from one checkpoint that works with both recorded audio and incoming chunks.
- Why it matters: A developer building meeting or call transcription can use the model to assign generic speaker labels to time intervals, including overlapping speech, then combine those intervals with a separate recognizer's words. The release widens the documented speaker limit for that task.
- What to watch: The practical choice is how much audio to buffer before each streaming result. NVIDIA's shortest recommended setting waits for 0.32 seconds of audio before computation begins, and its own accuracy tables show a cost to using that setting. Teams still need to measure the complete pipeline on their recordings.
What the model returns
The model card specifies 16 kHz, single-channel audio. It lists WAV, FLAC, Opus and MP3 files, and its NeMo interface accepts file paths, audio arrays or a line-delimited JSON manifest. Audio arrays need the correct sample rate passed to the diarize() call. Chunked inference has no fixed maximum recording duration in the card.
The model turns audio into a [T, 8] tensor of speaker activity probabilities, with a default output step of 10 milliseconds. Multiple channels can be active at once when people overlap. Postprocessing turns those probabilities into intervals with start and end times and generic speaker labels. The labels follow the order in which voices first appear; they do not identify people by name. NVIDIA uses a speaker cache for earlier chunks and a first-in, first-out queue for recent frames so the streaming model can retain context.
For a developer who needs a speaker-marked transcript, that distinction is consequential. NVIDIA's integration guide pairs diarization timestamps with a separate automatic speech recognition output. Its sample midpoint rule leaves a word unassigned when there is no active speaker and marks simultaneous activity as ambiguous. The diarization model alone does not decide which overlapping voice produced a recognized word.
Buffer time is not response time
NVIDIA lists recommended input-buffer settings of 30.4, 1.04, 0.64 and 0.32 seconds. The 0.32-second setting comprises three 80-millisecond processing frames and one 80-millisecond frame of right context. The card explicitly defines latency as audio held before inference; it excludes computation. A transcript system would also add recognition, transport and application time.
The documented NeMo path requires Python 3.12 or later, Cython, a recent PyTorch version and NeMo Speech. NVIDIA provides a SortformerEncLabelModel.from_pretrained() example, a diarize() call returning speaker segments, and a switch to include probability tensors. The card says a Hugging Face token is needed to load the hosted checkpoint through that NeMo call. It also shows an offline and streaming Transformers interface, with installation from the Transformers source repository, plus a NeMo-Speech.cpp command-line route. These are documented interfaces; BIG CHANGE has not run them. The model card lists Linux and NVIDIA Ampere, Hopper and Blackwell GPU families for its NeMo integration. Hardware needs and processing cost for a particular workload depend on the chosen route and machine; NVIDIA does not give a per-hour operating price in the card.
The weights are offered under OpenMDW 1.1. The license grants use, modification and distribution without a license fee, subject to its terms. Distribution must retain the license and applicable origin notices. The license does not restrict use or sharing of outputs, and it assigns users responsibility for securing rights and consents that may be needed for their audio. NVIDIA describes the model as available for commercial and noncommercial use.
What NVIDIA measured
NVIDIA's model-card evaluation compares Nemotron 3 with its earlier four-speaker streaming Sortformer v2.1. On the full DIHARD III evaluation set, NVIDIA reports a diarization error rate of 13.18% for Nemotron 3 at the 1.04-second input-buffer setting, against 19.60% for the earlier model at the same stated buffer setting. At 0.32 seconds, its Nemotron 3 result is 13.55%. Diarization error rate combines missed speech, false alarms and speaker confusion, divided by reference speaker time. Lower is better.
Those figures belong to a defined protocol. NVIDIA says all its evaluations score overlapping speech. DIHARD III uses a zero-second boundary collar, while CALLHOME-Part2 uses a 0.25-second collar. For AMI, AliMeeting and NOTSOFAR1, NVIDIA uses linked, forced-aligned reference annotations rather than the original segment labels; changing those labels changes the scores. The evaluation instructions identify the NeMo script and require a reference RTTM file, overlap setting, collar, streaming settings and postprocessing details for a comparable run. NVIDIA's inference-speed results used BF16 on an RTX PRO 5000, with and without torch.compile(). These are vendor-reported tests, not a BIG CHANGE reproduction.
The eight-speaker ceiling still matters. DIHARD III includes some recordings annotated with as many as nine speakers, and the card says speech beyond the model's eight channels can be missed or assigned to another channel. On AMI test recordings, NVIDIA reports lower diarization error for Nemotron 3 than for its earlier baseline while its exact speaker-count accuracy is lower. Accuracy therefore depends on the task and measure, as well as the audio. A team deciding whether to integrate the model can start from NVIDIA's published interfaces and test the complete speaker-and-word pipeline on recordings that resemble its intended use.
