NVIDIA Open-Sources Nemotron 3 Diarization for Real-Time Multi-Speaker Audio

NVIDIA published Nemotron 3 Diarization on September 23 in a Hugging Face blog post: an open-weight, 100-million-parameter model that labels who spoke when in both live and recorded conversations. NVIDIA says it ranks first on VoiceArena's Diarization-Bench with a 14.72 percent diarization error rate (DER) and supports up to eight speakers.

Diarization solves the other half of the transcription problem. Speech recognition supplies the words; diarization supplies the time intervals during which each speaker is active, including the stretches where people talk over one another. Only the two together produce a speaker-attributed transcript, and without that attribution a meeting, call or podcast transcript can be word-perfect yet useless for deciding who made a commitment, who raised an objection or who interrupted whom.

NVIDIA is explicit that standalone diarization and speaker-attributed speech recognition are different tasks: one returns speaker activity and timestamps without words, the other returns words without guaranteed attribution. The company advises evaluating both components and the combined pipeline on the audio they will actually face.

Architecturally, Nemotron 3 Diarization follows the Sortformer approach of ordering output speakers by arrival time, so the first new voice becomes the first speaker channel and generic labels stay stable across chunked inference. It accepts 16 kHz single-channel audio, converts it to Mel-spectrogram features on a 10 ms frame step, stacks them by a factor of eight into 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings, then upsamples predictions with a Conv1D layer. The default output is a T by 8 tensor of speaker-activity probabilities, which handles overlap naturally because two channels can be active in the same frame.

Streaming inference is supported by two forms of memory: an arrival-order speaker cache (AOSC) that retains information about speakers seen in earlier chunks, and a FIFO queue that supplies recent frame context, plus right context from the audio just after the current chunk. Recommended input-buffer latencies are 30.4, 1.04, 0.64 and 0.32 seconds, where shorter buffers respond sooner and more context generally improves accuracy.

In VoiceArena's initial Diarization-Bench, spanning 139 English-language conversations and roughly 22 hours of audio across 12 systems and 17 configurations, Nemotron 3 Diarization scored 14.72 percent DER with overlapping speech scored, system-generated speech activity detection and no boundary collar, against 19.3 percent for the next-ranked system — a relative reduction of about 24 percent. It also ranked first with 100 ms and 250 ms collars and across both in-person and online recordings.

Against NVIDIA's earlier four-speaker streaming model, diar_streaming_sortformer_4spk-v2.1, the new model reduces DER on all eight listed evaluation conditions at 1.04-second input-buffer latency, with relative reductions ranging from 9.0 percent on CALLHOME-Part2 to 65.2 percent on NOTSOFAR1 MHM. The unweighted mean of those eight relative reductions is about 40 percent.

NVIDIA also keeps the inconvenient numbers in view: on the two-speaker CALLHOME subset the new model records 5.98 percent DER versus 5.68 percent for the old baseline, even though CALLHOME-Part2 overall improves from 10.32 percent to 9.10 percent. DIHARD III aggregates five-to-nine-speaker recordings, and nine speakers exceed the model's supported maximum of eight. The model's speaker channels are anonymous labels rather than identities, so mapping them to real people requires meeting metadata, user profiles or speaker verification downstream.

The model was trained on public and licensed speech data, including multispeaker-annotated real conversations licensed from David AI and large-scale simulated English and multilingual mixtures spanning 21 languages; adding the David AI data cut compound DER by 0.77 absolute points, from 11.19 to 10.42 percent, at both offline-style and ultra-low-latency operating points. It is meant to be used with NVIDIA's NeMo Speech tooling, and the post walks through streaming diarization with streaming ASR, offline diarization, latency-quality trade-offs and diarization combined with offline ASR.

For products, this points to device-side, real-time speaker-aware transcription as a common layer: voice-agent memory, action-item ownership in meeting notes and responsibility splits in call quality review can all sit on top of it. The caveat is that VoiceArena's Version 1 evaluation and paired statistical analysis are still being completed, so the current ranking may shift.