Clinical speech needs more than accurate words. It needs to keep the people speaking apart, through short replies, interruptions and long conversations.

NVIDIA Nemotron 3 Diarization supports batch and streaming with up to eight speakers. NVIDIA gave us preview access and welcomed a public evaluation. Here we test the released model on 15 two-speaker mock consultations: 2.4 hours of clinical dialogue.

Batch results

Automatic speaker counts throughout. No supplied count and no forced two-speaker reassignment. Lower diarization error rate (DER) is better.

15 complete consultations · automatic speaker counts
SystemDER ±250 ms ↓Correct counts ↑Seconds/file ↓
Pyannote Precision-3 API2.891%14/1518.9 sAPI round trip
Nemotron 3 Diarization + Omi3.174%14/150.324 sL4
Nemotron 3 Diarization4.803%14/150.688 sL4
Pyannote Community-1 + Omi5.435%14/150.778 sL4
Pyannote Community-16.620%9/1518.691 sL4
Sortformer v1, whole-file6.778%14/153.869 sL4
Sortformer v2.17.974%3/151.077 sL4
VibeVoice-ASR8.233%15/15123 sASR + diarization server

+ Omi = Omi’s proprietary runtime. Same model weights, no retraining.

API round trips and local processing have different timing scopes. The Nemotron comparison uses the same L4, warm, batch size 1.

Meta Muse

Muse scored 13.042% DER, with the correct count in 18/20 clips and a median 92 s/request. It received separate clips rather than whole recordings.

Compare Muse on the same scoring intervals

All systems are scored on the same 20 intervals. Whole-file systems retain full-recording context; Muse receives each clip separately.

20 scoring intervals · different inference context
SystemDER ±250 ms ↓Correct interval counts ↑
Pyannote Precision-3 API2.895%19/20
Nemotron 3 Diarization4.804%19/20
Pyannote Community-16.622%13/20
Sortformer v1, whole-file BF166.630%19/20
Sortformer v2.1, FP326.832%4/20
VibeVoice-ASR8.235%20/20
Meta Muse Voice Transcribe13.042%18/20

Realtime results

Same 15 consultations · automatic speaker counts
SystemDER ±250 ms ↓Correct counts ↑Output
Pyannote live API3.959%10/15Live updates
Nemotron 3 Diarization + Omi4.575%14/15Live updates
Nemotron 3 Diarization *4.971%12/15Native preset
Sortformer v2.1 *6.958%0/15Native preset
VibeVoice streaming 1.5B17.210%15/15Live updates
VibeVoice streaming 7B18.032%13/15Live updates

+ Omi = Omi’s proprietary runtime.

* Native presets score the completed replay; other rows score delivered speaker updates. API inputs were paced; local replay was unpaced. This is an accuracy comparison, not a latency ranking.

With Omi’s runtime

Using the same Nemotron 3 Diarization weights, Omi lowers batch DER by 34% and processes recordings 2.13× faster.

Batch DER · ±250 ms
4.803%→3.174%
34% lower error
Processing · seconds/file
0.688→0.324
2.13× faster

Nemotron 3 Diarization leads the open models on this batch test. With Omi, it comes within 0.283 percentage points of Pyannote Precision-3, the leader at 2.891% DER.

At zero tolerance, DER moves from 12.720% to 13.203%. We publish both scores so the boundary-accuracy trade-off stays visible.

How we measured

DER counts missed speech, false speech and wrong-speaker attribution. We use ±250 ms boundary tolerance for the main comparison and retain strict DER in the full results. Correct counts show how many recordings had the right number of speakers.

Dataset and evaluation limits

15 frozen PriMock57 consultations, with reference turns intersected with voice activity detection. This is a two-speaker development set, not an eight-speaker validation or a production-capacity test. Settings were developed on this cohort. Full configurations, timing definitions and reference limitations are in the evaluation protocol.

Inspect the benchmark.

Audio, references, baseline inference code and scoring are public. Omi’s runtime remains proprietary.

View the evaluation on GitHub