Clinical speech needs more than accurate words. It needs to keep the people speaking apart, through short replies, interruptions and long conversations.
NVIDIA Nemotron 3 Diarization supports batch and streaming with up to eight speakers. NVIDIA gave us preview access and welcomed a public evaluation. Here we test the released model on 15 two-speaker mock consultations: 2.4 hours of clinical dialogue.
Batch results
Automatic speaker counts throughout. No supplied count and no forced two-speaker reassignment. Lower diarization error rate (DER) is better.
| System | DER ±250 ms ↓ | Correct counts ↑ | Seconds/file ↓ |
|---|---|---|---|
| Pyannote Precision-3 API | 2.891% | 14/15 | 18.9 sAPI round trip |
| Nemotron 3 Diarization + Omi | 3.174% | 14/15 | 0.324 sL4 |
| Nemotron 3 Diarization | 4.803% | 14/15 | 0.688 sL4 |
| Pyannote Community-1 + Omi | 5.435% | 14/15 | 0.778 sL4 |
| Pyannote Community-1 | 6.620% | 9/15 | 18.691 sL4 |
| Sortformer v1, whole-file | 6.778% | 14/15 | 3.869 sL4 |
| Sortformer v2.1 | 7.974% | 3/15 | 1.077 sL4 |
| VibeVoice-ASR | 8.233% | 15/15 | 123 sASR + diarization server |
+ Omi = Omi’s proprietary runtime. Same model weights, no retraining.
API round trips and local processing have different timing scopes. The Nemotron comparison uses the same L4, warm, batch size 1.
Meta Muse
Muse scored 13.042% DER, with the correct count in 18/20 clips and a median 92 s/request. It received separate clips rather than whole recordings.
Compare Muse on the same scoring intervals
All systems are scored on the same 20 intervals. Whole-file systems retain full-recording context; Muse receives each clip separately.
| System | DER ±250 ms ↓ | Correct interval counts ↑ |
|---|---|---|
| Pyannote Precision-3 API | 2.895% | 19/20 |
| Nemotron 3 Diarization | 4.804% | 19/20 |
| Pyannote Community-1 | 6.622% | 13/20 |
| Sortformer v1, whole-file BF16 | 6.630% | 19/20 |
| Sortformer v2.1, FP32 | 6.832% | 4/20 |
| VibeVoice-ASR | 8.235% | 20/20 |
| Meta Muse Voice Transcribe | 13.042% | 18/20 |
Realtime results
| System | DER ±250 ms ↓ | Correct counts ↑ | Output |
|---|---|---|---|
| Pyannote live API | 3.959% | 10/15 | Live updates |
| Nemotron 3 Diarization + Omi | 4.575% | 14/15 | Live updates |
| Nemotron 3 Diarization * | 4.971% | 12/15 | Native preset |
| Sortformer v2.1 * | 6.958% | 0/15 | Native preset |
| VibeVoice streaming 1.5B | 17.210% | 15/15 | Live updates |
| VibeVoice streaming 7B | 18.032% | 13/15 | Live updates |
+ Omi = Omi’s proprietary runtime.
* Native presets score the completed replay; other rows score delivered speaker updates. API inputs were paced; local replay was unpaced. This is an accuracy comparison, not a latency ranking.
With Omi’s runtime
Using the same Nemotron 3 Diarization weights, Omi lowers batch DER by 34% and processes recordings 2.13× faster.
Nemotron 3 Diarization leads the open models on this batch test. With Omi, it comes within 0.283 percentage points of Pyannote Precision-3, the leader at 2.891% DER.
At zero tolerance, DER moves from 12.720% to 13.203%. We publish both scores so the boundary-accuracy trade-off stays visible.
How we measured
DER counts missed speech, false speech and wrong-speaker attribution. We use ±250 ms boundary tolerance for the main comparison and retain strict DER in the full results. Correct counts show how many recordings had the right number of speakers.
Dataset and evaluation limits
15 frozen PriMock57 consultations, with reference turns intersected with voice activity detection. This is a two-speaker development set, not an eight-speaker validation or a production-capacity test. Settings were developed on this cohort. Full configurations, timing definitions and reference limitations are in the evaluation protocol.
Inspect the benchmark.
Audio, references, baseline inference code and scoring are public. Omi’s runtime remains proprietary.
View the evaluation on GitHub