Living document · last updated November 8, 2026
Why this page exists
Every benchmark we run lives here, versioned. We publish the audio and text sources, the scoring method and the raw transcripts, so you can rerun our tests or tell us where our maths is wrong. Results are updated when a meaningfully new model or runtime ships.
Speech-to-text (ASR) — Word Error Rate
Method: 10-minute audiobook chapters — Alice in Wonderland (English, LibriVox) and Chapter 1 of Le Tour du Monde en Quatre-Vingts Jours (French, LibriVox) — decoded at 16 kHz mono WAV, then scored word-by-word against the printed book from Project Gutenberg and Wikisource (NFD-normalised Levenshtein alignment). Same audio for every engine; numbers are ours, run on an Apple M5 Ultra (local) or vendor-hosted demos (cloud).
| Engine | Runs | EN WER | FR WER | Time / 10 min | Cost |
|---|---|---|---|---|---|
| NVIDIA Nemotron 3.5 Transcribe | Cloud demo | 1.32 % | 1.70 % | ~13 s | €0 demo |
| whisper.cpp large-v3-turbo | Local | 1.83 % | 4.39 % | 11.5 s (~55× realtime) | €0 |
| mlx-whisper large-v3-turbo | Local (MLX) | 1.87 % | 10.12 %† | 13.2 s | €0 |
| parakeet-tdt-0.6b-v3 (MLX) | Local (MLX) | 1.78 % | 7.43 % | 4.9 s (~130× realtime) | €0 |
| whisper.cpp large-v3 (full) | Local | 10.92 %* | 6.36 % | 40.5 s | €0 |
*Repetition-loop failure at the end of the English clip (the single real failure mode we observed). Full write-up: NVIDIA’s Nemotron Beat Whisper on Our Bench.
†The mlx-whisper French run skipped a 72-word passage silently — part of why its French WER is inflated and its time looks competitive. Parakeet’s French alignment was anchored on « en l’année » (not the year itself): it transcribed the novel’s opening date as 1861 instead of 1872. Full write-up: Parakeet 0.6B vs Whisper on Apple Silicon.
Write-ups
- NVIDIA’s Nemotron Beat Whisper on Our Bench. You Still Can’t Install It
- Whisper turbo vs large-v3: the smaller model won — then the full one looped 14 times
- mlx-whisper swallowed 72 words of Jules Verne — a silent-omission bench
- Parakeet 0.6B: fastest local model we measured — best English WER, different (milder) failure mode in French
Planned benches
- ASR round 2: noisy and accented audio (same scoring pipeline).
- Local LLMs: tokens/s and RAM at 4-bit vs 8-bit on identical long documents.
- Note apps: fidelity of importing a 500-note corpus.
- Streaming: bitrate and resolution actually delivered, via ffprobe.
- Smart home: standby watts at the wall (meter on order) and trigger-to-state latency.
Want a model or product benched? hello@blasting-reviews.com — we say so plainly if we can’t measure it.
| Test | Headline metric (FR) | Verdict | Machine | Date |
|---|---|---|---|---|
| NVIDIA’s Nemotron Beat Whisper on Our Bench. You Still Can’t Install It | 1.70 % | WAIT | Apple M5 Ultra 256GB | 2026-11-08 |
| Whisper large-v3-turbo vs large-v3: the smaller model won — then the full one looped 14 times | 4.39 % | YES | Apple M5 Ultra 256GB | 2026-11-08 |
| mlx-whisper swallowed 72 words of Jules Verne — a silent-omission bench on Apple silicon | 10.12 % | WAIT | Apple M5 Ultra 256GB | 2026-11-08 |
| Parakeet 0.6B vs Whisper large-v3-turbo on Apple Silicon: We Measured Both Languages | 7.43 % | WAIT | Apple M5 Ultra 256GB | 2026-11-08 |
Machine-readable: download CSV · protocol: How we test