Benchmarks

Living document · last updated November 8, 2026

Why this page exists

Every benchmark we run lives here, versioned. We publish the audio and text sources, the scoring method and the raw transcripts, so you can rerun our tests or tell us where our maths is wrong. Results are updated when a meaningfully new model or runtime ships.

Speech-to-text (ASR) — Word Error Rate

Method: 10-minute audiobook chapters — Alice in Wonderland (English, LibriVox) and Chapter 1 of Le Tour du Monde en Quatre-Vingts Jours (French, LibriVox) — decoded at 16 kHz mono WAV, then scored word-by-word against the printed book from Project Gutenberg and Wikisource (NFD-normalised Levenshtein alignment). Same audio for every engine; numbers are ours, run on an Apple M5 Ultra (local) or vendor-hosted demos (cloud).

EngineRunsEN WERFR WERTime / 10 minCost
NVIDIA Nemotron 3.5 TranscribeCloud demo1.32 %1.70 %~13 s€0 demo
whisper.cpp large-v3-turboLocal1.83 %4.39 %11.5 s (~55× realtime)€0
mlx-whisper large-v3-turboLocal (MLX)1.87 %10.12 %†13.2 s€0
parakeet-tdt-0.6b-v3 (MLX)Local (MLX)1.78 %7.43 %4.9 s (~130× realtime)€0
whisper.cpp large-v3 (full)Local10.92 %*6.36 %40.5 s€0

*Repetition-loop failure at the end of the English clip (the single real failure mode we observed). Full write-up: NVIDIA’s Nemotron Beat Whisper on Our Bench.

†The mlx-whisper French run skipped a 72-word passage silently — part of why its French WER is inflated and its time looks competitive. Parakeet’s French alignment was anchored on « en l’année » (not the year itself): it transcribed the novel’s opening date as 1861 instead of 1872. Full write-up: Parakeet 0.6B vs Whisper on Apple Silicon.

Write-ups

Planned benches

  • ASR round 2: noisy and accented audio (same scoring pipeline).
  • Local LLMs: tokens/s and RAM at 4-bit vs 8-bit on identical long documents.
  • Note apps: fidelity of importing a 500-note corpus.
  • Streaming: bitrate and resolution actually delivered, via ffprobe.
  • Smart home: standby watts at the wall (meter on order) and trigger-to-state latency.

Want a model or product benched? hello@blasting-reviews.com — we say so plainly if we can’t measure it.

Master results — all measured tests
TestHeadline metric (FR)VerdictMachineDate
NVIDIA’s Nemotron Beat Whisper on Our Bench. You Still Can’t Install It1.70 %WAITApple M5 Ultra 256GB2026-11-08
Whisper large-v3-turbo vs large-v3: the smaller model won — then the full one looped 14 times4.39 %YESApple M5 Ultra 256GB2026-11-08
mlx-whisper swallowed 72 words of Jules Verne — a silent-omission bench on Apple silicon10.12 %WAITApple M5 Ultra 256GB2026-11-08
Parakeet 0.6B vs Whisper large-v3-turbo on Apple Silicon: We Measured Both Languages7.43 %WAITApple M5 Ultra 256GB2026-11-08

Blasting Reviews

Independent tests of software, smart home gear, gadgets and everyday tech — run on our own bench, with the method and the numbers published. Verdicts stay ours; purchase links never change them.

Sections

AI & Models
Apps & Productivity
Streaming & Media
Smart Home & Energy
Gadgets & Hardware
Privacy & Services

Company

Benchmarks
About
How we test
Contact

Fair play

Some links may be affiliate links — they never influence a verdict, and prices are dated on publication.