Review · speech-to-text bench, part 3 — measured Nov 8, 2026
mlx-whisper is fine for English and unreliable for French. Same weights, same audio where whisper.cpp scored 4.39 % — it posted 10.12 % and silently dropped a 72-word passage. Situational at best; for French, use whisper.cpp.
On Apple Silicon, mlx-whisper is the tempting install: it uses the MLX array framework, feels native on M-series chips, and pip-installs in seconds. In our local bench it was competitive in English — then it ate Jules Verne. Not misheard him. Skipped him.
Same model, different runtime — and a hole in the transcript
The setup is the same bench as our whisper.cpp turbo-vs-full duel: the large-v3-turbo weights, the same WAV files, an Apple M5 Ultra, decoding at defaults. Only the runtime changed.
| Runtime (large-v3-turbo weights) | English WER | French WER | French transcript |
|---|---|---|---|
| whisper.cpp | 1.83 % | 4.39 % | 1,115 words |
| mlx-whisper | 1.87 % | 10.12 % | 1,043 words |
English: 1.87 % vs 1.83 % — statistical noise, both excellent. French: more than double the error rate. A word-count check is what flagged it — 72 words missing against the whisper.cpp transcript of the same audio. Diffing the two outputs located one contiguous 72-word passage that never reached mlx-whisper’s transcript at all. Not garbled. Absent.
The vanished passage is Phileas Fogg’s Reform Club pedigree:
« À qui s’étonnerait de ce qu’un gentleman aussi mystérieux comptât parmi les membres de cette honorable association, on répondra qu’il passa sur la recommandation de MM. Baring frères, chez lesquels il avait un crédit ouvert. »
…plus the sentences around it. Everything before and after decoded normally. The audio there is clean studio reading — whisper.cpp handled it at a normal error rate.
Why this failure is worse than a wrong word
A mistranscribed name is visible: you read « MM. Beaning Freres » and you know something is off. A silent omission is invisible — the transcript simply looks finished, and the missing chunk carries the one fact (Fogg’s money and connections) a reader or a downstream summariser would otherwise get right. This is the failure mode that actually breaks real pipelines: meeting notes that quietly lose a decision, interviews that lose an admission.
Two contributing factors we could verify from the run itself:
- Decoder configuration is not equivalent. Our mlx-whisper build runs greedy decoding —
beam_sizewas accepted and ignored in the CLI path we tested. whisper.cpp ran its default 5-beam search. Under-spec decoding is a plausible cause of the drop, but note the direction: the runtime that people choose for convenience shipped the accuracy loss silently. - The gap had no marker. No timestamp hole warning, no low-confidence flag at the splice point in the default output. Only the count/diff check caught it.
What to do about it if you are on Apple silicon
- Prefer whisper.cpp. Same Metal acceleration, 3.5× the transcript integrity in this test, and a plain CLI. Turbo scored 4.39 % where mlx scored 10.12 % on identical French audio.
- If you must use MLX, add an integrity gate. Word count vs duration (≈150 words/min of speech) and an overlap check against any second decode. Our whole “two decoder disasters” theme in part 1 reduces to this: never trust a single decode for anything that matters.
- Re-bench before upgrades. We will rerun mlx-whisper every release-train against the versioned /benchmarks/ table; if this regression is fixed, this verdict changes and we will say so at the top of this post.
Method, exactly
mlx-whisper (pip, MLX backend) vs whisper.cpp (Homebrew, Metal), both on ggml-large-v3-turbo weights, same 16 kHz mono WAVs, greedy (mlx, beam unsupported in tested path) vs 5-beam default (cpp) as shipped. Reference ground truth and scoring identical to part 1 (Gutenberg/Wikisource text, NFD-normalised global Levenshtein, WER = edits / reference words). The 69-word span was located with difflib against the whisper.cpp transcript (69 normalised words inside a 72-word reference span) and confirmed against the printed text; raw outputs are in our repo.
English-only Apple silicon users: mlx-whisper is acceptable. Anyone touching French — or any language where a missing paragraph is expensive — use whisper.cpp until mlx-whisper proves otherwise on this bench.