mlx-whisper swallowed 72 words of Jules Verne — a silent-omission bench on Apple silicon

By Blasting Reviews (independent tester, EU) · Published Nov 8, 2026 · Updated Nov 10, 2026 · Corrections & tips: hello@blasting-reviews.com

Bench resultSituationalverdict10.12 %FR word error rateApple-silicon bench · method · all results

MLX Whisper: 10.12% FR WER with 69-72 words silently dropped per 22-min excerpt — whisper.cpp turbo: 4.39% on the same audio.

mlx-whisper-silent-omission-bench

Review · speech-to-text bench, part 3 — measured Nov 8, 2026

Situational

mlx-whisper is fine for English and unreliable for French. Same weights, same audio where whisper.cpp scored 4.39 % — it posted 10.12 % and silently dropped a 72-word passage. Situational at best; for French, use whisper.cpp.

On Apple Silicon, mlx-whisper is the tempting install: it uses the MLX array framework, feels native on M-series chips, and pip-installs in seconds. In our local bench it was competitive in English — then it ate Jules Verne. Not misheard him. Skipped him.

Same model, different runtime — and a hole in the transcript

The setup is the same bench as our whisper.cpp turbo-vs-full duel: the large-v3-turbo weights, the same WAV files, an Apple M5 Ultra, decoding at defaults. Only the runtime changed.

Runtime (large-v3-turbo weights)English WERFrench WERFrench transcript
whisper.cpp1.83 %4.39 %1,115 words
mlx-whisper1.87 %10.12 %1,043 words

English: 1.87 % vs 1.83 % — statistical noise, both excellent. French: more than double the error rate. A word-count check is what flagged it — 72 words missing against the whisper.cpp transcript of the same audio. Diffing the two outputs located one contiguous 72-word passage that never reached mlx-whisper’s transcript at all. Not garbled. Absent.

The vanished passage is Phileas Fogg’s Reform Club pedigree:

« À qui s’étonnerait de ce qu’un gentleman aussi mystérieux comptât parmi les membres de cette honorable association, on répondra qu’il passa sur la recommandation de MM. Baring frères, chez lesquels il avait un crédit ouvert. »

…plus the sentences around it. Everything before and after decoded normally. The audio there is clean studio reading — whisper.cpp handled it at a normal error rate.

Why this failure is worse than a wrong word

A mistranscribed name is visible: you read « MM. Beaning Freres » and you know something is off. A silent omission is invisible — the transcript simply looks finished, and the missing chunk carries the one fact (Fogg’s money and connections) a reader or a downstream summariser would otherwise get right. This is the failure mode that actually breaks real pipelines: meeting notes that quietly lose a decision, interviews that lose an admission.

Two contributing factors we could verify from the run itself:

  1. Decoder configuration is not equivalent. Our mlx-whisper build runs greedy decoding — beam_size was accepted and ignored in the CLI path we tested. whisper.cpp ran its default 5-beam search. Under-spec decoding is a plausible cause of the drop, but note the direction: the runtime that people choose for convenience shipped the accuracy loss silently.
  2. The gap had no marker. No timestamp hole warning, no low-confidence flag at the splice point in the default output. Only the count/diff check caught it.

What to do about it if you are on Apple silicon

  1. Prefer whisper.cpp. Same Metal acceleration, 3.5× the transcript integrity in this test, and a plain CLI. Turbo scored 4.39 % where mlx scored 10.12 % on identical French audio.
  2. If you must use MLX, add an integrity gate. Word count vs duration (≈150 words/min of speech) and an overlap check against any second decode. Our whole “two decoder disasters” theme in part 1 reduces to this: never trust a single decode for anything that matters.
  3. Re-bench before upgrades. We will rerun mlx-whisper every release-train against the versioned /benchmarks/ table; if this regression is fixed, this verdict changes and we will say so at the top of this post.

Method, exactly

mlx-whisper (pip, MLX backend) vs whisper.cpp (Homebrew, Metal), both on ggml-large-v3-turbo weights, same 16 kHz mono WAVs, greedy (mlx, beam unsupported in tested path) vs 5-beam default (cpp) as shipped. Reference ground truth and scoring identical to part 1 (Gutenberg/Wikisource text, NFD-normalised global Levenshtein, WER = edits / reference words). The 69-word span was located with difflib against the whisper.cpp transcript (69 normalised words inside a 72-word reference span) and confirmed against the printed text; raw outputs are in our repo.

Situational

English-only Apple silicon users: mlx-whisper is acceptable. Anyone touching French — or any language where a missing paragraph is expensive — use whisper.cpp until mlx-whisper proves otherwise on this bench.

Get the next benchmark in your inbox

One email per measured test — the numbers, the method, the raw data. Twice a month at most, no third-party trackers.

By subscribing you agree to receive occasional emails from Blasting Reviews. One-click unsubscribe in every email; the list lives on our own EU server. Correct anything: hello@blasting-reviews.com.

Cite as: Blasting Reviews (2026-11-08). “mlx-whisper swallowed 72 words of Jules Verne — a silent-omission bench on Apple silicon” · 10.12 % FR word error rate · Apple M5 Ultra bench · BR-STT-2026-11-08-76 · https://blasting-reviews.com/mlx-whisper-french-silent-omission-bench

Blasting Reviews

Independent tests of software, smart home gear, gadgets and everyday tech — run on our own bench, with the method and the numbers published. Verdicts stay ours; purchase links never change them.

Sections

AI & Models
Apps & Productivity
Streaming & Media
Smart Home & Energy
Gadgets & Hardware
Privacy & Services

Company

Benchmarks
About
How we test
Contact

Fair play

Some links may be affiliate links — they never influence a verdict, and prices are dated on publication.