Faster inference and mitigating hallucinations in CrisperWhisper 2.0

Grounding the model in silence

The invented text is an artefact of how Whisper learned. A large share of its supervision came from subtitles and captions scraped from the web, where the transcript and the audio are only loosely aligned: captions linger over a musical intro, credits roll across silence, a "Thanks for watching" card sits on top of a stretch where no one actually says it. The model dutifully learned the association: audio with no speech in it tends to be paired, in the training data, with exactly those phrases. So when it later meets silence or noise, it reaches for the text that used to sit over that kind of audio and writes it down with confidence.

Once the cause is named, the fix reads as almost obvious: we are simply correcting that training-data flaw. If the problem is that the model was taught to pair non-speech with words, we reinforce the desired behaviour and actively associate silence and non-speech with empty output.

So alongside our speech data we mix in large, diverse non-speech audio and set the target to an empty transcript. The model is explicitly rewarded for staying quiet when no one is speaking. The non-speech sources:

• ESC-50 / ESC-10: 2,000 clips across 50 balanced environmental-sound classes; a standard, well-curated benchmark.
• FSD50K: a Freesound-derived collection of ~51k clips spanning 200 classes. Large, open, and acoustically varied.

The result is a model whose default behaviour on non-speech is to produce nothing, rather than to reach for the nearest familiar phrase. Nothing about this is exotic; it is just correcting a prominent faulty supervision pattern in the base model.

Breaking the repetition loop

Grounding handles "silence becomes a sentence." The repetition loop is different in kind: it is not learned from bad labels, it is baked into how autoregressive decoding works.

Repetition loops are most easily triggered on out-of-distribution audio: noisy recordings, disfluent or disordered speech, domain-shifted accents, long pauses. Atwany et al. (2025) report a 0.91 correlation between distribution shift and hallucination rate. The dynamics are autoregressive self-reinforcement (Xu et al., 2022): once a phrase has repeated a few times, that repetition is itself the strongest signal in the context window; the most likely next token becomes "more of the same," and each step deepens the context holding the model in. The loop is a shallow local attractor the model cannot escape under standard decoding.

We are not the first to notice Whisper loops, and it is worth being clear about how existing fixes work. They fall into three families:

• Global repetition penalties: logit processors such as no_repeat_ngram_size or a repetition penalty (as shipped in HuggingFace transformers' generation utilities, and borrowed earlier from Fairseq) that hard-block or down-weight any token extending a repeated n-gram. Cheap, but blunt: they apply everywhere, and genuine repetition ("no no no", a stuttered restart) gets penalised along with the pathological kind.
• Temperature fallback: Whisper's own heuristic. After decoding a segment it computes the gzip compression ratio (a looped transcript compresses suspiciously well) and the average token log-probability; if either crosses a threshold the segment is re-decoded from scratch at a higher temperature, and the temperature is stepped up until the checks pass. Sampling does often jostle the model out of the loop, but the intervention is global: it re-rolls the entire segment, trading the determinism of greedy decoding for randomness, costing one or more extra decodes, and with no guarantee of converging.
• Contrastive decoding: running additional decoding paths to actively suppress repetition and non-speech hallucination (e.g. multi-negative contrastive decoding). Effective, but it multiplies inference cost.

All three change how the model decodes everywhere, to fix something that goes wrong in one place. A loop is a local failure mode: the model hits a moment it is unsure about and gets stuck, while the rest of the transcript is fine. But penalties, temperature fallback and contrastive decoding all reshape decoding globally (every token, the whole segment, every probability), pulling inference away from the regime the model was trained in. We want to repair one specific failure mode and otherwise keep inference as close to training as possible, not alter the model's global decoding behaviour to patch a local problem.

We also notice that the looped token is usually grounded in the actual audio: the stutter or sound the model latched onto is really there, it just failed to move past it. So keeping a single realization of it gives the most faithful transcript. All that is needed is a small, surgical, local intervention to nudge the model back to paying attention to the audio at that one point.

So we let the model decode freely and only act once a loop has actually formed. We track consecutive n-gram repeats (sizes 1 through 5) with generous per-size thresholds: 1- and 2-grams may repeat up to 8 times, 3-grams up to 4, longer n-grams up to 3. Until a threshold is crossed we do not touch the model at all; normal speech, stutters and repeated words run exactly as they would without us. When one is crossed we rewind to where the loop started, keep a single copy of the phrase, ban the token that re-enters the loop for exactly one step, and let the model decode freely again. The decoder state (KV-cache) is reused across the rewind, so the repair is cheap.

Each repeat is counted as it is emitted. At the threshold the loop is flagged, the surplus copies are rewound to a single one, and the looping token is banned for exactly one step, just long enough for the model to pick its next-best token (liters) from the audio. Decoding then continues freely, with the token's likelihood already back to normal.

Why a one-step ban is enough

The loop is a shallow attractor: the only thing sustaining it is the repeated context the model just produced. Banning the loop-starter for one step breaks that self-reinforcement, and the model resumes from its acoustic evidence. Keeping one copy means the artifact that caused the loop stays in the transcript, but the model does not get stuck there.

That is the trade-off we are after: zero influence on normal transcription (stutters, restarts, genuine repetitions all pass through untouched) and surgical correction the moment a loop is confirmed.

Repair on, head-to-head

The SAPC2 corpus contains a reasonable density of pathological English speech (stutters, false starts, disordered articulation), making it a natural target for measuring looping and similar hallucinations. For the same audio we show the ground truth, our model decoding without repair, our model with repair, and Whisper large-v2. In the first two, Whisper falls straight into the same loop trap and never recovers.

Measured on the 1,000-clip SAPC2 set, 21 clips loop without the repair. With it, the repair removes the loop and the model resumes the real transcript in every one of them, cutting word error on those clips from the thousands of percent (a window flooded with repeated tokens) back to the normal range for hard disordered speech.

The numbers

We benchmarked CrisperWhisper 2.0 against OpenAI's whisper-large-v2 on the two hallucination failure modes: inventing text where the correct answer is no transcript at all (UrbanSound8K, pure Gaussian noise), and looping on hard speech (SAPC2).

Does it stay silent when it should?

On pure noise, Whisper writes something 92% of the time; our grounded model writes nothing. On real-world urban sounds, fabricated output drops from 47% to ~2%, more than a 20× reduction.

We never trained on UrbanSound8K; none of these urban recordings appear in our non-speech data. The ~2% is the grounding generalising to out-of-distribution noise it has never heard, not memorisation of non-speech sounds it already knows well.

Is it always the same sentence?

For Whisper, yes: its hallucinations are highly stereotyped. Across all the noise and silence clips it transcribed, a single word, you, accounts for 49% of every fabrication, and a handful of YouTube-subtitle phrases (thanks for watching, thank you for watching) make up most of the rest. This is the training-data contamination from the intro surfacing whenever the audio is ambiguous.

Does it loop?

To keep this an honest experiment rather than a foregone conclusion, we allow the repair to fire only once per clip: a single rewind-and-escape, then the model is on its own. The 0% is therefore not us suppressing loops by brute force: it shows that after one nudge the model does not re-enter the loop, but picks the acoustic evidence back up and finishes the real transcript.

Fast enough for production

To make CrisperWhisper 2.0 viable for latency-sensitive applications, we built on the excellent faster-whisper project (Whisper inference on the CTranslate2 runtime) and extended its engine to support speculative decoding and our word-level timing extraction.

Speculative decoding pairs the model with a smaller, faster draft model. The draft proposes a run of tokens; the large model then verifies all of them in a single batched pass and accepts the longest prefix it agrees with. Because verification is exact, the output is token-for-token identical to plain decoding: speculation only changes throughput, never the transcript (see HuggingFace's write-up for a fuller explanation). The trick is a draft that is both fast and rarely wrong, so we trained our own: CrisperWhisper 2.0 turbo, a fine-tuned Whisper turbo trained on exactly the same data and with all the same training configuration as CrisperWhisper 2.0. Because draft and target were shaped by the identical recipe, they agree most of the time and few drafted tokens are wasted. The number of tokens drafted per round self-tunes to the draft's running acceptance, so it stays near-optimal across easy and hard audio without any manual tuning.

The combined effect on longform audio is large. Moving CrisperWhisper 2.0 onto CTranslate2 already decodes several times faster than running it in HuggingFace transformers, or than regular Whisper in OpenAI's reference code. Adding CrisperWhisper 2.0 turbo as the speculative draft then roughly doubles that again.

Real-time factor = audio seconds processed per wall-clock second: 40× means an hour of audio in 90 seconds. All runs use large-v2 weights on an H100, longform decoding with word timings off; they differ only in the runtime. OpenAI is regular Whisper in OpenAI's official implementation; HuggingFace and the two CTranslate2 rows all run CrisperWhisper 2.0, the last adding CrisperWhisper 2.0 turbo as a speculative draft. CrisperWhisper on CTranslate2 is ~3× faster than either baseline runtime; speculation lifts it to 5–6×.