How we unlocked multilingual style-controlled ASR at scale

Verbatim or intended: one clip, two truths

Real speech is full of structure that clean text throws away. People hesitate, restart, repeat themselves, abandon a word halfway, and laugh mid-sentence. There are two equally legitimate ways to write that down, and they are not the same transcript:

• Verbatim keeps what was actually said: filled pauses ([uh], [um]), repetitions, self-corrections, cut-off word fragments, and vocal sounds like [laughter]. It’s still properly cased and punctuated (as grammatically clean as the audio allows), with numbers and dates left as they were spoken (March third, nine thirty).
• Intended keeps only the fluent content the speaker meant to convey, the readable sentence underneath, with numbers and dates normalized to their written form (March 3rd, 9:30).

That choice is not cosmetic. Intended transcripts are what you want for readability and downstream text processing. Verbatim transcripts are the signal itself in clinical and speech-science work: fillers, repetitions, and stutters are diagnostic markers for aphasia, stuttering, and cognitive decline, and they’re exactly what expressive text-to-speech needs to sound human rather than read-aloud. Neither is universally “the correct one”; which you want depends entirely on your use case.

The cost of never deciding

When a model is trained on a mix of verbatim and intended transcripts with no signal telling it which is which, transcription style becomes an uncontrolled latent variable. The model quietly learns competing strategies and picks between them randomly, with no user-facing control. That single unresolved choice cascades into three measurable failures.

Decoding becomes unstable. On disfluent speech, base Whisper’s five beam hypotheses for the same clip disagree, because the model is genuinely torn between writing the filler and dropping it. Add an explicit verbatim tag and that disagreement drops substantially. You can see the indecision directly: here are the beams the model proposes for one short clip.

Evaluation gets confounded. Verbatim and intended references of the same audio share 100% of their content words, yet word error rate can fluctuate widely purely based on the transcription style of the references. On the AMI benchmark, roughly 60% of a model’s reported WER is style mismatch, not recognition error. WER literally cannot tell you whether a model misheard or just made a different stylistic call.

To see this directly, we trained Whisper on data mixtures ranging from 0% to 100% verbatim-transcribed (versus intended-transcribed) data, and measured disfluency F1 at each ratio, both with and without mode tags. Each point on the x-axis is a separate training run at that verbatim share; the y-axis is how reliably the resulting model writes down the disfluencies that were actually spoken. Blue lines are models trained with mode tags, grey without; solid lines are English, dashed lines are German, which never saw additional training data in any of the finetunes:

Without tags (grey), the model only emits disfluencies in proportion to how much verbatim data it saw: the output is dictated by the data mix, with no knob for the user. With tags (blue), F1 stays high at every mixture, rising only slightly as the verbatim share grows. Strikingly, German (dashed) is never trained for verbatim yet tracks English (solid) almost exactly in both regimes; the controllable behaviour translates across languages on its own.

Whisper already knew. It just couldn’t be asked.

Here is the observation the whole method rests on. Whisper was trained on 680,000 hours of heterogeneously annotated audio. Buried in that data were plenty of verbatim transcripts. So the model already built internal representations of fillers, repetitions and cut-offs, it simply had no reliable way to be told to emit them. The capability was latent. What was missing was controlled activation.

We can prove this almost surgically, with three intervention steps of increasing strength: first nothing at all, just checking how verbatim the base Whisper model already transcribes; then training a handful of new token embeddings prepended to the decoder prompt; and finally a minimal finetune of the decoder plus those embeddings, to stabilize and sharpen the behaviour. Watch how much of the German capability is already unlocked by the smallest of them:

That middle bar is the heart of it. Freeze the entire pretrained network, add a few new token embeddings, train only those, and German disfluency F1 leaps from 10% to 79%, with zero updates to anything the model already learned. The English story is the same shape: 12%, then 53%, then 91%. The skill is progressively unlocked, not acquired from scratch.

Mode tags: a switch in the decoder prompt

Whisper’s decoder already starts every transcription with a little prompt of control tokens, things like the language and a “transcribe, no timestamps” instruction. We add our policy switch into exactly that slot: a small set of discrete mode-tag tokens prepended to the decoder input, before the model writes anything.

Three design decisions make this switch reliable rather than a fragile prompt trick.

1 · Paired supervision teaches a clean division of labor

For every training clip we have both transcripts. Each step, we randomly pick a mode, prepend its tags, and supervise with the matching reference. So the very same audio appears under both policies across training, distinguished by nothing but the tag. That forces a clean separation: the encoder learns to represent everything audible in the clip (words, disfluencies, sounds), while the mode tag learns only the emission policy, which of that content to actually write. The tag is a router, not a rewriter: watch the greyed-out tokens in the card above; switching tags never changes what the model heard, only what it writes.

2 · Several tokens per mode, not one

We use five tokens per mode rather than a single flag: [verbatim_1..3] plus [sound_1,2] for verbatim, [intended_1..5] for intended. Multiple prefix tokens give a stronger, more redundant conditioning signal, and crucially they enable compositional control, you can ask for disfluencies without sound events, or sounds without committing to full verbatim, by lighting up only the relevant tags.

3 · Coverage-aware partitioning so honest gaps don’t poison training

Real corpora are annotated inconsistently. Some mark fillers and repetitions but never note a cough; others have injected sound events but no reliable disfluency labels. Train all of it under one undifferentiated “verbatim” tag and you get contradictory gradients: the model is rewarded for emitting a laugh in one clip and punished for emitting the identical laugh in another, purely because one annotator wrote it down and the other didn’t.

So we split the verbatim tags by what each dataset actually covers. Disfluency tags and sound tags are separate, and every sample is trained only with the tags its annotations support: exactly the tag patterns the card above cycles through. A corpus with fillers and repetitions but no sound labels lights the disfluency tags alone; clean speech with injected sound events lights the sound tags alone; fully annotated data lights everything.

This hands the model a learnable signal that distinguishes incomplete annotation from absent phenomenon. A missing cough label becomes “this corpus didn’t annotate sounds,” not “there was no cough,” so honest gaps in the data stop fighting each other.

Two stages that protect what Whisper already knows

The reason this stays multilingual is the training schedule. Whisper’s multilingual knowledge is precious and easy to wreck with a heavy English-only finetune. So we move in two careful stages, each protecting the last.

Why one English switch works in German

This is where the design pays off. We did not include any German verbatim training data, German verbatim corpora barely exist. We train the verbatim policy on English alone. And yet the model produces faithful German verbatim transcripts, zero-shot, at 94% disfluency F1.

The reason follows directly from the separation of roles. The encoder’s job is to encode and preserve what’s audible in a structured way, and Whisper already learned to encode these phenomena, in German as everywhere else, during its large-scale multilingual pretraining on heterogeneous data and transcript conventions. The mode tag’s job, deciding which content to emit, is a language-agnostic policy. So when the tag learns “verbatim means keep the hesitations” from English audio, it’s learning a rule about emission, not about English. Point it at German encoder features and the same rule applies.

This is the payoff of keeping content and policy separate. The tag never has to learn anything about German; it only ever learns which of the encoder’s content to emit. The encoder, meanwhile, already understands German from Whisper’s pretraining. Bolt the two together and an English-trained verbatim policy reads German features just fine, which is why the same switch generalizes across most of the languages Whisper supports without a single verbatim example in them.

Adding 10,000 German verbatim samples nudges F1 from 93.8% to 95.0%, a marginal gain. The heavy lifting was already done by the tag and the multilingual backbone underneath it. The data is a polish, not the foundation.

Activation, not acquisition

The thread running through every result is the same: a large pretrained ASR model often already contains the capability you want, and behaves inconsistently only because it has no interface to express it reliably. The right intervention is frequently a control signal and a small amount of high-quality data displaying the correct behaviour, rather than simply more of the same.

We are likely not looking at a one-off, either. A recent paper, Prompting Whisper for Joint Speech Transcription and Diarization, suggests Whisper already possesses representations relevant for speaker diarization as well, but again lacks an interface along with more explicit annotations in training to express them properly.