Turning emergent cross-attention into a precise aligner

Verbatim transcription is what makes timing well-defined

Timing words means establishing a mapping between the transcript and the exact time each word was spoken in the audio. If the transcript omits things that were spoken, certain sections of the audio are unaccounted for, and the mapping becomes undefined. This makes timing fundamentally ill-defined for non-verbatim transcripts.

Collapse a repetition ("I, I think") to a single "I", and it is genuinely unclear which one should be timed: the first, the second, or should that one "I" occupy the timing of both and possibly the pause between them? Drop a filler, and the stretch of audio it occupied has no token to belong to: it bleeds into the previous word, the next one, or a silence. Every choice is an arbitrary convention. A verbatim transcript avoids all of this: every sound (filler, repetition, cut-off, laugh) has its own tokens, so every span of audio maps naturally to exactly where it belongs.

That is why we treat verbatimness as a precursor to well-defined timing. CrisperWhisper 2.0 is trained on parallel verbatim/intended pairs and switches between them on demand, and we read timing in verbatim mode, where "when did this word happen?" has a natural, well-defined answer.

What the decoder is doing as it predicts

With a stable, verbatim output policy, let's look at what the decoder is actually doing as it predicts the next token. Whisper sees audio as a strip of encoder frames, one every 20 ms. The decoder produces the transcript one token at a time, and to predict each token it consults the encoder through cross-attention: a distribution over those frames that answers "which slices of audio am I looking at right now?"

As each token is emitted, its cross-attention picks out the encoder frames where that word lives, and the rows stack up into a [tokens × frames] matrix. In a few heads the bright band already tracks the diagonal of time; that is the emergent alignment. Toggle to the untrained heads below and the same bands turn diffuse and, for the disfluent tokens, smear onto their neighbours.

Supervising the heads that already align

The key decision is where to push. We don't add an alignment module or retrain the decoder wholesale. We find the ten cross-attention heads whose untrained patterns already correlate best with real word alignments, and train only those. Because those heads already encode alignment-relevant information, supervising them is a gentle correction rather than a new skill bolted on, which both minimizes catastrophic forgetting of what the model can already do and directly sharpens the timing we care about.

We generate the training targets using the Montreal Forced Aligner over English speech: for each token we know which word it belongs to and where that word sits in the audio. From that we build a ground-truth target over the encoder frames: a binary indicator vector that is one on the frames the word occupies and zero everywhere else. The objective is simply that the selected heads' averaged cross-attention concentrate on that word's frames and stay off everything else.

We enforce this with a cosine-similarity loss applied at the word level. Every token belonging to a word shares the same ground-truth vector, so we compare that one-hot-style target against the predicted attention for the word: the averaged cross-attention of the supervised heads, taken over all of the word's tokens. Maximizing the cosine similarity between the predicted word-level attention and the binary target pulls the attention mass onto the word's frames and off everything else.

Why supervise the average, not each head

This is the part that reveals what is really going on. As the decoder commits to a word (say dog), it grows more and more certain of it through the deeper decoder layers, and at each step every head emits its own query that goes looking for the matching encoder frames. Those queries specialise: one effectively asks is there a "d" sound here?, another tunes to the "o", another to the closing "g". No single query captures the whole word, but together they cover its full acoustic realisation, and in doing so they locate exactly where in the audio that word was spoken.

Supervising the average of the ten heads is what allows that division of labour: the heads only have to be right together, so each is free to specialise. Force every head to match the target on its own and you forbid the specialisation: every head is dragged toward the same blurry compromise, and the collective loses exactly the acoustic sharpness that made the alignment good. Select a token below and compare the individual heads against their average.

From cross-attention to word times

CrisperWhisper 2.0 never emits Whisper's timestamp tokens. It reads timing straight from the supervised heads, in a few intuitive steps.

Collect the attention. As the model decodes, every token leaves one row of cross-attention over the encoder frames. Stacked, they form the [tokens × frames] matrix from the animation above: the raw material for timing.

Group tokens into words. Sub-word pieces are stitched back into words at the spaces, and a word made of several pieces simply owns all of their rows.

Pool and sharpen. The cross-attention rows belonging to a word are first added into a single distribution over frames, which is then sharpened so its peak is decisive: one timing signal per word.

Leaving room for silence and non-speech

Words alone can't explain the gaps between them, so the Viterbi also has a virtual blank state for silence and non-speech, non-vocal sound events: pauses, background noise, ambient sounds, anything that isn't a spoken word. To score it we lean on the mel energy as a rudimentary VAD: where the mel energy is low the frame is probably not speech, so the blank should be cheap. The intuition pairs nicely with the attention: because the supervised cross-attention is always firmly focused on the word being spoken, a frame should only fall to the blank where no word is strongly attending to it. So at each frame the path weighs the word emission (from attention) against a blank emission (from the mel energy), and the blank should win only where genuine speech focus is absent.

The catch is scale. Mel energy on its own is a poor discriminator: it isn't that speech is necessarily loud and everything else quiet; plenty of non-speech sound events carry energy too (background noise, music, ambient sound), while real speech can be modest. So we have to scale the two signals against each other: the cross-attention's speech focus must reliably cross above the energy-based blank wherever a word is actually being spoken, and the blank should only rise above it in true silence and non-speech. We reshape the mel energy into a blank probability that behaves that way: raising that blank probability to a power (blank_gamma=3) collapses it toward zero except where the mel energy is very low, and a fixed log-space penalty (blank_penalty=3) holds it under the attention until the non-speech evidence is strong. This pair of knobs is what worked well for us across our tests in various conditions across speakers and recordings without per-utterance tuning. Drag the controls to watch words shrink under a raw blank and recover under a shaped one.

Walking the path

Together, the word distributions and the blank give every state a per-frame cost: low where that state explains the frame well, high where it doesn't. A monotonic Viterbi search then finds the lowest-cost path through this state-by-frame grid, along the alternating sequence blank · word · blank · word · …: at each frame the path may stay in its current state or advance to the next, never going back. The minimal-cost path is the alignment: the frames a word claims become its start and end, and at 20 ms per frame that's seconds.

Because the alternating structure forces the Viterbi to assign some duration to the blank between every pair of words, we split these artefact pauses evenly between the two neighbours; empirically a 100 ms threshold works best across datasets.

How we measure timing

Every system is scored against ground-truth word timings on two held-out test sets, TIMIT and Buckeye, whose word boundaries were annotated by human experts. Since the transcription models produce their own words, we first align each system's words to the reference transcript by text and only score timing on the words both agree on, so timing accuracy is judged independently of transcription mistakes. On those matched words we report three numbers:

MAE: mean absolute boundary error. The average distance, in milliseconds, between a predicted word boundary and the true one, over every start and end. This is the headline number; lower is better.

mIoU: mean intersection-over-union. How much each predicted word interval overlaps the true one (shared time divided by combined time), averaged over words. A single 0–100% score for how well the spans line up; higher is better.

F1 @ collar: the share of words whose start and end both land within a tolerance of the truth, reported at 50, 100 and 200 ms collars. It reads as "what fraction of words are placed to within ±50 ms?"; higher is better.

How it scores against other speech transcription systems

We compared CrisperWhisper 2.0 against other leading speech-to-text providers, open and closed source alike, scoring every system on the same datasets. CrisperWhisper posts the best boundary error on both clean read speech (TIMIT) and conversational speech (Buckeye).

Types: forced aligner (given the words); transcription · local; transcription · hosted API. Word counts: TIMIT ≈ 12.6k, Buckeye ≈ 81k.