Longform transcription with conditional continuation

The seam is the hard part

Whisper's encoder consumes a fixed 30-second spectrogram. Longer audio must be cut into windows and the transcripts joined back together. Two joining mechanisms are established, and each makes specific failure cases at boundaries more likely.

Chunk and stitch

Slice the audio into fixed windows with a little overlap, transcribe each independently, then merge the transcripts by finding the longest common token sequence in the overlap; this is the approach in HuggingFace's transformers pipeline. It is simple and batches well, but the merge reconciles two transcripts produced independently, and it fails in both directions. When the windows render the overlap differently (a cold-started chunk may hear other words, or the same words with different casing and punctuation), no common run is found and words are duplicated or dropped. And when the overlap genuinely contains the same phrase twice, a stutter or a false start, the matcher cannot tell repetition from re-hearing: it aligns the two copies as one, and a copy that belongs in the transcript is swallowed. And each chunk is decoded with no knowledge of what preceded it, so it cannot use the words already spoken to disambiguate a name, keep a running number straight, or carry a sentence's grammar across the cut. Decoding that starts mid-sentence makes errors that the context of the preceding transcript could have prevented. The audio overlaps; the transcript does not.

The merge matches on token IDs, not meaning, so it fails in mirror images. Identical tokens from a genuine repeat look like one utterance heard twice and a copy is dropped; identical meaning under different tokens (casing, punctuation) looks like two utterances and both are kept. From our benchmark run: "because Because"; "how do we do this? How do we do this?"

Predict timestamps and seek

OpenAI's own longform method instead teaches Whisper to emit special timestamp tokens interleaved with the text, then advances the next window to the last timestamp. The seam is chosen by the model rather than a string match, but the choice has three costs. The decoder now spends steps emitting timestamp tokens, which slows generation. The hop varies from window to window: every window is 30 seconds, but where the next one begins is known only after the current one is decoded, so windows cannot be lined up and encoded as a batch. And when a predicted timestamp is slightly off, the next window is placed wrong: it can begin partway into a word, or cut away audio a word needs, so that word is transcribed from incomplete audio context. A timestamp mistake can easily become a transcription mistake at the boundary.

The model predicts that speech resumes at 27.60 s, but "board" begins at 27.30 s. Window 2 opens at the prediction, inside the word: it hears only …ard and writes ward. A fixed-stride boundary cuts words too, but there the fixed overlap guarantees the next window still contains the whole word, and the continuation drop recovers it. With seek, coverage itself depends on the prediction: after a small error, the word's full audio lies in no window at all.

Conditional continuation

Conditional continuation is best understood as a hybrid. It hands the stitching to the model, but gives it the previously transcribed text and guarantees the acoustic context around the seam: the grammar carries across the cut, and every word near it is transcribed from its complete audio. We keep a fixed 30-second window and advance it by a fixed k-second stride, so consecutive windows share a short overlap. Before decoding each new window we take the last n words already confirmed and hand them to the model, wrapped in two special tokens, <ctx> … <ectx>, placed right after the verbatim/intended mode tags. The model's task is then simply: given these words and this audio, produce the continuation beyond them. We find k = 27 s and n = 12 words work best across datasets, leaving a 3-second overlap.

Because the new window re-covers the final 3 seconds of the previous one, the tail of the context is normally present at the start of its audio (a window that ends in silence is the exception; the training section covers it). Conditioned on the context, the model emits only the words that follow it. There is no explicit alignment step: locating the continuation point is behaviour the model is trained for. The result is that the boundary is crossed with both kinds of context intact: the decoder continues a sentence it can read instead of one it must guess, and every word near the boundary has its complete audio inside the window.

The fixed stride also fixes the computation. Window positions do not depend on the model's output, so they are known before any decoding starts and the encoder can process all of a file's windows in a single batch, whereas a seek must finish decoding each window to learn where the next one begins. And since the handoff is carried by text, the decoder emits no timestamp tokens at all; word timings are recovered separately, from cross-attention.

Dropping the last words at the cut

A window has a hard right edge, and its last word may straddle it: only part of the word's audio is then inside the window, and the transcription of it is unreliable. So before committing a window we drop up to its last two words, under one condition: a word is dropped only if its start time (read from cross-attention, as in the timing deep dive) lies inside the overlap with margin to spare for timing error, which guarantees the next window contains the complete word and will re-transcribe it. A word that cannot be placed safely inside the overlap is committed and becomes part of the context instead. Nothing is lost or duplicated at the boundary.

Note the asymmetry with a timestamp seek: there, a timing error moves the window and can cost a word its audio. Here the window never moves (the stride is 27 seconds no matter what the timing says), and timing only decides whether a borderline word is committed now or re-transcribed by the next window.

Press play to watch a window slide, hand off its tail as context, and drop only the words the next window will reclaim.

Top: the audio and the sliding window; bottom: the sentence as it is predicted. Window 1's last word is clipped by the 30 s edge to bo-; because it starts inside the 3 s overlap, window 2 re-covers it, so we drop it. The last few words become the <ctx> … <ectx> context; the window slides forward and window 2 continues, transcribing the whole word, board, from full audio. Schematic: the real window is 30 s with a 12-word context.

A natural, trainable objective

Continuation works because we train for it, and training for it is easy: the task is ordinary next-token prediction, with no special data and no auxiliary loss. Any clip of speech with its transcript is already a continuation example: choose a random split point, place the words before it in the <ctx> … <ectx> prompt, and supervise the model to produce the words after it with cross-entropy. A single clip yields as many examples as there are places to split it, so the model learns to continue from real speech at no extra data cost. The model already predicts the next word; here it simply learns to keep predicting past a supplied prefix, so longform becomes a trained capability rather than a decode-time heuristic.

The same recipe extends to the imperfect contexts that occur at real seams: a word may have fallen just outside the window, or the overlap may not line up cleanly. So we mix three kinds of example, and the model learns to cope with each:

Because imperfect context is part of training, the model finds the right starting point when the context only partially matches the audio, and falls back to a clean from-scratch transcription when it doesn't match at all.

This is also why the 12-word context matters for accuracy and not just for stitching: handing the model a full clause of real preceding speech means every window is transcribed in context, the way the first 30 seconds are, instead of guessing at a sentence that started before the chunk began.

How it scores

We transcribed two longform English benchmarks end-to-end and scored whole-string word error rate against the reference. Meanwhile is 64 late-night-comedy monologues (44–89 s, every clip crosses at least one seam); the speech is very fast, so boundary errors accumulate more. TED-LIUM is 11 full recorded lectures. Conditional continuation has the lowest WER on both.

The baselines run their standard longform algorithm on strong whisper-large-v2 weights; the CrisperWhisper 2.0 rows use its own weights, a fine-tuned whisper-large-v2. Each system is compared as deployed, model and algorithm together. The two chunk-and-stitch rows isolate where the gain comes from: under the same stitching algorithm, CrisperWhisper 2.0 (8.01 / 4.77) scores about the same as base large-v2 (8.28 / 4.79), so the advantage of the top row is almost entirely attributable to the continuation algorithm rather than to better transcription weights.