Closing the verbatim data gap

The verbatim annotation gap

Speech transcripts are not all trying to do the same job. An intended transcript captures what the speaker meant to say. A verbatim transcript captures what the speaker actually produced.

That spoken-form layer is costly to annotate. It is not just transcription. Annotators have to mark every disfluency and vocal event, place each one correctly in time, and apply the same conventions consistently and accurately.

Those decisions are hard to standardize. Inter-annotator agreement is lower than for clean word transcription, review cycles are longer, and because these labels are so time-consuming to produce, the cost scales prohibitively.

The result is a severe imbalance. Clean or intended transcripts exist across many domains and languages in large quantities. Openly available verbatim-annotated data is scarce. In English, a few hundred hours of such verbatim-annotated data exist, with varying degrees of accuracy and annotation coverage. Outside English, there is often no meaningful verbatim-annotated speech data available at all. Either way, the scale is far below what would be needed to train better, more faithful systems.

STT and TTS fail differently when the layer is missing

The verbatim gap matters because speech models learn from the relationship between audio and text. If a corpus contains spontaneous speech but the transcript only contains intended content, the pair is incomplete: the audio contains events that the text does not match.

For speech-to-text, intended transcripts are valid targets. A model trained on intended labels can learn to produce clean intended transcripts. That is not wrong; it is a different transcript policy. But those labels cannot supervise verbatim STT. If fillers, repetitions, cut-offs, stutters, breaths, laughs, and vocal events are not labeled, the disfluency layer becomes something the model learns to ignore.

You can hear the gap directly. Each clip below shows how different providers handle the same input. Where available, we select the verbatim output style for every provider. The disfluencies and vocal events (cut-offs, fillers, laughs, lipsmacks) are exactly the spoken-form layer that intended-style systems drop. CrisperWhisper 2.0 recovers it; most others do not. For a principled, scored comparison across providers see our Nyra Verbatim Speech Benchmark.

For text-to-speech, the issue becomes a control and naturalness problem. If the audio contains disfluencies but the transcript does not, the model sees clean text paired with spontaneous, disfluent audio. The text does not explain why the speaker hesitated, repeated a word, laughed, breathed, or restarted a phrase at a particular moment. So the model may learn that these events occur in the data distribution, but not how to control them from text, and with less incentive to model them naturally and expressively. Disfluencies become unconditioned behavior: hallucinated when not wanted, omitted when wanted, or placed unpredictably.

Below are verbatim sentences fed directly to TTS systems. The prompt contains every disfluency explicitly (cut-offs, fillers, repetitions, vocal events), so a controllable system should reproduce them faithfully. Listen to how each provider handles the same input. The pattern tracks the STT results above: providers whose transcription keeps the verbatim layer also generate it best. Gradium and Cartesia, which drop disfluencies on the transcription side, render them poorly here too: flattened or skipped. ElevenLabs and xAI, which transcribe the spoken form more faithfully, also produce more natural disfluent speech and adhere more closely to the prompt, though even here, listening closely reveals instabilities and inaccuracies in how the output corresponds to the prompt.

So the verbatim transcription layer is not just a formatting choice. It is a foundational addition to the data layer to enable more faithful, natural, measurable, and steerable speech technology.

Upgrade the corpora we already have

The scarce asset is not the audio. Large spontaneous-speech corpora already exist: meetings, interviews, calls, podcasts, lectures, clinical conversations, broadcast speech, parliamentary proceedings, earnings calls, and other real-world recordings. Many contain exactly the speech behavior that modern models need to understand.

These corpora also often include valuable work that should not be thrown away: segmentation, speaker metadata, consent, domain coverage, quality control, human review, and often high quality intended transcripts produced by expert human annotators.

The intended transcript is often the most trusted part of the dataset. It may contain rare words (names, clinical terms, domain-specific spellings, or multilingual content) that an ASR model struggles to recover from audio alone.

Naive fully automatic relabeling throws away valuable and expensive work that has already been done and reintroduces errors that are hard to fix. The better path is to preserve it: let the given intended transcript remain the authority for what words were said, and let the audio supply the missing events around and between those words.

Verbatimize preserves content and inserts what was spoken

Verbatimize is a new trainable task: the model is supplied the audio together with the trusted intended transcript, and the target it learns to produce is the faithful verbatim transcript.

audio + trusted intended transcript faithful verbatim transcript

The key constraint is simple: the intended transcript must remain intact and in order. Verbatimize does not rewrite the supplied words. It does not reorder them. It does not replace a rare term with a more acoustically plausible guess. It copies the prompted content and inserts only the disfluencies and vocal events that are grounded in the audio.

Because the operation is insertion-only, Verbatimize fits naturally into hybrid labeling pipelines. A human annotator, expert model, or domain-specific process can provide the intended transcript. Verbatimize then adds the costly error-prone disfluency layer on top without disturbing the content annotations.

That is especially important for rare words, which are often exactly the words most likely to be mistranscribed from audio alone. Verbatimize is designed so those words are copied from the prompt, not rediscovered from sound.

A small training trick reinforces this behavior. During training, some longer content words are uppercased in both the prompt and the target, like the CLEAN in the card above. Casing is not audible. The only way for the model to reproduce it is to read the prompt. This encourages the model to rely on the supplied transcript for exact surface form, while using the audio to decide where fillers, repetitions, cut-offs, stutters, laughs, breaths, and other events belong.

Evidence that it works

The central question is whether Verbatimize really preserves the trusted transcript while adding the missing spoken-form layer. Rare words provide a strong test. If a word is rare, domain-specific, or absent from training data, and if it is hard to recover from acoustics alone, then preserving it in the output shows that the model is using the prompt as intended.

On the ICSI meeting corpus rare-word set, the evaluation used 1,843 manually verified rare-word occurrences across 1,342 samples, covering 1,591 unique word types.

Two metrics matter: rare-word recall measures how often those rare words are preserved, and content loss rate measures how often words from the intended transcript change in the output. Crucially, both settings use the same model weights: the same model builds the verbatim layer once from acoustics alone, and once with Verbatimize, conditioned on the intended transcript. Any difference therefore reflects the effect of the prompt, not a different model.

The contrast is the point. Audio-only transcription recovers only 6.8% of the rare words. Conditioning on the prompt lifts rare-word recall to 94.1%. With casing perturbation, it reaches 96.1%, while content loss falls from 9.4% to 1.3%.

The residual errors are mostly benign. For vWER, manual inspection shows the remaining differences are largely cut-off realizations: the same partial word written slightly differently (for example t- trr- transformer vs. d- tr- transformer), which are inherently ambiguous. The small content loss is similar, and partly irreducible: some of it comes from mistakes in the base annotations themselves. When the acoustic evidence strongly contradicts the prompt, either because a content word was never actually spoken or because it is mispronounced so severely that it cannot reasonably be matched to its spelling, the model still leans toward trusting the audio rather than blindly copying the prompt. That is a feature, not a bug: it keeps the output grounded in what was actually said.

That shows the desired behavior: the model uses the transcript as the content authority and the audio as evidence for insertions. Verbatimize does not trade content fidelity for disfluency recovery. It preserves the content layer and adds the spoken-form layer on top.

The gap is coherence, not speech

The bottleneck was never the speech or the transcripts; both already exist in abundance, with metadata, segmentation, and review already done. What is missing is the layer that makes a transcript faithful to the full waveform at scale, and that is exactly what Verbatimize adds: it preserves the words you trust and inserts the disfluencies, vocal events, and timing the audio contains.

This compounds in practice. CrisperWhisper 2.0 Pro, an evolution of the model from the paper, uses Verbatimize exactly as described above on the available multilingual spontaneous-speech corpora that already have intended transcripts, with beam search and filtering, to iteratively bootstrap high-quality verbatim datasets that train each next iteration. That loop both demonstrates the method's effectiveness and shows how CrisperWhisper 2.0 Pro can scale cost-effective verbatim annotation.

Upgrade existing corpora into coherent verbatim data

Preserve the transcript you trust. Recover the spontaneous events the audio contains. Build faithful speech datasets for STT, TTS, clinical analysis, and speech science.