Measuring verbatimness

One number, many hidden decisions

Every speech-to-text system has to make a choice it almost never states out loud: does it write down what was said, or what was meant? Most never decide, and leave the choice implicit, inheriting it from their training data and applying it inconsistently, keeping a filler here, dropping a stutter there, swallowing a laugh, silently repairing a false start.

WER cannot see any of this. It compares a hypothesis to one reference and counts substitutions, insertions, and deletions. But on spontaneous speech, the reference itself encodes an unstated policy. The two transcripts below describe the same audio, side by side. Ask which one a low WER is rewarding.

Both can be "correct." A system tuned for clean captions is penalized against the verbatim reference for every disfluency it correctly dropped; a verbatim system is penalized against the intended reference for every hesitation it faithfully kept. A single WER score can't tell a careful transcriber from a careless one; it only tells you how well a model matched whichever transcription policy the reference happened to use.

One thing the verbatim line does not drop is ordinary writing: it keeps normal capitalization, proper nouns, and punctuation: it is as grammatically correct a transcript as the speech allows, just with the disfluencies left in. The single deliberate exception is numbers and dates, which it spells out as spoken (March third at nine thirty) while the intended line normalizes them for reading (March 3rd at 9:30). That mismatch is a second, orthogonal source of noise: punish a model for writing "3rd" instead of "third" and you are measuring formatting, not recognition. To keep the benchmark's normalization simple and unambiguous, the evaluation sets deliberately contain no numeric or date material: every gold transcript across both languages is digit-free, so number formatting never confounds a score. Models that do emit digits are still mapped to spoken forms by a conservative normalizer, but the datasets are built so that path is never load-bearing.

The annotation convention

Before any of that can be measured, we have to pin down what the two transcripts actually are. Each follows a small, strict convention: the verbatim track is almost entirely "write what you hear," while the intended track is "write what they meant." Here are the two rule sets side by side:

The two tracks describe one and the same audio; the verbatim/intended example near the top of this article is exactly this convention applied. Models that emit a different spelling, such as a - instead of * for cut-offs or [laughs] instead of [laughter], are mapped onto this canonical set by a conservative, idempotent normalizer before any scoring happens.

Two transcripts, and the rest follows

With the convention fixed, the benchmark's input is just two transcripts of the same utterance: the verbatim one, written to the convention above, and the intended one, the clean, readable version of what the speaker meant, with disfluencies removed and numbers and dates normalized. The verbatim/intended pair at the top of this article is exactly such an input.

The verbatim transcript does carry real annotation: its fillers, sounds, and cut-offs have to be marked correctly under the convention: genuine labeling effort, not something the benchmark gets for free. What it avoids is a second layer on top of that. The intended transcript is, by construction, a subset of the verbatim one (produced only by deleting and collapsing tokens, never by inventing new words), so aligning the two reveals exactly which verbatim tokens were dropped. Those dropped tokens are the disfluencies, and their type can be read straight off their surface form: brackets mean a filler or a sound, a trailing marker means a cut-off, an unmatched repeat means a repetition.

That is the whole framework, and it is a compact one. Two transcripts in; every typed label and metric in this benchmark then falls out of the alignment between them: no separate per-disfluency tagging pass, and nothing to re-annotate when you add a new dataset or language.

Deriving gold labels from alignment

Here is the heart of the benchmark. The verbatim transcript already carries its surface markers (brackets for fillers and sounds, a cut-off marker for fragments), so labeling reduces to two steps: align, then read off the labels. The figure below plays both steps on a loop; hover any token to see why it got its tag.

The logic is a two-pass rule set. Pass one assigns each token by surface form and alignment, in priority order: a cut-off marker wins first, then bracketed fillers, then bracketed sound events, then any token that aligns to an intended word with a matching normalized form becomes Fluent. Anything left over is Other: the bucket for false starts, substitutions, and discourse markers like the "I mean" above. Pass two finds repetitions: walking backward from each kept fluent token, consecutive unmatched copies of the same word become Rep. The repeated "we" is caught exactly this way.

The alignment itself is a standard minimum-edit-distance alignment over normalized tokens, the same machinery behind WER, but used here to discover structure rather than to score it.

Scoring a model, tag-aware

Once the gold labels exist, a model's verbatim prediction is aligned against the gold verbatim transcript a second time. This alignment is deliberately tag-aware: matching a fluent token is preferred over matching a disfluent one, and deleting a disfluent token is cheaper than deleting a fluent one. A tiny epsilon nudges the edit-distance costs so that ties always break the linguistically sensible way.

Why it matters is clearest with repetitions. Consider a stutter "I I think…" where the model only writes "I" once:

The single predicted "I" is aligned to the Fluent gold "I", and the Rep token is treated as deleted, exactly the right read. A naïve aligner would have matched the prediction to the first "I" and called the keeper a deletion, scrambling both the repetition and fluency metrics; the epsilon is what tips the tie the right way. Tag-aware alignment is what keeps the typed scores honest.

Transcript-level accuracy (vWER, vCER, iWER, iCER) is computed separately from raw normalized text: the verbatim prediction against the verbatim reference, the intended prediction against the intended reference. The intended comparison is its own kind of test: it asks whether a model removes the right disfluencies while keeping all the real content intact.

The cut-off problem

Cut-offs are the one place where there often isn't a single right answer, not never, but often enough that the benchmark can't pretend otherwise. The trouble is acoustic. Imagine a speaker stumbling toward the word "bird": pi- pi- bird, or b- bir- bird, or be- beer- bird. A broken-off fragment is, by definition, only a sliver of a word, and those slivers are genuinely confusable: pi-, bi-, be- and beer- can sound nearly identical on their own. So even a perfect transcriber is left under-determined here: there is no acoustic fact that fixes exactly how much of the word came out, or which letters to commit to.

That doesn't make typed cut-offs worthless; quite the opposite. A system that emits a fragment as a cut-off (bir- rather than a confidently wrong full word) is telling you something true and useful: it heard a break, not a complete word. That signal is worth having. But because the exact spelling is underdetermined, it can't be the only thing we score: penalizing a model for writing bir- when the gold says b- would be measuring guesswork. That tension is exactly why the benchmark reports four cut-off metrics, from strict to lenient.

Two cases are worth separating, because they are easy to confuse. A wrong fragment at the right place (bir- for a gold b-) still aligns to the cut-off position, so marker, lenient, and recall all credit it; only strict, which compares stems, holds the spelling against it. A spurious marker at the wrong place (wor- on a word that was never broken off) is different: it is a clean false positive that costs precision on every F1, but it cannot touch recall, because recall only asks whether the genuine gold cut-offs were acknowledged. Hallucinating cut-offs hurts you, just not by pretending you missed a real one.

The leaderboard's headline Cutoff F1 is the lenient one: a gold cut-off counts as covered if the model places any aligned token there, while hallucinated markers still cost precision. It is the fairest single number, because some excellent systems preserve interrupted speech perfectly well without ever emitting a dedicated cut-off symbol. The stricter variants are still reported, so you can see whether a model genuinely marks fragments or merely happens to land a token nearby.

What gets reported

Rather than one opaque score, the benchmark emits typed metrics so you can see exactly how a model fails: which phenomena it silently drops, which it hallucinates, which it mislabels.

The two datasets

The benchmark ships with two open, verbatim-annotated speech datasets (English and German) together carrying over 7,500 labeled disfluencies (6,120 in English, 1,385 in German). They are the first of their kind released together for the two languages. Each sample carries audio plus the verbatim/intended transcript pair the evaluator needs.

Both are conversational speech datasets containing spontaneous speech rich in disfluencies: the regime where disfluencies actually occur, and where clean read-speech benchmarks conflate accuracy with annotation convention and become uninformative and potentially misleading. Here is the full typed inventory, derived by the same alignment described above:

What matters is that each dataset contains enough events of every type to draw statistically meaningful conclusions about how a system handles fillers, repetitions, cut-offs, and sounds. Any HuggingFace audio dataset exposing id, audio, verbatim_transcript, and intended_transcript drops straight into the same evaluator; the design is meant to grow to new languages as verbatim data becomes available.

The leaderboard

Sixteen systems, scored on the same audio. Tables are sorted by Disfluency F1, the headline metric. Switch languages with the toggle. Nyra Health models are highlighted. The gap between general-purpose ASR and verbatim-aware systems is stark: several strong commercial models score in the single digits because they silently clean speech before you ever see it.

How to read it

Two patterns dominate. First, vocal-sound detection is where most systems fall off a cliff: a column of 0.0% marks every model that simply never writes down a laugh or a cough. Second, the spread on verbatim WER tracks disfluency handling closely: models that drop fillers and repetitions don't just miss those events, they accumulate ordinary word errors against a verbatim reference. CrisperWhisper 2.0 leads both languages on the headline metric while keeping vWER at the bottom of the field; the full ten-language leaderboard lives in the overview post.

Run the benchmark

Every prediction, metric, and dataset is public. Reproduce the tables from cached predictions, or score your own system in three steps. Any HuggingFace audio dataset exposing id, audio, verbatim_transcript, and intended_transcript drops straight into the same evaluator.