PINT: content is what remains
- 1.2%
- speaker probe accuracy on PINT tokens, down from 93.1% on HuBERT
- 27–30%
- lower LM perplexity than HuBERT and WavLM tokens, same model and data
- 56 b/s
- after dedup and BPE, approaching character-level text BPE at 48
PINT (parallel-invariant tokenization) is a speech tokenizer that keeps linguistic content and only content. It learns from parallel utterances: many voices saying the same words under different conditions. Whatever survives the comparison is content; everything else falls away.
Language models work on speech the same way they work on text: turn the signal into a sequence of discrete tokens, then predict the next one. For that to work well, the tokens have to behave like text tokens: the same content should produce the same sequence, every time.
Today's semantic speech tokens do not behave that way. They are built by clustering features from self-supervised encoders like HuBERT or WavLM, and those features remember far more than the words: who spoke, how they spoke, and what microphone they spoke into.
PINT is our fix. It is the first published piece of the representation line of our research program: compressing speech into representations where each attribute lives in its own place, starting with content.
The full write-up is coming soon!
The deep dive into PINT, the nuisance-leakage problem, the parallel-invariance training recipe, and the full results, is on its way.
Related research
CrisperWhisper 2.0
The most accurate verbatim speech recognition you can run in production: controllable, multilingual, and timed to the word.
Closing the verbatim data gap
How to upgrade existing spontaneous-speech corpora into faithful verbatim datasets while preserving the transcripts you already trust.
The nyra forced aligner
Phoneme-level forced alignment for real speech: robust to the spontaneous and imperfect, and explicit about the fillers and vocal sounds that break other aligners.