This is Part 14 of a series walking through my book Voice and AI. In the previous chapter, neural models learned to turn text into rich acoustic representations. But a spectrogram is still silent — it's an instruction, not sound. The component that makes it audible is the vocoder.
Vocoder quality often decides whether synthetic speech sounds convincing or artificial. Many TTS improvements came not from better text understanding but from better waveform generation. A good vocoder disappears; a bad one ruins everything upstream.
What a Vocoder Does — and Why It's Hard
A vocoder converts an abstract acoustic representation (spectral envelopes, pitch, mel-frequency energy) into a time-domain waveform — turning parameters into air vibrations that sound like speech. It's difficult because humans are exquisitely sensitive to waveform errors: small mistakes produce buzzing, metallic ringing, or lost clarity. Classical vocoders assumed simplified source–filter models of speech — a periodic impulse train or noise as the source, a filter for the vocal tract — efficient but crude, losing fine detail and sounding muffled or robotic. That limitation defined the ceiling of classical TTS.
Why Phase Matters
Spectrograms usually store magnitude and discard phase. For recognition that's fine; for synthesis it's not. Phase governs how frequencies align in time, and getting it wrong smears and distorts the sound. Griffin–Lim, a classic algorithm, reconstructs a waveform from a magnitude spectrogram by iteratively estimating phase — no trained model required, handy for prototyping, but slow and often buzzy. Its real contribution was historical: it underlined just how much phase matters to perceived quality.
Neural Vocoders: From WaveNet to HiFi-GAN
Neural vocoders drop the simplified assumptions and learn the representation-to-waveform relationship straight from data, capturing structure classical models can't. WaveNet was the first widely known example, achieving unprecedented quality by modeling raw audio sample by sample — but slowly. Later models pursued efficiency through parallel generation, producing many samples at once to make real-time synthesis possible, trading some flexibility for speed at comparable perceptual quality. HiFi-GAN exemplifies the modern, deployable class: trained adversarially, a generator produces waveforms while a discriminator judges realism, and through that competition the generator learns high-quality audio — fast, lightweight, real-time on modest hardware.
What Chapter 14 Sets Up
Listeners don't analyze speech technically — they react emotionally, and a single artifact breaks immersion instantly while a good vocoder can mask imperfections elsewhere. That asymmetry makes vocoders disproportionately important: improving them often yields larger perceptual gains than improving upstream models. With the vocoder, text has finally become sound. But sounding human takes more than accurate waveforms — it takes expression.
Next up — Chapter 15: Prosody and Emotion. The final frontier of synthetic speech — what prosody and emotion are, why they resist formalization, and how systems learn not just to speak but to express. It's also the bridge to personal voices.