Shaving 400ms off voice-AI turn latency

The single number that decides whether a voice agent feels human is turn latency: the gap between the caller finishing and the agent starting to speak. Past ~400ms it reads as a bad phone tree. Here’s where that budget actually goes, and how to claw it back.

Don’t wait for the full sentence

The biggest win is refusing to serialize the pipeline. Start the LLM on stabilized partial transcripts, and start speaking on the first TTS tokens instead of the last:

stt.on('partial', (t) => llm.prime(t));     // warm the model early
llm.on('token', (tok) => tts.push(tok));     // speak as we generate
tts.on('frame', (f) => outbound.write(f));    // stream audio, don't buffer

Make interruption a first-class event

A pipeline you can’t interrupt feels dead. Watch the inbound channel while speaking and cancel both playback and the in-flight generation on speech onset. That one behavior does more for perceived latency than any single optimization — because the fastest turn is the one the caller doesn’t have to wait through.

Result on Switchboard: p50 around 280ms, barge-in under a frame.