Switchboard

Problem

Voice AI breaks the instant latency creeps past a human beat. If the gap between a caller finishing a sentence and the agent starting to speak drifts over ~400ms, the conversation stops feeling like a conversation. Worse, callers interrupt — and a system that can’t be interrupted mid-sentence feels like a phone tree, not an agent. Switchboard exists to make that turn feel instant, for many tenants, on one pipeline.

Approach — architecture

The core is a full-duplex media pipeline. Audio streams in from PSTN/SIP, gets transcribed incrementally, drives the LLM as partial transcripts stabilize, and is spoken back as tokens arrive — all while a barge-in detector watches the inbound channel to cut playback the moment the caller talks over the agent.

Full-duplex: inbound audio never stops being watched, even while the agent speaks.

Everything is budgeted against a turn-latency envelope. Each hop publishes timing to Redis so a slow tenant config or a cold model can be spotted in the histogram rather than in an angry email.

What I built

A TypeScript/Node service holding the duplex socket, a streaming STT adapter, an LLM client that starts generating on partial transcripts, and a TTS stage that emits audio frames as they render. Session and turn state live in Redis; per-tenant routing and config live in Postgres.

// Barge-in: if inbound speech resumes while we're speaking, cancel playback.
stt.on('speechStart', () => {
  if (session.state === 'speaking') {
    tts.cancel();               // stop emitting frames immediately
    llm.abort(session.turnId);  // drop the in-flight generation
    session.transition('listening');
  }
});

Hard parts

• Full-duplex streaming — inbound and outbound audio move at once; nothing blocks.
• Barge-in — sub-frame cancellation of both TTS and the in-flight LLM turn.
• Per-tenant routing — voice, prompt, model and telephony config resolved per call.
• Latency budgeting — every hop timed and attributed so regressions are visible.

Outcome

p50 turn latency settled around 280ms end-to-end, with barge-in cancellation under a single audio frame. The pipeline runs multiple concurrent calls per node with per-tenant isolation.

Links

Repo and live demo are placeholders until this is public — see the contact page to ask for a walkthrough.