Prism

Problem

Switchboard produces a firehose of transcript fragments. To route, escalate and summarize calls in real time you need structured signal out of that stream — intent, sentiment, entities — fast enough to act on during the call, not in a nightly batch. A naive “one HTTP request per fragment” design melts under load and blows the latency budget.

Approach — architecture

Prism serves a fine-tuned transformer behind FastAPI, exported to ONNX for inference speed. Incoming fragments are coalesced by a dynamic batcher: requests that arrive within a few milliseconds of each other are run as one padded batch, which is where GPUs and ONNX Runtime actually earn their keep.

Micro-batching turns bursty single requests into efficient padded batches.

What I built

A Python service with a Keras/transformer model trained for the domain, exported via tf2onnx, and served through ONNX Runtime. A small async batcher sits in front; a Redis cache dedupes repeat fragments.

async def infer(text: str) -> Prediction:
    fut = loop.create_future()
    queue.append((text, fut))          # join the current micro-batch window
    await schedule_flush()             # flush at ≤5ms or when batch is full
    return await fut                   # resolved when the batch completes

Hard parts

• Model serving with dynamic batching — bounded-latency coalescing under bursty load.
• Streaming partial results — emit early predictions, refine as text stabilizes.
• Export-to-ONNX — parity between the Keras graph and the runtime, quantized for latency.

Outcome

Batched inference lands under 50ms at the p50 the pipeline needs, with intent classification around F1 ≈ 0.9x on the held-out set. The batcher keeps GPU utilization high without letting any single fragment wait too long.

Links

Model card, repo and demo are placeholders until this is public.