Serving a TensorFlow model behind FastAPI with request batching

A model that’s fast in a notebook can still be slow in production if you serve it one request at a time. GPUs and ONNX Runtime want batches. The trick is getting batching without making any single caller wait too long.

A bounded micro-batch window

Collect requests that arrive within a few milliseconds, run them as one padded batch, then fan the results back out:

async def infer(text: str) -> Prediction:
    fut = loop.create_future()
    queue.append((text, fut))
    await schedule_flush()      # flush at ≤5ms OR when the batch is full
    return await fut

Why it works

The window is capped, so tail latency stays bounded; the batch is capped, so memory stays predictable. Under bursty load you get near-full utilization; under light load a request basically runs alone. On Prism this held batched inference under 50ms while keeping the model busy.