ONNX in production: from Keras notebook to sub-50ms inference

A model that trains well in Keras isn’t a production model yet. Getting to sub-50ms inference is mostly unglamorous plumbing: export cleanly, prove parity, quantize, and serve on a runtime built for it.

Export and verify

tf2onnx gets you an ONNX graph. The step people skip is verifying numeric parity between the original graph and the exported one before trusting it:

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("model.onnx")
onnx_out = sess.run(None, {"input": x})[0]
np.testing.assert_allclose(keras_out, onnx_out, rtol=1e-3, atol=1e-3)

Then make it fast

ONNX Runtime with the right execution provider, dynamic quantization where the accuracy budget allows, and batching in front (see the FastAPI batching post). That combination is what took Prism’s model from “fine in a notebook” to under 50ms on the serving path.