Prompt regression testing: treat your LLM prompts like code

If a prompt change can break production, a prompt is code — and code gets tests. The goal isn’t to test the model, it’s to catch the day your “small wording tweak” quietly regresses intent detection.

Deterministic first, judged last

Most of what you care about is checkable without a model: schema validity, required fields, forbidden content. Only reach for an LLM-as-judge on genuinely fuzzy criteria, and give it a rubric.

@evalcase(fixture="refund_request")
async def test_refund(agent, judge):
    out = await agent.run(fixture.input)
    assert out.intent == "refund"                       # deterministic
    assert await judge.scores(out.reply, "empathetic") >= 4

Gate it in CI

Run the prompt × fixture matrix on every change and fail the build on regression. Now a bad prompt edit is a red check on a pull request — with a diff showing which case moved — not a support ticket next week. That’s what Anvil automates.