Anvil

Problem

Prompts and agent flows are code — but they’re usually shipped like folklore. Change a system prompt, cross your fingers, notice the regression in production. Anvil treats prompt and agent behavior as something you can test, diff and gate in CI, the same as any other unit.

Approach — architecture

Fixtures define inputs and expectations. Each case runs through deterministic checks (schema, must-contain, must-not-contain) and, where judgment is needed, an LLM-as-judge with a rubric. Results roll up into an HTML report and a pass/fail exit code the CI can block on.

Deterministic checks first; the model judge only for genuinely fuzzy criteria.

What I built

An async, pytest-style Python harness. Prompts are versioned; an eval matrix runs each prompt version against each fixture so you can see exactly which change moved which metric.

@evalcase(fixture="refund_request")
async def test_refund_intent(agent, judge):
    out = await agent.run(fixture.input)
    assert out.intent == "refund"                      # deterministic
    assert await judge.scores(out.reply, rubric="empathetic") >= 4  # judged

Hard parts

• Prompt versioning — track which prompt produced which result.
• Eval matrix — prompt × fixture, with diffs between runs.
• Deterministic scoring — keep as much as possible out of the model’s hands.
• CI integration — a clean gate that fails the build on regression.

Outcome

Prompt changes now come with evidence. A regression shows up as a red check in a pull request, not a support ticket a week later.

Links

Repo and sample report are placeholders until this is public.