Every other part of your stack has a gate before production. Unit tests, integration tests, a staging soak, a canary. LLM features usually have none of that. They have a demo, a vibe check, and a deploy. Then a prompt changes, a model version rolls forward, and quality drifts without a single alarm firing.
The fix is not exotic. It is the same discipline you already apply everywhere else: define what "working" means, measure it automatically, and refuse to ship when the measurement regresses. That is an eval gate.
Why LLM features slip through the process
Conventional software fails loudly. A null pointer throws, a schema mismatch 500s, a failing assertion turns the pipeline red. LLM features fail quietly. The model returns a fluent, well-formatted, confidently wrong answer, and every layer of your infrastructure reports success.
Three properties make this worse:
- Non-determinism. The same input can produce different outputs. A single manual test proves almost nothing.
- Silent dependency drift. Providers update models behind a stable-looking endpoint. Your code did not change; your behaviour did.
- Diffuse failure. Quality degrades by degrees rather than breaking outright, so it shows up in churn and support tickets long before it shows up in monitoring.
None of this means LLM features are unshippable. It means the gate has to measure behaviour statistically rather than assert on a single result.
What an eval gate actually is
An eval gate is a CI job that runs a fixed dataset through the feature, scores the outputs, and compares the score against a committed baseline. If the score drops beyond a defined tolerance, the build fails. It is a test suite whose assertions are thresholds instead of equalities.
Three parts, all boring by design:
1. A dataset that reflects reality
Start with fifty to two hundred real cases, not synthetic ones. Pull them from production logs, support tickets, and the queries your users actually type. Deliberately over-sample the awkward cases: ambiguous phrasing, missing context, adversarial input, the long tail your demo never covered.
Version this dataset in git alongside the code. When it changes, that change gets reviewed like any other. A dataset that drifts silently is as dangerous as a model that does.
2. Scorers that match the failure you care about
Pick the cheapest scorer that catches the failure mode, and layer upward only when you must:
- Deterministic checks — valid JSON, required fields present, values inside an allowed set, no PII in the output, latency and token budget within bounds. Fast, free, and they catch a surprising share of real regressions.
- Reference-based scoring — exact match, F1, or embedding similarity against a known-good answer. Works well for extraction, classification, and retrieval.
- Model-graded scoring — a second model judges the output against a written rubric. Necessary for open-ended generation, but it is itself a model with its own drift. Pin the grader version, and periodically check the grader against human labels.
Weight these by business impact. A malformed JSON payload that breaks a downstream system matters more than a slightly clumsy sentence, and your aggregate score should say so.
3. A threshold, and the nerve to enforce it
Commit a baseline score. Fail the build when a change drops it by more than your tolerance — a couple of points absolute is a reasonable starting point. Run the suite several times to establish natural variance before you set that tolerance, or you will spend your first month chasing noise.
The gate only works if it can actually block a merge. A gate everyone routinely overrides is documentation, not a control.
Where the gates sit in the pipeline
| Stage | What runs | Budget |
|---|---|---|
| Pre-commit | Deterministic checks on a 20-case smoke set | Seconds |
| Pull request | Full dataset, all scorers, diff against baseline | Minutes |
| Pre-release | Adversarial and safety suites, cost and latency profile | Tens of minutes |
| Production | Sampled online scoring, drift alerting | Continuous |
The last row is the one teams skip, and it is the one that catches provider-side model updates. An offline suite tells you your change was safe. Only online scoring tells you the world changed underneath you.
Treat prompts and models as versioned dependencies
A prompt is code. It belongs in version control, it gets reviewed, and it ships through the same pipeline. Two rules make this concrete:
- Pin model versions explicitly. Never point production at a floating alias. Upgrade deliberately, behind the gate, as its own reviewable change.
- Record the full context of every eval run — prompt hash, model version, dataset version, scores. When quality moves, you want to answer "what changed" from an artifact, not from memory.
What this costs, and what it returns
Standing up a first gate is typically one to two engineer-weeks: assembling the dataset is most of it, and the CI wiring is straightforward. Running it costs inference on a few hundred cases per pipeline run — real money, but small against a single quality incident that reaches customers.
The return is not just fewer regressions. It is that the team can move faster, because changing a prompt stops being an act of faith. Refactoring, model migration, and cost optimisation all become tractable once you can measure whether they hurt.
A pragmatic first month
- Week one. Collect fifty real cases. Write down what a good answer looks like for each. This is the hard part and it cannot be skipped.
- Week two. Add deterministic scorers only — structure, schema, safety, latency. Run in CI as a report, not yet a gate.
- Week three. Establish variance across repeated runs. Set a baseline and a tolerance you can defend.
- Week four. Turn on blocking. Add model-graded scoring for the open-ended cases. Start sampling production traffic.
Common questions
How many test cases do I need?
Fifty is enough to catch gross regressions. Two hundred gives reasonable confidence on a narrow feature. Breadth of failure modes matters far more than raw count — twenty well-chosen adversarial cases beat two hundred variations of the happy path.
Is a model grading another model trustworthy?
Well enough, with guardrails: use a written rubric rather than a vague "is this good", pin the grader version, and validate the grader against human labels on a sample every so often. Treat grader agreement with humans as a metric you monitor, not an assumption.
Won't running evals in CI be expensive?
Tier them. Deterministic checks on every commit cost nothing. Reserve model-graded scoring for pull requests and releases. Most teams find the bill is a rounding error next to the engineering time saved on a single avoided incident.
How does this interact with EU AI Act obligations?
Documented, repeatable testing with recorded results is close to what the Act expects for higher-risk systems. If you are building the gate anyway, retaining the run artifacts turns an engineering practice into audit evidence at almost no extra cost.
The short version
LLM features are not exempt from engineering discipline; they need a different assertion. Define the dataset, score it automatically, commit a threshold, and let CI block regressions. Pin your models, version your prompts, and sample production so provider drift cannot surprise you. That is the whole idea, and it is what separates a feature that survives contact with users from a demo that did not.