"Human in the loop" is the answer everyone gives when asked how an AI system is controlled, and it is usually the least examined part of the design. Implemented as a queue, it becomes a bottleneck that grows with volume until someone quietly widens the auto-approve threshold. Implemented as a policy, it scales — because most items never need a human at all.
The queue that eats the benefit
The default implementation routes every AI output to a person for confirmation. It feels safe and it is easy to explain to a regulator. It also fails predictably.
If a system processes ten thousand items a day and review takes ninety seconds, full review costs roughly 250 person-hours daily. Nobody staffs that, so one of two things happens. Either the queue grows until the system is abandoned, or reviewers start approving in bulk without reading — which is worse than no review, because it manufactures an audit trail of oversight that did not occur.
The second failure is the dangerous one. It looks compliant right up until someone samples the decisions.
Reframe: what decision is the human actually making?
Oversight is not one thing. Distinguishing the modes clarifies where a person adds value:
| Mode | Human role | Appropriate when |
|---|---|---|
| In the loop | Approves before the action takes effect | Irreversible, high-value, or legally significant |
| On the loop | Monitors in aggregate, intervenes on exception | High volume, reversible, well-measured |
| Over the loop | Sets policy and reviews outcomes periodically | Low individual impact, strong aggregate metrics |
Most systems need all three, applied to different slices of the same traffic. The mistake is choosing one for everything.
Routing by policy, not by default
A workable design routes on explicit, written conditions. Four families cover most cases:
- Consequence. Value above a threshold, irreversibility, regulated categories, or anything meeting the Article 22 test for decisions with legal or similarly significant effects.
- Confidence. Model uncertainty, weak retrieval support, or disagreement between an ensemble. Calibrate this against observed error rates — raw model confidence is often poorly calibrated and should be validated before it gates anything.
- Novelty. Input unlike anything in the evaluation set. New patterns deserve a look even when the model is confident, because confidence on out-of-distribution input is uninformative.
- Sampling. A random slice of auto-approved decisions, always. This is what tells you whether the thresholds are still right, and it is the first thing teams cut when under pressure.
The policy is a business artefact. It belongs in version control, it gets reviewed when it changes, and each condition should have a stated rationale. That document is also, conveniently, most of what an auditor wants to see.
Design the reviewer's surface as a product
Throughput is determined less by the model than by the review interface. Reviewing a decision well requires the input, the model's proposal, the evidence it relied on, and the specific reason it was routed — presented together.
What matters in practice:
- Show the reason for routing. "Low confidence" and "value above €50,000" call for different scrutiny.
- Show the evidence, not just the answer. Retrieved passages with the relevant part highlighted. Verifying a citation is far faster than reconstructing a judgement.
- Make disagreement cheap and structured. Capture why the reviewer overrode the model, in categories. This becomes your best training and evaluation data.
- Resist pre-filled defaults. A pre-selected "approve" turns review into clicking. Automation bias is well documented; the interface should work against it.
Close the loop
Review generates a labelled dataset for free. Most organisations discard it.
Feed overrides back into the evaluation set, so cases a human corrected become regression tests. Track the override rate by routing condition: if reviewers approve 99% of items routed by one rule, that rule is miscalibrated and is consuming attention for nothing. If they override 40% on another, the model has a real weakness there and the threshold should tighten.
This is what makes the system get cheaper over time instead of more expensive. Without it, review volume grows linearly with traffic forever.
Guard against the failure modes of oversight itself
- Automation bias. People under-scrutinise confident-sounding machine output. Counter with sampling, blind spot-checks, and measurement of reviewer accuracy — not just reviewer throughput.
- Rubber-stamping under load. Watch review time per item. A falling average is an early warning, well before quality metrics move.
- Threshold creep. Every widening of auto-approval should be a reviewed change with a stated justification, not an operational tweak.
- Accountability without authority. The EU AI Act expects overseers for high-risk systems to have the competence and standing to actually override. A reviewer who cannot realistically say no is not oversight.
Common questions
What review rate should we target?
There is no universal number — it follows from consequence and measured error rate. A useful discipline is to work backwards: decide the acceptable rate of unreviewed errors, then set thresholds so the expected volume of missed errors stays under it. That makes the trade-off explicit rather than emergent.
Does GDPR require a human in every decision?
No. Article 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects, and provides for a right to obtain human intervention in those cases. Most operational decisions do not meet that bar. Classify which of yours do rather than applying the strictest reading everywhere.
Can the reviewer be another model?
For triage and pre-filtering, yes, and it is often effective. It does not satisfy a human-oversight obligation where one applies, and a model grading a model shares failure modes. Use it to reduce what reaches a person, not to replace the person where the law expects one.
How do we start if everything is currently reviewed?
Instrument first. Measure the override rate on what is already being reviewed. The categories where humans essentially never disagree are your first candidates for automation, and you will have the evidence to justify it.
The short version
Stop treating oversight as a queue. Write a routing policy based on consequence, confidence, novelty, and sampling; version it like code. Build the review surface as a real product, because throughput lives there. Feed overrides back into evaluation so the system needs less review over time. Human-in-the-loop scales precisely when it stops being a step in every transaction and becomes a policy about which transactions deserve one.