The number gets quoted constantly and rarely interrogated: most enterprise AI pilots never reach production. The interesting question is not whether the figure is exactly right. It is that the failures cluster — and they cluster around architecture decisions taken in the first fortnight, long before anyone notices the pilot is doomed.
Pilots do not usually fail because the model is bad. They fail because the pilot was never built to become a system. Five decisions separate the two.
Why the demo was never the hard part
A pilot optimises for one thing: showing that the capability exists. It runs on a curated dataset, in a notebook, with a human deciding which outputs to show. Production optimises for entirely different properties — predictable cost, bounded latency, auditable behaviour, graceful failure, and someone whose job it is to own it at 3am.
Nothing in a successful pilot demonstrates any of those. So the gap between demo and deployment is not a gap in model quality. It is the entire set of properties the pilot deliberately deferred.
Decision 1: define the evaluation before you build
Teams that ship decide, up front, what "good enough" means numerically and for whom. Teams that stall rely on a stakeholder's impression, which is unfalsifiable and therefore never satisfied.
Write down the metric, the threshold, and the dataset before the first prompt. Base it on real inputs, over-weighted toward awkward cases. Then run it automatically on every change, so quality becomes something you can regress-test rather than argue about.
The practical tell: if you cannot answer "how would we know this got worse?", you are not close to production regardless of how the demo looks.
Decision 2: put a contract on the data
Pilots run on an extract someone exported once. Production runs on live systems that change without warning — a field is renamed, a nullable column starts arriving null, an upstream team changes an enum.
Specify the interface: schema, semantics, freshness, and ownership. Validate at the boundary and fail loudly when the contract breaks, rather than letting malformed input flow silently into a model that will produce fluent nonsense from it.
This is unglamorous and it is where a large share of stalled projects actually die — not in a decision to cancel, but in months of unexplained quality drift nobody can attribute.
Decision 3: decide where the human sits, and why
"Human in the loop" is usually asserted rather than designed, and it collapses under volume. The system either routes everything to review — creating a queue that grows faster than the team — or routes nothing, and the oversight is theatre.
Make it a policy decision instead. Define the conditions that require review: confidence below a threshold, financial value above a limit, a regulated category, a novel pattern. Everything else proceeds automatically with an audit trail. Design the reviewer's interface as a real product surface, because throughput depends on it more than on the model.
Decision 4: bound cost and latency at design time
Pilots have no cost model. Production discovers one when a finance business partner asks why inference spend tripled.
Decide early: what does one transaction cost, and what is the ceiling? That single question drives model selection, context length, caching strategy, and whether retrieval or fine-tuning is the cheaper path. Retrofitting a cost constraint usually means re-architecting.
The same applies to latency. A three-second response is fine in a demo and unusable inside a call-centre workflow where an agent is waiting.
Decision 5: name the owner before you build
The most reliable predictor of whether a pilot survives is whether a named team owns it in production with a budget line, an on-call rotation, and a roadmap. Pilots run by innovation functions with no path to an owning team have a poor record, however good the technology.
Decide who runs it, who pays for it, and who is accountable for its behaviour — before the first sprint, not after the demo lands well.
What this looks like in practice
| Dimension | Pilot posture | Production posture |
|---|---|---|
| Quality | Reviewed by eye | Scored in CI against a committed baseline |
| Data | One-off extract | Contracted interface, validated at the boundary |
| Oversight | Someone checks the output | Policy-driven routing with audit trail |
| Cost | Unmeasured | Cost per transaction with a ceiling |
| Ownership | Project team | Named team, budget, on-call |
A sequencing that works
- Pick a decision, not a technology. Choose a recurring, high-volume decision with a measurable outcome. "Improve customer service with AI" is not a project; "reduce the time to resolve a billing dispute" is.
- Build the evaluation harness first. Before the feature. It forces the definition of success into the open while it is still cheap to change.
- Ship narrow, to real users, early. One workflow, one team, real traffic. A narrow production deployment teaches more in a fortnight than a broad pilot does in a quarter.
- Instrument, then widen. Expand scope only where the measurements support it.
Common questions
Is the 95% figure real?
Estimates vary widely and depend heavily on how "pilot" and "production" are defined; treat any single number with suspicion. The consistent finding across studies is directional — most enterprise AI experiments do not become operational systems, and the reasons are organisational and architectural far more often than they are about model capability.
Should we start with a build or buy decision?
Buy the commodity, build the differentiator. If the capability is generic — transcription, translation, standard document extraction — buying is almost always right. Build where the value comes from your own data, process, or regulatory position.
How long should a pilot run?
Weeks, not quarters. If it cannot demonstrate measurable value against a defined metric within a few weeks, the problem is usually the framing rather than the technology, and more time rarely fixes framing.
What is the most common single mistake?
Starting without a definition of success that a machine can check. Everything else — drift, cost surprises, stalled rollouts — becomes far harder to diagnose when nobody can say what "working" means numerically.
The short version
Pilots stall for architectural and organisational reasons, and those reasons are locked in early. Define evaluation before building. Contract the data. Make oversight a policy rather than a queue. Bound cost and latency at design time. Name the owner before the first sprint. None of it is about the model, which is exactly the point.