Why most AI pilots never reach production — and what the successful ones do differently

The pilot worked. The demo impressed the steering committee. Eighteen months later, the system is still not in production, and the reasons are mundane: it was built against an extract rather than the live system; nobody designed how it would inherit the permissions of the source; there was no test set, so nobody could say whether the last change made it better or worse; and no one had agreed who would own it once the project team dispersed.

Where pilots actually stall

In our experience the failure points cluster into four categories, and the model is almost never one of them.

• Integration. The pilot read from a CSV. Production has to read from — and often write to — an ERP, a case system, a patient record. Those systems have transaction semantics, batch windows, audit obligations and support contracts that a pilot never encounters.
• Entitlements. A pilot runs as a superuser. A production system has to show each person only what they may see, which means either duplicating the access model (which drifts) or filtering at retrieval time against the source (which is harder but correct).
• Evaluation. Without a maintained test set and scoring rubric, quality is anecdote. Every change becomes a risk, so changes stop, and the system decays as the world moves on around it.
• Ownership. Someone must be accountable at three in the morning, must approve the next model version, must answer the auditor. Pilots do not have this person. Production requires them.

What the successful programmes do differently

They treat the production engineering as the project rather than as a phase after the project. They assess the data foundation before choosing a use case, because feasibility is decided there. They build the evaluation harness before the system, so quality is measured from the first version. They resolve ownership and the operating model before build starts. And they scope the first release narrowly enough that all of the above can actually be done.

None of this is glamorous. All of it is the difference between an AI system and a slide about one.