Where AI automation breaks (and how to catch it first)

Abstract. Most AI automation that fails in production doesn't fail loudly. It produces plausible wrong output until someone notices a metric has slipped. This piece draws on research from Google engineers on the hidden costs of machine learning systems and on production readiness testing, and applies it to automation built on large language models. Three failure patterns stand out: no evaluation set, failures that are silent rather than visible, and no ownership after launch.

Background

Sculley and colleagues' 2015 paper “Hidden Technical Debt in Machine Learning Systems” made a point that has held up well: in a real ML system, the model code is a small part of the whole. Most of the work and most of the risk sit in data dependencies, configuration, monitoring and the surrounding infrastructure. They described how changing one input can change everything downstream, and how systems quietly degrade as the world they model changes. LLM-based automation inherits all of this, and adds the problem that its wrong outputs read just as fluently as its right ones.

Failure pattern 1: no evaluation set

Breck and colleagues' “ML Test Score” rubric treats testing the model against known cases, before every change, as basic production readiness. For LLM automation that means a set of real inputs with expected outputs, run whenever a prompt, model or data source changes. It is the single most common thing missing from builds that need rescuing.

Failure pattern 2: silent failure

A workflow that throws an error gets fixed quickly. One that returns a confident wrong answer can run for weeks. Both papers stress monitoring as part of the system, not an afterthought: log inputs and outputs, flag low-confidence results, track a small number of quality signals over time, and make spot-checks easy to do.

Failure pattern 3: nobody owns it after launch

Sculley's team described ML systems as carrying ongoing maintenance costs that are easy to ignore at launch. Inputs drift, model providers update or retire models, connected tools change their APIs, and new edge cases appear. An automation without a named owner and a regular review is likely to degrade without anyone noticing.

Scroll sideways to see the whole diagram.

Discussion

None of the fixes are novel. Evaluation sets, logging, review gates and a regular check on what has changed are standard practice in mature ML teams. What's new is how easy it has become to ship an LLM automation without any of them, because the demo works on day one.

Limitations

The cited papers describe large-scale ML systems at Google, which are more complex than most business automations. The principles transfer; the amount of process should scale down with the size and risk of the system.

Conclusion

Automation fails in the gap between demo data and real data, and in the months after launch when nobody is watching. Building for that gap is unglamorous, and it's what keeps a system working a year later.

Where X-GenAI fits

Our evaluation and monitoring service builds the eval set and drift checks for existing automations, whoever built them.

References

Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28.

Breck, E., Cai, S., Nielsen, E., Salib, M. and Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE International Conference on Big Data.