Is this task worth automating? What the field studies say

Abstract. “Can AI do this task?” is now almost always answered yes, which makes it a poor way to choose what to automate. This piece reviews three widely cited studies of generative AI in real work: a field study of customer support agents, a controlled experiment with management consultants, and an exposure analysis of the US labour market. Together they suggest that AI helps most on repeatable tasks with checkable outputs, and can make results worse on tasks that look similar but sit outside what the model does well. We turn those findings into five practical screening questions.

Background

Most discussion of AI and work happens at the level of jobs. Researchers have moved to the level of tasks, because a single job is a bundle of very different activities. Eloundou and colleagues scored every task in the US O*NET occupational database for exposure to large language models. They estimated that around 80% of US workers have at least 10% of their tasks exposed, and about 19% have at least half their tasks exposed. Exposure is not the same as benefit, though. It says a model could touch the task, not that it should.

Evidence from the field

Customer support. Brynjolfsson, Li and Raymond studied more than 5,000 support agents at one company as an AI assistant was rolled out in stages. In the working paper version, agents with access resolved 14% more issues per hour on average, with gains of around 34% for newer and less experienced agents and little change for the most experienced. The published version in the Quarterly Journal of Economics reports a 15% average gain. The work was high volume, repetitive in kind and easy to score, which is exactly where you'd expect AI to do well.

Knowledge work. Dell'Acqua and colleagues ran a pre-registered experiment with 758 Boston Consulting Group consultants using GPT-4. On tasks inside what they called the “jagged frontier” of AI capability, consultants with AI completed 12.2% more tasks, worked 25.1% faster and produced work rated over 40% higher in quality. On a task deliberately chosen to sit just outside that frontier, consultants with AI were 19 percentage points less likely to reach the correct answer than those working without it. The tasks looked similar from the outside. The model's competence did not.

Discussion

Read together, the studies point to the same conditions for a good automation candidate. The task happens often enough that the gain adds up. The inputs are the same kind of thing each time. And, most importantly, someone can tell a right output from a wrong one, because that's the only defence against the outside-the-frontier failure Dell'Acqua's team measured. A few practical constraints sit on top of that: what a mistake costs, and whether the data is somewhere a tool can reach.

That gives five screening questions:

• Volume. Does it take at least two hours a week across everyone who does it? Below that, building and maintaining an automation tends to cost more time than it saves.
• Repeatability. Are the inputs the same kind of thing each time, even if the details vary?
• Checkability. Could someone tell a right output from a wrong one in a few seconds?
• Cost of error. Do you know what a wrong output costs, and who catches it before it matters?
• Data access. Do the inputs already live in email, a shared drive, a CRM or a spreadsheet?

A weekly report built by copying numbers from three dashboards into slides passes all five. Negotiating terms with a new supplier passes one at best: it's rare, every case differs, and a good outcome is a judgment call.

Limitations

Each study covers one setting, one period and, in two cases, one company or one model generation. The support study measured throughput, not long-run quality. The consulting experiment used tasks designed by researchers. The exposure estimates come partly from model-generated ratings. None of this tells you the exact payoff for a particular task. It does tell you which questions to ask first. The two-hour threshold is our own rule of thumb, not a finding from these papers.

Conclusion

The evidence supports a narrow, practical view: automate the repetitive, checkable middle of the week, and be most careful with tasks that look automatable but where nobody can quickly verify the answer. Those are the tasks where AI can quietly make things worse.

Where X-GenAI fits

Our Discovery Audit runs this kind of screen across a business's processes and ranks the candidates before anything gets built.

References

Brynjolfsson, E., Li, D. and Raymond, L. (2025). Generative AI at Work. Quarterly Journal of Economics, 140(2). Working paper version: NBER w31161 (2023).

Dell'Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper 24-013.

Eloundou, T., Manning, S., Mishkin, P. and Rock, D. (2023). GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. arXiv:2303.10130. Later published in Science (2024).