Should you get better at prompting, or better at judging the output?

Abstract. Prompt engineering was a real advantage when models needed coaxing. As models follow plain instructions better, that advantage shrinks. This piece looks at two bodies of evidence: decades of research on how people over-trust automated systems, and a recent experiment showing that AI can make skilled workers less accurate on the wrong kind of task. Both point to the same conclusion. The durable skill is judging output, and in automated workflows that judgment has to be built into the system.

Background

Early large language models were sensitive to phrasing, and a whole craft grew up around templates, role prompts and step-by-step cues. Many of those tricks matter much less with current models, which are trained to follow ordinary instructions. What hasn't changed is that models still produce fluent, confident text whether or not it's correct.

Evidence

Automation bias. Long before generative AI, human factors researchers documented a consistent pattern: people supervising an automated aid tend to accept its output and miss its errors, especially when the aid is usually right and they are busy. Parasuraman and Manzey's review found this complacency among novices and experts alike, and found that simply telling people to stay alert didn't remove it. Fluent AI text, which looks finished, is a near-perfect trigger for the same effect.

The jagged frontier. In Dell'Acqua and colleagues' experiment with 758 consultants, GPT-4 lifted quality by more than 40% on tasks within its capability. On a task just outside it, consultants using AI were 19 percentage points less likely to get the right answer than those without it. A group given prompt-engineering guidance did not escape that drop. The people who did best were the ones who could tell which kind of task they were facing.

Discussion

Taken together, the evidence says the bottleneck is verification, not instruction. In practice, judgment looks like three habits:

• Knowing which parts of a task have a checkable right answer and which don't.
• Looking for the single confident, wrong detail inside an otherwise good answer, which is far easier if you know the subject.
• Asking “what would this look like if it were wrong?” before trusting it, not after something breaks.

Once an AI step runs inside a workflow, nobody reads every output, so personal vigilance stops being enough. Judgment has to move into the system: a review gate on outputs that matter, an evaluation set built from known failure cases, and scheduled spot-checks. That is the automation-bias literature's main lesson applied to a new tool. Design for the lapse rather than hoping people won't have one.

Scroll sideways to see the whole diagram.

Limitations

The automation-bias research mostly predates generative AI and comes from settings like aviation and process control. The consulting experiment used one model and a small set of tasks. Neither shows that prompting is useless. Clear instructions still help. The claim is narrower: the marginal return on clever prompting is falling, and the return on verification isn't.

Conclusion

Pick a task AI already helps with and write down how you'd know if the output were subtly wrong. If that's hard to answer, it deserves more attention than any prompt template.

Where X-GenAI fits

When we build workflows, the review gates and evaluation sets come as standard. Our AI education work covers the same judgment skills for the people working alongside them.

References

Dell'Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper 24-013.

Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3), 381–410.