The context window trap: why a bigger model doesn't fix bad retrieval

Abstract. As context windows have grown to hundreds of thousands of tokens, a common shortcut is to skip retrieval and put an entire document set into the prompt. This piece reviews two lines of research on how models actually use long inputs. Both find that accuracy depends heavily on where relevant information sits and how much irrelevant text surrounds it, and that advertised context length overstates usable context length. The practical conclusion is that retrieval quality still matters, however large the window.

Background

Retrieval-augmented generation (RAG) narrows a large corpus down to a few relevant passages before the model sees anything. Long-context models seem to make that step optional: if the whole corpus fits, why filter it? In a demo with a small, clean dataset, the shortcut often works.

Evidence

Lost in the middle. Liu and colleagues tested models on multi-document question answering and key-value lookup, moving the relevant passage around within the input. Performance followed a U-shape: best when the answer sat at the start or end of the context, and noticeably worse when it sat in the middle. This held for models built specifically for long context. Adding more retrieved documents also showed diminishing and sometimes negative returns.

Effective versus claimed length. Hsieh and colleagues' RULER benchmark went beyond simple “needle in a haystack” tests, which most models pass, to harder tasks such as multi-hop tracing and aggregation. Almost every model showed large drops as input length grew, and only about half of the models claiming 32,000 tokens or more held up satisfactorily at 32,000.

Discussion

The findings explain a pattern that shows up in production: a workflow that answers well over twenty documents gets steadily worse at two thousand. The key passage is more likely to land in the middle, the noise around it grows, and the model is working further from the length where it performs reliably. Cost and response time rise with every token whether it helps or not, and when an answer is wrong it's hard to tell which passage misled it.

Good retrieval addresses each of these. Chunking that follows the real structure of the content, a solid embedding search and a reranker that puts the most relevant passages first give the model a short, focused context. That keeps answers more accurate, and it makes failures traceable to a specific passage.

Scroll sideways to see the whole diagram.

Limitations

Long-context performance improves with each model generation, and the studies here test models available at the time. Newer models may narrow the gap. The underlying point, that relevant information is easier to use when there's less irrelevant information around it, is unlikely to disappear, but the size of the effect should be measured on your own data rather than assumed.

Conclusion

If a workflow's answers get worse as you add documents, the task usually isn't too hard for AI. The retrieval step was skipped in favour of a bigger window.

Where X-GenAI fits

We build and repair retrieval pipelines, with an evaluation set from your own documents so you can see whether answers improve.

References

Liu, N. F. et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.

Hsieh, C.-P. et al. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654.