Context Windows Explained: Why Bigger Is Not Always Better

The context window is the maximum number of tokens a model can consider at once. It is a hard architectural limit, and it covers more than people expect.

What counts toward the window

Everything:

• your system prompt
• the conversation history
• any documents or retrieved context
• tool definitions and tool outputs
• the response the model generates

That last item is the one that catches people. The window is shared between input and output. Fill 200,000 tokens of a 200,000-token window with your document and the model has no room to reply.

Always reserve space — 2,000-4,000 tokens is a reasonable default, more if you expect long answers. Our context window calculator shows this reservation explicitly across models.

Windows have grown enormously

Early models handled 2,000-4,000 tokens, roughly a few pages. Current models range from 128,000 to over 1,000,000 — the difference between a chapter and a small library.

This changed what is possible. You can pass an entire codebase, a full contract, or a long transcript without chunking. Many problems that previously required a retrieval pipeline can now be solved by pasting the document.

But capacity is not attention

Here is the part that matters and gets less coverage: models do not use their full window evenly.

The documented pattern is called lost in the middle. Accuracy at retrieving a fact from a long context follows a U-shape — high when the fact appears near the beginning or end, noticeably lower when it sits in the middle. This has been measured across models and tasks.

So a 200,000-token window does not give you 200,000 tokens of reliable attention. It gives you a large capacity with uneven quality across it.

Designing around it

Put critical instructions at the edges. Beginning or end, not buried. If a prompt is long, state the task at the top and restate it at the bottom.

Order retrieved passages by relevance, not document order. If you are passing ten chunks, the most relevant should be first or last, not fifth.

Retrieve rather than paste. Passing the relevant 5% of a document reliably beats passing all of it and hoping the model finds the right part. This is the core argument for RAG even in an era of huge windows.

Summarise conversation history. In long sessions, replace the full transcript with a running summary every few turns. Cheaper, and keeps the model focused on what is still relevant.

Watch for silent truncation. Some interfaces drop the oldest content when you exceed the window rather than erroring. The model then answers confidently without material it never saw. If quality degrades mysteriously in a long conversation, suspect this.

When a big window genuinely helps

Large windows are transformative for specific things: analysing a long document as a whole where the relationships between distant sections matter, working across a codebase where you cannot predict which files are relevant, and processing long transcripts or video.

They are less useful than they appear for question answering over a document collection, where targeted retrieval is both cheaper and more accurate.

Practical checklist

1. Estimate your total input with the token counter.
2. Add 2,000-4,000 for the response.
3. Check it against your model's limit.
4. If you are near the ceiling, retrieve rather than paste.
5. Put the important instructions at the very start or very end.

Our RAG & Custom AI Knowledge course covers the retrieval side in depth, including chunking strategy and why it matters more than window size.