Context window

It is a shared budget

The window is not "how much you may send" — it is everything at once. A 32k window holding a 2k system prompt, a 20k document and 8k of conversation has 2k left for the answer, and a longer answer is simply impossible. Most APIs let you reserve output space explicitly, and forgetting to do so is a common cause of replies that stop mid-sentence.

Because a model is stateless between calls, the appearance of memory comes from re-sending the transcript every turn. That is why long chats get gradually more expensive: you pay for the whole history again on each message. When the transcript no longer fits, something has to go, and the strategy — drop oldest, summarise the middle, retrieve only relevant parts — is a design decision in your application, not something the model handles.

What to watch out for

A bigger window does not mean better reasoning, and it does not mean the whole window is used well. This is the single most common misunderstanding. Three specifics:

1. Attention is uneven. Models reliably use the beginning and the end of a long input and are measurably weaker on material in the middle. Putting the important instruction last usually beats burying it on page forty.
2. Cost and latency grow with what you actually send. Filling a million-token window because it exists is an expensive way to make responses slower and often worse.
3. Advertised length is not effective length. Vendors state the architectural maximum. Retrieval accuracy at that maximum is a separate measurement, and it is usually worse than the number suggests.

The practical conclusion is that a large window is a convenience, not a substitute for selection. Sending the three relevant pages beats sending the whole manual — which is the reason retrieval-augmented generation did not become obsolete when windows grew.

Small local models make this concrete. Browser models running on WebGPU often have only 4k to 8k tokens, which is why tools built on them tend to send targeted diffs of an existing file rather than the whole document.