Skip to content
LUIINTERACTIVE

Insights

Why does AI forget what you told it? Context windows explained

LUI Interactive
  • AI
  • Technology
  • Small business

The short answer

A context window is the amount of text an AI model can read in a single request, measured in tokens at roughly three quarters of a word each; because every reply re-reads the whole conversation, anything past that limit is dropped and the model cannot recall it.

Show prices in

An AI model can only work with the text inside its context window, the block of text it reads before writing each reply. Once a chat or a document runs past that limit, the earliest material is dropped and the model has no way to look back at it.

What is a context window in plain terms?

A context window is the maximum amount of text a model can take in on a single request, measured in tokens. A token is a fragment of text rather than a word, roughly three quarters of an English word on average. The word “marketing” is one token; “optimisation” is often two or three.

A 100,000-token window holds around 75,000 words, which is a 250-page report. Smaller and cheaper models often offer 8,000 to 32,000 tokens, or roughly 6,000 to 24,000 words. Larger ones commonly offer 128,000 to 200,000, and a few advertise a million or more.

The window has to hold everything else in the request too: the system instructions, any retrieved documents, and the reply itself. A 32,000-token window given a 24,000-token document leaves 8,000 tokens for the answer.

Nothing inside the window is stored once the reply is written. That is why the same question asked in a new chat gets a fresh answer with no knowledge of the earlier conversation.

Why does AI forget what I told it earlier in a chat?

Each reply is generated from scratch using only the text sent in that one request. A chat interface works by resending the whole conversation, oldest message first, every time you press send.

When the thread grows past the window, something has to be cut. Usually the oldest exchanges are dropped, or compressed into a brief summary. The model is not being forgetful in the human sense: the text has physically left the input.

This explains a familiar pattern. An assistant quotes a detail from message two while answering message five, then loses it entirely by message thirty. Nothing changed in the model. The thread crossed the limit.

How much can an AI read at once in practice?

Less than the headline figure suggests. Recall is uneven across a long input: material at the very start and the very end of a long block is used more reliably than material buried in the middle. Anyone who tests this a few times sees the same shape.

If the one sentence you need sits on page 90 of a 200-page document, a large window is no guarantee it will be found. A modest window with the right five pages attached is often more accurate.

The test is easy to run yourself. Take a long document, ask a question whose answer sits in the middle, and see what comes back. Then ask for the page or section reference and check it.

Why do long documents get worse answers than short excerpts?

A model does not search a document. It reads it in one pass, with no index, no contents page and no way to flip back.

The analogy is handing a 300-page manual to a capable temp who has twenty seconds and no pen. The literal mechanism is attention. The model weighs every token against every other token it is allowed to see, so the computation grows roughly with the square of the input length. Doubling the document roughly quadruples that part of the work, and the useful signal thins out as more unrelated text competes for the same weight.

In practice, five relevant pages usually beat 200 mixed ones. For a reliable answer about a refund policy, supply the refund policy.

What does a bigger context window cost?

Money and time, both scaling with length. Pay-per-token pricing for mid-range models typically sits somewhere between £1 and £10 per million input tokens, with output tokens often costing three to five times more. Those rates move often, so treat the range as a shape rather than a quote.

The arithmetic matters more than the price list. If a thread is 5,000 tokens and each exchange adds 400, the tenth request reads about 8,600 tokens. Because every reply re-reads the whole thread, cost per message climbs through the conversation, and a long chat can cost several times more per reply than a short one.

Latency follows the same curve. More input tokens means a longer wait before the first word appears, and in a customer-facing tool that wait is often the cost people notice.

Token counts also vary by language. A page of English might be 500 tokens; the same page in German or Japanese can take 1.5 to 2 times as many, which raises the cost for multilingual teams.

Does a small business need a huge context window?

Usually not. Most business questions concern a price list, a set of terms, a service description or a week of email, and all of those fit inside a modest window.

A large window earns its price when a task genuinely needs one long connected input: reviewing a full contract, comparing twenty supplier quotes, or summarising a transcript where the key detail could sit anywhere.

It is poor value when the real problem is that your documents are duplicated, out of date or badly organised. Paying a premium for a million tokens while feeding the model an unedited 2019 policy manual buys speed and very little else.

How can you tell whether a tool’s context window is enough?

Ask for the figure in tokens, and ask whether it covers input and output combined or input alone. Vendors often quote the largest window the underlying model supports, which is not the same as what their product will accept. A tool that takes 20 pages but reads 10 is worse than its own specification.

Then test with your own material. Load a 40-page document, ask a question whose answer sits around page 33, and request the section it came from. If the answer is right and the reference checks out, the tool is doing real work. If the answer is confident and sourceless, it is producing plausible text, which is a different product.

What should you do differently?

  • Put the task at the top of the prompt, then repeat the single most important instruction at the end.
  • Send the five pages that matter rather than the 200 that do not.
  • When a chat passes about twenty exchanges, ask for a short summary and start a fresh chat with it.
  • Ask for page or section references, then spot-check two of them.
  • Keep a one-page brief of stable facts about your business instead of re-explaining it each time.

Some assistants now keep a separate stored profile of facts about you between chats. That memory is not the context window, and it can be wrong, so it is worth reviewing what has been saved.

A context window is a working limit, not a capability. Most disappointing AI answers in small businesses come from a vague question or a bloated document, not from a window a few thousand tokens too small.

Find out how visible you actually are

A free written audit of how search engines and AI assistants currently read your website. No obligation, and no sales call required to receive it.

Prefer to talk first? Email contact@luiinteractive.com or message +44 7349 961542 on WhatsApp.