Context Windows: The Short-Term Memory Ceiling

Introduction

In the previous lesson, you learned that the model has no memory between sessions. But within a single session, it does hold information — everything you have said and everything it has replied with is accessible right up until the moment it falls off the edge of the context window.

The context window is the model's working memory. It is powerful and it has a ceiling. Understanding what that ceiling is, why it exists, and what happens when you hit it will change how you design every AI workflow you build.

Core Concept

What a Context Window Is

When you send a message to an LLM, the model does not process just your message. It processes your entire conversation — every message you have sent, every response it has given — assembled into a single long string of text. That entire string is called the context.

The context window is the maximum size of that string, measured in tokens.

A token is not quite a word. It is more like a word-fragment. In English, a rough rule of thumb is 1 token ≈ 3/4 of a word. So:

1,000 tokens ≈ 750 words
100,000 tokens ≈ 75,000 words ≈ a short novel

Current frontier models have context windows ranging from a couple hundred thousand to over 1,000,000 tokens. That sounds enormous. It fills faster than you think.

What Eats Context

Every token you consume counts against the limit:

System prompt. The instructions you give the model at the start of a session ("You are a helpful assistant focused on Python development...") live in the context window. A detailed system prompt can easily run 500-2,000 tokens.

Conversation history. Every back-and-forth exchange accumulates. A typical detailed message is 200-500 tokens. A code block can be 1,000+. After 20 exchanges, you may have consumed 20,000-40,000 tokens in conversation history alone.

Injected documents. Paste a long file, a README, or a spec into the chat? That is added to the context. A 3-page specification might be 3,000-5,000 tokens.

The model's own responses. The model's previous answers are part of the context. A detailed, multi-step explanation might generate 800 tokens. That stays in the window for the rest of the session.

What "Losing Context" Really Means

When the total context — system prompt + history + documents + model responses — exceeds the window limit, something has to give. There are two distinct overflow modes, and it pays to keep them separate. If a single request is oversized from the start — say, one document larger than the entire window — the request is simply rejected: raw API calls return a validation error rather than truncating your input. But in a long-running conversation that grows past the limit gradually, the chat application typically handles the overflow for you by truncating the oldest content. The model simply stops receiving it.

This is not a clean failure. The model does not say "I can no longer see your first message." It continues responding as if it has a coherent picture of the conversation, but it is now missing the foundation. You will notice this as:

The model contradicts something it agreed to earlier
It forgets a constraint you specified at the start
It reverts to a behavior you had corrected
It answers a follow-up question as if it lacks the prior context that would make the answer obvious

This is not the model malfunctioning. It is working correctly on an incomplete picture of the conversation. The beginning of the conversation literally fell outside the window.

A Practical Mental Model

Think of the context window as a scroll of paper with a fixed length. As the conversation grows, you add to the bottom. When the scroll is full, you have to unroll from the top — the earliest content falls off. The model can only see what is currently on the scroll.

Your job as an operator is to be intentional about what is on that scroll. Important instructions belong near the bottom (most recent), not at the top where they may eventually fall off. Documents that are needed throughout belong in a position where they will survive the session length you need.

Practical Application

Here is a concrete scenario: you are using an AI to help review a codebase. You start by pasting three files — 4,000 tokens total — and saying "Review these for security issues." After a long back-and-forth (40 exchanges, ~30,000 tokens), you ask a follow-up about the first file.

By now, the first file has been pushed out of the context window. The model responds as if it cannot quite recall the details you started with, because it genuinely cannot see them anymore.

The fix is not to use a bigger model. The fix is architectural:

Do not paste all three files at once. Address one at a time, keep sessions focused.
If you need the model to remember a key conclusion ("File A has an SQL injection risk in the login() function"), summarize it explicitly and re-inject it at the start of the next relevant exchange.
Better yet: build a system that stores key conclusions externally and re-injects only the relevant ones as needed. That is what RAG and structured memory systems do — covered in the next lesson.

The context window is not a problem to route around by upgrading to a larger model. It is a constraint that teaches you to be precise about what information the model actually needs at each step.

Common Mistakes

Pasting everything at once. Beginners dump every relevant document into the chat at the start of a session. This consumes the context budget immediately and leaves little room for the actual conversation. Only inject what is needed for the current task.

Ignoring token counts. If you do not know roughly how many tokens are in your conversation, you are flying blind. Most AI interfaces show you a token count or at least indicate when you are approaching the limit. Pay attention to it.

Relying on "just use a bigger context window." A 1M-token context window does not solve the underlying problem — it delays it. The model's ability to attend to all parts of a very long context degrades as the context grows. Relevant information buried deep in a million-token context may still be missed. Architecture matters.

Repeating corrections instead of re-anchoring. If you corrected the model on something and it seems to have forgotten, you are probably seeing context loss. Repeating the correction mid-stream helps for a few exchanges but will lose again. The right fix is to re-state key constraints in a compact form near the start of each new work block.

Summary

The context window is everything the model can see: system prompt, conversation history, injected documents, and its own prior responses
It is measured in tokens (roughly 3/4 of an English word each)
When the window fills, the oldest content is truncated — the model loses access to it silently
"Losing context" manifests as forgotten constraints, contradictions, and reverted behavior
The fix is architectural: keep context lean, inject what is needed when it is needed

What's Next

The context window is working memory — fast, limited, and temporary. In the next lesson, we look at the two main approaches to building something more durable: RAG (fuzzy semantic recall from a vector database) and structured fact stores (precise key-value memory). These are the tools that let AI agents operate across sessions and across long conversations without losing the thread.