Context Engineering Is the New Prompt Engineering

There is a phrase floating around AI engineering circles right now that I think is the single most useful mental model anyone building on top of these systems can adopt: behavior belongs in the weights, knowledge belongs in the context. Say it a few times. Almost every expensive mistake I see teams make with language models is a violation of that one line.

The classic version of the mistake goes like this. A company wants their chatbot to know their product catalog and their support policies, so they decide to fine-tune a model on all of it. Months later the catalog has changed, the policies have changed, and the model is confidently quoting prices that no longer exist. They fine-tuned their FAQ. That was never the right tool. Knowledge that changes does not belong baked into a model any more than your inventory belongs hard-coded into your app binary.

The discipline that gets this right has a name now, and it is quietly eating prompt engineering. Anthropic calls it context engineering, and in its September 2025 engineering post it defines the term as “the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference.”¹ That is a mouthful, so let me translate it.

Why prompt engineering stopped being enough

Prompt engineering made sense when the unit of work was a single turn. You typed a question, the model answered, done. In that world the only lever you had was the wording of the prompt, so of course a cottage industry grew up around phrasing tricks and magic incantations. I wrote plenty of them myself.

That world is gone. The unit of work now is an agent that runs for dozens or hundreds of turns, calls tools, reads files, holds a conversation, and accumulates state the whole way. Anthropic draws the line cleanly: prompt engineering is about “writing and organizing LLM instructions,” while context engineering is about “managing the entire context state,” including system instructions, tools, Model Context Protocol data, external data, and message history.¹ It calls context engineering the natural progression of prompt engineering, and I think that framing is exactly right. Prompt engineering did not die. It shrank into being one small part of a much bigger job.

Here is the part that people who have not built agents underestimate. In a long-running agent, the context window is not a place you write a nice prompt into. It is a live, mutating buffer that fills up with tool outputs, half-finished reasoning, retrieved chunks, and stale intermediate results. Left alone it turns to garbage. Managing that buffer is the actual engineering problem, and no amount of prompt-wording cleverness touches it.

The attention budget is real, and it is finite

The reason context engineering matters at all comes down to how transformers work. Anthropic frames it in terms of an “attention budget” that every new token depletes.¹ The mechanism is not mysterious. A transformer computes pairwise relationships between tokens, so for n tokens you get on the order of n-squared relationships. Double the context and you roughly quadruple the attention work spread across the same fixed capacity. Recall quality degrades as context grows because attention gets diluted across more and more pairs.

So the goal is not “give the model everything.” Anthropic states it precisely: find “the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.”¹ Smallest. High-signal. That is the whole discipline in one sentence, and it is the opposite of the instinct most people have, which is to stuff the window full because they can.

This is where the marketing and the reality split hard. As of mid-2026 the leading models, Claude Opus, OpenAI GPT-5.x, and Google Gemini 3.x Pro, all ship 1M-token context windows, and Meta's Llama 4 Scout advertises 10M tokens.⁶ Sounds like the context problem is solved, right? Just dump the whole codebase in there. Except the effective, high-quality recall in practice often falls off somewhere around 300,000 to 400,000 tokens, far below the advertised ceiling.⁶ The window is a billboard. The usable region is a lot smaller.

Context rot: more tokens can make it worse

The clearest evidence that bigger is not better came out of Chroma Research in July 2025, in a study called “Context Rot: How Increasing Input Tokens Impacts LLM Performance.”² The authors, Kelly Hong, Anton Troynikov, and Jeff Huber, tested 18 leading models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. Their headline finding is one every builder should tattoo somewhere visible: accuracy drops non-uniformly as input length grows, and it starts dropping well before the model hits its advertised context limit.

Two specific results from that study changed how I think about assembling context. First, the lost-in-the-middle effect. Models attend well to the beginning and the end of the context but poorly to the middle, and this contributes to accuracy drops of 30% or more.² If you bury the one fact that matters in the middle of a 200-page dump, you have effectively hidden it from the model. Position is not neutral.

Second, and this is the one that really lands: semantically similar but irrelevant content actively misleads the model.²Chroma calls these “distractors.” It is not that extra tokens are harmless dead weight. They are worse than dead weight. Retrieve five chunks that look relevant but only one actually is, and the other four do not just take up space, they pull the answer toward the wrong thing. More retrieval can lower accuracy. That runs against every instinct that says gather more evidence.

Why you should not fine-tune your knowledge in

Now back to the line I opened with. If knowledge belongs in context, the corollary is that you should stop trying to teach models facts by fine-tuning. This is not a taste argument. There is hard data.

Stanford ran a study called FineTuneBench that did exactly the experiment everyone should want to see.³ They fine-tuned five commercial models, GPT-4o, GPT-4o-mini, GPT-3.5-turbo, Gemini 1.5 Pro, and Gemini 1.5 Flash, and measured how well the fine-tuning actually infused new knowledge. The result: an average generalization accuracy of just 37% when learning new information through fine-tuning. Thirty-seven percent. You would not ship a knowledge base that is wrong nearly two-thirds of the time.

Updating knowledge the model already had was even worse. FineTuneBench found that incorporating changes, like updated medical guidelines, landed at an average generalization accuracy of only 19%.³ And two of the five models, Gemini 1.5 Flash and Gemini 1.5 Pro, could not learn new knowledge or update existing knowledge through their fine-tuning APIs at all.³ Not poorly. At all.

Compare that to retrieval. Research in 2025 comparing knowledge injection methods found that RAG, pulling the right facts into the context at inference time, outperformed fine-tuning of weights on both previously-seen and new knowledge.⁴ And it does it without retraining. When facts change frequently and you need transparency about where an answer came from, retrieval wins on every axis that matters. You change a database row, and the model is instantly current. You do not retrain anything.

So what is fine-tuning actually for?

I want to be fair here, because “never fine-tune” is the wrong takeaway. Fine-tuning is a great tool. It is just a tool for behavior, not for facts. A 2026 decision framework popularized the split cleanly, and it matches everything above: behavior, meaning tone, output schema, refusal patterns, and reasoning style, is a candidate for fine-tuning into the weights, but only once it is stable. Facts that change, product details, policies, current pricing, and document contents, belong in retrieval and context, not weights.⁵

The word doing the work in there is stable. Behavior is a good fine-tuning target precisely because you do not change it every Tuesday. How your assistant should sound, what JSON shape it should always return, when it should refuse, how it should reason through a problem: that is durable. Bake it in. But the moment something has a shelf life, a price, a policy, this quarter's roadmap, it belongs in a place you can edit without a training run.

The actual craft: assembling context for agents

Once you accept that the window is a scarce, finite resource and knowledge has to be fed in at inference time, the real engineering starts. How do you keep a long-running agent's context lean and high-signal over hundreds of turns? Anthropic lays out four concrete techniques, and they are worth knowing by name because they generalize far beyond any one framework.¹

• Compaction. When the context fills up, summarize what happened and reinitiate a fresh window seeded with that summary. You throw away the raw transcript and keep the distilled state. It is garbage collection for context.
• Structured note-taking. Write important state out to persistent memory that lives outside the context window, then pull it back in only when it is needed. The window stops being the only place the agent remembers things.
• Just-in-time retrieval. Do not preload documents. Carry lightweight identifiers, file paths, links, IDs, and fetch the actual content only at the moment you need it. This is the direct answer to context rot: you hold pointers, not payloads.
• Sub-agent architectures. Spin off a focused task into its own sub-agent with a clean context window, let it do its narrow job without your accumulated clutter, and return just the result. Each agent gets the small, high-signal context it needs instead of one giant shared mess.

Notice what all four have in common. Every one of them is about keeping the context small and relevant, not large and complete. That is the opposite of how people instinctively use a 1M-token window. The window is not a blessing that lets you stop thinking about context. It is a bigger canvas that makes the discipline matter more, because now you have enough room to pollute it badly.

The way I think about it

Put the whole thing together and a clean architecture falls out. Weights hold stable behavior. Retrieval and context hold everything that changes. The prompt is one carefully chosen input inside a context you are actively managing token by token. And the size of your context window is a constraint to respect, not a bucket to fill, because past a few hundred thousand tokens the model starts losing the plot no matter what the spec sheet says.

Prompt engineering was a real skill for the single-turn era, and the phrasing intuitions still help. But the center of gravity has moved. The people getting the most out of these models in 2026 are not the ones with the cleverest prompt wording. They are the ones who treat the context window like the scarce, leaky, position-sensitive resource it actually is, and who know which information belongs in the weights, which belongs in a database, and which belongs in the window right now. That is context engineering, and it is the job.