Context Engineering for AI Agents: A Practical Guide

You deploy an agent, it runs fine for a few turns, then it starts contradicting itself. It ignores an instruction it followed perfectly ten minutes ago. It retrieves a document that has nothing to do with the current task and confidently acts on it anyway. You blame the model. The model is not the problem.

Most production agent failures trace back to context — specifically, to nobody having thought carefully about what the agent knows at each step, where that knowledge comes from, how old it is, and when it gets thrown away. That is what context engineering addresses. This guide covers the full picture: what context engineering is, how to structure it, where it breaks, and what to do about it.

Why Most Agent Failures Are Context Failures

A language model is stateless. Between calls, it remembers nothing. Every decision an agent makes is a function of exactly the tokensTokenThe unit models actually read and generate — roughly a word-piece. Context limits and API pricing are both measured in tokens. in its context windowContext windowThe maximum amount of text a model can consider at once, counted in tokens. Everything outside it is invisible to the model. at that moment — not what it "learned" in previous turns, not what you told it yesterday, not the task it was halfway through completing.

This means the quality of an agent's behavior is bounded by the quality of its context. A capable model with poor context will produce incoherent, stale, or misdirected outputs. A modest model with well-engineered context will outperform it consistently.

The binding constraint for agents in production is not model capability. It is context design.

What Context Engineering Is — and How It Differs from Prompt Engineering

Context engineering is the practice of curating and maintaining the optimal set of tokens in an agent's context window at every point in its execution. As Anthropic describes it, "Context is a critical but finite resource for AI agents." Every token that occupies the window either earns its place by improving the agent's decision-making or costs you something for nothing.

Google Cloud defines it straightforwardly: "Context engineering is the practice of managing information for an AI."

Prompt engineering is a subset of this. It governs how you phrase instructions — the wording of a system promptSystem promptThe standing instruction that sets a model's role, rules, and tone for an entire session, separate from the user's individual messages.Read: System Prompts: What They Are and How to Write Them →, the structure of a user message, the format you request for the output. Context engineering governs what the agent knows when those instructions are executed: which facts are present, which memories have been retrieved, which tool outputs have been injected, and which history has been preserved or discarded.

The distinction sharpens as agents become more autonomous. A single-turn assistant only needs a well-written prompt. An agent running a 24-step workflow across multiple tool calls and sessions needs a coherent, continuously updated information environment — otherwise, by step 12, it is effectively operating blind.

How Agent Context Is Structured

Designing agent context well starts with understanding its layers. Not all context is the same: some elements are fixed when you build the agent, others are assembled fresh at runtime, and others grow and decay across the conversation. Here is how those layers fit together.

The Five Layers of Context

A well-designed agent context has five distinct layers, each with a different update frequency and purpose.

The system prompt is the stable foundation. Everything above it is variable. The more clearly you separate these layers in your design, the easier it is to update one without corrupting another.

Static vs. Dynamic Context: What Lives Where

Static context is assembled once and held constant: the system prompt, role definition, behavioral rules, output format instructions. Dynamic context is assembled at runtime: retrieved documents, API responses, tool outputs, the current turn's message history.

The principle that matters here is determinism. Context assembled by code and logic — explicit retrieval calls, structured state objects, programmatic history trimming — is predictable and debuggable. Context assembled improvisationally, by letting the model decide what to include or by appending everything available, is neither.

Deterministic context assembly reduces the variance in agent behavior. When an agent surprises you, you want to be able to look at exactly what was in its context at that moment. If context assembly is ad hoc, you cannot reconstruct it.

The Agent Harness and Model Context Protocol (MCP)

The Agent Harness is a structural pattern: a wrapper around a model call that assembles and injects context before the call and processes the output after it. It is where context management actually lives — the code that decides what goes in, in what order, and how much.

Model Context ProtocolModel Context ProtocolAn open standard for connecting AI models to external tools and data sources through one consistent interface instead of bespoke integrations. (MCP) extends this to tool integration. It standardizes how tools expose their schemas and how tool outputs are returned and injected into the agent's context. Without a standardization layer, every tool integration is a bespoke context-injection problem.

OpenAI's Responses API is a concrete implementation of this pattern. Its agents Python library frames it directly: "Context is a dependency-injection tool." The context object is passed through runs, tools, and handoffs — it is not an afterthought appended to a prompt but a first-class object in the execution flow.

The Context Window Problem

Every agent operates within a finite token budget. Understanding how that budget gets consumed — and misused — is the difference between an agent that works at scale and one that degrades after a few turns. This section covers the failure modes that most implementation guides underserve.

Finite Attention Budget: What It Means in Practice

The raw token limit of a model (128K, 200K, 1M tokens) is not the same as the effective context window. The effective context window is the portion of the window that the model reliably attends to and acts on. In practice, models attend less precisely to information buried in the middle of a long context than to information at the beginning or end — a well-documented phenomenon sometimes called the "lost in the middle" problem.

Filling the window is not using it well. A 128K context stuffed with marginally relevant retrieval results will produce worse outputs than a 20K context containing only the information the agent actually needs for the current step.

Token count monitoring is an operational necessity, not an optimization. Track context length across turns and flag when an agent's context exceeds a threshold you set based on observed degradation — not the model's maximum.

Context Rot, Context Pollution, and Context Distraction

These three failure modes are distinct and worth naming precisely.

Context rot is the accumulation of stale information across turns. An agent that carries its full message history forward will eventually be operating with instructions, facts, and tool outputs from twenty turns ago that no longer reflect the current state of the task. The model cannot distinguish "this is old" from "this is current" unless you tell it explicitly.

Context pollution is noise introduced by over-broad retrieval. When a retrieval query is loosely specified, the returned documents are only partially relevant. Those partially relevant tokens occupy the window and subtly bias the model's reasoning toward content that was not the right answer.

Context distraction is the sharper version: a retrieved document that is irrelevant to the current step but similar enough to the query that it lands in context anyway, and the model acts on it. This is not a retrieval failure in isolation — it is a context management failure. The fix is not always better retrieval; sometimes it is filtering retrieved results before injecting them.

All three are visible in agent logs as coherence degradation: the agent starts referencing facts from earlier in the conversation that are no longer true, or it produces outputs that would make sense given a different version of the task than the one currently active.

Compaction Cycles and Summarization Strategies

A compaction cycle is triggered when the context window approaches its capacity and the agent (or the harness around it) must reduce the size of the accumulated history to continue operating. The naive approach is truncation: drop the oldest turns. This works until the oldest turns contain task setup information the agent still needs.

The common alternative is LLMLarge language modelA model trained on vast amounts of text to predict the next token, which is what lets it write, summarise, and reason over language. summarization: use a model call to compress older history into a summary, then replace the raw turns with the summary. This preserves more semantic content but adds latency and cost.

A 2025 NeurIPS workshop paper from JetBrains Research compared these approaches and found that "Simple Observation Masking Is As Efficient As LLM Summarization for Agent Context Management" — meaning selectively hiding (masking) irrelevant observations from earlier steps often performs as well as generating a new summary, at a fraction of the cost.

Structured note-taking is a third option: instead of compressing history, the agent maintains an explicit state object (a short structured document) that is updated as facts change. The state object stays in context; old turns that contributed to it are dropped. This is the most token-efficient approach and produces the most debuggable context.

Pre-rot threshold: set a turn count or token count at which you proactively trigger compaction, before the window is full. Reactive compaction (triggered at the limit) forces tradeoffs under pressure; proactive compaction gives you time to do it cleanly.

Retrieval and Memory Systems

Beyond what the agent carries in its live context window, it needs access to information it cannot hold there permanently. Retrieval and memory systems are how agents access knowledge that exceeds the window — and how they build continuity across sessions. The design decisions here are as important as anything in the context window itself.

Short-Term, Long-Term, and Persistent Memory — Defined

These three terms are often conflated. They are not the same thing.

Short-term memory is the live context window: what the agent is actively attending to right now. It is fast and immediately actionable, but finite and ephemeral.

Long-term memory is stored outside the context window and retrieved as needed — typically via a vector database or a structured knowledge store. As Google Cloud notes, "Long-term memory is the foundation for an agent's intelligence, grounding, and personalization — and is distinct from the fast, short-term context of a live conversation."

Persistent memory is a subset of long-term memory that survives across sessions and across agent instances. It is how an agent "remembers" that a user prefers a certain output format, or that a particular data source was unreliable, or what the last completed step of a multi-day task was.

Working memory is the scratchpad: intermediate results, in-progress reasoning, tool outputs that have been produced but not yet acted on. It lives in the context window and should be treated as temporary.

How Retrieval Works: Vector Databases and Query Augmentation

The standard retrieval pattern for agents is retrieval-augmented generation (RAG): the agent formulates a query, the query is used to search a vector database (a store of embedded document chunks), and the top-k most semantically similar chunks are returned and injected into the context window.

The critical failure point is query quality. A query that matches the surface form of the user's request but not the semantic content of the relevant documents will return poor results. Query Augmentation addresses this: before issuing the retrieval query, the agent (or a pre-retrieval step) rewrites or expands the query to improve precision. This can mean generating multiple query variants, extracting key entities, or using the agent's current state to add context to the query.

Retrieved documents do not belong in context raw. Filter for relevance after retrieval, not before. Inject the most relevant passages, not full documents. Include source metadata (document ID, date) so the model can reason about freshness.

File-Based and Hybrid Memory Setups

Vector databases are not always the right retrieval mechanism. For large structured data (tables, audit logs, configuration files), file-based memory is often more efficient: store the data externally, retrieve specific rows or sections via tool calls, and inject only what is needed.

Hybrid setups combine vector retrieval for unstructured knowledge, structured queries for tabular data, and in-context state objects for the agent's current task status. The Anthropic enterprise architecture guide describes this combination as the basis for reliable long-horizon agents: tools, memory systems, and explicit state working together.

Reflection loops fit here too. An agent that periodically writes a summary of what it has learned to persistent memory — and retrieves that summary at the start of the next session — maintains continuity without requiring the context window to carry everything forward.

Tool Discovery, Selection, and Argument Formulation

Every tool registered to an agent occupies context space. The tool schema (name, description, parameters) must be present in the context for the model to know the tool exists and how to call it. A registry of 50 tools is not free — it consumes tokens that could hold task-relevant information.

Two design principles follow from this. First, only expose tools that are relevant to the current task scope. A context-aware state machine can dynamically load and unload tool schemas as the agent moves through task phases. Second, write tool schemas for the model, not for the developer: precise, unambiguous descriptions of what the tool does and what its parameters expect. Ambiguous tool schemas produce incorrect argument formulation, which produces failed tool calls, which produce error messages in the context, which consume more tokens.

Practical Implementation: What Actually Works

The principles above translate into a set of concrete implementation practices. These are drawn from real deployments and from the research literature — not general advice, but specific patterns with known tradeoffs.

Build Context Deterministically, Not Improvisationally

Assemble context in code, not by feel. Write the function that builds the system prompt. Write the logic that selects which retrieved documents get injected and in what order. Write the trimming rule that drops message history beyond N turns.

The openai-agents-python library makes this explicit: context is a dependency-injection object, passed through the agent's execution graph. Treat it that way. When context assembly is a defined function with inputs and outputs, you can test it, log it, and debug it. When it is implicit, you cannot.

Choose High-Signal Tokens — Everything Else Is Noise

Every token in the context window is a claim on the model's attention. Ask of every element: does this change what the agent should do right now?

What earns its place in context: the current task definition, recent turns that are still causally relevant, retrieved facts that bear on the immediate decision, tool outputs from the current execution step, and explicit state objects.

What should be cut: turns from earlier in the session that have been superseded, redundant instructions that appear in both the system prompt and the user message, decorative prose in system prompts, tool schemas for tools not relevant to the current phase.

A system prompt that is 2,000 tokens of careful instruction will outperform one that is 6,000 tokens of thorough-but-unfocused documentation.

Use Few-Shot Examples and Chain-of-Thought as Context Tools

Few-shot examples are compressed behavioral context. Instead of writing a 500-word instruction explaining the output format you want, three input-output pairs demonstrate it directly and consume fewer tokens while being more precise.

Chain-of-thought (CoT) prompting works similarly: by asking the agent to reason step by step before producing an answer, you give it a structured path through complex tasks that it would otherwise have to derive on its own each time. This is context shaping behavior without requiring the context to explicitly encode every rule.

The ReAct pattern — reasoning and acting interleaved — combines these: the agent thinks, acts, observes the result, and thinks again. Each thought step is written into the context, creating a running record of the agent's reasoning that subsequent steps can build on.

Manage Cost with KV-Cache and Token-Efficient Tool Design

KV-cache hit rate is one of the most underused levers in production agent economics. When the prefix of a prompt (the system prompt, static context, stable instructions) stays identical across calls, the model provider can cache the computed key-value representations for those tokens and skip recomputing them. The result is lower latency and lower cost.

To maximize cache hit rate: put the stable elements (system prompt, persona, fixed instructions) at the beginning of the context, before any dynamic content. Never vary the prefix unnecessarily. Keep the stable prefix as long as possible and append dynamic content at the end.

Token-efficient tool schemas follow the same logic: shorter, unambiguous parameter descriptions that leave no room for model interpretation reduce both the token cost of the schema and the error rate on tool calls.

Distribute Context Load Across Sub-Agents

Some tasks are simply too large for a single agent's context window. A task that requires maintaining awareness of 50 documents, 10 tool states, and a 200-turn history simultaneously is not a context engineering problem — it is an architecture problem.

Multi-agentMulti-agent systemSeveral specialised agents working together on one job, each owning a step, coordinated by a planner or a shared workflow. designs solve this by breaking the task into scoped sub-agent workstreams. Each sub-agent carries only the context relevant to its piece of the problem. An orchestrating agent manages task state at a higher level of abstraction, receiving summarized outputs from sub-agents rather than raw detail.

Microsoft's Agent Framework provides open-source infrastructure for exactly this pattern: multi-agent orchestration with session state management across Python and .NET, handling the handoffs between agents without requiring each agent to carry the full shared state.

OpenAI's AgentKit takes this further with a visual workflow builder for multi-agent orchestration, a connector registry, and built-in evaluation tools. Ramp, the financial automation company, reported that using AgentKit's Agent Builder "transformed what once took months of complex orchestration, custom code, and manual optimizations into just a couple of hours."

Common Failure Modes and How to Fix Them

Knowing the failure modes by name makes them much faster to diagnose. Here are the four most common, with their root causes and fixes.

Context Clash: Conflicting Instructions

This happens when the system prompt says one thing, a tool output says another, and a user instruction says a third — and the agent has no rule for resolving the conflict. The result is inconsistent behavior: the agent follows a different instruction each time depending on which one happens to be most recently attended to.

The root cause is missing instruction hierarchy. The fix is to make the priority order explicit in the system prompt: "If this instruction conflicts with a tool output, follow this instruction. If the user contradicts this rule, explain why it cannot be overridden." Explicit override rules produce consistent behavior; implicit priority produces variance.

Context Confusion: Ambiguous State

An agent that has been running for many turns through a compacted history may lose track of where it is in the task. It re-asks questions it already answered, repeats steps it already completed, or continues a subtask the user already cancelled. The cause is not a model failure — it is implicit state.

The fix is explicit state objects: a short structured document (JSON or a bulleted list) that records the current step, completed steps, outstanding decisions, and any state that must survive compaction. This object lives in context, gets updated after each meaningful action, and survives the compaction of the raw message history that contributed to it.

Context Distraction: Irrelevant Retrieval

A retrieval call returns three documents: two are highly relevant, one is semantically similar but about a different entity or time period. The model incorporates all three into its reasoning, and the irrelevant one pulls the output off target.

This failure mode is subtle because the agent's output looks plausible — it is not a hard error. The root cause is no post-retrieval filtering. The fix is to evaluate retrieved results for task-relevance before injection, either with a lightweight scoring step or by including the retrieved document's source and date in context so the model can reason about whether it applies.

Context Overload: When the Window Can't Hold the Task

When a task requires more context than a single window can reliably support, expanding the window is the wrong answer. Models attending to 500K tokens spread across a complex task are less reliable, not more. The signal to watch for is output quality degrading as context grows, not just errors.

The fix is architectural. Refactor the task into sub-agents with scoped contexts, each responsible for a portion of the problem. Promote facts that must persist across the full task into a persistent memory store rather than holding them in the live context. Revisit the task design to identify what information is actually load-bearing at each decision point.

What This Looks Like in Practice: A Marketing Agent Example

Abstract principles are easier to hold when you can see them operating in a real workflow. Consider a marketing agent running 24/7 to monitor keyword rankings, generate weekly SEO reports, and push alerts to Slack when a tracked page drops in position.

This is the kind of workflow that Letaido manages for marketing teams. The context design it uses is worth walking through because it illustrates every principle above.

The system context is fixed at build time: the agent's role, the Ahrefs data it has access to, the Slack workspace it routes to, and the alert thresholds it uses to decide when to escalate. This never changes between runs, which means it is a strong candidate for KV-cache optimization — the prefix is stable across every scheduled execution.

The session guidance layer is assembled fresh at the start of each run: which keywords to check, which pages are under active monitoring, and the current reporting period. This is pulled from a structured configuration stored in persistent memory, not re-specified in each prompt.

The tool-retrieved data layer is populated by live Ahrefs pulls during the run: current ranking positions, traffic changes, newly discovered backlinks, technical audit findings. These are injected as structured snippets, not raw API dumps — only the fields the agent needs for its current task, filtered before injection.

Message history from prior runs is not carried forward raw. Instead, Letaido writes a short state document at the end of each run: what changed, what was reported, what is pending investigation. The next run starts with that state document, not with 200 turns of prior conversation. This is the structured note-taking pattern in production.

When a rank drop exceeds the threshold, Letaido calls the Slack connector, routes the alert to the right channel, and logs the event to the state document. It does not hold the full Slack message history in context — that would be context pollution. It holds the threshold rules and the current alert state.

The result is an agent that stays coherent across weeks of continuous operation, because the context design gives it exactly what it needs at each step and nothing it does not.

The Rules of Good Context Engineering

These six principles summarize everything above. Apply them in order.

1. Assemble context in code, not by feel. Deterministic context assembly is testable and debuggable. Ad-hoc context stuffing is neither.

2. Separate static from dynamic context. Fixed elements (system prompt, rules, persona) belong at the start and should never vary between calls. Dynamic elements (retrieved data, history, tool outputs) are appended at runtime.

3. Every token earns its place. If a context element does not change what the agent should do right now, remove it. Shorter, more relevant context outperforms longer, comprehensive context.

4. Monitor context length as an operational metric. Set a pre-rot threshold. Trigger compaction proactively — before the window is full — using observation masking or structured state objects rather than raw truncation.

5. Match memory type to information lifecycle. Live decisions belong in the context window. Facts that must survive turns belong in explicit state objects. Knowledge that must survive sessions belongs in persistent memory and is retrieved, not carried.

6. When a task outgrows a single context, change the architecture. Distribute load across scoped sub-agents. An orchestrator that summarizes results beats a single agent that tries to hold everything.

If you want to see these principles already implemented in a marketing workflow — rank tracking, content gap analysis, site audits, Slack alerts, all running on a shared agent with structured context management — Letaido is worth a look. It runs on Ahrefs data with flat pricing at $99/month, and the first trial is free.