Context Engineering and Agent Memory · AI Engineering

Budget the window, order the context, page memory in and out, fuse memory stores and persist agent state.

The context window budget

Nodes and connections

Node	Role	Kind
System prompt	~500 tokens, goes first
Tool definitions	~2K-8K tokens
Retrieved context	~2K-10K tokens
Conversation history	~2K-20K tokens
Few-shot examples	~1K-3K tokens
User query	~100-500 tokens, goes last
Generation budget	~2K-8K tokens
Context window	128K tokens	model

From	To	Connection
System prompt	Tool definitions
Tool definitions	Retrieved context
Retrieved context	Conversation history
Few-shot examples	User query
User query	Generation budget
Generation budget	Context window	fits or truncates

Sources

phases/11-llm-engineering/05-context-engineering/docs/en.md

Numbers that matter

Lost in the middle: 85-90% accuracy with the document first or last. 60-70% at position 10 of 20.
Low zone: positions 40-70% of the context lose 10-20% accuracy.
Curated beats dumped: a 10K token context often beats a dumped 100K token context.
Tool cost: 50-200 tokens per tool. 50 tools at 150 tokens is 7,500 tokens.
Tool pruning: cuts tool tokens by 60-80%.
History: 50 turns at 200 tokens is 10,000 tokens.
Summary: 100 tokens replace 10 turns that took 2,000 tokens.
Token estimate: word count times 1.3 in count_tokens.

Sources

phases/11-llm-engineering/05-context-engineering/docs/en.md

Assemble context per query

Classifier classify_intent(query) counts keyword hits per intent. No hit gives ["code"].
Tool selector select_tools(query, token_budget=2000) keeps tools whose category matches.
Retriever score_relevance(query, documents) scores word overlap. Docs below threshold drop out.
Reorder reorder_lost_in_middle(items, scores) puts the best items first and last.
History ConversationManager summarizes the oldest 2 turns while history is over the cap.
Budget ContextBudget truncates each component to its cap and to the space left.

Sources

ContextEngine.assemble

budget = ContextBudget(max_tokens=128000, generation_reserve=4000)

budget.allocate("system_prompt", self.system_prompt, max_tokens=1000)

tools, tool_tokens = select_tools(query, token_budget=2000)

budget.allocate("tools", tool_text, max_tokens=2000)

budget.allocate("retrieved_context", doc_text, max_tokens=3000)

budget.allocate("conversation_history", history_text, max_tokens=5000)

budget.allocate("user_query", query, max_tokens=500)query goes last

print(budget.report())tokens and percent per component

Sources

Memory tool surface

Command	What it does
`core_memory_append(section, text)`	Writes to a pinned section of the prompt
`core_memory_replace(section, old, new)`	Edits a pinned section
`archival_memory_insert(text)`	Writes to the searchable external store
`archival_memory_search(query, top_k)`	Pages facts back into the next turn
`conversation_search(query)`	Scans past turns, evicted turns too
`block_append(label, text)`	Adds text to a memory block
`block_replace(label, old, new)`	Edits a memory block
`block_summarize(label)`	Condenses a block near its limit

Sources

Memory designs side by side

Factor	MemGPT two tiers	Letta blocks	Mem0 hybrid	Repo memory
Stores	Main context plus archival	Core, recall, archival	Vector, KV, graph	JSON files under a schema
Write	Memory tool call	Block tools and sleep-time rewrites	`add(text, user_id, metadata)` to all three	Validate, then atomic write
Read	`archival_memory_search`	Core blocks stay in the prompt	`search(query, user_id)` fuses scores	Load and validate
Stale facts	Consolidation, invalidation	Sleep-time agent invalidates	Old edge marked invalid, not deleted	`schema_version` refuses unknown files
Main risk	Memory rot, poisoning	Block bloat, silent drift	Embedding drift, graph explosion	Partial writes
Use it when	Data is larger than the window	Some facts must stay in the prompt	Queries mix similarity, lookup, relations	State must outlive the chat

Sources

Tasks in code

Task	Code
Create a block	`create(label, description, limit=300)`
Test a block for summary	`block.near_limit()`
Consolidate off the critical path	`sleep.run(contradictions=[...])`
Soft delete a stale fact	`self.archival.invalidate(record.rid)`
Write to all three stores	`mem.add(text, user_id=..., kv_triples=...)`
Fused, scoped search	`mem.search(query, user_id=..., top_k=3)`
Persist state safely	`atomic_write(path, content)`
Refuse a bad write	`mgr.commit(state)` raises `SchemaError`

Sources

Memory server project

Command	What it does
`python3 scripts/project_test.py memory-server --init <dir>`	Creates your workspace. A fresh starter fails
`python3 scripts/project_test.py memory-server --all --path <dir> --strict`	Grades all four stages
`node cli.ts --data-dir memory-data --put samples/memory.json`	Appends revision 1 of `cache-policy`
`node cli.ts --data-dir memory-data --query "cache policy"`	Searches the `docs` namespace
`node cli.ts --data-dir memory-data --put samples/memory-updated.json --revision 1`	Appends revision 2. Revision 0 again is a conflict
`node cli.ts --data-dir memory-data --history cache-policy --html history.html`	Prints all revisions and writes a comparison table
`node cli.ts --data-dir memory-data --serve --port 8788`	Serves REST and MCP. Set MEMORY_TOKEN first

Sources

Defaults

Setting	Default	Note
`max_tokens`	128000	Minus `generation_reserve` 4000
`max_history_tokens`	5000	Summarize old turns above it
`token_budget`	2000	Cap for tool definitions
Relevance `threshold`	0.05	The docs version uses 0.1
`max_messages`	4	MemGPT main context. Older messages go to `evicted`
Block `limit`	300	Characters. `near_limit` fires at 0.8
`w_relevance`	0.6	`w_importance` 0.2, `w_recency` 0.2
`recency_halflife_s`	86400.0	Recency is 0.5 per half-life

Sources

Troubleshooting

Symptom	Cause	Fix
Model ignores a fact in the middle	Lost in the middle	Key facts first, query last. Repeat the key point at the end
Tools fill the window	Every tool sent on every call	Classify intent and prune tools
Retrieval drowns in stale facts	Memory rot	Consolidate and invalidate in a sleep pass
Agent obeys an old injected note	Memory poisoning	Treat retrieved memory as untrusted text
Agent recalls but cannot cite	Citation loss	Store session ID and turn ID with each write
Block hits its cap	Block bloat	Summarize before the write that crosses the cap
One user sees another user's facts	Scopes mixed	Pick one scope for each write
Session resumes on corrupt state	Non-atomic write	Temp file, `fsync`, `os.replace`

Sources

Questions

Bigger window, no memory?: 128k-window baselines still miss long-horizon facts that a 4k-window agent with external memory catches.
Why sleep-time compute?: Primary turns do not wait. The sleep agent can use a stronger, slower model.
Why three stores?: A single store is wrong for two of three query classes.
How does fusion rank?: A weighted sum of relevance, importance and recency. Weights change per product.
What goes in repo memory?: Task id, touched files, assumptions, blockers, next action. Not raw chat transcripts.

Sources

Original source

---
id: context-engineering
title: Context Engineering and Agent Memory
code: CE
subtitle: Budget the window, order the context, page memory in and out, fuse memory stores and persist agent state.
hue: 280
sources: phases/11-llm-engineering/05-context-engineering, phases/14-agent-engineering/07-memory-virtual-context-memgpt, phases/14-agent-engineering/08-memory-blocks-sleep-time-compute, phases/14-agent-engineering/09-hybrid-memory-mem0, phases/14-agent-engineering/34-repo-memory-and-state, projects/memory-server
source_hash: de3d6fbb8902e981
---

## The context window budget
type: architecture
from: phases/11-llm-engineering/05-context-engineering/docs/en.md

```arch
node | sys | 1,1 | System prompt | ~500 tokens, goes first
node | tools | 2,1 | Tool definitions | ~2K-8K tokens
node | ret | 3,1 | Retrieved context | ~2K-10K tokens
node | hist | 4,1 | Conversation history | ~2K-20K tokens
node | shots | 1,2 | Few-shot examples | ~1K-3K tokens
node | query | 2,2 | User query | ~100-500 tokens, goes last
node | gen | 3,2 | Generation budget | ~2K-8K tokens
node | win | 4,2 | Context window | 128K tokens | model
edge | sys | tools |
edge | tools | ret |
edge | ret | hist |
edge | shots | query |
edge | query | gen |
edge | gen | win | fits or truncates
```

## Numbers that matter
type: concepts
from: phases/11-llm-engineering/05-context-engineering/docs/en.md

- **Lost in the middle**: 85-90% accuracy with the document first or last. 60-70% at position 10 of 20.
- **Low zone**: positions 40-70% of the context lose 10-20% accuracy.
- **Curated beats dumped**: a 10K token context often beats a dumped 100K token context.
- **Tool cost**: 50-200 tokens per tool. 50 tools at 150 tokens is 7,500 tokens.
- **Tool pruning**: cuts tool tokens by 60-80%.
- **History**: 50 turns at 200 tokens is 10,000 tokens.
- **Summary**: 100 tokens replace 10 turns that took 2,000 tokens.
- **Token estimate**: word count times 1.3 in `count_tokens`.

## Assemble context per query
type: flow
from: phases/11-llm-engineering/05-context-engineering/docs/en.md, phases/11-llm-engineering/05-context-engineering/code/main.py

1. **Classifier**: `classify_intent(query)` counts keyword hits per intent. No hit gives `["code"]`.
2. **Tool selector**: `select_tools(query, token_budget=2000)` keeps tools whose category matches.
3. **Retriever**: `score_relevance(query, documents)` scores word overlap. Docs below `threshold` drop out.
4. **Reorder**: `reorder_lost_in_middle(items, scores)` puts the best items first and last.
5. **History**: `ConversationManager` summarizes the oldest 2 turns while history is over the cap.
6. **Budget**: `ContextBudget` truncates each component to its cap and to the space left.

## ContextEngine.assemble
type: annotated-code
lang: python
from: phases/11-llm-engineering/05-context-engineering/code/main.py, phases/11-llm-engineering/05-context-engineering/docs/en.md

```python
budget = ContextBudget(max_tokens=128000, generation_reserve=4000)
budget.allocate("system_prompt", self.system_prompt, max_tokens=1000)
tools, tool_tokens = select_tools(query, token_budget=2000)
budget.allocate("tools", tool_text, max_tokens=2000)
budget.allocate("retrieved_context", doc_text, max_tokens=3000)
budget.allocate("conversation_history", history_text, max_tokens=5000)
budget.allocate("user_query", query, max_tokens=500)  # query goes last
print(budget.report())  # tokens and percent per component
```

## Memory tool surface
type: command-table
from: phases/14-agent-engineering/07-memory-virtual-context-memgpt/docs/en.md, phases/14-agent-engineering/07-memory-virtual-context-memgpt/code/main.py, phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/docs/en.md

| Command | What it does |
|---|---|
| `core_memory_append(section, text)` | Writes to a pinned section of the prompt |
| `core_memory_replace(section, old, new)` | Edits a pinned section |
| `archival_memory_insert(text)` | Writes to the searchable external store |
| `archival_memory_search(query, top_k)` | Pages facts back into the next turn |
| `conversation_search(query)` | Scans past turns, evicted turns too |
| `block_append(label, text)` | Adds text to a memory block |
| `block_replace(label, old, new)` | Edits a memory block |
| `block_summarize(label)` | Condenses a block near its limit |

## Memory designs side by side
type: comparison
span: 2
from: phases/14-agent-engineering/07-memory-virtual-context-memgpt/docs/en.md, phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/docs/en.md, phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md, phases/14-agent-engineering/34-repo-memory-and-state/docs/en.md

| Factor | MemGPT two tiers | Letta blocks | Mem0 hybrid | Repo memory |
|---|---|---|---|---|
| Stores | Main context plus archival | Core, recall, archival | Vector, KV, graph | JSON files under a schema |
| Write | Memory tool call | Block tools and sleep-time rewrites | `add(text, user_id, metadata)` to all three | Validate, then atomic write |
| Read | `archival_memory_search` | Core blocks stay in the prompt | `search(query, user_id)` fuses scores | Load and validate |
| Stale facts | Consolidation, invalidation | Sleep-time agent invalidates | Old edge marked invalid, not deleted | `schema_version` refuses unknown files |
| Main risk | Memory rot, poisoning | Block bloat, silent drift | Embedding drift, graph explosion | Partial writes |
| Use it when | Data is larger than the window | Some facts must stay in the prompt | Queries mix similarity, lookup, relations | State must outlive the chat |

## Tasks in code
type: task-table
from: phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/code/main.py, phases/14-agent-engineering/09-hybrid-memory-mem0/code/main.py, phases/14-agent-engineering/34-repo-memory-and-state/code/main.py, phases/14-agent-engineering/34-repo-memory-and-state/docs/en.md

| Task | Code |
|---|---|
| Create a block | `create(label, description, limit=300)` |
| Test a block for summary | `block.near_limit()` |
| Consolidate off the critical path | `sleep.run(contradictions=[...])` |
| Soft delete a stale fact | `self.archival.invalidate(record.rid)` |
| Write to all three stores | `mem.add(text, user_id=..., kv_triples=...)` |
| Fused, scoped search | `mem.search(query, user_id=..., top_k=3)` |
| Persist state safely | `atomic_write(path, content)` |
| Refuse a bad write | `mgr.commit(state)` raises `SchemaError` |

## Memory server project
type: command-table
span: 2
from: projects/memory-server/README.md, projects/memory-server/solution/cli.ts

| Command | What it does |
|---|---|
| `python3 scripts/project_test.py memory-server --init <dir>` | Creates your workspace. A fresh starter fails |
| `python3 scripts/project_test.py memory-server --all --path <dir> --strict` | Grades all four stages |
| `node cli.ts --data-dir memory-data --put samples/memory.json` | Appends revision 1 of `cache-policy` |
| `node cli.ts --data-dir memory-data --query "cache policy"` | Searches the `docs` namespace |
| `node cli.ts --data-dir memory-data --put samples/memory-updated.json --revision 1` | Appends revision 2. Revision 0 again is a conflict |
| `node cli.ts --data-dir memory-data --history cache-policy --html history.html` | Prints all revisions and writes a comparison table |
| `node cli.ts --data-dir memory-data --serve --port 8788` | Serves REST and MCP. Set MEMORY_TOKEN first |

## Defaults
type: defaults
from: phases/11-llm-engineering/05-context-engineering/code/main.py, phases/11-llm-engineering/05-context-engineering/docs/en.md, phases/14-agent-engineering/07-memory-virtual-context-memgpt/code/main.py, phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/code/main.py, phases/14-agent-engineering/09-hybrid-memory-mem0/code/main.py

| Setting | Default | Note |
|---|---|---|
| `max_tokens` | 128000 | Minus `generation_reserve` 4000 |
| `max_history_tokens` | 5000 | Summarize old turns above it |
| `token_budget` | 2000 | Cap for tool definitions |
| Relevance `threshold` | 0.05 | The docs version uses 0.1 |
| `max_messages` | 4 | MemGPT main context. Older messages go to `evicted` |
| Block `limit` | 300 | Characters. `near_limit` fires at 0.8 |
| `w_relevance` | 0.6 | `w_importance` 0.2, `w_recency` 0.2 |
| `recency_halflife_s` | 86400.0 | Recency is 0.5 per half-life |

## Troubleshooting
type: troubleshooting
from: phases/11-llm-engineering/05-context-engineering/docs/en.md, phases/14-agent-engineering/07-memory-virtual-context-memgpt/docs/en.md, phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/docs/en.md, phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md, phases/14-agent-engineering/34-repo-memory-and-state/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Model ignores a fact in the middle | Lost in the middle | Key facts first, query last. Repeat the key point at the end |
| Tools fill the window | Every tool sent on every call | Classify intent and prune tools |
| Retrieval drowns in stale facts | Memory rot | Consolidate and invalidate in a sleep pass |
| Agent obeys an old injected note | Memory poisoning | Treat retrieved memory as untrusted text |
| Agent recalls but cannot cite | Citation loss | Store session ID and turn ID with each write |
| Block hits its cap | Block bloat | Summarize before the write that crosses the cap |
| One user sees another user's facts | Scopes mixed | Pick one scope for each write |
| Session resumes on corrupt state | Non-atomic write | Temp file, `fsync`, `os.replace` |

## Questions
type: qa
from: phases/11-llm-engineering/05-context-engineering/docs/en.md, phases/14-agent-engineering/07-memory-virtual-context-memgpt/docs/en.md, phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/docs/en.md, phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md, phases/14-agent-engineering/34-repo-memory-and-state/docs/en.md

- **Bigger window, no memory?** 128k-window baselines still miss long-horizon facts that a 4k-window agent with external memory catches.
- **Why sleep-time compute?** Primary turns do not wait. The sleep agent can use a stronger, slower model.
- **Why three stores?** A single store is wrong for two of three query classes.
- **How does fusion rank?** A weighted sum of relevance, importance and recency. Weights change per product.
- **What goes in repo memory?** Task id, touched files, assumptions, blockers, next action. Not raw chat transcripts.