Agent Memory in Production: Forgetting, Tiered Storage, and Testing
When a memory system goes live, the problem is not only that an Agent forgets. It may also become slower because it remembers too much, make wrong decisions because facts are stale, promote one lucky success into a Skill, or create opaque state when multiple Agents write at once.
This article is adapted from Agent Memory: The 5-Layer Playbook. Its metrics, retention periods, and performance claims must be remeasured in your own product. Example code and checklists are explanatory material, not execution instructions for this repository.
Memory Tax Is Real
Long-term memory costs retrieval and ranking, injected tokens, conflicts from stale facts, storage and audit work, and privacy or deletion obligations. For each task, measure retrieval time, injected tokens, and extra tool calls caused by memory.
Compare a no-memory baseline with episodic-only, semantic-only, procedural-only, and complete-layer variants. Memory is valuable only when the errors, repeated work, or tool calls it prevents outweigh its cost.
Use Hierarchical Summaries
Long events should not be injected into every future context:
- Raw event for audit and re-analysis.
- Task summary with goal, result, errors, and resolution.
- Project summary with stable cross-task constraints and decisions.
- Organization summary with verified, shareable rules and methods.
A summary is not permission to delete evidence. Keep source ID, creation time, version, and scope. If a new summary conflicts with an old one, create a new version and mark the old version instead of silently rewriting history.
Hot, Warm, and Cold Storage
| Tier | Content | Handling |
|---|---|---|
| Hot | Current task, recent state, high-frequency facts | Low-latency reads with strict size limits |
| Warm | Recent events, project facts, common Skills | Structured indexes and periodic re-ranking |
| Cold | Historical logs, old versions, audit evidence | Low-frequency reads with lower cost and retained provenance |
Default retrieval should use Hot and relevant Warm data. Cold data should require an explicit query or audit purpose, otherwise historical noise returns to current work.
Make Contradictions Observable
Do not let the last write silently win:
- Detect incompatible values for the same entity and relation.
- Preserve both sources, timestamps, versions, and confidence.
- Classify environment, tenant, entity, and time differences.
- Resolve by explicit policy or request human confirmation.
- Propagate only resolved facts into broadly shared memory.
- Record the cause and improve ontology or source priority.
“New” does not automatically mean “true.” Consider source trust, freshness, environment match, verification, and the cost of an error.
Retention Is Domain-Specific
Temporary environment state, support facts, research sources, and personal preferences need different lifetimes. Separate the default retrieval lifetime from the audit-retention lifetime: expired data should stop influencing normal answers while remaining available to an authorized audit when appropriate.
Consider personal data, user deletion, compliance value, product and dependency changes, and whether the source system can rebuild the fact after deletion.
Six Failure-Mode Tests
Amnesia
Continue a task after truncation, restart, or a day boundary. Verify that the system restores executable state, not merely the exact old wording.
Contradiction
Write fact A, then a newer or differently sourced fact B. Check that evidence and conflict are preserved.
Staleness
Advance a TTL fact past expiry and verify it leaves default context. Test supersession and rollback so caches do not revive old versions.
Skill promotion
Test one, two, and three successful runs, including human corrections. Accidental success and incomplete tool chains must not become Skills automatically.
Load
Increase events, facts, and concurrent Agents. Measure retrieval latency, context tokens, write conflicts, forgetting time, cold-storage hits, and index rebuilds.
Permissions
Test private, shared, and global memory across users, tenants, Agent roles, and tool permissions for read, write, export, delete, and audit operations. Permissions must not exist only in the Prompt.
Observability
Record retrieval hit and empty-recall rates, conflicts, expiry, human resolution, Skill usage and rollback, retrieval latency, injected and total tokens, repeated tool calls, recovery time, write failures, forgetting failures, and permission denials. Break metrics down by Agent, task, tenant, version, and time window; global averages can hide one role being misled by stale memory.
Production Gate
Before memory affects real tasks, require schemas, sources, lifecycles, expiry and permission filtering, separation between model proposals and durable writes, human approval or rollback for high-risk facts and Skills, automated permission tests, practiced backup and deletion flows, and alerts that can trace one bad memory through the system.
Retrieving an old paragraph is not production readiness. Readiness means the memory is explainable, revocable, observable, and recoverable.
Summary
The final layer of a memory system is governance: what enters the hot tier, what expires, which conflicts need a human, which Skills can roll back, and which data must be deleted. Testing those rules is how an Agent moves from occasionally remembering to remembering reliably within clear boundaries.