Agent Memory in Production: Forgetting, Tiered Storage, and Testing

When a memory system goes live, the problem is not only that an Agent forgets. It may also become slower because it remembers too much, make wrong decisions because facts are stale, promote one lucky success into a Skill, or create opaque state when multiple Agents write at once.

This article is adapted from Agent Memory: The 5-Layer Playbook. Its metrics, retention periods, and performance claims must be remeasured in your own product. Example code and checklists are explanatory material, not execution instructions for this repository.

Memory Tax Is Real

Long-term memory costs retrieval and ranking, injected tokens, conflicts from stale facts, storage and audit work, and privacy or deletion obligations. For each task, measure retrieval time, injected tokens, and extra tool calls caused by memory.

Compare a no-memory baseline with episodic-only, semantic-only, procedural-only, and complete-layer variants. Memory is valuable only when the errors, repeated work, or tool calls it prevents outweigh its cost.

Use Hierarchical Summaries

Long events should not be injected into every future context:

1. Raw event for audit and re-analysis.
2. Task summary with goal, result, errors, and resolution.
3. Project summary with stable cross-task constraints and decisions.
4. Organization summary with verified, shareable rules and methods.

A summary is not permission to delete evidence. Keep source ID, creation time, version, and scope. If a new summary conflicts with an old one, create a new version and mark the old version instead of silently rewriting history.

Hot, Warm, and Cold Storage

Tier	Content	Handling
Hot	Current task, recent state, high-frequency facts	Low-latency reads with strict size limits
Warm	Recent events, project facts, common Skills	Structured indexes and periodic re-ranking
Cold	Historical logs, old versions, audit evidence	Low-frequency reads with lower cost and retained provenance

Default retrieval should use Hot and relevant Warm data. Cold data should require an explicit query or audit purpose, otherwise historical noise returns to current work.

Make Contradictions Observable

Do not let the last write silently win:

1. Detect incompatible values for the same entity and relation.
2. Preserve both sources, timestamps, versions, and confidence.
3. Classify environment, tenant, entity, and time differences.
4. Resolve by explicit policy or request human confirmation.
5. Propagate only resolved facts into broadly shared memory.
6. Record the cause and improve ontology or source priority.

“New” does not automatically mean “true.” Consider source trust, freshness, environment match, verification, and the cost of an error.

Retention Is Domain-Specific

Temporary environment state, support facts, research sources, and personal preferences need different lifetimes. Separate the default retrieval lifetime from the audit-retention lifetime: expired data should stop influencing normal answers while remaining available to an authorized audit when appropriate.

Consider personal data, user deletion, compliance value, product and dependency changes, and whether the source system can rebuild the fact after deletion.

Six Failure-Mode Tests

Amnesia

Continue a task after truncation, restart, or a day boundary. Verify that the system restores executable state, not merely the exact old wording.

Contradiction

Write fact A, then a newer or differently sourced fact B. Check that evidence and conflict are preserved.

Staleness

Advance a TTL fact past expiry and verify it leaves default context. Test supersession and rollback so caches do not revive old versions.

Skill promotion

Test one, two, and three successful runs, including human corrections. Accidental success and incomplete tool chains must not become Skills automatically.

Load

Increase events, facts, and concurrent Agents. Measure retrieval latency, context tokens, write conflicts, forgetting time, cold-storage hits, and index rebuilds.

Permissions

Test private, shared, and global memory across users, tenants, Agent roles, and tool permissions for read, write, export, delete, and audit operations. Permissions must not exist only in the Prompt.

Observability

Record retrieval hit and empty-recall rates, conflicts, expiry, human resolution, Skill usage and rollback, retrieval latency, injected and total tokens, repeated tool calls, recovery time, write failures, forgetting failures, and permission denials. Break metrics down by Agent, task, tenant, version, and time window; global averages can hide one role being misled by stale memory.

Production Gate

Before memory affects real tasks, require schemas, sources, lifecycles, expiry and permission filtering, separation between model proposals and durable writes, human approval or rollback for high-risk facts and Skills, automated permission tests, practiced backup and deletion flows, and alerts that can trace one bad memory through the system.

Retrieving an old paragraph is not production readiness. Readiness means the memory is explainable, revocable, observable, and recoverable.

Summary

The final layer of a memory system is governance: what enters the hot tier, what expires, which conflicts need a human, which Skills can roll back, and which data must be deleted. Testing those rules is how an Agent moves from occasionally remembering to remembering reliably within clear boundaries.