The Write Path: Search Before You Store

The Duplicate Spiral

An agent fleet ran for several months with a memory store that had no dedupe logic on the write path. Every time a session processed a post-task retro, it extracted lessons and wrote them as new memories. The first time the agent learned that a particular project uses kebab-case for filenames, it wrote one memory. The next session, it re-extracted the same lesson and wrote another. Within eight weeks, recall for any query about file naming returned five near-identical memories, each worded slightly differently, all with high confidence. Every other kind of memory — the corrections, the project context, the useful retros — was buried.

This is the duplicate spiral, and it is the most common failure mode in naive memory system implementations. The spiral does not require bugs. It requires only a write path that appends without checking.

The fix is architectural: the write path is a gate, not a log. Before any new memory enters the store, the path searches for near-duplicates. If a match exists above the similarity threshold, the new memory supersedes the old one instead of joining it. The number of memories about any given topic stays bounded — one live row at any time, with the full history preserved in the soft-deleted supersede chain.

The Gate: Embed, Search, Decide

The write path has four stages. Each is necessary. None is optional.

Stage 1: Capture. Not every session event is worth storing. The write path should only be triggered for specific signals: corrections received from a human, surprising failures, post-task retros that surface a lesson. Session trivia, intermediate reasoning steps within a task, and content derivable from the live repo do not belong in the memory store. Filtering at capture time is cheaper than cleaning up a polluted store later.

Stage 2: Embed the candidate. Embed the combined heading and content using the same model and dimensions used for recall: text-embedding-3-small, 1536 dimensions. Consistent vector space is critical — if the write path embeds differently than the recall path, near-duplicate detection produces incorrect matches.

Stage 3: Search the store. Run a cosine distance query against the live memories in the same namespace:

SELECT memory_id, heading, embedding <=> $1::vector AS distance
FROM memories
WHERE namespace = $2 AND deleted_at IS NULL
ORDER BY embedding <=> $1::vector ASC
LIMIT 3

Then filter the results: keep only rows where distance < 0.08 (similarity > 0.92). A distance below 0.08 means the vectors are very close — these are near-paraphrases of the same fact.

Stage 4: Dedupe gate. If any near-duplicates are found, take the closest one and supersede it. If none are found, insert new.

The distance threshold requires judgment. A threshold too strict (0.03) will miss true duplicates phrased differently. A threshold too loose (0.20) will collapse distinct but related memories into a single row. For most knowledge types, 0.08 is a reasonable starting point. Semantic memories about the same system component often cluster tightly; episodic memories about distinct events cluster loosely. Namespacing by project prevents cross-project false positives.

The Supersede Chain

The supersede pattern is how the write path makes updates auditable without losing history. When a new memory replaces an existing one:

1. INSERT the new memory with supersedes = existing_memory_id
2. Only after a successful insert: UPDATE the old memory, setting deleted_at = NOW()

Order matters. Insert first. If the insert fails, the old memory is intact. If you delete first and the insert fails, the fact is gone — permanently, unless you have a backup. The supersedes column creates a forward link from new to old. Walking the chain from the current live row back through supersedes gives you the complete history of a belief.

Recall queries always filter WHERE deleted_at IS NULL. The superseded ancestors are invisible to normal recall. But a history query — "what did the agent believe about the deploy pipeline in January?" — can walk the supersedes chain backward to find the answer. Hard deleting destroys this trail. Soft delete preserves it at negligible storage cost.

What Not to Write

The write path has a discipline on the inbound side. Certain categories of content should never enter the memory store:

Facts derivable from the live repo or docs. If the answer is findable in the current API documentation or the project's source code, it belongs to the RAG corpus — not the memory store. The RAG corpus is designed to ingest, version, and retrieve corpus documents. The memory store is for lessons the agent earned through experience. Putting API documentation in the memory store creates staleness problems: the documentation is updated in the repo but the memory is not re-embedded, so the agent gets contradictory signals.

Session trivia. The fact that the agent ran a particular query at 14:33 UTC on a given day is not a lesson. The memory store should contain distilled knowledge, not a raw event log.

Secrets and credentials. Never. Regardless of encryption or namespace isolation. If a key rotation is needed, the memory store is not a safe or appropriate location.

Intermediate reasoning steps. If the agent is reasoning through a multi-step problem within a single task, that reasoning belongs in working memory with a short TTL — or it belongs in the context window alone, never persisted. Storing every thought produces noise that drowns out signal.

The write triggers are: post-task retros, corrections received from humans, surprising failures worth learning from, and patterns observed across multiple sessions. These are the signals that compound into a better agent. Everything else is noise.