Memory Ops: Health, Drift & Observability
A process can run while the store rots — health means the artifact is working, not just the service; four metrics tell you the difference.
The Gap Between Running and Working
A 24/7 agent platform was producing accurate recall metrics. The API responded to every request. Session hydration completed in under 150ms. Health checks passed. The service was running.
Then someone looked at the store directly. Working memories with TTLs from six weeks prior were still live — 84 of them. A batch of procedural rules had been updated with corrected wording, but the embed job had silently failed three weeks ago. The embedding column was three weeks behind the content — the stored vectors still encoded the old wording. Recall was matching against those stale vectors, missing the updated rules entirely.
The process was healthy. The artifact was not.
This is the central distinction in memory ops: a process check and an artifact check are different things. A process check tells you the service is running. An artifact check tells you the store is correct, the index is current, and the recall it produces is accurate. You need both. Most systems only build the first.
The Four Health Metrics
Four metrics provide a complete picture of memory store health. None of them are process metrics. All of them measure the artifact.
Staleness distribution measures the age of the last verification for each live memory, grouped by type. Semantic memories (facts about how the world works) decay faster than procedural memories (rules for how to act) — the world changes, and a semantic memory that was accurate when written may no longer be. Staleness is not automatically a problem, but a memory in the critical bucket (> 90 days unverified) is a candidate for manual review. A semantic memory about a specific API endpoint or a tool's behavior deserves scrutiny after 90 days.
Duplicate rate measures near-match density: what fraction of live memories with embeddings have at least one other live memory within cosine distance 0.12. The sweep threshold is deliberately looser than the write path's 0.08 dedupe gate: the gate must be conservative because it auto-supersedes inline at write time, while this offline sweep is a detection net — it should also surface the borderline pairs (distance 0.08–0.12) that a healthy gate would have let through, so an operator can review them. A rate above 5% signals a broken write path. The duplicate spiral from the write-path lesson — where the same lesson gets written three times across three sessions — produces exactly this signal at the store level. The health metric surfaces what the write-path telemetry missed.
Recall hit rate requires a recall_log table that records which memories were hydrated at session start and which were actually referenced during the session. A hit rate below 40% means the hydration ranking is returning memories that are not relevant to the actual session work — the semantic search is not being given a specific enough query, or the recency weighting is dominating over relevance. A hit rate above 80% can signal the opposite: the hydration budget is too conservative and relevant memories are being left out.
Index drift measures the gap between what recall should see and what it actually sees. It has three components: rows with null embeddings (invisible to recall — there is no vector to search), rows updated after the last successful embed job (stale embeddings — the stored vector still encodes the old text), and expired memories still live (expires_at has passed but the hygiene job has not yet soft-deleted them, so any recall path that omits the expires_at guard keeps returning them as current facts). Note what is not on this list: the HNSW index itself falling behind the table. pgvector's HNSW index is a regular Postgres index, maintained transactionally with every INSERT and UPDATE — it cannot lag the store. Drift lives in the embedding column (content changed, vector did not) and in lifecycle state (rows that should be retired but are not), never in index lag. One column contract keeps the stale-embeddings check trustworthy: only content changes touch updated_at. The hygiene job's confidence decay records its passes on the dedicated confidence_updated_at column (the hygiene lesson) — if decay bumped updated_at, every decay pass would falsely flag aged rows as stale embeddings and burn the embed job on content that never changed.
The drift detection flow scans the store against these four categories and produces a structured report. Missing embeddings and expired-but-live rows are immediate remediation priorities. Stale embeddings queue for the next embed job. Soft-deleted rows surfacing in results mean some recall path is missing the WHERE deleted_at IS NULL filter — soft-deleted rows stay in the table and in the index by design, and the filter is the only thing that excludes them. Fix the query, not the index.
Logging Recall
Recall observability starts with logging. Every recall call should produce a structured log entry:
interface RecallLog {
session_id: string
query: string
memory_ids_returned: string[]
memory_ids_used: string[] // populated after session ends
timestamp: string
}
The memory_ids_used field is populated during the post-session retro pass. The retro pipeline that extracts lessons (the compound-learning lesson) can also annotate the recall log — if a hydrated memory influenced a decision or was explicitly cited in the agent's work, it counts as used.
This log is what makes the recall hit rate metric computable. Without it, you can measure hydration but not utility.
The recall log also surfaces a subtler problem: the agent citing memories that were not hydrated. This means the agent is reasoning from training data rather than from the store — the recall path is being bypassed. Log the divergence; it signals either a hydration gap (the relevant memory is in the store but scored below the retrieval threshold) or a confidence calibration problem.
The Backup and Reindex Cycle
Two operational routines complete the ops picture.
Backup is straightforward for Postgres: logical backups via pg_dump, scheduled to a durable storage target. The pgvector embeddings are stored as vector(1536) columns in the same table — they back up with the row data. The HNSW index is rebuilt on restore (it cannot be logically dumped). Schedule backups ahead of any destructive hygiene operation.
Reindex is a maintenance operation, not a drift fix. Because pgvector's HNSW index is maintained transactionally, REINDEX INDEX exists for index bloat or suspected corruption — and note that a rebuild re-indexes every row in the table, soft-deleted rows included; only the WHERE deleted_at IS NULL filter keeps those out of results. A rebuild also cannot repair drift: it reads the same stale embedding column the old index pointed at. Drift remediation is the embed job's work — re-compute embeddings for rows whose content changed since the last successful run, and the index updates transactionally as each row is written. For small stores (< 50K memories), a nightly REINDEX for bloat is cheap; just do not confuse it with freshness.
The trigger for remediation is the drift detection flow: when missing or stale embeddings exceed a threshold, schedule an embed job run.
Connecting to the RAG Track
If you completed the rag-in-production track, the health patterns here are directly analogous. The RAG re-embed lifecycle (the re-embed lifecycle lesson) used checksums to detect changed chunks and trigger incremental re-embedding. The same principle applies here: when a memory row's content changes, its embedding becomes stale, and the recall returns a vector that no longer matches the current text.
The difference is the write pattern. RAG chunks are ingested; memory rows are written by the agent itself. The ingest trigger for RAG was a document update; the embed trigger for memories is any row update. In both cases, freshness is the health metric — and a stale index is invisible to the service health check.
A production tutor running over a corpus like this one would schedule these four metrics the same way: staleness distribution checked nightly, duplicate rate checked after every batch write, index drift checked before every embed job, and recall hit rate computed from the session log after each day's interactions. None of those checks ask whether the service is up — all four ask whether the artifact is healthy.
Building the Health Metrics Job
The challenge for this lesson implements all four health computations against a sample in-memory store. The staleness distribution groups live rows by memory type and buckets them by age. The duplicate rate iterates each row and checks for near-matches. The index drift check compares updated_at against the last embed job completion timestamp. The expired-but-live count filters rows where expires_at is in the past and deleted_at is null.
The mock store is pre-populated with rows that exercise every metric: a row with a null embedding, a duplicate pair, an expired working memory, and rows with a range of ages. The solution assembles all four into a structured HealthReport with an overall_health classification.