Level 6: Memory That Survives the Session

Introduction

Every session ends. The context window clears. All the decisions you made, all the preferences you expressed, all the operational facts the agent learned — gone.

This is the default state of every LLM-based agent. It is also one of the most significant limits on what agents can do in practice. An agent that does not remember cannot learn. And an agent that cannot learn gives you the same output quality on day 300 as it did on day 1.

Level 6 is persistent memory — but not the kind you might assume. It is not a vector database that semantically searches your conversation history. It is something more specific and more powerful: a structured fact store where discrete knowledge persists across sessions, carries a , and can detect when you contradict yourself.

Core Concept

Why Pure RAG Is Not Enough

RAG (retrieval-augmented generation) is the standard approach to giving agents access to prior knowledge. You embed your conversation history or documents into a vector database. When the agent needs context, it retrieves semantically similar chunks and stuffs them into the prompt.

RAG is excellent for some things. It is not the right tool for all memory problems.

Consider this scenario: you told your agent last week that your file backup runs "every 5 days." Today you say it should run "every 3 days." In a RAG system, both statements exist as chunks in the vector store. When the agent queries for backup configuration, it might retrieve either one — or both — and have no systematic way to know which is current.

A structured fact store solves this. The fact "backup_frequency" exists as a named, typed entry. When you say "3 days," the store finds the existing entry, detects the conflict, and surfaces it: "You previously stored backup_frequency = 5 days. You are now saying 3 days. Which is correct?" You resolve it; the store updates. Now there is one source of truth.

This distinction is not academic. For agents that make operational decisions based on stored knowledge — scheduling tasks, routing work, configuring systems — having stale or contradictory facts in memory produces real errors.

The Structured Fact Memory Architecture

The Agent Framework calls its level-6 implementation "Structured Fact Memory." The name reflects the design goal: every fact is independently accessible and verifiable, not buried inside a long document that requires semantic search to find.

The core data model is straightforward:

@dataclass
class Fact:
    key: str            # e.g., "backup_frequency"
    value: str          # e.g., "every 3 days"
    trust_score: float  # 0.0 (uncertain) to 1.0 (high confidence)
    source: str         # "user_stated" | "agent_inferred" | "system_verified"
    created_at: datetime
    updated_at: datetime
    superseded_by: Optional[str] = None  # points to newer fact key if deprecated

The trust_score encodes how confident the agent should be in a fact:

1.0 — user explicitly stated and confirmed; system verified via test
0.8 — user stated once, not tested; high confidence but not verified
0.5 — agent inferred from behavior patterns; reasonable but unconfirmed
0.2 — agent guessed based on limited evidence; should be verified before acting

When the agent makes a decision based on a low-trust-score fact, it should surface that uncertainty: "I am scheduling the backup for every 3 days based on your message, but I have not confirmed this is still correct — do you want to verify?"

Session-End Auto-Extraction

The agent should not require you to manually say "remember this" for every important piece of information. Session-end auto-extraction handles this automatically.

At the end of each session, the agent scans the conversation and identifies:

Explicit preferences stated ("I prefer X to Y")
Configuration values set ("set the threshold to 0.75")
Decisions made ("we decided to retire the old bot")
Named entities established ("the new repo is called trading-bot")

Each identified fact is stored with an appropriate trust score based on how it was established. Facts stated explicitly by the user get 0.8; facts confirmed by the user in response to agent inference get 1.0; facts the agent inferred without explicit confirmation get 0.5.

In practice, a production semantic memory layer performs this role across an entire ecosystem: 2,968 chunks indexed across 43 repos, with semantic search for retrieval but also structured memory for key operational facts. The session-end pattern is the foundation — every significant session adds to the knowledge base rather than evaporating when the context window closes.

Contradiction Detection

The mechanism is what separates this pattern from a simple key-value store.

When a new fact is stored, the system checks for semantic near-matches among existing facts. If it finds one with a conflicting value, it pauses and asks:

Memory conflict detected:

Stored: backup_frequency = "every 5 days" (trust: 0.8, stored 2026-04-12)
New:    backup_frequency = "every 3 days" (source: current session)

Which is correct?
  A) Keep "every 5 days"
  B) Update to "every 3 days"
  C) Both are correct (different contexts)

This surfaces the conflict before it becomes a silent error. In a system that schedules backups based on the stored frequency, a stale fact is not just incorrect — it is operationally dangerous.

Seeding the system is fast: in the Agent Framework documentation, a user loaded 15 facts from prior sessions in a single pass and had a functional memory foundation immediately. You do not need hundreds of stored facts to get value — 15-30 operational facts covering your core preferences and configuration values are enough to change how the agent operates.

Practical Application

Here is the minimal SQLite schema for a local structured fact store:

CREATE TABLE facts (
    id          TEXT PRIMARY KEY,
    key         TEXT NOT NULL,
    value       TEXT NOT NULL,
    trust_score REAL NOT NULL DEFAULT 0.8,
    source      TEXT NOT NULL,
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL,
    active      INTEGER NOT NULL DEFAULT 1
);

CREATE UNIQUE INDEX idx_facts_key_active
    ON facts (key) WHERE active = 1;

CREATE TABLE fact_history (
    id          TEXT PRIMARY KEY,
    fact_id     TEXT NOT NULL REFERENCES facts(id),
    old_value   TEXT,
    new_value   TEXT,
    changed_at  TEXT NOT NULL,
    change_reason TEXT
);

The UNIQUE INDEX ... WHERE active = 1 ensures only one active value per key while preserving history. The fact_history table gives you an audit trail for every update — essential for debugging contradiction resolution.

Query pattern for agent context assembly:

SELECT key, value, trust_score
FROM facts
WHERE active = 1
  AND trust_score >= 0.5
ORDER BY trust_score DESC;

High-trust facts go first. Low-trust facts go last or are omitted from context to save tokens.

Common Mistakes

Treating memory as an append-only log. If you only add facts and never update or supersede them, your memory fills with contradictions. Contradiction detection and update mechanics are requirements, not optional features.

Storing everything. Not every session produces facts worth persisting. Storing "the user said hello" or "the user asked about the weather" generates noise that degrades retrieval quality. Be selective: persist preferences, decisions, configuration values, and named entities. Filter out conversational filler.

Setting trust scores to 1.0 uniformly. If everything has maximum trust, the scores carry no information. Use the gradient: verified-by-system facts earn 1.0; user-stated facts earn 0.8; agent-inferred facts earn 0.5. The gradient is the signal.

Skipping the RAG layer entirely. Structured fact memory and RAG are complementary, not competing. Use structured memory for discrete, named facts where contradiction detection matters. Use RAG for retrieving relevant context from past conversations and documents. Both layers belong in a complete memory architecture.

Summary

RAG retrieves semantically similar text; structured fact memory stores discrete facts with metadata — both belong in a complete memory architecture
Trust scores (0.0-1.0) encode confidence: verified facts get 1.0, agent-inferred facts get 0.5
Contradiction detection surfaces conflicts before they become silent operational errors
Session-end auto-extraction populates memory without requiring manual "remember this" commands
Seed with 15-30 core operational facts to immediately change agent behavior
Do not treat memory as append-only; update and supersede facts as ground truth changes

What's Next

The next lesson brings it all together at level 7: exposing your persistent agent as an MCP server. This is the capstone — your coding tools can call the agent, destructive operations route through your phone for approval, and long-running jobs report progress while you are away from your laptop.