ASK KNOX
beta
Learn/Agent Memory 101/Lesson 346
LESSON 346

RAG vs Structured Facts: Two Kinds of Memory

Vector search and key-value stores are both 'memory,' but they solve fundamentally different problems — knowing which to reach for changes the quality of every AI system you build.

9 min read·Agent Memory 101

Introduction

You know the model does not remember between sessions, and you know the context window has a ceiling. The obvious next question is: where does memory actually live? The answer splits into two fundamentally different systems, and the most common beginner mistake is using one where the other belongs.

RAG (Retrieval-Augmented Generation) and structured fact stores are both memory systems for AI agents. They solve different problems. Getting this wrong produces either slow, expensive systems or unreliable recall that costs you trust in the AI.

Core Concept

RAG: Fuzzy Semantic Recall

RAG stands for Retrieval-Augmented Generation. Here is the mechanics stripped to essentials:

  1. You have a large collection of text — documents, notes, conversation summaries, knowledge articles.
  2. At storage time, each piece of text is converted into a vector embedding: a list of numbers that captures the semantic meaning of the text.
  3. At retrieval time, your query is also converted into an embedding. The system finds stored chunks whose embeddings are mathematically close to your query's embedding.
  4. The closest matches are returned and injected into the model's context window.

The key word is semantic. RAG does not match on exact words. It matches on meaning. A query for "how do I handle login errors" might retrieve a chunk that says "authentication failure handling" — different words, same concept.

This is powerful for:

  • Searching large bodies of text you cannot load into context at once
  • Answering questions when you do not know exactly how the relevant information was worded
  • Surfacing relevant knowledge from hundreds of past conversations or documents
  • Any "what did we say about X?" type of recall

The semantic memory layer — the "knowledge OS," a persistent memory system you will build in this track's final lesson — is built on this pattern. It stores thousands of chunks across 40+ repositories and surfaces the right ones when an agent queries "agent memory structured facts trust scores" even though no stored document uses those exact words together.

Structured Fact Stores: Precise Key-Value Recall

A structured fact store is what it sounds like: a database of facts where you look things up by a specific key. Think of it as a dictionary for your AI agent.

Examples of entries in a structured fact store:

  • owner_of_task_123 → "sarah@company.com"
  • api_retry_policy → "max 3 retries, exponential backoff, cap at 60s"
  • preferred_date_format → "ISO 8601 (YYYY-MM-DD)"
  • project_status → "in review, blocked on legal approval since 2026-05-15"

The retrieval is exact. You ask for owner_of_task_123, you get sarah@company.com. No fuzziness. No semantics. No chance of a wrong answer due to similarity scoring.

This is the right tool for:

  • Configuration and preferences that must be recalled precisely
  • Ownership and assignment records
  • Decisions that must be respected exactly ("always use USDC.e, not native USDC")
  • Status flags and state that the agent acts on

The Decision Rule

When you are designing what kind of memory a fact should live in, ask one question: does the agent need to find this when it does not know the exact key?

If yes → RAG. The agent is exploring: "what do we know about authentication?" It cannot predict the exact key or the exact words. Semantic search finds relevant chunks.

If no → structured fact store. The agent knows exactly what it needs: the owner of this task, the configured timeout, the API base URL. Exact lookup is faster, cheaper, and more reliable.

A mature memory system uses both. Structured facts handle configuration and precise assignments. RAG handles knowledge retrieval from larger corpora of text.

Practical Application

Suppose you are building a personal AI assistant that helps you manage your consulting practice. You have:

  • 200 past meeting notes (each 500 words)
  • 15 client configuration records (each client's billing rate, preferred contact, project status)
  • 80 decisions you have made over two years about how you run the business

The 15 client configuration records belong in a structured fact store. When your AI prepares a client brief, it should look up client:acme_corp and get exact values: billing rate, contact name, current project status. Fuzzy semantic search here would be dangerous — imagine the AI confusing two similar-sounding client names and pulling the wrong billing rate.

The 200 meeting notes belong in a RAG store. When you ask "what did I decide about project deadlines for Acme?" the AI cannot know which specific meeting note to retrieve. It embeds the query and finds semantically relevant notes from across all 200 documents.

The 80 business decisions are a judgment call: if they are structured rules ("I always charge a 20% rush fee for next-day turnaround"), they belong in the structured store as explicit facts. If they are contextual narratives ("in early 2025 I experimented with retainer pricing and found it worked better for long-term clients"), they belong in RAG where the full context is part of the value.

Common Mistakes

Using RAG for everything. RAG is the more exciting technology, so beginners reach for it by default. But RAG on a fact like "the API retry limit is 3" is overkill and introduces error. A vector search might return a slightly different document that says "retry up to 5 times" depending on query phrasing. For precision, use a structured store.

Using a structured store for unstructured knowledge. The inverse: trying to look up "what did the team decide about authentication" with a fixed key when you do not know the key. You will get nothing back or need to manually index every possible query in advance. That is not scalable.

Conflating embedding quality with retrieval quality. RAG is only as good as your chunking strategy and your embeddings. Dumping entire documents in as single chunks produces poor retrieval — the semantic signal gets diluted. Splitting documents into coherent 200-500 token chunks dramatically improves RAG quality.

Not updating structured facts. A structured fact store that is not kept current becomes a liability. If project_status says "in review" when the project shipped last month, the AI will act on stale data. Structured stores need a clear ownership and update process.

Summary

  • RAG retrieves by semantic meaning — good for searching large text corpora when you cannot predict the exact words
  • Structured fact stores retrieve by exact key — good for precise, stable facts that must be recalled reliably
  • The decision rule: does the agent need to find this without knowing the exact key? Yes → RAG. No → structured store
  • Most real memory systems use both: structured facts for configuration and assignments, RAG for knowledge retrieval
  • RAG quality depends on chunking strategy — smaller, coherent chunks outperform large blobs

What's Next

Knowing the tools is the start. In the next lesson, we cover the most important discipline in memory-augmented AI work: the query-first habit. Before the agent does anything, it recalls what it already knows. This single workflow change eliminates the most common failure mode — re-solving problems that were already solved.