Grounded Generation: Answers That Show Receipts

The Citation Contract

An AI Q&A assistant on a personal site once had this property: it produced fluent, confident answers. Users trusted those answers. None of the answers showed which sources they came from.

The system did actually support citations. The answer model returned a structured JSON payload: an answer field and a citations array. But somewhere in the parsing layer, a one-character bug had turned citations into an empty array for every response — the catch block swallowed the JSON parse error, defaulted citations to [], and returned the answer unchanged. The UI showed answers without source chips. Every mocked test passed because the mocks returned well-formed payloads that the real model never produced under load. For weeks, users received answers with no receipts.

Citations are not a nice-to-have. They are a contract. Every claim in an AI answer should be traceable to a chunk that was retrieved and passed to the model. The contract has three parts: (1) the prompt assembles chunks with their source ids inlined, (2) the model returns a structured output that maps claims to source ids, and (3) the parser verifies citations are present — alerting immediately if they are not.

Prompt Assembly Order

The assembly order of the prompt is load-bearing. Get it wrong and citation accuracy degrades significantly.

1. System contract     (stable, cacheable — ~0.1× cost on cache reads)
2. Retrieved chunks    (each labeled with source id and chunk id)
3. User question       (last — query goes here, after all context)

The system contract establishes the rules the model must follow. The retrieved chunks provide the evidence the model must cite from. The user question comes last — after the model has processed all context. Placing the question first causes the model to "answer" before reading the evidence, which degrades both accuracy and citation fidelity.

Inline the source ids directly with their chunks. The following format gives the model what it needs to cite accurately:

// Assemble chunks with inline source ids
const chunkContext = chunks
  .map((c) =>
    `[source: ${c.source_id}, chunk: ${c.chunk_id}] ${c.content}`
  )
  .join("\n\n")

const systemPrompt =
  "You are a knowledge assistant. Answer ONLY from the provided sources. " +
  "If the sources do not contain the answer, say so explicitly. " +
  'Return JSON: { "answer": string, "citations": [{ "source_id": string, "chunk_id": string, "quote": string }], "confidence": "high"|"medium"|"low" }'

When the source id is adjacent to the content it describes, the model's attention can connect them naturally. A separate list of ids at the end of the prompt forces the model to maintain a cross-reference across the full context window — a much harder task, especially at long context lengths.

The Structured Output Schema

The response shape is a contract, and it must satisfy the structured-output validator of the model API you generate with (the schema constraints each provider enforces are documented in its structured-output / response-format reference). For the API used in this track, the schema must be validator-legal: additionalProperties: false everywhere, no minItems, maxItems, minLength, maxLength, minimum, or maximum constraints — the validator rejects those. An enum on a field (for example, constraining confidence to "high" | "medium" | "low") is permitted; it is the count- and range-bound keywords above that are not. The capstone uses exactly such an enum on its confidence field.

// Validator-legal schema for grounded answers
const ANSWER_SCHEMA = {
  type: "object",
  additionalProperties: false,
  properties: {
    answer: { type: "string" },
    citations: {
      type: "array",
      items: {
        type: "object",
        additionalProperties: false,
        properties: {
          source_id: { type: "string" },
          chunk_id:  { type: "string" },
          quote:     { type: "string" },
        },
        required: ["source_id", "chunk_id", "quote"],
      },
    },
    confidence: { type: "string" },
  },
  required: ["answer", "citations", "confidence"],
}

The confidence field encodes what the retrieved sources actually support — "high" means the sources directly answer the question, "medium" means the sources are relevant but partial, "low" means the sources touch the topic but do not conclusively answer it. Confidence is not the model's self-reported certainty about its own generation — it is a signal about source coverage.

Defensive Citation Parsing

The parsing layer is where the contract is either upheld or broken. The rules are simple but not optional:

// lib/parse-citations.ts
interface ParseResult {
  answer: string
  citations: Citation[]
  confidence: "high" | "medium" | "low"
  parseOk: boolean
}

function parseCitedAnswer(
  rawText: string,
  retrievedSourceIds: Set<string>,
  query: string
): ParseResult {
  // 1. Extract JSON from the response text
  const jsonMatch = rawText.match(/\{[\s\S]*\}/)
  if (!jsonMatch) {
    console.error("citation-parse-failure: no JSON found", { query: query.slice(0, 80) })
    return { answer: "", citations: [], confidence: "low", parseOk: false }
  }

  // 2. Parse defensively
  let parsed: { answer: string; citations: Citation[]; confidence: string }
  try {
    parsed = JSON.parse(jsonMatch[0])
  } catch (err) {
    // NEVER silently swallow — log immediately
    console.error("citation-parse-failure: JSON.parse threw", { query: query.slice(0, 80), err: String(err) })
    return { answer: "", citations: [], confidence: "low", parseOk: false }
  }

  // 3. Alert on empty citations — never silently return zero receipts
  if (!parsed.citations || parsed.citations.length === 0) {
    console.warn("empty-citations", {
      query: query.slice(0, 80),
      answer: parsed.answer?.slice(0, 80),
    })
  }

  // 4. Validate source ids against what was actually retrieved
  const validCitations = (parsed.citations ?? []).filter((c) => {
    const valid = retrievedSourceIds.has(c.source_id)
    if (!valid) {
      console.warn("hallucinated-citation", { source_id: c.source_id, query: query.slice(0, 80) })
    }
    return valid
  })

  return {
    answer: parsed.answer ?? "",
    citations: validCitations,
    confidence: (parsed.confidence as "high" | "medium" | "low") ?? "low",
    parseOk: true,
  }
}

Step 4 — validating citation source ids against the set of chunk ids that were actually sent to the model — catches hallucinated citations before they reach the UI. If the model invents a source_id that sounds plausible but was not in the retrieved context, this check removes it and logs the hallucination. An answer with no citations after validation is returned to the caller clearly labeled as such, not silently rendered as a receipts-free response.

The Thin-Retrieval Refusal Path

The retrieval-quality lesson introduced the similarity threshold gate. When no chunks pass that threshold, there is nothing reliable to generate from. The generation layer must be wired to respect this signal:

async function generateWithReceipts(
  query: string,
  chunks: RetrievedChunk[],
  minChunks: number
): Promise<AnswerWithReceipts> {
  // Wire the threshold from retrieval into the generation refusal path
  if (chunks.length < minChunks) {
    return {
      answer: "I don't have that in my sources. Try rephrasing or check if the topic is covered.",
      citations: [],
      confidence: "low",
      refused: true,
      refusalReason: "insufficient-sources",
    }
  }

  // ... proceed with generation only when retrieval is adequate
}

"I don't have that in my sources" beats a confident hallucination every time. Users who receive a refusal can rephrase, consult another source, or recognize the limits of the system. Users who receive a confident wrong answer may act on it. The refusal is the honest answer.

Testing Against Real Payloads

The dead-source-chips incident happened because every test used mocks that returned well-formed JSON — the exact format that never caused a parse error. Real model responses under load, at edge cases, or after a model version change sometimes deviate from the expected format. The only tests that catch the real failure mode are tests that run against captured real payloads from the production endpoint.

// test/citation-parser.test.ts
// Load these from actual captured responses — not from constructed mocks

import { parseCitedAnswer } from "../lib/parse-citations"

const REAL_PAYLOAD_WITH_TRAILING_BRACKET =
  '{"answer":"The threshold controls...", "citations":[{"source_id":"lesson-580","chunk_id":"3","quote":"similarity >= threshold"}]'
  // Note: missing closing } — real truncation seen in production

test("trailing-bracket payload does not silently return zero citations", () => {
  const result = parseCitedAnswer(
    REAL_PAYLOAD_WITH_TRAILING_BRACKET,
    new Set(["lesson-580"]),
    "what is a threshold?"
  )
  // Should return parseOk: false, not silently succeed with empty citations
  expect(result.parseOk).toBe(false)
  expect(result.citations).toHaveLength(0)
  // The failure must be logged, not swallowed
})

The academy's own AI tutor maintains a fixture file of edge-case real payloads captured from production. These run in CI on every prompt template change. If the template change produces a response shape that breaks parsing, the test fails before the change reaches users.

What's Next

The re-embed lifecycle lesson closes the loop on the re-embed lifecycle. You have retrieval quality gates and citation contracts. But if your index is stale — if documents were edited days ago and the vectors were never updated — all of this machinery is retrieving from outdated content. The index is a cache of the corpus, and every cache needs invalidation. The re-embed lifecycle lesson is that invalidation story.