Evaluating Prompts — Golden Sets & Regression Testing

The prompt library lesson established that prompts need versioning. But versioning without evaluation is just a changelog with no way to know whether the new version is better, worse, or subtly broken. You can roll back to v2.3 — but how do you know v2.4 regressed in the first place?

The answer is the same one software engineering arrived at decades ago: tests. A golden set of representative fixtures, a runner that scores each output against declared assertions, and a CI gate that blocks the merge if anything that was passing is now failing.

What a Golden Set Is

A golden set is a fixed corpus of input→expected-property pairs that represent the task your prompt is supposed to handle. It is not a record of what the model once happened to output — it is a specification of what the output must satisfy.

Each fixture contains:

1. A unique ID — fixture-001, summarize-earnings-q1, classify-complaint-billing. Human-readable so that a failing fixture name tells you immediately which part of your prompt's behavior broke.

2. An input — a representative example of real production input. The broader the coverage across input types, the more regression signal the set provides.

3. Assertions — not expected verbatim text (with rare exceptions), but properties the output must satisfy:

   • contains: "quarterly revenue" — the output mentions a required concept
   • max_words: 150 — the output respects a length constraint
   • not_contains: "I cannot" — the model does not refuse the task
   • schema_valid: true — for structured output, the JSON is schema-valid
   • min_items: 3 — a list output contains at least three items

The fixture file is versioned in the same repository as the prompt. When you intentionally change what the output should look like, you update the fixtures in the same PR — that is the deliberate update workflow, not silent automation.

[
  {
    "id": "summarize-earnings-brief",
    "input": "Q3 revenue reached $4.2B, up 12% YoY. Operating margin expanded to 23%...",
    "assertions": [
      { "type": "contains",     "value": "revenue" },
      { "type": "max_words",    "value": 150 },
      { "type": "not_contains", "value": "I cannot summarize" }
    ]
  }
]

Diffing Prompt Versions

Running a golden set against a single prompt version tells you the pass rate. Diffing two versions tells you what changed — which is the regression signal.

The diff operation is: fixtures that passed in version A minus fixtures that passed in version B. The result is the regression set — fixtures that were working and now fail.

This definition is important. A fixture that was already failing in version A is not a regression introduced by version B. It is pre-existing debt. The regression gate only cares about newly broken behavior — things that were working before your change.

def detect_regressions(report_a, report_b):
    # passed_ids: fixture IDs that passed all assertions
    return sorted(list(report_a.passed_ids - report_b.passed_ids))

The symmetric complement — fixtures that were failing in A and now pass in B — are improvements. The eval runner should surface these too, separately, so you have evidence for the changelog.

Exact-Match vs. Property-Based Checks

The assertion type you choose determines how brittle your regression gate is.

Exact-match assertions compare the output to a verbatim expected string. They catch any deviation — including acceptable rephrasing, punctuation changes, and temperature-driven variance. A gate built entirely on exact-match assertions will fail on every minor prompt iteration even when the substantive quality is unchanged. This produces alert fatigue and leads teams to quietly disable the gate.

Property-based assertions check specific properties of the output regardless of wording. contains: "revenue" passes whether the model says "quarterly revenue reached" or "revenue figures show." The assertion captures the intent — the concept must be present — without mandating the exact phrasing. These are more robust and more maintainable.

The practical recommendation: use property-based assertions for everything except the rare cases where verbatim output is genuinely required (specific legal language, exact error codes, precise structured fields). Reserve exact-match for fields that must never change by even one character.

LLM-as-Judge: Capabilities and Failure Modes

LLM-as-judge is a technique where you use a separate model to score your prompt's output for quality. The judge model receives the input, the output, and a scoring rubric, then returns a score (typically 0.0–1.0) or a pass/fail verdict.

It can be useful for dimensions that are hard to capture with mechanical assertions — tone, factual accuracy, helpfulness. But it has well-documented failure modes that disqualify it as a sole regression gate:

Sycophancy. The judge tends to rate outputs higher when the output confidently agrees with the prompt framing, regardless of factual accuracy. A prompt change that makes the model more confident but less accurate can receive a higher judge score while being objectively worse.

Length bias. Most judge models prefer longer outputs to shorter ones, all else equal. A prompt change that adds padding increases the judge score without improving content.

Format bias. Outputs with structured formatting (bullet points, headers, bold text) often score higher than equally good prose, because the judge was trained on human feedback that associated formatting with quality.

Self-referential evaluation. If you are evaluating the prompts that drive your own system with a judge that uses similar system prompts, the judge and the evaluated system can share the same blind spots. Bad outputs in both systems' failure modes receive high scores from each other.

The correct use of LLM-as-judge is as a secondary signal: run it alongside your property-based checks, look for large divergences between the two, and use it to surface candidates for human review — not as the gate itself.

Wiring the Eval into CI

The eval pipeline becomes a regression gate only when it runs automatically and blocks merges on failure. The wiring is simple:

Step 1: Add the eval step to your CI workflow.

# .github/workflows/prompt-eval.yml
- name: Run prompt regression tests
  run: |
    python prompt_eval.py \
      --golden fixtures/summarize_golden.json \
      --prompt-a prompts/summarize_v_baseline.txt \
      --prompt-b prompts/summarize_v_candidate.txt

Step 2: Make the exit code the gate.

The runner exits 0 (pass) or 1 (regression). CI interprets a non-zero exit as a failed check. A failed check blocks the PR. No configuration beyond that is needed.

Step 3: Enforce that fixture updates are deliberate.

The golden set must be committed in the same PR as the prompt change that changes its expected behavior. Silent auto-updating — where CI rewrites the fixtures to match whatever the new prompt outputs — eliminates the gate entirely. The whole point of fixtures is that they encode what you previously declared correct.

Handling Variance and Tolerance Bands

Some prompts have inherently variable output — summaries that might be 140 or 180 words, lists that might have 3 or 5 items. Exact-count assertions on variable outputs produce false regressions.

The solution is tolerance bands in the assertion specification:

{ "type": "max_words", "value": 200 }
{ "type": "min_words", "value": 100 }

This declares: the summary must be between 100 and 200 words. Outputs of 140 or 180 both pass. An output of 50 words fails the min constraint. An output of 300 words fails the max constraint. The tolerance band is documented in the fixture — not inferred from whatever the baseline happened to produce.

The same principle applies to list-length assertions, schema-version tolerance, and numeric field ranges. Declare the acceptable range explicitly. If the range is hard to specify, that is often a signal that the prompt's output spec needs tightening.

Lesson Drill

Apply this to a prompt you already own:

1. Identify one prompt from your library that runs in production.
2. Write 5 fixture inputs that represent real inputs that prompt receives.
3. For each fixture, write 2–3 property-based assertions that capture what the output must satisfy.
4. Run the fixtures against the current prompt and verify all 5 pass.
5. Make a deliberate improvement to the prompt — change one thing. Re-run. Check whether anything regressed.

That five-fixture exercise is a working regression suite. Add it to your repository. Wire it to CI this sprint.

Bottom Line

Prompts are code. They need tests. A golden set of representative input→property pairs is the test suite. Property-based assertions tolerate acceptable variance while still catching functional regressions. The regression gate is the set difference between what passed before and what passes now — not all failures, only newly introduced ones. LLM-as-judge provides a signal but cannot serve as the sole gate because of sycophancy, length bias, and format bias. CI wiring converts this from a manual quality check into an automated merge gate. Change a prompt, run the fixtures, read the exit code. That is the professional standard for prompt engineering at scale.