Building an Eval Harness for LLM Apps

You would not ship a feature without running the test suite. Yet most LLM-powered features ship without any systematic quality check at all — just a developer running a few prompts and deciding it "feels right." That gap is where quality regressions live.

An eval harness is the test suite for your LLM layer. It gives you the same confidence interval for prompt changes and model upgrades that a unit-test suite gives you for logic changes: either it passes, or you have a regression to fix.

The Four Stages of an Eval Harness

Every eval harness, regardless of framework or provider, has the same four stages:

1. Golden set — a frozen corpus of input/expected-output pairs that encodes the behaviour you care about
2. Run — call the model on every input in the corpus and capture the actual output
3. Score — compare actual vs expected using a rubric, producing a numeric score per case
4. Gate + report — aggregate scores, compare against a baseline, block the release if quality dropped

The concepts are provider-agnostic. The runnable example below uses the OpenAI Responses API — the current recommended path for OpenAI integrations (see the "Assistants API" and "Building a Production OpenAI Integration" lessons in the ChatGPT track for context on why Assistants API was deprecated in favour of this). Verify the current model string and SDK method at platform.openai.com/docs before production use.

Stage 1: Building Your Golden Set

A golden set is the foundation of the whole system. Get this wrong and your scores measure the wrong thing. Get it right and every prompt change, model upgrade, or system-prompt tweak gets a quantified quality signal.

What belongs in a golden set:

• Happy-path cases — the inputs you handle well and want to keep handling well
• Edge cases — ambiguous inputs, corner cases, short queries, very long queries
• Adversarial cases — inputs that trip up the current prompt (these are the most valuable)
• Regression seeds — any case that has ever produced a bad output in production

Corpus size: 20 cases is a minimum viable golden set. 50–100 cases gives you a signal you can actually trust. Fewer than 20 and a single case carries too much weight in the aggregate score.

What a case looks like:

{"input": "Summarise this in one sentence: The sky is blue.", "expected": "sky blue", "tags": ["summarisation", "short"]}
{"input": "Extract the customer name from: Hi, I am Alice, I need help with my order.", "expected": "Alice", "tags": ["extraction"]}
{"input": "Classify as positive or negative: The product is excellent!", "expected": "positive", "tags": ["classification"]}

Stage 2: Running the Model via the Responses API

The Responses API (client.responses.create) is OpenAI's current recommended path for new integrations. It replaces the deprecated Assistants API (sunset August 26, 2026) and bare Chat Completions for stateful, tool-capable use cases.

Basic call shape (verify at platform.openai.com/docs):

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

response = client.responses.create(
    model="gpt-4o",  # verify current model IDs at platform.openai.com/docs
    input="Summarise this in one sentence: The sky is blue.",
)

actual_output = response.output_text  # SDK convenience property

Running the full golden set:

def run_golden_set(cases: list[dict], model: str = "gpt-4o") -> list[dict]:
    results = []
    for case_ in cases:
        response = client.responses.create(
            model=model,
            input=case_["input"],
        )
        actual = response.output_text or ""
        results.append({**case_, "actual": actual})
    return results

Stage 3: Defining Your Scoring Rubric

Not all outputs can be scored the same way. A classification label has a right answer. A summary does not. A scoring rubric combines multiple methods to cover the full range of tasks your application handles.

Exact-Match

Exact-match is the simplest scoring method: strip and lowercase both strings, then compare. Binary result — pass or fail.

def score_exact_match(expected: str, actual: str) -> dict:
    if expected.strip().lower() == actual.strip().lower():
        return {"method": "exact", "score": 1.0, "pass": True}
    return {"method": "exact", "score": 0.0, "pass": False}

Use exact-match when the output is a controlled value: a classification label, a JSON field value, a boolean answer. It breaks down the moment the model rephrases a correct answer.

Assertion-Based Scoring

Assertion-based scoring runs programmatic checks against the output — keyword presence, schema validation, numeric range checks, regex patterns. More flexible than exact-match and more explicit than asking a judge.

def score_assertions(expected: str, actual: str, assertions: list[callable]) -> dict:
    passed = [a(actual) for a in assertions]
    score = sum(passed) / len(passed)
    return {"method": "assertion", "score": score, "pass": score >= 0.7}

# Example assertions
assertions = [
    lambda out: "Alice" in out,
    lambda out: len(out.split()) <= 10,
    lambda out: not out.lower().startswith("i'm sorry"),
]

A common and cheap assertion is word-overlap: check whether every key word from the expected output appears in the actual output. Score = matching words / expected words.

def score_word_overlap(expected: str, actual: str, threshold: float = 0.7) -> dict:
    expected_words = set(expected.lower().split())
    actual_lower = actual.lower()
    hits = sum(1 for w in expected_words if w in actual_lower)
    score = round(hits / len(expected_words), 4) if expected_words else 0.0
    return {"method": "assertion", "score": score, "pass": score >= threshold}

LLM-as-Judge

LLM-as-judge uses a second model call to evaluate the output semantically. You give the judge a rubric — a set of criteria to evaluate — and it returns a score.

def score_with_judge(input_: str, expected: str, actual: str, rubric: str) -> dict:
    prompt = f"""You are an evaluator. Score the following model output on a scale of 0 to 1.

Rubric: {rubric}

Input: {input_}
Expected: {expected}
Actual: {actual}

Return a JSON object: {{"score": float, "reason": str}}"""

    response = client.responses.create(
        model="gpt-4o",  # use a different model from the one being evaluated when possible
        input=prompt,
    )
    import json
    result = json.loads(response.output_text)
    return {"method": "llm-judge", "score": result["score"], "pass": result["score"] >= 0.7}

Layering All Three

In practice, a well-designed harness layers all three methods:

def score_case(case_: dict) -> dict:
    expected = case_["expected"]
    actual = case_["actual"]

    # 1. Try exact-match first — cheap and unambiguous
    if expected.strip().lower() == actual.strip().lower():
        return {"method": "exact", "score": 1.0, "pass": True}

    # 2. Assertion-based — word-overlap as a fallback
    score_result = score_word_overlap(expected, actual)
    if score_result["pass"]:
        return score_result

    # 3. LLM-as-judge for open-ended cases tagged as such
    if "open-ended" in case_.get("tags", []):
        rubric = "Is the actual output semantically equivalent to the expected output?"
        return score_with_judge(case_["input"], expected, actual, rubric)

    return score_result

Stage 4: The Regression Gate

The regression gate compares the current run's average score against a stored baseline and blocks the release if quality dropped.

def build_report(scored_cases: list[dict], baseline_score: float = 1.0) -> dict:
    total = len(scored_cases)
    passed = sum(1 for c in scored_cases if c["score_result"]["pass"])
    avg_score = round(sum(c["score_result"]["score"] for c in scored_cases) / total, 4)

    return {
        "total": total,
        "passed": passed,
        "failed": total - passed,
        "avg_score": avg_score,
        "regression": avg_score < baseline_score,  # strict less-than
        "cases": scored_cases,
    }

Choosing a baseline: A baseline of 1.0 means "every previous run was perfect." That is appropriate when you start — all your golden-set cases should pass on the current system. As the harness matures and you add harder cases, the baseline may settle lower. Store the baseline in version control alongside the golden set.

Running Evals in CI

Wire the harness into your CI pipeline so it runs on every PR that touches a prompt, a model parameter, or any code that affects the LLM output path.

# .github/workflows/eval.yml
name: LLM Evals
on:
  pull_request:
    paths:
      - "prompts/**"
      - "lib/llm/**"
      - "eval/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: pip install openai
      - name: Run eval harness
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python eval/run_evals.py --baseline 0.9 --fail-on-regression

# eval/run_evals.py
import sys, json, argparse
from eval_harness import run_eval

parser = argparse.ArgumentParser()
parser.add_argument("--baseline", type=float, default=1.0)
parser.add_argument("--fail-on-regression", action="store_true")
parser.add_argument("--golden-set", default="eval/golden_set.jsonl")
parser.add_argument("--model", default="gpt-4o")  # verify at platform.openai.com/docs
args = parser.parse_args()

report = run_eval(args.golden_set, model=args.model, baseline_score=args.baseline)

print(json.dumps(report, indent=2))
print(f"\nResult: {report['passed']}/{report['total']} passed · avg_score={report['avg_score']}")

if args.fail_on_regression and report["regression"]:
    print(f"REGRESSION DETECTED: avg_score {report['avg_score']} < baseline {args.baseline}")
    sys.exit(1)

Cost Management for CI Evals

LLM evals cost money. A golden set of 100 cases at gpt-4o rates (input and output tokens — verify current pricing at platform.openai.com/docs) adds up across dozens of PRs per day. Strategies to keep costs manageable:

Run a subset for quick checks. Tag golden-set cases as smoke or full. The CI gate runs only smoke cases on every PR and the full set nightly.

def load_golden_set(path: str, tags: list[str] | None = None) -> list[dict]:
    cases = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            case_ = json.loads(line)
            if tags is None or any(t in case_.get("tags", []) for t in tags):
                cases.append(case_)
    return cases

Use a smaller model for CI scoring. Your scoring judge does not need to be the most capable model. A faster, cheaper model is fine for pass/fail classification on well-defined rubrics. Verify current model options at platform.openai.com/docs — the lineup changes faster than this lesson does.

Exact-match and assertion first. Only escalate to LLM-as-judge for cases that genuinely require it. Most golden-set cases have deterministic or rule-checkable expected outputs.

What Good Coverage Looks Like

A complete golden set covers at least three dimensions:

Dimension	Examples
Task type	Classification, extraction, summarisation, Q&A, generation
Input length	Short (< 50 tokens), medium, long (near context limit)
Edge cases	Empty input, ambiguous request, refusal-triggering input

A golden set that only covers happy-path, medium-length inputs will miss the regressions that actually matter in production.

Bottom Line

An eval harness is not a nice-to-have — it is the mechanism that turns "I think this works" into "I can prove this works at this quality level." The golden set encodes what matters. The scoring rubric measures it without ambiguity. The regression gate blocks releases that would degrade the user experience. Running it in CI means you know immediately, not after a customer complaint.

The Responses API call at the centre of the harness is straightforward: client.responses.create(model=..., input=...) returns response.output_text. The hard part is building the golden set and calibrating the rubric — but that investment compounds. Every case you add makes the harness more sensitive to exactly the failures your application has actually encountered.