Ask Knox

A model that says it is 90% confident is making a testable claim: in a large sample of cases where it expressed 90% confidence, approximately 90% of those predictions should be correct. If the actual accuracy is 65%, the confidence score is lying — not from malice, but from miscalibration.

Miscalibration is not a corner case. Most models exhibit systematic overconfidence in the high-confidence range and sometimes underconfidence in the low-confidence range. Without measuring calibration, you cannot know whether the confidence scores your system produces are signal or decoration.

What Calibration Measures

Calibration is the degree to which predicted probabilities correspond to observed frequencies. A perfectly calibrated model — when it says 70% confident — is correct 70% of the time on that subset of predictions.

The reliability curve makes this visible: plot predicted confidence on the x-axis, actual accuracy on the y-axis. Perfect calibration is the diagonal line from (0,0) to (1,1). Overconfidence pushes the curve below the diagonal at high confidence values — the model says it is 90% sure, but it is only correct 70% of the time. Underconfidence pushes the curve above the diagonal — the model hedges to 40% confidence when it is actually correct 60% of the time.

The reliability curve is diagnostic, not a single number you can gate on. For a CI gate, you need ECE.

Expected Calibration Error

Expected Calibration Error (ECE) is the calibration metric most suited to CI integration. It produces a single scalar that measures the aggregate gap between predicted confidence and actual accuracy across the full confidence range.

The calculation:

Divide all predictions into N confidence buckets (typically 10, covering [0.0, 0.1), [0.1, 0.2), …, [0.9, 1.0]).
For each bucket: compute the mean predicted confidence and the fraction of predictions that were correct.
ECE = Σ (bucket_count / total_count) × |mean_confidence − actual_accuracy|

An ECE of 0.10 means the model's confidence scores are off by 10 percentage points on average, weighted by how many predictions fall in each bucket. ECE < 0.10 is a reasonable gate for a calibration harness — it indicates the confidence scores are useful signal, not just noise.

The Brier Score

The Brier score is a complementary metric that measures both calibration and sharpness (how confident the model is overall). For binary classification:

Brier = (1/N) × Σ (predicted_probability − correct_label)²

A model that always predicts 0.5 has a Brier score of 0.25. A perfect model (predicts 1.0 when correct, 0.0 when wrong) has a Brier score of 0. Lower is better.

The Brier score rewards a calibrated model that also expresses appropriate confidence. A model that says 0.9 on correct predictions and 0.1 on wrong ones scores better than one that hedges everything to 0.5 — even if the hedging model has lower ECE (hedging everything to 0.5 is technically calibrated only when the task's own accuracy is itself about 50% — a model that is actually 90% accurate but hedges to 0.5 is badly miscalibrated, with ECE ≈ 0.4).

Minimum Sample Thresholds

Calibration measurement is a statistical estimate. With fewer than 30 predictions, the per-bucket accuracy estimates are too noisy to be meaningful — a bucket with 3 predictions where 2 were correct reports "67% accuracy," but the true accuracy for that confidence range might be anywhere from 0% to 100%.

The sample thresholds are:

< 30 samples: do not report calibration metrics. The estimates are worthless.
30–100 samples: directional signal only. You can see if the model is grossly overconfident, but you cannot trust the per-bucket breakdown.
100–500 samples: CI-worthy. ECE is stable enough to gate on. Per-bucket accuracy is meaningful for buckets with ≥ 10 predictions.
> 500 samples: production-grade. You can compute confidence intervals on ECE and identify specific buckets that need attention.

These thresholds apply to the per-bucket breakdown. If your evaluation set has 200 samples but your model only expresses high confidence (0.8+) on 5 of them, the high-confidence bucket is still too sparse to trust.

Integrating the Harness into CI

A calibration harness in CI looks like this:

Maintain a labeled evaluation set with known correct answers (at least 100 samples).
On every prompt version change, run all predictions through the model with confidence scoring enabled.
Compute ECE. If ECE ≥ 0.10, fail the CI gate.
Write the per-bucket reliability table and the Brier score to the PR as an artifact.
Alert when any single bucket drops > 15 percentage points in accuracy versus the previous run.

The CI gate catches the most common miscalibration failure mode: a prompt change that shifts the model's confidence expression (making it more or less confident in ways that do not correspond to actual accuracy changes).

What Calibration Does Not Tell You

Calibration measures whether confidence scores are meaningful, not whether the model is correct. A perfectly calibrated model that is correct 50% of the time on every confidence level is calibrated — and useless.

Calibration is a prerequisite for trusting confidence-gated pipelines, not a measure of the pipeline's performance. Before calibration: confidence scores are decoration. After calibration: confidence scores are a gateable signal you can rely on to route predictions to human review when the model is genuinely uncertain.