Rebalance Without Moving the Threshold
PR #28 walkthrough. Grok 30 to 35, Perplexity 30 to 50, Calibrator 25 to 20. Threshold held at 70. Zero signals clearing became eight in 24 hours. Why restructuring weights is always safer than lowering the bar.
The Agent Framework calibrator was dead. The threshold was 70. Zero signals had cleared the bar in weeks. There were two ways to fix it.
Option A: Lower the threshold. Drop 70 to 50. Signals start clearing. Everyone goes to bed happy.
Option B: Restructure the weights so the effective ceiling rises above 70 without the calibrator.
PR #28 chose Option B. This lesson is about why — and what the walkthrough looks like.
The Case Against Lowering the Threshold
The threshold in a scoring system is the quality gate. It is the line above which a signal is considered strong enough to act on. Lowering it does not make signals better. It makes weaker signals eligible.
That is exactly the wrong intervention. The signals that were failing to clear 70 were failing because the composite could not represent their quality — not because they were weak. The correct fix is to reshape the composite, not to admit lower quality.
The deeper reason is the second-order effect: once you lower the threshold, you have lost the reference point. Future performance comparisons become impossible ("did the bot get better, or did we just lower the bar again?"). The threshold is your quality anchor. Do not touch it unless the underlying signal quality has actually changed.
The PR #28 Walkthrough
The rebalance had four changes:
| Component | Before | After | Delta | Reasoning |
|---|---|---|---|---|
| Grok | 30 | 35 | +5 | Strong historical fire rate (82%), worth a small boost |
| Perplexity | 30 | 50 | +20 | Highest signal quality per point, highest fire rate — scale it up |
| News | 15 | 15 | 0 | Already calibrated, no change needed |
| Calibrator | 25 | 20 | -5 | Load-bearing bug fixed by reducing its max — even if it stays dead, the other components can now clear 70 |
| Total | 100 | 120 | +20 | Headroom bought |
| Threshold | 70 | 70 | 0 | Unchanged |
The key move is the Perplexity boost. Before PR #28, Perplexity's 30-point max meant that even a near-perfect Perplexity score (say 28) combined with a strong Grok score (say 28) and a typical News score (~10) would get you to 66 — still short of 70 — unless the calibrator also fired or News scored near its own max. After the rebalance, a near-perfect Perplexity score (47) plus a strong Grok score (32) gets you to 79 on those two components alone. Threshold cleared without the calibrator firing at all.
The calibrator's max dropped from 25 to 20. On the surface this looks like a demotion. But the real purpose was to shrink its share of the total so that the other components could realistically cover the threshold without it. A dead load-bearing component is a footgun; a dead additive component is just dead weight. The rebalance moved the calibrator from the first category to the second.
The Validation
PR #28 was validated via the arithmetic backtest described in the “Arithmetic Backtesting from Stored Components” lesson:
SELECT COUNT(*) FROM (
SELECT
(grok_score * 35.0 / 30.0)
+ (perplexity_score * 50.0 / 30.0)
+ news_score
+ (calibration_score * 20.0 / 25.0) AS new_composite
FROM agent_signals
WHERE created_at > NOW() - INTERVAL '14 days'
) rescaled
WHERE new_composite > 70;
Result on historical data: 8 signals over the trailing 14-day window would have cleared, compared to 0 under the old weights. That count — 8 across two weeks of stored signals — was the merge criterion. (Do not confuse it with the live result: once shipped, a backlog of strong signals that had been queuing against the unreachable old bar flushed through on day one, so eight cleared in the first 24 hours. The 14-day backtest count and the day-one live burst are two different measurements that happen to land on the same number.) If the rescale had produced too many (say 200) or too few (say 1) over the 14-day window, the rebalance would have needed another round.
Inline Diagram — Before / After Distribution
The Rule
When signals are not clearing, fix the math, not the threshold. Restructure component weights until the effective ceiling at realistic fire rates sits comfortably above the threshold. Validate with an arithmetic backtest. Ship with a target range for clearance count — not a target number. Preserve the bar.