// Track

Silent Failure Modes in Production ML

When broken looks exactly like healthy

Dead components, wrong addresses, stale configs, backtest/live drift, proxy/funder footguns. The failure modes that don't throw errors, don't log warnings, and don't page oncall — they just silently return zero and let the system keep running on empty. Detection patterns for each.

6 lessons~44 min total

Lessons are shown in recommended order. Complete them in sequence for the best experience — or jump to any lesson.

Lesson 278·8 min read

The Dead Component Pattern

The Hermes calibrator story end-to-end. Silence is not health. A component that always returns zero is indistinguishable from a broken one without fire-rate tracking — and the canonical silent failure mode that every scoring system eventually hits.

Lesson 279·7 min read

Wrong-Address Debugging

The Hermes wallet mismatch case. How 'USDC balance: $0.00' lied. The Doppler key derived the wrong EOA. Detection: log derived address, compare to expected. Prevention: startup derivation check. Why 'balance = 0' is one of the most ambiguous signals in an EVM bot.

Lesson 280·7 min read

Stale Config Drift

Deploy targets accumulating untracked edits. The broker_bridge.py file that blocked git pull on Tesseract. Why deploy targets must be pure git reflections. Pre-flight git status as reflex — not an afterthought.

Lesson 281·8 min read

Backtest vs Live Divergence

The InDecision 54% to 21.5% drift. Three root causes. Why alignment tests should be a deploy gate on every scoring system. Detection: compare backtest PnL distribution to live PnL distribution nightly, before the divergence compounds into months of bad trades.

Lesson 282·7 min read

Proxy / Funder Footguns

Polymarket's signature_type matrix. When your EOA is the funder, when it is not, and how to detect the misconfiguration before it costs you. The balance check that lied because it polled the wrong address — extended to proxy wallets.

Lesson 283·7 min read

Fire-Rate Alerting

The auto-alert pattern. Every scoring component emits fire_rate_24h. Below 20% warning, below 5% critical. The dashboard that would have caught Hermes weeks earlier — and the template for every scoring bot in the ecosystem.