ASK KNOX
beta
LESSON 679

Execution Integrity: Reconciliation & Confirm-or-Halt

Almost every catastrophic agent loss happens at the intent-to-reality seam, not in the signal — the exchange is the source of record, and local state is just a cache.

12 min read·Trading Agent Fleets

Where losses actually happen

The single most common finding when you trace a catastrophic autonomous trading agent loss to its root cause is not a bad signal. It is not a wrong entry. It is a gap between what the agent believes about the world and what is actually true on the exchange.

The agent believes a position closed. The exchange still has it open. The agent stops managing the position. Price moves adversely. The position has no stop. The exchange liquidates it.

Or: the agent believes a stop was placed. The stop submission failed silently. The confirmation never arrived. The agent's retry logic was not robust. Price moves through where the stop would have been. The position is unprotected and the loss is larger than the agent's risk parameters allowed.

These are not edge cases. They are the main event. And they share a single structural cause: the agent's local state — its database, its in-memory representation of positions and orders — drifted from the exchange's actual state, and the agent had no mechanism to detect the drift.

Local state is a cache, not the ledger

The exchange is the source of record. Everything else — your database, your in-memory position tracker, your order log — is a cache. Caches are useful because they are fast and do not require a round-trip. Caches are dangerous because they go stale.

A process restart drops in-memory state. A network failure during order submission means the order may or may not have been processed — the submission side has no way to know. An exchange maintenance window may have cleared or altered state while the agent was offline. Any of these events creates a gap between local belief and exchange truth.

The Execution Integrity Layer exists to close that gap continuously.

The reconciler runs on two triggers: startup (before the agent begins autonomous activity) and periodically (every N seconds while running). On each run, it queries the exchange for the live positions and open orders, diffs them against local state, and emits actions for each discrepancy.

There are four discrepancy types: a position the exchange has that local does not know about (adopt it); a position local believes is open that the exchange shows as flat (close the orphaned local record); an open position that has no live stop order (reconfirm the bracket immediately); and an ambiguous state where local and exchange disagree on direction (halt and require operator resolution).

The seven guards

The reconciler is the first and most important guard, but it is not the only one. A production-grade execution layer wraps the core order executor with seven distinct controls, each catching a different class of failure.

L1: Startup and periodic reconciler. Closes the local-state-vs-exchange-truth gap. Runs before any autonomous activity and on a timer.

L2: Confirm-or-halt protective stops. After every bracket order submission, the agent waits for exchange confirmation before proceeding. If confirmation is not received within a timeout, the agent halts. It does not retry optimistically — it halts and surfaces the uncertainty.

L3: Idempotent client order ID. Every order submission includes a unique client-generated ID. If the agent retries a submission after a network failure, the exchange deduplicates using the same ID rather than processing the order twice. Without this, a network error can cause duplicate orders on retry.

L4: Attempt and fee breaker. Caps the number of entry attempts per signal and the total fees consumed in a session. This catches enter→abort loops where the agent repeatedly enters and exits without holding a position, bleeding fees each cycle.

L5: Fail-loud configuration. Bad or missing configuration causes the agent to halt on startup with an explicit error, never to silently default to a dangerous value. A missing maximum leverage setting should halt the agent, not default to 10x.

L6: Liveness monitor. A heartbeat mechanism detects when the agent has stopped processing. A silent process death is more dangerous than a loud crash — it leaves positions open and unmonitored with no alert.

L7: Deploy-verify. After deploying a new version of the agent, a verification step confirms that the running process is actually the new version — not a cached or zombied old process. A deploy that fails silently to replace the running process is a class of bug that can make a fix take longer than the original incident.

Equities are different: reconcile harder

For equities agents, the reconcile step is not just important — it is mandatory even if the agent has submitted no orders since its last run.

Equities positions can change without any order from the agent. A stock split changes the share count. An options assignment creates or closes an equity position. Dividends on certain structured products change basis. A corporate restructuring may merge, spin off, or convert holdings. An equities agent that only tracks its own submissions will have stale local state any time one of these corporate actions occurs — and corporate actions do not generate order confirmations that would trigger the agent's normal state-update path.

The reconciler must treat every discrepancy between local state and exchange state as potentially caused by a corporate action, not just a submission failure. The response — adopt the exchange truth, update local state — is the same. The implication is that the reconciler must run on startup every time, regardless of how recently the agent last ran or how confident the developer is that "nothing happened."

Build it: implement the reconciler

Implement the function that diffs local agent positions against exchange positions and emits a list of ReconcileActions. The four action types directly encode the four discrepancy classes described above.

The seam that matters

The intent-to-reality seam is everywhere in autonomous systems: the line between what the agent intends to happen and what actually happened on the exchange. Every guard in the Execution Integrity Layer addresses a specific way that seam can open.

The reconciler closes the state-drift gap. Confirm-or-halt closes the submission-confirmation gap. The idempotent client order ID closes the retry-duplication gap. The fee breaker closes the enter→abort-loop gap. Fail-loud config closes the silent-misconfiguration gap. Liveness closes the silent-death gap. Deploy-verify closes the stale-process gap.

None of these guards are optional for a production agent. They are the difference between a system that you can observe, understand, and trust, and one that occasionally surprises you with a loss you cannot fully trace. The signal and the entry are the glamorous parts of trading agent design. The Execution Integrity Layer is the part that keeps the agent alive long enough for the signal to matter.