Managing the Fleet: Orchestration, Halts & Observability

From one agent to a fleet

A single autonomous trading agent is operationally straightforward: one process, one set of positions, one control plane to manage. The leap to a fleet — multiple agents running simultaneously, each managing its own symbols or strategies — introduces a qualitatively different class of problems that individual-agent design does not prepare you for.

The first problem is account collision. The second is alert fatigue from silent failures. The third is configuration autonomy: giving an intelligent supervisor enough capability to be useful without giving it the ability to act on its own recommendations. This lesson covers all three, plus the observability infrastructure that makes a fleet operable at 2 AM when something goes wrong.

The collision problem

Perpetual-futures exchanges operate on an account-level model. Margin is pooled: all positions on the account share the same available capital. Orders reduce the available margin account-wide, and external-close detection operates at the account level, not the position level.

When multiple agents share one account, they collide in both of these dimensions.

Agent A opens a position that consumes 40% of available margin. Agent B's next entry is sized based on what it believes is available margin — which has not been updated to reflect Agent A's new position. Agent B's entry is either sized incorrectly (too large relative to remaining capital) or rejected outright. More insidiously, when Agent A closes its position, Agent B's external-close detection may fire — because the account-level position change looks, from Agent B's perspective, like an unexpected external closure.

The correct architectural pattern for a fleet on a single exchange account is one of two approaches. The first is isolated sub-accounts: each agent gets its own sub-account with its own margin pool, eliminating shared-state collisions entirely. Most major perpetual-futures exchanges support sub-accounts or portfolio margin modes that enable this. The second is a centralized position manager that acts as the single coordinator for all entries and exits on the shared account, with agents submitting intent to the coordinator rather than directly to the exchange. This is more complex but allows sophisticated portfolio-level constraints (overall account exposure, correlated position detection) that sub-account isolation does not.

The halt-watcher contract

A silent halt is more dangerous than a loud crash. A crash produces a traceable event: a log entry, a process exit, a monitoring alert. A silent halt — where the agent is technically running but not processing or updating positions — can persist for hours without any operator visibility.

The halt-watcher is a separate monitoring process that polls each agent's halt state and last-trade time on a fixed interval (typically 30–60 seconds). When it detects a halted agent or a stale last-trade timestamp, it fires an immediate alert to the operator: a push notification, a message in a monitoring channel, or an entry in the control surface's alert feed.

The critical design constraint: the halt-watcher must be a separate process from the agents it monitors. A halt-watcher that runs inside the agent process it is monitoring does not fire when the agent process itself hangs. The monitoring boundary must be at the process level, not the function level.

The halt-watcher also differentiates between halt types: a deliberate operator halt (the agent was manually stopped) is not an alert condition, but an unexpected halt from an unconfirmed stop or an exchange error is. This requires the halt state to carry a reason that the watcher can classify.

Observability per agent

A fleet without per-agent observability is operationally blind. The minimum instrumentation required to operate a fleet safely:

Last trade time: the timestamp of the most recent completed trade cycle. Staleness beyond a threshold (accounting for expected signal frequency) indicates either a stale market or a silent agent failure.
Balance and open positions: the account's available margin and each agent's current open positions, queried from the exchange on a fixed interval.
Halt state: whether each agent is halted, and if so, the reason — unconfirmed stop, daily loss cap, operator halt, exchange error.
Position fire rate: the ratio of signals generated to positions actually opened over a rolling window — distinct from a signal engine's component fire rate (the signal-engines lesson), which measures how often an individual filter contributes. A position fire rate near zero in a normally active market suggests the agent's filters are too restrictive or its entry conditions are broken.

These four metrics, available per agent and refreshed on a short interval, are what makes a fleet operable without constant manual inspection.

The regime supervisor: read-only by design

The most dangerous temptation in fleet design is giving the regime supervisor — the component that reads macro conditions and proposes configuration adjustments — the ability to apply its own proposals.

A regime supervisor that can self-approve and self-apply is not a supervisor. It is an autonomous configuration system with no human check. In a high-volatility environment, a supervisor that can act unilaterally can reduce position limits, change signal thresholds, and adjust risk parameters faster than any human operator — which sounds useful, until the supervisor's macro model is wrong and the agents are now misconfigured with no audit trail.

The read-only constraint is not a limitation of the supervisor's analytical capability. It is the load-bearing safety property of the entire architecture. The supervisor reads everything, proposes everything, and decides nothing. Every proposed change — even a conservative one, even a change that reduces exposure — must pass through the human gate before being applied to the control surface.

The practical implementation: the supervisor returns a Proposal object with approved=false. The operator reviews the proposal's rationale and marks it approved. Only then does applyApproved() run — and it throws if the proposal is not approved, making the gate enforced by code rather than by convention.

The control surface contract

Every agent in the fleet exposes a control surface: a set of operations the operator can invoke to change the agent's behavior without restarting it. The minimum control surface for a production agent:

arm: transition from a standby or post-halt state to active trading
halt: stop autonomous activity and surface the halt reason
clear-halt: acknowledge a halt condition and allow the agent to resume (does not necessarily arm)
PATCH config: update configuration values at runtime (signal thresholds, position limits, enabled symbol sets)

The PATCH config operation is what the applyApproved() function calls after operator approval. The control surface is the only path from a proposed change to a live configuration change — it is not a suggestion, it is the mechanism.

Control surface operations should be idempotent and should emit audit log entries: who called, what changed, what the previous value was, what the new value is. Without this, configuration drift becomes untraceable.

Build it: implement the fleet supervisor

Implement the two-component fleet supervisor: a pure proposeAdjustment method that returns a proposal without mutating anything, and a guarded applyApproved function that enforces the human gate at the code level.

Operating a fleet is a discipline, not a feature

The fleet patterns in this lesson — sub-account isolation, halt-watcher monitoring, per-agent observability, read-only supervisor with a human gate — are not features you add after the fleet is running. They are the operating model. A fleet without them is not a fleet; it is several agents sharing an account and hoping they do not interfere with each other.

The discipline of fleet operations is the discipline of knowing the state of every agent at every moment, being alerted immediately when that state changes unexpectedly, and having a clear path from observation to intervention that does not require the supervisor to act unilaterally.

The signal work, the risk management, the execution integrity layer — all of it is in service of positions that close profitably. Fleet operations is what keeps the infrastructure underneath those positions reliable enough that the signal work has a chance to matter.

This is the end of the Trading Agent Fleets track. The capstone project ahead asks you to design, build, and operate a minimal but complete trading agent — signal engine through execution integrity through fleet control surface — and document every architectural decision. The lessons in this track are the toolkit. The capstone is where you build the real thing.