Fleet Observability: Event-Log Querying

The content pipeline in the “A Production Multi-Agent System” lesson logs every stage to events.jsonl. That file is not decoration. When the pipeline goes silent at 2 AM and you need to know whether the researcher ran at all, whether the writer stalled, or whether the publisher opened the PR — the event log is the only witness. Its usefulness is entirely determined by one design decision you made weeks before the incident: did you write structured events, or log lines?

This lesson is about that decision, and everything that follows from it.

Log Lines vs Structured Events

A log line looks like this:

2026-01-15T14:22:01Z researcher-1 completed research for topic X

You can grep it. You can count it. You cannot query it by agent, join it to another agent's output, filter it by status, or aggregate across a delegation chain. It is a string. Its meaning is embedded in the format of the text, not extractable programmatically.

A structured event looks like this:

{
  "ts": "2026-01-15T14:22:01Z",
  "event_type": "completion",
  "run_id": "run-2026-01-15",
  "agent_id": "researcher-1",
  "trace_id": "tr-abc123",
  "span_id": "sp-7f9a2b",
  "parent_agent_id": "orchestrator",
  "stage": "research",
  "status": "complete",
  "detail": "artifact=research.json bytes=4821"
}

Every field is a queryable dimension. You can filter, join, group, and aggregate. You can ask: which agents completed but produced empty output? Which events share this trace_id? What is the sequence of events between 14:00 and 14:30 for this run? None of those questions require parsing text.

The Universal Event Envelope

Every event in a fleet event log needs a minimum set of fields — the universal envelope. These are not optional. Every field that is missing is a query you cannot write.

The four correlation IDs deserve special attention:

run_id is the pipeline-run anchor. Every event from a single content pipeline run shares the same run_id. It answers: what did this run do, in order?

agent_id is the stable agent identity. Not a PID, not a process name — a stable string like "researcher-1" that survives restarts and remains consistent across runs. It answers: what did this specific agent do, across all runs?

trace_id is the top-level request lineage key. When an orchestrator dispatches a researcher, which dispatches a validator, which dispatches a sub-checker — all three agents inherit the same trace_id from the original request. It answers: what did the entire delegation tree do in response to this single user request?

parent_agent_id is the direct-parent pointer. Each agent records the ID of the agent that spawned it. This lets you reconstruct the delegation chain as a tree — who dispatched whom, in what order, at what depth.

The Queries That Answer Operational Questions

There are five questions that every fleet operator needs to answer at 2 AM when something goes wrong. The schema above was designed specifically to make them answerable.

Breaking Down the Queries

Query 1 — which agent touched this file: Filter on event_type = 'tool_call' and filename in detail. Return the latest by ts. This is a two-column filter and a max-aggregation. If your detail field is a raw string blob rather than a structured string containing artifact=filename, this query becomes a regex match. Use structured detail values: artifact=output_report.json bytes=4821.

Query 2 — full fleet timeline for a window: Filter on run_id and ts range. Sort by ts ascending. This is the literal transcript of what the fleet did during the incident window. Without run_id on every event, you cannot isolate one run from another. Without ts as an ISO-8601 string, you cannot do string comparison for range filtering.

Query 3 — cascade origin: Filter on trace_id and status in ('failed', 'timeout'). Return the minimum ts. The first failure is the cause; all subsequent failures are effects. This is the query where trace_id does the most work — without it, you cannot group failures by causal lineage. You are looking at a list of individual agent failures with no way to connect them.

Query 4 — silent failures: Filter on event_type = 'completion', ts >= since_ts, and 'bytes=0' in detail. This is the query that catches the failure mode the “A Production Multi-Agent System” lesson identified as the most dangerous — the agent that appears to succeed but produces nothing. The detection mechanism is encoding output artifact size into the detail field at completion time. If you log detail: "stage complete" instead of detail: "artifact=article.mdx bytes=0", this query is impossible.

Query 5 — delegation chain reconstruction: This is a recursive query. It walks the parent_agent_id tree starting from the root dispatcher (where parent_agent_id IS NULL) and follows each agent's children using a recursive CTE. The full delegation tree is encoded in the event log itself — no external dependency required.

Retention and Aggregation Tiers

A raw JSONL event log works for a small fleet generating a few thousand events per day. At tens of thousands of events per day, JSONL range queries become I/O-bound. The solution is a tiered retention architecture.

Tier 1 — Hot storage (JSONL, last 7 days): Every event is written to a rolling JSONL file. This is the ground truth. It is append-only, human-readable, and suitable for ad-hoc queries on recent events.

Tier 2 — Warm index (SQLite or DuckDB, last 90 days): A background process reads the hot JSONL and inserts events into a structured store. Queries on agent_id, trace_id, ts range, and status become indexed reads rather than full-file scans. SQLite handles fleets up to ~10 million events; DuckDB handles 100M+ with columnar efficiency.

Tier 3 — Cold archive (compressed JSONL, full history): Rolled-up JSONL files, gzip-compressed, retained indefinitely. Used for audits, post-mortems, and anomaly investigation across long time horizons. Not queried in real time.

Hot JSONL  →  Tier-2 indexer (cronjob, every 5 min)  →  SQLite/DuckDB
                                                          ↓
                                    Dashboard queries read from Tier 2
                                    Alert queries read from Tier 2
                                    Ad-hoc queries read from Tier 2
                                    Hot JSONL read only for < 5-min-old events

The JSONL → Tier 2 ingestion is a one-way pipeline. Events are never updated in place — the JSONL is append-only and the indexed store is insert-only. Immutability is what makes the event log trustworthy as an accountability surface.

Turning Recurring Queries into Alerts

A query that you run manually after an incident is a lagging indicator. The same query, run on a schedule, is a leading indicator.

The five queries above map directly to five health checks:

Query	Alert condition	Frequency
Silent failures	Any completion with `bytes=0` in last 60 min	Every 15 min
Cascade origin	Any trace with 3+ failure events	Every 5 min
Run timeline	No events for active `run_id` in 30 min	Every 10 min
Delegation chain	Max delegation depth > configured limit	Every run
Last agent on file	Expected artifact not written within SLA	Per-stage

The alert for "no events for active run_id in 30 min" is the event-log equivalent of the Watchdog Service's log-staleness check from the “A Production Multi-Agent System” lesson. But it is more precise: instead of checking whether any log was written, it checks whether the specific run you care about produced an event in the expected window. A fleet that is generating log noise from one pipeline while another is silently hung will pass the staleness check. It will fail the run-specific query.

What This Track's Pipeline Would Look Like

The content pipeline from the “A Production Multi-Agent System” lesson — orchestrator, researcher, writer, image agent, metadata agent, publisher — produces the following structured events on a healthy run:

{"ts":"2026-01-15T09:00:00Z","event_type":"dispatch","run_id":"run-20260115","agent_id":"orchestrator","trace_id":"tr-abc","parent_agent_id":null,"stage":"research","status":"start","detail":"topic=Agent fleets"}
{"ts":"2026-01-15T09:00:01Z","event_type":"dispatch","run_id":"run-20260115","agent_id":"researcher-1","trace_id":"tr-abc","parent_agent_id":"orchestrator","stage":"research","status":"start","detail":""}
{"ts":"2026-01-15T09:04:22Z","event_type":"completion","run_id":"run-20260115","agent_id":"researcher-1","trace_id":"tr-abc","parent_agent_id":"orchestrator","stage":"research","status":"complete","detail":"artifact=research.json bytes=4821"}
{"ts":"2026-01-15T09:04:23Z","event_type":"handoff","run_id":"run-20260115","agent_id":"orchestrator","trace_id":"tr-abc","parent_agent_id":null,"stage":"write","status":"start","detail":"from=researcher-1 to=writer-1 artifact=research.json"}
{"ts":"2026-01-15T09:04:24Z","event_type":"dispatch","run_id":"run-20260115","agent_id":"writer-1","trace_id":"tr-abc","parent_agent_id":"orchestrator","stage":"write","status":"start","detail":""}
{"ts":"2026-01-15T09:08:11Z","event_type":"completion","run_id":"run-20260115","agent_id":"writer-1","trace_id":"tr-abc","parent_agent_id":"orchestrator","stage":"write","status":"complete","detail":"artifact=article.mdx bytes=9203"}

The silent-failure version of the writer completion looks like this:

{"ts":"2026-01-15T09:08:11Z","event_type":"completion","run_id":"run-20260115","agent_id":"writer-1","trace_id":"tr-abc","parent_agent_id":"orchestrator","stage":"write","status":"complete","detail":"artifact=article.mdx bytes=0"}

Status is complete. Event type is completion. The trigger file exists. The metadata agent fires. The publisher opens a PR. And bytes=0 is in the detail field — the one signal that catches this class of failure, detectable in real time by the silent-failure query, four hours before you would discover it at PR review.

Lesson Drill

Take the event log from any multi-agent system you operate (or design a fictional one):

Audit every event against the universal envelope. Which correlation IDs are missing? For each missing one, name the query it prevents.
Write the cascade origin query for your log. Does your schema have trace_id on every event? If not, what is the cost — in manual steps — of reconstructing a cascade without it?
Write the silent-failure query. Does your detail field include artifact size on completion events? If not, add it to your event spec.
Pick one recurring manual query you run after incidents. Turn it into a scheduled check with a concrete alert threshold.
Design the JSONL → Tier 2 indexer for your fleet. What are the indexed columns? What is the query that drives your most important health check?

Bottom Line

The event log is not logging. It is the accountability surface for your fleet. Log lines answer questions about individual events; structured events answer questions about relationships — between agents, between stages, across time.

The schema you design before an incident determines what you can debug during it. The five fields that matter most — run_id, agent_id, trace_id, parent_agent_id, and status — are the difference between debugging in five minutes and debugging in two hours. Encode them into every event type, for every agent in the fleet, from day one.

The silent-failure query is the one that matters most. It catches the failure mode that passes every status check, writes every trigger file, and completes every stage — while producing nothing.