Pipeline Observability — Logs, Metrics & Alerts

The Gap This Track Left Open

The previous lessons in this track taught you to alert on failure. The “Self-Healing & Failure Alerting” lesson built Discord webhooks that fire when the pipeline crashes, when an image provider fails, when a manifest goes stale. That is necessary infrastructure. But it covers only one failure mode: the hard stop.

The failure mode this track did not cover is slower and more dangerous: a pipeline that keeps running, keeps exiting 0, and keeps looking healthy — while quietly producing worse content, spending more money, or burning through image provider fallbacks. You discover it on a Friday when you wonder why the last three articles sounded generic, or on a Monday when you check the API bill.

This lesson covers the other half of pipeline reliability: measuring health itself.

The Heartbeat vs Artifact Growth Distinction (Again)

The “Self-Healing & Failure Alerting” lesson established this, but it is worth restating in the context of observability:

• A heartbeat tells you the process is alive. pgrep -afl blog-autopilot returns 0. launchd shows status 0. The last run exited without error.
• A health check tells you the pipeline is producing output. SELECT COUNT(*) FROM run_log WHERE completed_at > NOW()-24h returns ≥ 1. The PR was opened. The article is live.

A process can be alive and producing nothing. This is the zombie pipeline scenario — covered in the “Self-Healing & Failure Alerting” lesson — and it is the most common monitoring mistake. A heartbeat that passes while the pipeline publishes zero articles is not a monitoring system. It is a false sense of security.

But even artifact-growth checks only catch the binary case: published or not published. They do not catch quality drift, cost inflation, latency regression, or provider degradation. For those, you need stage-level metrics.

The Four-Layer Observability Stack

Every stage in a production pipeline should be instrumented with four layers, not one:

The four layers answer four different questions:

1. Structured log event: What happened in this stage, right now, in this run? Answerable from the run logs without querying a database.
2. Stage metric: How is this stage performing over time? Answerable from the metrics store: latency trend, cost trend, quality trend.
3. Health check: Is the pipeline producing the expected downstream artifact? Answerable from the artifact store: PRs opened, images generated, words published.
4. Alert rule: When should I be notified? Answerable by threshold crossing on any of the above.

Most pipelines implement only the first layer, partially — they log errors. A production observability layer implements all four, for every stage.

Structured Logging Per Stage

A structured log event is a single JSON line emitted when a stage completes. It contains every field that is observable at stage exit: identifiers, time measurements, quality signals, cost signals, and the completion timestamp.

The format is intentional: JSON is machine-parseable. A log aggregator can query it. A cron health check can search it. You can grep it. print("DRAFT done") is not structured logging.

The minimal structured event for the DRAFT stage:

import json, logging, datetime as dt

log = logging.getLogger("blog_autopilot.draft")

log.info(json.dumps({
    "event": "draft_stage_complete",
    "topic_id": topic_id,
    "model": "claude-sonnet-4-6",
    "word_count": len(draft_text.split()),
    "voice_score": voice_score,
    "cost_usd": cost_usd,
    "stage_ms": stage_ms,
    "completed_at": dt.datetime.now(dt.timezone.utc).isoformat(),
}))

Every field in this event can be queried, trended, and alerted on. word_count below 600 is a quality signal. cost_usd 30% above baseline is a cost signal. stage_ms above 60 seconds is a latency signal. None of these are failure signals — the stage succeeded — but all of them are health signals.

The Metrics That Matter

Not every metric is worth collecting. A metric earns its place in the dashboard by answering a specific operational question you have asked or will ask. If you cannot name the question a metric answers, drop the metric.

The anti-metric column is the important one. For every metric in this table, there is a weaker proxy metric that looks similar but misses the failure mode. Process heartbeat looks like output_volume. Exit code 0 looks like stage_latency. PR count looks like voice_score. The proxy passes when the real metric fails — that is what makes silent degradation silent.

Detecting Silent Degradation (Not Just Hard Failures)

Silent degradation is the state where the pipeline is functioning but declining. It does not trigger error handlers. It does not set exit code 1. It does not fail the health check. It just makes the pipeline slightly worse every week until Knox notices the articles have become mediocre.

The three most common forms of silent degradation in a content pipeline:

Quality drift: The voice score trends downward over 2–3 weeks. Usually caused by a model provider silently upgrading their default model, a voice profile that is no longer loading correctly, or a system prompt that has grown too long and is crowding out the voice context.

Cost regression: The cost per article trends upward. Usually caused by a prompt that was edited and is now longer, a model routing regression that switched from a cheaper to a more expensive model, or retry storms from a degraded provider that are burning quota.

Provider degradation: The image fallback_depth trends upward — the pipeline is still publishing images, but it is using its second or third provider more often than its primary. Usually caused by quota exhaustion, rate limiting accumulation, or a silent API change in the primary provider.

All three degradation patterns share one property: they are detectable from stage metrics before they become hard failures. The catch is that you need a rolling window, not just a threshold. A single bad voice score is variance. Three consecutive bad voice scores is a trend. A threshold on a single value fires on noise; a threshold on a rolling window fires on signal.

def check_quality_trend(db: sqlite3.Connection, floor: float = 0.70, window: int = 3) -> None:
    """Fire a degradation alert if the last `window` voice scores are all below `floor`."""
    rows = db.execute(
        "SELECT value FROM stage_metrics "
        "WHERE stage='draft' AND metric='voice_score' "
        "ORDER BY id DESC LIMIT ?",
        (window,)
    ).fetchall()

    scores = [r[0] for r in rows]
    if len(scores) == window and all(s < floor for s in scores):
        discord_alert(
            level="warning",
            summary=(
                f"Draft quality degradation — {window} consecutive voice scores "
                f"below {floor}: {[round(s,2) for s in scores]}"
            ),
        )

This check does not require the stage to fail. It runs after every successful DRAFT completion. If the last three scores are all below 0.70, Knox gets a warning before the next three articles are published.

Turning Recurring Questions Into Dashboards

A dashboard that nobody uses is not a dashboard — it is a monument to good intentions. The way to build a dashboard that gets used is to start from the questions Knox actually asks, not from the metrics that are easy to collect.

Knox asks:

• "Did the pipeline run today?" → output_volume, checked hourly on publish days
• "Is the content quality holding up?" → voice_score_trend, last 10 runs
• "Is the pipeline getting expensive?" → cost_usd_per_run, trended weekly
• "Is the image provider degrading?" → fallback_depth_trend, trended daily
• "How long is a typical run?" → pipeline_latency_p50 and p95, trended

Every question maps to exactly one metric. That metric maps to exactly one alert rule. The alert fires when the metric crosses a threshold for a rolling window that eliminates noise. Knox sees the alert before the failure propagates.

The Heartbeat-to-Health-Check Upgrade

If your current monitoring is a process heartbeat — pgrep, launchd status, or exit code checks — you can upgrade it to an artifact growth check in one afternoon.

The upgrade has three steps:

1. Add a run_log table to your pipeline's SQLite database (or a flat file if you do not use SQLite):

# One row per completed pipeline run
db.execute("""
    CREATE TABLE IF NOT EXISTS run_log (
        id INTEGER PRIMARY KEY,
        topic_id TEXT,
        stage TEXT,
        completed_at TEXT,
        pr_url TEXT
    )
""")

2. Write a completion record at the end of every successful run:

db.execute(
    "INSERT INTO run_log (topic_id, stage, completed_at, pr_url) VALUES (?,?,?,?)",
    (topic_id, "publish", dt.datetime.now(dt.timezone.utc).isoformat(), pr_url),
)
db.commit()

3. Check artifact growth in a separate health check cron (runs every 2 hours on publish days):

from datetime import datetime, timezone, timedelta

def artifact_growth_check(db: sqlite3.Connection) -> None:
    today_start = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0).isoformat()
    count = db.execute(
        "SELECT COUNT(*) FROM run_log WHERE completed_at >= ? AND stage='publish'",
        (today_start,),
    ).fetchone()[0]

    # On a publish day (Mon/Wed/Fri), expect at least one completed run by 11:00 AM
    now = datetime.now(timezone.utc)
    is_publish_day = now.weekday() in (0, 2, 4)
    is_late_enough = now.hour >= 11

    if is_publish_day and is_late_enough and count == 0:
        discord_alert(
            level="warning",
            summary=f"Pipeline health check FAILED — no completed publish run today ({now.date()})",
        )

This check is independent of the pipeline process. It fires whether the process ran or not. It catches the zombie scenario (process ran, produced nothing) and the missing run scenario (process never ran) with the same query.

The Complete Observability Checklist

Before declaring a pipeline production-observable, verify these are in place:

Per stage:

• Structured JSON log event on stage completion (not just on failure)
• At least one quality/performance metric stored in the metrics table
• An alert threshold for the most dangerous per-stage failure mode

Pipeline-wide:

• Artifact growth check running every 2 hours on publish days
• A cost_per_run trend tracked weekly and alerted if > 1.30× baseline
• A rolling window degradation check for the most important quality signal

Alert delivery:

• All alerts route to the Discord webhook, not just to logs
• Degradation alerts are level warning; hard failure alerts are level error
• Recovery notifications fire when the degradation window clears

What this is not:

• A full Prometheus + Grafana stack. SQLite metrics tables and a Discord webhook are sufficient for a solo-operated pipeline.
• Logging every function call. One structured event per stage completion is the right granularity.
• A replacement for the artifact growth check from the “Self-Healing & Failure Alerting” lesson. Structured metrics extend that check — they do not replace it.

Why This Matters: The Compound Effect

The previous lessons in this track built the pipeline. This lesson instruments it. The gap between the two is the difference between a pipeline you operate and a pipeline that operates itself.

When Knox is on vacation, the pipeline runs. When Knox returns, he checks the metrics dashboard: voice scores held above 0.75 for all 6 runs, cost per run stayed within 10% of baseline, zero fallback events on the image stage. The pipeline produced six articles while he was away and did it without any of the warning signs that precede a degradation event.

That confidence requires the observability layer. Without it, Knox returns from vacation and has to manually read six articles to check quality, pull the API bill to check cost, and dig through logs to check provider health. With it, he checks a dashboard for 90 seconds and confirms everything is green.

The observability layer is the instrumentation that makes the pipeline trustworthy for unsupervised operation — not just functional, but verifiably healthy.