Self-Healing & Failure Alerting

The Silent Failure Problem

The most dangerous failure mode in a production pipeline is not a crash. It is a pipeline that runs, produces nothing, and exits 0. Knox won't know until he checks why Monday's article never appeared.

This failure mode — the zombie pipeline — is common because most pipeline code handles the happy path well and handles failure by logging and exiting. "Logged and exited" is not the same as "alerted and recovered." The logging is for debugging. The alerting is for Knox. The recovery is for the pipeline itself.

The Five Failure Modes and Their Responses

Knox's blog-autopilot has encountered all five of these failure modes in production. The current response protocols are the result of learning from each:

The most instructive failure mode is silent crash. A gather.py crash mid-execution leaves the run manifest open, the intake entry in processing state, and the Discord channel silent. Knox discovers the failure on Wednesday when the pipeline runs normally and he notices Tuesday's article is missing.

The fix required two changes: 1) the manifest staleness check (any open manifest > 15 minutes triggers an alert) and 2) the Discord webhook on failure. Now, within 60 seconds of a crash, Knox sees: ⚠️ Pipeline stalled — manifest open for 15min — topic_id: xyz — last stage: DRAFT. He can respond or ignore knowing the next scheduled run will auto-recover.

Heartbeat vs Health Check: The Most Common Mistake

The most common monitoring mistake in pipeline operations is confusing process liveness with pipeline health. They are not the same metric.

The concrete example from Knox's memory: heartbeat is not pipeline health — prove downstream artifact growth, not just process liveness. This lesson came from a period where the blog-autopilot process was running (confirmed by pgrep -afl blog-autopilot) but not publishing. The process was alive but the draft generation was failing silently and the failure handler was exiting 0.

The fix was adding a last_published timestamp to the run state database and checking it against the expected cadence: "If today is Monday and the last PR was not opened today, something is wrong." This check fires regardless of whether the process ran — it checks the output, not the process.

The Discord Alert Architecture

Every failure mode in the pipeline has a corresponding Discord webhook call. The webhook sends an embed with:

• Color: red for failures, yellow for warnings, green for recovery
• Title: the failure mode and stage
• Fields: topic_id, topic_title, stage where failure occurred, error message
• Timestamp: exact UTC time of failure

The alert fires before the intake entry is reset to pending. Knox receives the alert immediately. If he is available, he can investigate and decide whether to intervene. If he is not available, the next scheduled run will auto-recover.

The webhook URL is stored as DISCORD_WEBHOOK_URL in the plist EnvironmentVariables. If the webhook fails (Discord outage, deleted channel), the failure is logged but does not cascade — the pipeline's failure handling does not itself fail. The webhook failure is logged as a separate event.

The Intake Reset Pattern

The devlog-engine pattern that Knox uses for failure recovery is: intake items consumed on generation are reset to pending on failure. This means:

1. Topic is claimed (status: processing)
2. Run manifest opened
3. Generation begins
4. On any unrecoverable error: status reset to pending, manifest closed, Discord alert fired
5. On success: status set to done, manifest closed, delivery receipt logged

The reset pattern makes the pipeline idempotent on failure — every failed run leaves the system in a clean state for the next run. No manual cleanup required.

The one case where reset-on-failure is not sufficient is permanent failures: a topic that fails because of a permanent content policy violation will be retried indefinitely. Knox's pipeline handles this with a failure_count field: after 3 failures, the entry transitions to permanently_failed and requires manual review.

The Complete Self-Healing Loop

The full self-healing sequence for a production pipeline failure:

1. Stage fails — exception thrown or quality gate returns false
2. Manifest updated — failure stage logged to run manifest
3. Intake entry reset — status reverts to pending if retryable; permanently_failed after 3 attempts
4. Discord alert fires — within 60 seconds, Knox sees the failure details
5. Pipeline exits — exit code 1 for retryable failures, exit code 2 for permanent failures
6. launchd observes — if exit code 0 expected and job is periodic, launchd waits for next interval
7. Next run recovers — next Mon/Wed/Fri slot: idempotency checks detect the reset entry, processes it cleanly

This loop means that for any transient failure, the recovery is automatic and Knox's only required action is reading the Discord alert to confirm the failure was transient.

Implement that loop in a watchdog: detect a stall by manifest staleness, alert Discord exactly once, restart, and re-arm only on recovery so a long outage doesn't bury the signal under repeat pings.

Self-healing is infrastructure debt you pay once. The alternative — manual Monday morning pipeline debugging — is operational debt you pay every time the pipeline fails. The pipeline will fail. The question is whether you discover it in 60 seconds or on Wednesday.