ASK KNOX
beta
LESSON 280

Stale Config Drift

Deploy targets accumulating untracked edits. The broker_bridge.py file that blocked git pull on Tesseract. Why deploy targets must be pure git reflections. Pre-flight git status as reflex — not an afterthought.

7 min read

Deploying Hermes PR #26 to Tesseract should have been a git pull away. It was not. The pull failed because a broker_bridge.py file existed locally on Tesseract but was not tracked in git. A prior agent had copied it in manually during an earlier session, hoping to smoke-test before the PR landed. The file was never committed. Two days later, the real PR merged with a different version of the same file. The pull refused to continue.

This is stale config drift. It is the silent failure mode of deploy targets, and it accumulates invisibly until the next pull — which then fails at the worst possible moment.

Why Drift Happens

Three patterns produce drift on deploy targets:

  1. Emergency hotfix. Something is broken in production. An operator SSHes in and edits a file directly. The fix works. The edit is never ported back to git. The next deploy either overwrites the fix (reintroducing the bug) or conflicts with the fix (blocking the deploy).

  2. Pre-merge smoke test. Someone copies a file onto the target to test a change before the PR lands. The file is now untracked. When the PR eventually merges with the same file, the deploy tries to create a file that already exists.

  3. Manual dependency install. An operator installs a package, creates a config file, or adds a cron entry to make something work. The step is not in the deploy script. The next time the environment is rebuilt from scratch, the manual step is missing and the system is subtly broken.

None of these are malicious. Each one starts as a reasonable action under time pressure. They accumulate because there is no alarm bell — git does not tell you "a file drifted" until you try to pull, and by then the deploy is already blocked.

Inline Diagram — Drift Accumulation

DEPLOY TARGET DRIFT — HOW IT ACCUMULATESGIT SOURCEcanonical stateday 0DEPLOY TARGETmatches gitNEXT git pullBLOCKS on untracked fileday N

The Pre-Flight Check

Every deploy script should run git status on the target before attempting a pull. Any untracked files or local modifications surface as errors, not warnings:

#!/bin/bash
cd /opt/hermes
status=$(git status --porcelain)
if [ -n "$status" ]; then
    echo "ERROR: deploy target is not clean"
    echo "$status"
    echo "refusing to deploy until working tree is clean"
    exit 1
fi
git pull origin main
./restart.sh

Ten lines. Every deploy. No exceptions. If the working tree is dirty, the deploy refuses to proceed, the operator investigates the drift, and the environment is restored to a clean git reflection before the deploy continues.

The Recovery Pattern

When a pre-flight check fails, the recovery is:

  1. Inspect the untracked or modified files. What are they? Why are they there?
  2. Port the changes back to git if they are legitimate. Open a PR, review, merge.
  3. Delete the local changes if they are obsolete. git clean -fd and git checkout -- ..
  4. Re-run the pre-flight check. It should now pass.
  5. Proceed with the deploy.

Never resolve a drift case by force-overwriting. Force-overwrites destroy the very information needed to understand why the drift happened.

The Rule

Stale config drift is preventable with one git status call per deploy. Every deploy target is pure. Every manual edit is a PR. Every untracked file is investigated before it blocks a deploy under time pressure. The Hermes Tesseract deploy cost extra minutes because this check did not exist. It exists now — and so does the discipline.