Write-Path Verification: The Probe Write

A health endpoint returning 200 is necessary. It is not sufficient.

The SSL certificate mismatch on the new host only affects writes. The missing environment variable only breaks the database connection pool on write commits. The firewall rule that wasn't carried over only blocks the write path to the downstream queue. None of these affect the health endpoint. All of them mean users are losing data.

The pattern that catches this class of failure is the probe write.

What Read-Path Checks Cannot See

A health endpoint verifies that the process is running and can respond to HTTP. A dashboard showing green verifies that the metrics pipeline is working. A log with no ERRORs verifies that nothing has raised an exception yet. None of these checks require the write path to work.

A write requires every layer to succeed in sequence: authentication validates the token, routing delivers the request to the correct handler, application logic validates the input, the ORM constructs the query, the database connection accepts the commit, and any downstream dependencies the write triggers are reachable. A single failure anywhere in this chain silently drops the write. The health endpoint stays green. The users lose data.

The Probe Write

The probe write is a minimal, reversible end-to-end write that confirms the entire write path is alive. The design is four steps:

1. Create a probe record with minimal value. One row, one byte, the smallest legal payload. Include a flag — is_probe: true — so it can be filtered out of production queries if cleanup fails.

2. Verify read-after-write. After the create returns its ID, immediately GET that ID and confirm the record is returned. This catches the class of bug where the 201 response is genuine but the database transaction rolled back.

3. Delete the probe record. Call DELETE with the ID from step 1. Confirm it returns the expected status.

4. Confirm the deletion. Call GET again and verify a 404 is returned. This proves the delete actually ran, not just acknowledged.

# Minimal probe write — Python + urllib (no dependencies beyond stdlib)
import urllib.request, json

def probe_write(base_url: str, token: str) -> bool:
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json",
    }
    # Step 1: Create
    body = json.dumps({"title": "probe", "is_probe": True}).encode()
    req = urllib.request.Request(
        f"{base_url}/api/records",
        data=body, headers=headers, method="POST"
    )
    with urllib.request.urlopen(req) as resp:
        record_id = json.loads(resp.read())["id"]

    # Step 2: Read-after-write
    req = urllib.request.Request(
        f"{base_url}/api/records/{record_id}", headers=headers
    )
    urllib.request.urlopen(req)  # raises on non-200

    # Step 3: Delete
    req = urllib.request.Request(
        f"{base_url}/api/records/{record_id}",
        headers=headers, method="DELETE"
    )
    urllib.request.urlopen(req)

    # Step 4: Confirm deletion
    try:
        req = urllib.request.Request(
            f"{base_url}/api/records/{record_id}", headers=headers
        )
        urllib.request.urlopen(req)
        return False  # Still exists — DELETE failed silently
    except urllib.error.HTTPError as e:
        return e.code == 404  # 404 = confirmed deleted

When the Probe Write Is Unsafe

Not every write path can accept a live probe. Recognize these cases and use the appropriate alternative:

Financial transactions. A $0.01 order on a live exchange is real money on the exchange's books. Use the service's sandbox or test environment for the external call. The probe still confirms the service's internal write path (record created in the service's own database); it just does not send a real order to the exchange.

Irreversible side effects. A write that sends a real email, fires a webhook to an external system, or charges a payment cannot be undone. Probe up to the external system boundary — confirm the record was created in the service's own database — then mock or skip the outbound call. The write path through the service is confirmed; the external delivery is a separate concern.

No delete endpoint. If there is no way to remove the probe record, add an is_probe=true flag to the schema and a cleanup cron that removes flagged records. The probe still runs; cleanup happens asynchronously.

Declaring an Integration Live

The rule is simple: do not declare an integration live until the probe write passes.

This applies to every cutover: migrating to a new database host, upgrading the ORM, switching to a new queue provider, rotating credentials. The read path may work immediately after any of these changes. The write path may not. The probe confirms it does.

# Post-deploy integration check — fails fast on write-path failure
python3 probe_write.py --base-url "$NEW_HOST_URL" --token "$SERVICE_TOKEN"
echo "Write path confirmed. Integration is live."

Running the probe takes under 10 seconds. Discovering a silent write failure at 2 AM because a user filed a bug report takes hours.

The Relation to the “Last Real Trade as a Metric” lesson

The "last real trade" metric from the “Last Real Trade as a Metric” lesson is a liveness metric — it measures how long the system has been silent. The probe write is a cutover check — it confirms a specific change did not break the write path. They are different tools for different moments:

• Probe write: run once at deployment time to confirm the new version works.
• Liveness metric: run continuously to detect gradual failures and long-term silence.

Both are necessary. The probe write answers "did the deploy break writes?" The liveness metric answers "has the system stopped producing output?"