ASK KNOX
beta
Learn/Your First AI Content Pipeline/Lesson 361
LESSON 361

The Four-Stage Pipeline at a Glance

SOURCE feeds DRAFT, DRAFT feeds IMAGE, IMAGE feeds PUBLISH — four clean handoffs that turn a topic queue into a live article.

7 min read·Your First AI Content Pipeline

Introduction

Most pipeline discussions drown you in implementation details before you have a mental model of the whole system. This lesson does the opposite: it gives you the map first.

The jeremyknox.ai content pipeline has four stages: SOURCE, DRAFT, IMAGE, PUBLISH. Every automated content system I have seen — from solo blogs to newsroom tools — fits this same backbone. The implementations differ. The stages do not.

Understanding this architecture before you write a single line of code means every subsequent decision has a home. When you hit a failure in week three, you know immediately which stage failed and what it was supposed to hand off.

Core Concept

Stage 1: SOURCE

The SOURCE stage answers one question: what should we write about today?

It consumes a topic queue — a list of potential article ideas, creator channels, trend signals, or manual priorities — and produces one structured topic record. That record contains at minimum:

  • A working title
  • The angle or thesis
  • The target category (AI, engineering, strategy, etc.)
  • Any supporting context (links to source material, existing related articles, notes)

The SOURCE stage does not write. It selects and enriches. Its output is a compact JSON object or plain text file that the DRAFT stage reads.

In the jeremyknox.ai pipeline, gather.py handles this stage. It rotates across categories using a round-robin schedule tracked in state, pulls from a manually maintained priority queue, and writes a structured record to disk before exiting.

Stage 2: DRAFT

The DRAFT stage answers: what should this article say, and in whose voice?

It consumes the topic record from SOURCE and produces a finished article draft — typically as MDX or Markdown. The DRAFT stage is where the AI model does the heavy lifting, but it needs guidance: a voice profile, a structural template, an audience definition, a word count target.

The output of DRAFT is a text file. It should be close to publishable. It does not need to be perfect — that is what the human review gate is for — but it should require editing, not rewriting.

In the jeremyknox.ai pipeline, this stage calls Claude with a system prompt that includes existing article samples, explicit style rules, and the topic record from gather.py.

Stage 3: IMAGE

The IMAGE stage answers: what visual represents this article?

It consumes the draft (or just the title and excerpt) and produces one or more image files. Most pipelines use an AI image generation API here. The IMAGE stage needs a fallback chain — if your primary provider is rate-limited, down, or rejects the prompt, you need a secondary and tertiary option. More on this in the “Hero Images and the Fallback Chain” lesson later in this track.

The output is a file path to the generated image, recorded alongside the article draft.

In the jeremyknox.ai pipeline, generate_image.py calls Leonardo AI Phoenix 1.0 as the primary provider, with fallbacks defined for cases where that API returns an error.

Stage 4: PUBLISH

The PUBLISH stage answers: how does this article get to readers?

It consumes the draft and image path, assembles the final MDX file with correct frontmatter, and delivers it to the publishing target. For most static sites, this means opening a pull request on GitHub with the assembled file. The human merges the PR after review.

The output of PUBLISH is not a live article — it is a staged article awaiting approval. That distinction is intentional. Full automation is a goal you reach after your pipeline proves consistent quality over dozens of cycles.

In the jeremyknox.ai pipeline, deliver.py handles this stage. It creates the MDX file, commits it to a feature branch, and opens a PR. A Discord notification fires with a preview link.

The Handoff Contract

The most important architectural rule: each stage must be independently runnable. Stage 2 should not care how Stage 1 found the topic. Stage 3 should not care how Stage 2 generated the prose. This is .

The handoff between stages is a defined contract: a file on disk, a JSON object, a set of environment variables. Keep these contracts simple and explicit. If a stage fails, you can re-run that stage in isolation without touching the others.

Practical Application

Here is the full flow for one article on jeremyknox.ai, end to end:

9:00 AM  Cron fires
9:00 AM  gather.py runs → reads topic queue → selects "AI memory systems" (AI category, round-robin)
           → writes topic.json: { title, angle, category, context }
9:02 AM  Claude drafting step runs → reads topic.json → generates 1200-word article
           → writes draft.mdx (no frontmatter yet, raw prose)
9:04 AM  generate_image.py runs → reads draft title + excerpt → calls Leonardo API
           → writes image to /public/images/blog-autopilot/ai-memory-systems.png
9:06 AM  deliver.py runs → assembles final MDX with frontmatter + image path
           → git commit on feature branch → gh pr create → Discord notification fires
9:07 AM  Pipeline exits successfully
Noon     Knox opens PR link from Discord → reads article → minor edits → merges
12:05 PM Vercel deploys → article is live

Total pipeline runtime: under 10 minutes. Human time: 15 minutes of review. Article from idea to live: same morning.

Common Mistakes

Coupling stages tightly. If your DRAFT stage directly calls your IMAGE stage, which directly calls your PUBLISH stage, you have one long script — not a pipeline. When Stage 3 breaks, you cannot diagnose it without running all of Stage 2 again. Keep stages as separate scripts with explicit inputs and outputs.

No intermediate state. If a stage runs and produces no output file, the next stage has nothing to consume. Store intermediate artifacts — topic.json, draft.mdx, image_path.txt — so you can inspect what each stage produced when debugging.

Treating the pipeline as one atomic operation. A cron that runs all four stages in sequence and exits with code 0 on success is fine. But log each stage separately. If Stage 3 fails, your log should say "Stage 3 failed: Leonardo API rate limit" — not "pipeline failed."

Skipping failure handling at each stage. Each stage needs its own error handling and its own failure notification. Do not assume a stage will succeed because the previous one did.

Summary

  • The four stages are SOURCE (select topic), DRAFT (generate article), IMAGE (generate cover), and PUBLISH (deliver for review).
  • Each stage has one input and one output; those handoffs are the pipeline's contracts.
  • Loose coupling — keeping stages independently runnable — is the key architectural rule.
  • The PUBLISH stage opens a PR for human review; it does not publish autonomously until quality is proven.
  • Intermediate artifacts (JSON files, draft files, image paths) are your debugging surface when stages fail.

What's Next

The next lesson goes deep on the SOURCE stage — where good topics come from, how to build a topic queue that stays fresh, and how to avoid the garbage-in problem that produces generic, low-value articles nobody reads.