ASK KNOX
beta
Learn/Your First AI Content Pipeline/Lesson 362
LESSON 362

Sourcing: Where Good Topics Come From

The quality of every article your pipeline generates is bounded by the quality of what you fed the source stage — garbage in, garbage out, at scale.

9 min read·Your First AI Content Pipeline

Introduction

The SOURCE stage is the quietest part of the pipeline and the most important. Nobody sees it. It runs in a few seconds. It produces a small JSON file that most operators treat as an afterthought.

It is not an afterthought. It is the ceiling.

Every article your pipeline generates will be bounded by the quality of the topic input it received. Feed the pipeline shallow, repetitive, or off-strategy topics and you get shallow, repetitive, off-strategy articles — at scale, automatically, three times a week. That is a , not a compounding asset.

This lesson covers where good topics come from, how to structure your sourcing layer, and how to prevent the common failure mode of a pipeline that technically works but produces content nobody wants to read.

Core Concept

Three Input Types

The most reliable sourcing systems combine three types of inputs:

1. Trusted creator feeds. Identify three to eight creators in your domain whose content is consistently high quality and whose audience overlaps with yours. Subscribe to their RSS feeds, monitor their YouTube uploads, or pull their recent posts via API. These creators have already done editorial filtering — they decided the topic was worth publishing. Your job is to find your own angle on it, not copy theirs.

For jeremyknox.ai, this means monitoring AI engineers, operators, and builders whose work is signal-dense rather than hype-heavy. When someone like that publishes a thread or video about a specific tool or architectural pattern, that is a legitimate sourcing signal.

2. Trend signals. Tools like Google Trends, Exploding Topics, and X/Twitter search surface what is gaining traction in your space right now. These are good for timing — knowing when a topic is peaking so your article lands while the audience is actively searching. The risk is that trending topics attract a lot of competing content, which makes differentiation harder. Use trend signals to sharpen timing, not as your only sourcing method.

3. Manual priority queue. A manually maintained list of topics you know you want to cover — things no trend tool would surface yet. Upcoming product launches. Evergreen concepts competitors have covered badly. Questions your audience asks repeatedly in Discord or email. The manual queue is where your strategic intent lives.

The jeremyknox.ai gather.py script reads from a JSON file of manually curated topics as its primary source. Trend signals inform when to promote certain items in that queue. Creator feeds add fresh inputs weekly.

The Rotation and Category Logic

If your pipeline can cover multiple categories, you need rotation logic — otherwise the pipeline will default to whatever is easiest to generate and your content mix becomes lopsided.

The jeremyknox.ai pipeline tracks state across runs. After each article, it records which category was covered. The next run selects the category that has been least recently published. If you run Monday/Wednesday/Friday and cover AI on Monday, the Wednesday run picks from a different category (engineering, strategy, crypto). This keeps the mix balanced and prevents any single topic from dominating.

State tracking is a simple JSON file: { "last_category": "ai", "last_run": "2026-05-30", "published_topics": [...] }. Each run reads this file at start, and writes to it on success.

The Deduplication Problem

Without deduplication, a pipeline running for six months will inevitably republish the same angles. Search engines penalize thin duplicate content. Readers notice. The pipeline's authority erodes.

The fix is simple: maintain a log of published topic slugs and topic titles. Before selecting a topic from the queue, check whether a similar title already exists in the log. A fuzzy match (checking if key terms overlap) is better than an exact match — "How AI Memory Works" and "Understanding AI Memory Systems" are the same topic.

At scale, some teams use embedding-based similarity search to detect near-duplicate topics. For a solo pipeline, a keyword overlap check is sufficient to start.

Practical Application

Here is a minimal gather.py logic flow for a pipeline like jeremyknox.ai:

1. Load state.json → read last_category, published_topics[]
2. Select next_category via round-robin (exclude last_category)
3. Load topic_queue.json → filter to next_category items
4. Remove topics where title overlaps with published_topics (deduplication)
5. Select top item (highest priority score or most recently added)
6. Enrich the topic record:
   - Pull related URLs from creator feeds
   - Add competitor article links for context
   - Note word count target and audience level
7. Write topic.json with full record
8. Exit 0 (success) or exit 1 with Discord alert (failure)

The output topic.json might look like:

{
  "title": "Why AI Agents Need Persistent Memory",
  "angle": "Most agents are stateless — here is why that breaks in production and how memory layers fix it",
  "category": "ai",
  "target_word_count": 1200,
  "audience": "technical operators",
  "related_links": ["https://...", "https://..."],
  "priority": 1
}

This is what the DRAFT stage consumes. The richer this record, the better the draft.

Common Mistakes

Sourcing from low-quality or repetitive feeds. If your creator feeds publish five posts per day, most of them are filler. Quantity is not a quality signal. Be selective about which sources you monitor. Fewer, better sources outperform a wide, noisy firehose.

No priority mechanism. If the pipeline selects topics randomly from the queue, strategic items get buried. Assign a priority field to each topic. Time-sensitive items (tied to a product launch, a breaking trend) should have higher priority so the pipeline surfaces them before they become stale.

Treating the topic queue as a set-and-forget list. The queue needs regular maintenance. Topics become stale. Competitors cover them. Your audience's interests shift. Review and prune the queue monthly. A pipeline sourcing from a stale queue is quietly degrading.

Overly broad topic framing. "The future of AI" is not a topic — it is a category. A good topic record has a specific angle: "Why chain-of-thought reasoning fails on multi-step planning tasks and what structured generation does instead." Specificity produces better drafts and more useful articles.

Missing "already ran today" guard. If your pipeline can be triggered manually or by multiple cron events, add a guard that checks whether today's date already has a completed run in state. The gather.py script on jeremyknox.ai exits with "Already ran today" if state shows a same-day completion. This prevents duplicate articles from a double-trigger.

Summary

  • Good sourcing combines trusted creator feeds, trend signals, and a manually maintained priority queue.
  • State tracking (last category, published topics) enables category rotation and deduplication across runs.
  • The topic record must be specific — angle, audience, context — not just a working title.
  • Topic queue maintenance is monthly work; a stale queue quietly degrades pipeline output.
  • A "already ran today" guard prevents duplicate articles from pipeline double-triggers.

What's Next

The next lesson covers the DRAFT stage — specifically how to get an AI model to write in your voice rather than generic AI prose, what a voice profile contains, and how to structure your editing pass so reviews stay light rather than becoming rewrites.