Sourcing at Scale — Feeds, Transcripts & Topic Queues

The Sourcing Problem at Scale

The beginner pipeline sourced one topic at a time, manually. You opened a browser, found an interesting article, copied the URL, and passed it to Claude. That process does not scale to three articles per week, every week, indefinitely.

Production sourcing has two problems: volume and quality. You need enough candidates to always have three good topics ready for the weekly schedule. And you need a filter system that ensures those three topics are not repetitive, not already covered, and not stale.

The Three Source Types

Knox's blog-autopilot ingests from three source types, each with different fetch mechanisms, freshness cadences, and volume characteristics:

RSS feeds are the highest-volume source. AI lab announcements (Anthropic, OpenAI, Google DeepMind), tech blogs, and domain-specific feeds generate 40–200 candidates per week. The signal-to-noise ratio is low — most items are either not worth covering or are duplicates of items that have already been published.

YouTube transcripts are the highest-quality source. Knox follows specific AI engineering channels where the content is deep, original, and not widely covered in text form. The Supadata API (extract.py transcript <video_url>) fetches the full transcript. Volume is low (5–15 per week) but signal is high — a 45-minute deep dive by Andrej Karpathy is worth a dedicated article.

Manual queue entries are the highest-priority source. When Knox has a specific topic he wants covered — based on something he encountered, a community question, or a strategic content gap — he adds it directly to the queue via CLI: python gather.py add-topic --title '...'. Manual entries always receive priority 1.

The Topic Queue Flow

All three source types funnel into the same queue system via five sequential steps:

The most critical step is CLAIM — the atomic state transition from pending to processing with an open run manifest. This is the exactly-once guarantee. If the pipeline crashes after claiming but before completing, the run manifest remains open. On restart, the pipeline detects the open manifest, resets the topic to pending, and claims it cleanly. Without this mechanism, a crash mid-run leaves the topic stuck in processing forever.

The Dedup Filter in Detail

Deduplication runs at three levels, applied in sequence:

Level 1 — URL hash dedup. Every source URL is hashed on ingestion and checked against the seen_urls set in the pipeline state database. An exact URL match is an immediate hard reject. This catches 60–70% of duplicates — the same story appearing in your RSS feed from The Verge, TechCrunch, and VentureBeat.

Level 2 — Title similarity. For the items that survive URL dedup, the title is compared against all titles in the last 30 days using fuzzy string matching. A similarity score above 85% is a soft reject — the item is marked as similar_to: <existing_topic_id> and skipped unless Knox overrides manually. This catches "Five Ways AI Will Change X" variants across different publications.

Level 3 — Topic cluster saturation. If three or more items in the same semantic topic cluster (e.g., "AI coding tools") have been processed in the last 72 hours, new items in that cluster are pushed to a future slot. This prevents a single news cycle from dominating the weekly schedule.

intake.json as the Single Source of State

The intake.json file is the canonical state for the entire sourcing layer. Every topic that enters the pipeline passes through it. Every transition — pending, processing, done, failed — is recorded here. Every retry, every skip, every manual override writes to this file.

This makes intake.json the correct place to inspect when something goes wrong with sourcing. Is the pipeline selecting the wrong topics? Check the priority fields. Is the pipeline re-processing old topics? Check the status fields. Is the pipeline missing new topics? Check the last_fetched timestamp on the source feeds.

The devlog-engine pattern adds one critical behavior: intake items consumed on generation are reset on failure. If the DRAFT stage fails mid-generation, the intake entry reverts to pending. The pipeline never marks an entry as done until all stages have completed and the delivery receipt has been written.

Scaling the Source Without Scaling the Queue

The production sourcing goal is not to process more articles per week. It is to maintain consistent quality at a fixed cadence (Mon/Wed/Fri) while the source volume grows. The dedup filter is what makes this possible — it absorbs volume growth without changing the output rate.

When Knox adds a new RSS feed, he does not need to adjust the pipeline's output rate. The dedup layer handles the increased candidate volume. When a major AI announcement generates 50 articles in one day, the pipeline selects one and defers the rest to future slots or rejects them as low-priority duplicates.

This is the sourcing architecture that makes the Mon/Wed/Fri cadence sustainable: not a bigger pipeline, but a smarter filter.