Ask Knox

Introduction

You have built the four stages. The cron fires three times a week. Articles are going live. PRs are getting reviewed and merged. The pipeline is working.

Now what?

Most content pipeline tutorials stop here. They treat "the pipeline is running" as the finish line. It is not. A pipeline that publishes without measuring is a — high output, unknown direction. What you want is a — one that learns from what works and uses that learning to get progressively better at what it produces.

Closing the feedback loop is the difference between those two things. This is how you do it.

Core Concept

What to Measure

Not all metrics are equally useful for a content pipeline. Start with three:

1. Engagement rate by article. Time on page and scroll depth, measured by an analytics tool like PostHog, are the most reliable early signals of content quality. An article with a 4-minute average time on page is resonating with readers who stay. An article with a 45-second average is being skipped after the intro. This signal is available within 72 hours of publish — fast enough to inform the next week's topic selections.

2. Traffic source by topic category. Where do readers come from for each category — organic search, direct, social? A category where 80% of traffic is organic search is building sustainable authority. A category where 80% of traffic is from social shares is dependent on your promotion and will drop when you stop promoting. Knowing this tells you which categories compound and which require active maintenance.

3. Topic-level performance variance. Within a category, do some topic angles consistently outperform others? Articles about specific tools tend to outperform articles about abstract concepts. Tutorials outperform opinion pieces for search traffic. How-to formats outperform "why X matters" formats for time on page. These patterns are actionable: they tell you how to sharpen your topic queue.

The Feedback Loop Mechanism

The feedback loop has three steps:

Step 1: Export. At the end of each month, export performance data from PostHog (or your analytics tool) at the article level. You want, per article: publish date, category, title, page views (30-day), avg time on page, scroll depth, top traffic source.

Step 2: Score. Apply a simple scoring model to each article. For jeremyknox.ai, a reasonable scoring formula is:

score = (avg_time_on_page / 4.0) * 0.4
      + (scroll_depth / 0.75) * 0.4
      + (organic_traffic_share) * 0.2

This weights engagement (time + depth) at 80% and organic share at 20%. Adjust the weights for your priorities. The output is a 0 to 1 score per article.

Step 3: Feed back. Update the topic queue using the performance scores. Specifically:

Topics that outperformed (score > 0.7): generate related angles, add them as priority queue items
Topics that underperformed (score < 0.3): flag the angle as weak, add it to an exclusion list or drop its priority
Categories with consistently strong performance: increase their representation in the round-robin
Categories with consistently weak performance: investigate topic specificity before reducing their slot

This is a monthly process, not a daily one. The feedback loop runs on a slow cycle that allows enough data to accumulate for each article before drawing conclusions.

What to Do With Underperformers

An article that scores poorly is data, not failure. The questions to ask:

Was the topic too broad? ("The future of AI" performs worse than "Why chain-of-thought reasoning fails on planning tasks.")
Was the angle off for this audience? (A crypto-heavy angle in the engineering category may not find readers who came for technical depth.)
Was the publish timing wrong? (An article published the same week as a major competing piece in the same space gets crowded out.)
Was the voice profile drifting? (If multiple underperformers in a row have a generic AI tone, the voice profile needs updating.)

Each of these has a different fix. Do not optimize for the metric before diagnosing why the metric is where it is.

When the Pipeline Gets Smarter

After six months of running the feedback loop, you will notice something: the pipeline starts to feel like it has opinions. The topic queue has converged toward topic types and angles that consistently work for your audience. The round-robin allocates more cycles to the categories that compound. The voice profile has been tuned based on patterns from review edits.

This is the compounding effect from the sourcing side. The pipeline is not just publishing faster — it is publishing more accurately toward what your audience responds to.

The jeremyknox.ai pipeline is still early in this maturity curve. The data is accumulating. The feedback loop is a quarterly process right now, not a monthly one. That is fine — consistency comes first, optimization comes second.

Practical Application

Here is a concrete monthly feedback workflow for a solo operator running a pipeline like jeremyknox.ai:

Week 1 of each month (30 minutes):

Export PostHog data for the previous month: articles published, time on page, scroll depth, traffic source breakdown
Apply the scoring formula in a spreadsheet
Identify top 3 performers and bottom 3 performers

Week 2 of each month (45 minutes):

For top performers: what do they have in common? (Specificity? Format? Category? Time of publish?) Add three similar topic angles to the manual priority queue.
For bottom performers: diagnose the failure mode (see the questions above). Flag the angle pattern in the topic queue notes.
Review category balance: is any category over- or underrepresented in the top performers?

Ongoing (5 minutes after each PR merge):

Record the article slug, title, category, and your review time (how long the editing pass took) in a simple log. Review time is a proxy for draft quality — longer reviews indicate voice profile drift or topic misalignment.

The review time log is the one measurement most operators skip. It is also one of the most useful. If your editing pass time increases from 15 minutes to 35 minutes over three months, something in the pipeline is degrading — most likely the voice profile or the topic specificity.

Common Mistakes

Measuring volume instead of engagement. Total articles published is a pipeline health metric, not a content quality metric. Do not optimize for output count. Optimize for the engagement metrics that predict reader value.

Drawing conclusions from too-small samples. An article that published two days ago with 12 page views and a 2-minute average time on page is statistically meaningless. Wait 30 days before scoring an article's performance. Wait three months before drawing category-level conclusions.

Optimizing the pipeline for metrics you cannot control. Social shares depend on your audience size and promotion habits. Backlinks depend on other sites' editorial decisions. Optimize for time on page and scroll depth — metrics that are almost entirely a function of content quality, which is the pipeline's job.

Not updating the voice profile from editing data. Every editing pass you do is a signal: what did AI produce that you changed? Keep a quick log of the types of edits you make. After 20 articles, patterns will emerge. Those patterns are the voice profile updates you should make.

Treating measurement as a separate project. Measurement works best when it is built into the pipeline workflow from the start — not bolted on later. Add the article-level tracking fields to your state file from day one. Add the review time log from day one. Retroactively reconstructing six months of data from scattered notes is painful.

Summary

Measure engagement rate (time on page, scroll depth), traffic source by category, and topic-level performance variance.
The feedback loop is monthly: export data, score articles, update the topic queue with what works and what does not.
Review time (how long your editing pass takes) is a proxy for draft quality — track it per article.
Top performers reveal angles to replicate; underperformers require diagnosis before optimization.
After six months of consistent loop execution, the pipeline develops a compounding accuracy advantage — it publishes better, not just more.

What's Next

You have completed the "Your First AI Content Pipeline" track. You have the full architecture: SOURCE selects topics using creator feeds, trend signals, and a manual priority queue. DRAFT generates articles using a voice profile that teaches the model to write like you. IMAGE creates hero visuals with a three-provider fallback chain. PUBLISH stages the article as a pull request for human review and notifies via Discord. A cron runs the whole sequence three times per week. And the feedback loop closes the circle — turning performance data into better sourcing decisions each month.

The pipeline behind this very blog is the working example. Now go build yours.

Measuring What Actually Works