Skill Provenance: Knowing Which Skills to Trust

Accumulating skills is easy. Knowing which ones to trust is the hard part.

After enough months of building, your ~/.claude/skills/ directory looks healthy on paper — 40, 50, maybe 60 files covering deployments, test workflows, PR processes, API patterns, cron triggers. But file count is not the same as reliability. Some of those skills have been battle-tested across dozens of real executions. Others were written from memory one afternoon and never run against a live system. Without provenance tracking, you can't tell which is which — and neither can the agent.

40+ skills without provenance tracking is a liability, not an asset.

The Hollow Skill Problem

A hollow skill is one that has never been executed in production. It may be structurally correct — proper Markdown, clear description, reasonable steps. But every step in a hollow skill is an assumption about how the system works, not an observation of how it actually behaves.

The failure mode is not dramatic. Hollow skills don't crash loudly. They produce output that looks approximately right, uses the right terminology, follows the right structure — but gets some detail wrong that only a real execution would have caught. The wrong PATH prefix. The missing --no-pager flag. The step that assumes a service is already running when it isn't.

Worse: when an orchestrator dispatches a hollow skill that fails mid-execution, the failure often looks like model failure. The agent produced the wrong command, used the wrong flag, hit an unexpected error. Without provenance tracking, there is no way to distinguish "model made a mistake" from "skill contained an untested assumption." The debugging surface is the entire execution, not the specific skill.

The Provenance Schema

The fix is a frontmatter block at the top of each skill file that records its lineage:

---
skill: deploy-to-mac-mini
built_from: memory://session/2026-03-28-deploy-retro
iteration: 3
last_failure: "2026-04-01 — wrong PATH in non-interactive SSH context"
---

Each field carries information:

• built_from — the source of truth for this skill's content. A Semantic Memory Layer session ID means the skill was built from a real execution retro. A commit SHA means it was derived from a code change. An empty field means it was hand-authored with no execution record.
• iteration — how many times this skill has been through the recursive build loop. Iteration 1 is first draft. Iteration 3+ means it has survived real failures and been updated from them.
• last_failure — the most recent failure this skill produced, with date and description. This is not a shame record — it is operational intelligence. A skill with a documented last_failure is a skill that got tested and improved. A skill with no last_failure record either has never failed (unlikely) or has never been executed (common).

The Three Provenance Categories

Every skill in your library falls into one of three categories:

Battle-tested. built_from populated with a real session or commit. iteration greater than 1. last_failure documented (even if dated). These skills have been stress-tested against real systems and updated from what they revealed. You can dispatch them from an orchestrator with confidence.

Hand-authored. Written from memory or spec, no built_from entry. These skills may be perfectly correct — but they haven't been proven. They carry unvalidated assumptions. Before dispatching from an orchestrator, they need at least one real execution pass.

Unknown. No frontmatter at all. No history. These are the most dangerous category because their failure mode is indistinguishable from any other failure. Every skill without provenance frontmatter is, by definition, unknown.

The Provenance Audit Workflow

Run this once to categorize your skill library, then maintain it as part of your skill update cycle:

1. List all skills. ls ~/.claude/skills/*/SKILL.md
2. Grep for built_from. Skills with this field are categorized. Skills without it are hand-authored or unknown.
3. Flag hollow skills. Any skill that is hand-authored with iteration 0 or 1 is hollow.
4. Prioritize for validation. Which hollow skills are your orchestrators actually dispatching? Those need real-execution passes first.

The output is a simple three-column list: skill name, category (battle-tested / hand-authored / unknown), and priority for validation.

What to Do With Hollow Skills

Three paths:

Recursive rebuild. Run the skill against a real task on a real system. Capture the failure. Update the skill. Add provenance frontmatter. Repeat until iteration reaches 2+. This is the right path for any skill your orchestrators rely on.

Retire. If a hollow skill is superseded by a newer battle-tested skill covering the same procedure, delete the hollow one. Dead skills in the library create routing ambiguity — two skills with similar descriptions means the agent has to choose, and it may not choose the right one.

Accept-with-flag. For hollow skills that describe rarely-used procedures, document them explicitly as unvalidated speculation. Add a warning at the top of the skill content: "This skill has not been validated in production. Treat steps as a starting point, not a procedure." This is honest about the skill's state and prevents silent dispatch from orchestrators.

Why This Matters for Orchestrators

The provenance problem is most acute in orchestrated systems. An orchestrator that dispatches 5 skills per run, 50 runs per day, against a library of 40 skills is executing against thousands of skill invocations. A single hollow skill in a high-frequency dispatch path silently introduces errors at scale.

With provenance, you can enforce a dispatch policy: orchestrators may only dispatch battle-tested skills (iteration > 1). Hand-authored skills require manual invocation until validated. Unknown skills are blocked from dispatch entirely.

This is not over-engineering. It is the difference between an orchestration layer you can trust and one that occasionally produces subtly wrong results with no clear cause.

Track where your skills came from. It is the cheapest investment with the highest debugging return in the entire skill architecture.