PII Leak Paths in AI Pipelines

The Leak That Survived the Fix

A public learning platform ran a content pipeline powered by an AI agent. The agent processed user-submitted support transcripts to build a knowledge base — useful for the AI tutor that helped learners debug their code. A routine privacy audit discovered that a user's real email address was appearing in tutor responses. The security team traced it back to a transcript that had been included in the RAG corpus — the retrieval-augmented generation store the tutor searches for relevant context before answering — six weeks earlier.

Here is the part that stings: the fix had already been merged. Fourteen days before the audit, a developer noticed the email in a lesson file, redacted it, and pushed a PR that passed CI. The fix was correct. The source file was clean. But nobody re-indexed the RAG corpus after the merge — and the old embedding was still there, still serving the leaked value on every relevant semantic query.

The transcript was fixed. The leak was not.

How PII Travels

PII does not stay where you put it. An AI pipeline is a propagation machine: data enters the context window and copies of it end up in every artifact the pipeline touches.

A support transcript containing a customer's name and email address enters as context. The model sees the full value in its prompt — which means every log of that prompt now contains the PII. The model generates a summary that may reproduce the name verbatim. That summary gets embedded into the RAG corpus. An agent reads from the corpus and puts the name into a generated doc. The repo gets forked for a side project. Four artifacts, one source record.

The consequence is that you cannot fix a PII leak by patching one file. You need to know every artifact the value has propagated into, and you need to fix each one independently.

The three surfaces most teams miss:

RAG corpora and embeddings. This is the highest-risk surface because it is invisible. Embeddings are not text you can grep — they are vector representations of text that was there when the index was last built. The value lives in the index until you re-embed. A merged fix does not trigger a re-index unless you have wired that step explicitly.

Agent transcripts and logs. An agent transcript is a complete record of everything the agent saw, wrote, and was told. If the agent processed a CRM record containing a customer's home address, that address is now in the transcript log. Treat agent transcripts as PII-containing logs — apply the same retention, access control, and purge policies you apply to your application logs.

Forks and downstream consumers. A fork is a copy. When you fix a leak in the canonical repo and push a new version, the fork is still running the old version with the old corpus. The canonical fix stops new leaks from the canonical source; it does not reach anything downstream. Propagate the gate — not just the fix.

The Three Hard-Won Rules

These come from real remediation work on a production content pipeline. Each one was learned the hard way.

Rule 1: Redaction completes at re-embed. Merging the fix is step three of a six-step process. You still need to trigger the re-index, verify the live corpus no longer returns the leaked value, and close the ticket. Build the re-embed trigger into your CI/CD pipeline so it runs automatically when content files change. A manual step that depends on someone remembering is not a gate; it is a hope.

Rule 2: Forks re-leak what the canonical repo fixed. When you remediate a leak, audit downstream consumers: forks, mirrors, dependent services, and any system that ingested the original content. For each one, apply the fix and re-embed independently. More importantly, propagate the CI gate — the scanner that now blocks the pattern in the canonical repo should also be running in every fork that contains the same content type.

Rule 3: Scanners need allowlists or they get disabled. A PII scanner that flags every intentional example in a teaching document will generate so much noise that the team disables it. When a scanner goes silent, the next real leak sails through. Build allowlist support into the scanner from day one: pattern example@example.com, domain example.com, and any other intentional demonstration values get whitelisted with a comment explaining why. Everything else still triggers. The scanner stays active and trusted.

Placeholder Discipline

There is a subtler failure mode: realistic-looking fake values. A developer writing a tutorial uses sarah.johnson@techcorp.example as a fake email address to illustrate a form validation example. The email is not real — but it looks real. A learner copy-pastes it into a production system. Another learner submits it to a real email provider to test deliverability. A third includes it in a feature request that gets emailed to support.

Fake-but-realistic values escape into production constantly. The discipline is to use placeholder shapes that are structurally recognizable as non-real: <your-email-here>, user@example.com (the RFC-reserved domain), test-user-1234 — forms that are clearly not real values and that a downstream system will reject rather than process.

The same principle applies to API keys, credentials, and any other sensitive value used in teaching content. A key shaped like sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with 32 x characters is obviously a placeholder. A key with a real prefix and realistic mixed-case characters — sk-aB3k... continuing for another thirty characters — looks real enough to be copy-pasted, which is exactly why this lesson will not print one in full.

At a content pipeline level, enforce this through your PII scanner: any string that matches a real email pattern, a real key pattern, or a real IP pattern — and is not in the allowlist — fails CI. Real-looking fake values fail the same gate as real values, which is exactly the right behavior.

Wiring It in Practice

A PII-safe AI content pipeline has four gates, not one:

1. Ingestion gate: before content enters the context window, scan it for PII patterns. Strip or redact before the model sees it. This prevents the leak from propagating into transcripts and embeddings in the first place.

2. Generation gate: after the model generates output, scan the output before writing it to any persistent store. A model that received a redacted record may still reproduce the original value if it inferred it from context — the output gate catches this.

3. Corpus gate: CI check on every content commit. Pattern match for email shapes, phone number shapes, and IP addresses (using documentation ranges like 203.0.113.0/24 in examples, never real IP allocations). Block the merge if the gate fails. Allowlist intentional examples explicitly.

4. Re-embed trigger: wire corpus changes to automatically trigger a re-index. The trigger is part of the deployment pipeline — not a post-deploy TODO comment.