The Avatar Layer: A Lip-Synced Digital Twin

The Layer That Changes Everything

The AI content production pipeline produces a clean output: ElevenLabs voice, Leonardo image, assembled in ffmpeg. But there's a layer that transforms it from "AI video" into something that looks like a real creator: the avatar.

A circular picture-in-picture corner-cam. Lip-synced. Your face. Moving to every word in the voiceover.

It sounds like a cosmetic upgrade. It isn't. It's the layer that makes viewers stay. It also hides more failure modes per square centimeter of UI than anything else in the pipeline.

This lesson is the war story: what works, what burned credits, and the exact steps that prevent both.

The Audio Path: Why Single Source Matters

The most important thing to understand about HeyGen lip-sync: you don't pair the avatar video with your ElevenLabs audio. You extract the avatar's own audio and use that.

Here's why. When you upload the ElevenLabs VO to HeyGen and render with voice.type: "audio" + audio_asset_id, HeyGen drives the avatar's lips directly from that audio file. The rendered video contains its own audio track — the same audio, now baked in and frame-accurate to the lip movements.

If you then compose the avatar video against a separate copy of the ElevenLabs VO in ffmpeg, you have two audio sources. Render timing introduces drift. The lips and the words go out of phase.

The correct path:

ElevenLabs VO
    ↓  upload to HeyGen /v1/asset
    ↓  render with voice.type="audio" + audio_asset_id
HeyGen avatar.mp4 (contains its own audio)
    ↓  extract avatar's own audio: ffmpeg -i avatar.mp4 -vn avatar_audio.aac
    ↓  composite: avatar_video (no audio) + avatar_audio.aac → final

Single source of truth. Frame-accurate by construction.

The Upload Step

Before HeyGen can render against your ElevenLabs VO, the audio needs to live in HeyGen's asset system:

# Upload the ElevenLabs VO to HeyGen
response = requests.post(
    "https://upload.heygen.com/v1/asset",
    headers={"X-Api-Key": HEYGEN_API_KEY, "Content-Type": "audio/mpeg"},
    data=open("voiceover.mp3", "rb").read(),
)
asset_id = response.json()["data"]["id"]

Then reference that asset_id in the generate call:

payload = {
    "video_inputs": [{
        "character": {
            "type": "avatar",
            "avatar_id": AVATAR_ID,
            "avatar_style": "circle",
        },
        "voice": {
            "type": "audio",
            "audio_asset_id": asset_id,
        },
    }],
    "dimension": {"width": 720, "height": 720},
}
response = requests.post(
    "https://api.heygen.com/v2/video/generate",
    headers={"X-Api-Key": HEYGEN_API_KEY, "Content-Type": "application/json"},
    json=payload,
)

The avatar_style: "circle" tells HeyGen to present the avatar as a circle crop on their end. The 720×720 dimension gives you a square canvas to work with in ffmpeg.

The Chromakey Step

The avatar comes back as a green-screen recording. For a clean circular PiP, you need two steps: strip the green, then composite onto a disc.

# Step 1 — key the green
ffmpeg -i avatar.mp4 \
  -vf "chromakey=0x00B050:0.18:0.06,despill=type=green" \
  -c:v prores_ks -alpha_bits 16 \
  avatar_keyed.mov

# Step 2 — composite onto a dark disc with a cyan ring
# (disc.png: 720x720, dark background, transparent inside the circle)
ffmpeg -i avatar_keyed.mov -i disc.png \
  -filter_complex "[0:v][1:v]overlay=0:0" \
  avatar_disc.mp4

The disc image is the mask. Without it, the chroma-keyed avatar sits on whatever is beneath it with a rectangular bounding box. The disc and the ring are what create the circular PiP appearance — not the avatar_style parameter.

The Credit Pool Trap

This one cost us a full session of debugging.

HeyGen has two separate credit pools. They do not share balance. They are not interchangeable:

Pool	What fills it	What draws from it
plan_credit	Studio / Creator subscription	Studio video rendering
api	Direct API credit purchase	/v2/video/generate endpoint

We hit "insufficient 'api' credits" repeatedly. Checked the account: 2,000 plan_credit units. Topped up the subscription again. Still failing. That's because topping up the subscription adds plan_credit — it does nothing for the api pool.

Check the right pool before rendering:

quota = requests.get(
    "https://api.heygen.com/v2/user/remaining_quota",
    headers={"X-Api-Key": HEYGEN_API_KEY},
).json()

api_credits = quota["data"]["details"]["api"]   # ← this is what matters
plan_credits = quota["data"]["details"]["plan_credit"]  # ← irrelevant for API use

A short 10-second test render succeeded (proving API access worked), while the full 90-second render failed — that isolated it as a credit-quantity issue, not a permissions issue. Once we purchased API credits directly, it cleared.

The Stale Avatar ID Trap

The second trap: the avatar id in your environment config goes stale.

HeyGen avatar ids are opaque strings that map to a specific avatar "look" in your account. They don't self-expire, but they can become invalid when an avatar is updated, regenerated, or removed from the account. When that happens, the render endpoint returns 404: avatar look not found.

Preflight check — resolve the live id before rendering:

avatars = requests.get(
    "https://api.heygen.com/v2/avatars",
    headers={"X-Api-Key": HEYGEN_API_KEY},
).json()

live_ids = {a["avatar_id"]: a["avatar_name"] for a in avatars["data"]["avatars"]}

if AVATAR_ID not in live_ids:
    raise ValueError(f"Avatar {AVATAR_ID} not found. Live avatars: {live_ids}")

This runs before any credit is spent. If the id is stale, the error surfaces before the render, not during it.

The Preflight Gate

A HeyGen render is the one step in the AI content production pipeline that costs real money and uses a likeness. It's also the one step where most failure modes are detectable in advance.

The preflight gate runs three checks, in order, before spending a single API credit:

1. api credit pool funded — GET /v2/user/remaining_quota → data.details.api; fail if below estimated render cost
2. avatar id live — GET /v2/avatars → verify AVATAR_ID exists in the list
3. ElevenLabs asset uploaded — verify the audio_asset_id returned a valid id before passing it to the render call

Failures that happen before the spend cost nothing but a few API reads. Failures that happen during a render cost credits and time.

def avatar_preflight(avatar_id: str, audio_asset_id: str) -> None:
    """Raise before spending credits if any preflight check fails."""
    # Check 1: API credit pool
    quota = heygen_get("/v2/user/remaining_quota")
    api_credits = quota["data"]["details"]["api"]
    if api_credits < MIN_API_CREDITS_REQUIRED:
        raise RuntimeError(f"Insufficient API credits: {api_credits}. Buy more at heygen.com/api")

    # Check 2: Avatar id is live
    avatars = heygen_get("/v2/avatars")
    live_ids = {a["avatar_id"] for a in avatars["data"]["avatars"]}
    if avatar_id not in live_ids:
        raise RuntimeError(f"Avatar {avatar_id} not found. Update HEYGEN_AVATAR_ID in .env")

    # Check 3: Asset id looks valid (not None/empty from a failed upload)
    if not audio_asset_id:
        raise RuntimeError("ElevenLabs audio_asset_id is empty — upload failed upstream")

What This Looks Like in the Pipeline

Once the preflight clears, the avatar layer slots into the content pipeline's existing ffmpeg composition as the final overlay:

ElevenLabs VO (approved)
    ↓  upload → HeyGen /v1/asset → audio_asset_id
    ↓  preflight(avatar_id, audio_asset_id)
    ↓  POST /v2/video/generate → video_id
    ↓  poll /v2/video/status until complete
    ↓  download avatar.mp4
    ↓  extract avatar's own audio: avatar_audio.aac
    ↓  chromakey + despill → avatar_keyed.mov
    ↓  composite onto disc → avatar_disc.mp4
    ↓  ffmpeg: main_video + avatar_disc (PiP, bottom-right) + avatar_audio.aac → final.mp4

The avatar video's audio is the only audio source in the final composition. The ElevenLabs file was the input to HeyGen — its job is done after the upload.

What's Next

The next lesson closes the track with a production retro: a reel that took three rounds of human feedback before it cleared the bar — desync, dense captions, AI b-roll where diagrams belonged — and how to convert each recurring, human-caught defect into a cheap build-time guardrail. The lesson you're finishing now is exactly the kind of step those guardrails protect.