Render Video From Code, Not a Timeline

Tools You'll Meet in This Arc

The lessons up to this point have been about calling video APIs — Veo, Runway, HeyGen. From here through the end of the track, the arc shifts to programmatic rendering: building reels from code instead of a timeline editor. A handful of tools appear throughout this arc that the track hasn't introduced yet, so here is the one-line version of each:

• Remotion — a React framework for making videos in code. Each frame is a React render; a headless browser captures the frames and encodes them into an MP4. It is the incumbent renderer in the production pipeline these lessons draw from.
• FFmpeg — the command-line workhorse of video processing: it encodes, decodes, concatenates, and overlays tracks. Every time it re-encodes a video, a little quality is lost — called generation loss — which drives several architecture decisions ahead.
• GSAP — a JavaScript animation library used to script motion (slides, fades, counters) inside HTML-rendered videos.
• Caption formats (SRT / ASS) — subtitle file formats. ASS supports styling (fonts, colors, screen position) and is what FFmpeg burns into the frame.
• PiP (picture-in-picture) — a small overlay video, like a talking-head avatar, layered over the main footage.
• Docker — a container runtime; it matters here only because pinning the render environment is what makes output byte-reproducible.

You don't need hands-on experience with any of these to follow the arc. You need to know what each one is so the benchmarks and tradeoffs below make sense.

The Debate You Don't Need to Have

Every tool evaluation eventually produces the same dynamic: one person argues for the new tool, another argues for the incumbent, and both are drawing from incomplete information. The new tool's README describes what it's designed to do. The incumbent's behavior is known from production. Neither tells you which one wins for your specific use case on your specific hardware with your specific pipeline requirements.

I ran into this evaluating HyperFrames — HeyGen's open-source HTML-to-MP4 framework — against Remotion in the content pipeline's video stage. HyperFrames describes itself as "write HTML, render video, built for agents." That framing is compelling if you're running an agentic video pipeline. Remotion is React-based, fast, and production-proven. The instinct is to spend two weeks debating the tradeoffs. The correct move is to run a bounded spike in two days.

What HyperFrames Actually Is

HyperFrames renders video from HTML using headless Chrome's BeginFrame API, then muxes frames via FFmpeg. The key architectural claim: because the render engine is a browser, any agent that can write HTML can author a video. You don't need React. You don't need JSX. You write <div>, add styles, and get frames.

The byte-determinism claim is the other headline: run the same HTML twice under --docker, get identical bytes. This matters for agentic pipelines where you need reproducibility without storing every rendered artifact.

The cost: determinism requires --docker. Without Docker, HyperFrames uses host GPU and produces non-deterministic output, same as Remotion. With Docker, you get byte-identical output — but the render time jumps from ~9.4s to ~209s on the same component.

Remotion doesn't offer byte-determinism. It's faster on bare metal and requires React/JSX. Those are the known inputs. Now measure.

Spike 1: Data-Viz Layer (StatCard)

The first spike rebuilt a StatCard — one of the content pipeline's data visualization primitives — in both tools. The measurement criteria:

• Render time: wall-clock from input to MP4 frame
• Visual fidelity: frame-diff against the Remotion reference
• Determinism: does the same input produce the same output twice?

Results: Remotion rendered the StatCard in ~3.7s on host GPU. HyperFrames rendered it in ~9.4s on host GPU — roughly 2.5x slower. Fidelity was near-identical; both tools hit browser-quality rendering. Determinism: Remotion is non-deterministic by design (font rasterization, anti-aliasing variance). HyperFrames is non-deterministic on host, byte-identical under --docker at ~209s — a 56x overhead.

Verdict: keep Remotion for data-viz primitives. There is no payoff to switch a 3.7s render to a 9.4s render for identical visual output, and the determinism benefit only materializes at a render cost that blows the pipeline budget.

The spike took half a day. The decision is permanent until something changes about the pipeline requirements.

Spike 2: Post-Production / Multi-Track Composition

The second spike was a different question entirely. The content pipeline's post-production stage assembles multiple tracks: rendered clips, avatar overlay, captions, audio. The incumbent Remotion-based approach required approximately 4 compounding FFmpeg re-encodes to get from individual tracks to final output. Each re-encode adds generation loss and latency. The PiP avatar overlay had been strapped because the multi-pass approach couldn't handle it cleanly.

HyperFrames' architecture supports capturing multiple tracks in a single pass via its composition layer — the BeginFrame API captures everything layered in the DOM in one shot. The spike rebuilt the multi-track timeline: clips + avatar + captions + audio, assembled and rendered.

Result: HyperFrames assembled the full composition in ONE capture pass. No intermediate re-encodes. The PiP avatar, previously strapped, rendered correctly inside the single-pass composition.

Verdict: adopt HyperFrames for post-production / composition. The single-pass advantage is structural — it eliminates the cascading quality loss from multiple FFmpeg operations — and it recovered a feature that had been disabled.

The Composition Insight

The two spikes produced something more interesting than "use tool A" or "use tool B." They produced a clear layer map:

• Remotion wins for primitive rendering: component-level data viz, stat cards, charts. Fast, React-native, no re-tooling needed.
• HyperFrames wins for composition: multi-track timelines, avatar + captions + audio assembly. Single-pass, HTML-authorable, agent-friendly for the assembly layer.

The two tools compose. Remotion renders the primitives, HyperFrames assembles them. The pipeline gains the speed advantage in the render layer and the single-pass advantage in the assembly layer — a better outcome than replacing one with the other.

This is the kind of insight that only emerges from measuring. You can't derive "they compose" from reading two READMEs.

The Spike Rubric

Every bounded spike should measure against the same four criteria that determined the content pipeline verdict:

1. Render time     — wall-clock from input to usable output
2. Determinism     — does the same input produce the same output twice?
3. Visual fidelity — frame-diff against the incumbent reference
4. Authorability   — how hard is it for an agent to generate valid input?

These four axes expose the tradeoff space. For the content pipeline's data-viz layer, render time was decisive. For the composition layer, authorability and pass-count were decisive. A spike that measures all four produces a complete picture in one day.

What Makes a Good Spike

A spike is bounded by three constraints:

1. ONE component. Rebuild a single, real component from your existing pipeline — not a toy example, not a hello world, but the actual thing. A StatCard, not "a card." A multi-track timeline, not "a video."
2. Measure, don't eyeball. Record the numbers. Wall-clock render time. Frame diff score. Pass count. These numbers are the deliverable of the spike.
3. Throw it away. The spike code is not production code. It may hardcode paths, skip error handling, or use dev credentials. Record the verdict and the measurements, then build the production integration from scratch with that knowledge.

Violating constraint 3 is the most common mistake. "The spike works, let's just ship it" is how you inherit prototype debt. The spike proves the decision; the production integration implements it correctly.

Applying This Beyond Video

The spike methodology is domain-agnostic. The pattern is: pick a decision criterion, rebuild one real artifact in both options, measure, decide. It works for database choices (rebuild one query in Postgres and the challenger, measure latency and query complexity), for embedding providers (run the same corpus through two providers, measure recall quality), for inference providers (send the same prompt to both, measure latency and cost per token).

The bounded part matters. A spike that tries to rebuild the entire pipeline is not a spike — it's a migration. Keep it to one component, one measurement, one decision.

# Content pipeline spike workflow — pseudocode
spike_component="stat_card"
build_in_remotion "$spike_component"      # ~3.7s
build_in_hyperframes "$spike_component"   # ~9.4s host, ~209s docker

measure render_time fidelity determinism authorability

record_verdict "KEEP_REMOTION" --criteria "speed,no-docker-overhead"
discard_spike_branch

The spike should cost one day. The verdict should last until the pipeline requirements change. And the production integration — built afterward, informed by the measurement — should be clean.

What's Next

The next lesson covers the HyperFrames HTML authoring model in depth: how to structure layouts for BeginFrame capture, the CSS constraints that affect frame accuracy, and how to template multi-track compositions so agents can author them reliably without manual JSX.