Streaming & the Batch API: Delivery Modes

The Question Nobody Asks Until the Bill Arrives

Most Claude integrations start with client.messages.create() and stay there. It works. It is simple. And for a surprising range of use cases, it is exactly right.

The problem arrives when you are running a nightly pipeline that processes 600 article summaries, or when you are building a chat interface where users watch Claude think, or when a user complains their request timed out after 45 seconds on what should have been a long response. At that point, the delivery mode you chose at day one is costing you money, causing errors, or damaging the user experience — and refactoring the plumbing while the system is live is painful.

The Claude API offers three delivery modes. Each one is optimal for a different combination of latency requirement, volume, and cost tolerance. Understanding the tradeoffs upfront is one of the highest-leverage decisions you make in a Claude integration.

Synchronous: The Default Case

The simplest mode: call client.messages.create(), await the result, read response.content. One HTTP request, one response. The model generates the complete message server-side before sending anything back to you.

This is the right choice for most short, non-interactive requests. If you are building a classification service that takes a support ticket and returns a priority label, synchronous is correct — the latency is low, the code is simple, and there is no reason to add streaming complexity.

The hard limit to know: at roughly 16,000 max_tokens and above, SDK HTTP timeouts become a real risk. The model is still generating, but the client-side timeout fires first. This is not a configuration you can tune your way around — the fix is to stream.

One other constraint that applies specifically to Opus 4.8: sampling parameters (temperature, top_p, top_k) are removed on this model family and will return a 400 if you send them. Behavior steering happens through prompting, not sampling params. This is not a streaming-specific issue, but it bites developers who have copy-pasted examples written for older model families.

Streaming: First Token in Milliseconds

Streaming (client.messages.stream()) opens a persistent SSE connection and delivers the response incrementally as it is generated. The model starts sending within hundreds of milliseconds. Your user sees something immediately instead of waiting for the full response.

The wire protocol sends typed events in sequence: message_start (containing model, usage prefixes), then content_block_start / content_block_delta / content_block_stop for each content block, then message_delta (which carries stop_reason and final usage), then message_stop.

In practice, you usually do not need to handle these events individually. await stream.finalMessage() gives you the complete assembled message once the stream ends — exactly the same shape as a synchronous response. Use the individual events when you need to display partial text as it arrives, which is the primary reason to stream in the first place.

Streaming is also mandatory for agentic work with thinking enabled at xhigh or max effort, where max_tokens should be set to 64,000 or above. At those lengths, waiting for a synchronous response is not viable — the stream is both a technical requirement and a UX necessity.

import Anthropic from "@anthropic-ai/sdk"

const client = new Anthropic()

const stream = await client.messages.stream({
  model: "claude-sonnet-4-6",
  max_tokens: 32000,
  messages: [{ role: "user", content: "Explain how transformer attention works." }],
})

// Display partial text as it arrives
stream.on("text", (text) => process.stdout.write(text))

// Get the complete message with stop_reason and usage
const message = await stream.finalMessage()
console.log("\nStop reason:", message.stop_reason)
console.log("Usage:", message.usage)

The Batch API: 50% Off Everything

The Batch API is the delivery mode most developers discover late and then wish they had known about from day one. The value proposition is stark: every token you send through the Batch API costs 50% less. Input tokens, output tokens, and cache reads all qualify. On a workload of any size, this is meaningful money.

The model: you submit an envelope of requests (up to 100,000, up to 256 MB) via POST /v1/messages/batches. The API processes them asynchronously — most batches complete in under an hour, with a maximum of 24 hours. Results are available for 29 days.

This is the right mode for eval suites, nightly enrichment pipelines, content generation at scale, re-embedding jobs, and any other workload that can tolerate hours of latency. If no user is waiting for the result, the Batch API should be your default.

The correlation mechanism is the custom_id field on each request. You set it to whatever identifier makes downstream joins trivial — typically your source record's ID. Results return out of order, so custom_id is the only reliable way to match a result back to its input.

import Anthropic from "@anthropic-ai/sdk"

const client = new Anthropic()

// Submit a batch — the Batches API is GA, so it lives at client.messages.batches
const batch = await client.messages.batches.create({
  requests: articles.map((article) => ({
    custom_id: article.id,
    params: {
      model: "claude-haiku-4-5",
      max_tokens: 150,
      messages: [{
        role: "user",
        content: "Summarize this in one sentence: " + article.text,
      }],
    },
  })),
})

console.log("Batch ID:", batch.id)

// Poll until ended
let current = batch
while (current.processing_status !== "ended") {
  await new Promise((r) => setTimeout(r, 5000))
  current = await client.messages.batches.retrieve(batch.id)
}

// Collect results — handles succeeded, errored, canceled, expired per item
for await (const result of await client.messages.batches.results(batch.id)) {
  if (result.result.type === "succeeded") {
    const block = result.result.message.content[0]
    if (block.type === "text") {
      console.log(result.custom_id, "→", block.text)
    }
  } else {
    console.warn(result.custom_id, "→", result.result.type)
  }
}

Notice the per-result type switching. Batch results have four possible outcomes: succeeded (the request completed, .message contains the full response), errored (that specific request failed, .error has the details), canceled (the batch was cancelled mid-processing via client.messages.batches.cancel(batch.id) before this item finished — resubmit it if you still want it), and expired (the batch hit the 24-hour limit before processing this item). Your code must handle all four — a batch loop that only handles the happy path will silently drop failed items in production. The else branch above catches errored, canceled, and expired together by logging result.result.type.

The Batch API also stacks with prompt caching. A cache read on a batched request costs 0.1× of the already-discounted batch price. For pipelines that share a large frozen system prompt across hundreds of requests, the combined discount can reduce token costs by 90%+.

Handling Per-Request Failures in Batches

One of the most common mistakes in first batch implementations: treating the batch as atomic. It is not. The Batch API processes each request independently — a single failed request does not fail the batch. The other 99,999 requests continue and complete.

This means your result-processing code must be designed for partial success. A typical production pattern looks like this: collect all succeeded results into your main output store, collect errored results into a separate log for investigation, and collect expired results for re-submission in a follow-up batch. Never assume all results are succeeded just because the batch ended with processing_status: "ended".

The error on a failed result follows a taxonomy parallel to synchronous errors — a 400-class failure on one item means that item's params were invalid, not that the whole batch was malformed. On an errored result, check the result.result.error.type field: if it is invalid_request, that specific request needs to be fixed before re-submitting. If it is api_error, it is worth retrying in the next batch.

The Batch-Plus-Cache Stack

The Batch API's 50% discount stacks with prompt caching. This is not a minor detail — it is the combination that makes certain workloads economically compelling.

Consider a nightly enrichment job that processes 1,000 documents with the same 5,000-token frozen system prompt. (The prompt has to clear the per-model minimum cacheable prefix — 4096 tokens on Haiku 4.5 and Opus 4.8, 2048 on Sonnet 4.6 — or the breakpoint is silently ignored and nothing caches. 5,000 tokens clears every tier.) The first request in the batch pays for a cache write (the 1.25× write cost). Every subsequent request reads from cache at 0.1× of the already-halved batch price. For those 999 cache reads, the effective cost for the cached prefix is roughly 5 cents per 1M tokens instead of the full rate.

For production pipelines where the economics of AI-augmented content matter — and they do, at any scale — this stacking effect is worth designing for explicitly. Freeze your system prompt, set a cache breakpoint, run the workload as a batch, and verify the cache hit rate in your usage tracking. The combination of these two features is one of the highest-leverage cost optimizations available in the Claude API.

A Concrete Scenario: Two Services, Two Modes

Consider a team running two Claude-powered services: an interactive AI tutor for students, and a nightly content pipeline that enriches 500 articles with structured metadata.

The tutor streams unconditionally. Students type a question and see Claude's answer build letter by letter. Response times feel instant. At typical Sonnet pricing, the cost is predictable and reasonable.

The nightly pipeline uses the Batch API. No student is awake at 2am waiting for a summary. The 500 requests go into a single batch submission before midnight. By morning, results are ready and cost half of what they would have cost synchronously. Over a month, the savings fund a month's worth of server costs.

This is not a contrived optimization. It is the obvious architecture once you understand what the three modes exist for.