Ask Knox

The standard Gemini API is a request-response system. You send a payload, you wait for the response, you process it. That cycle takes one to three seconds on typical prompts — acceptable for most generation tasks, completely wrong for real-time conversations with actual human beings.

The Live API is a different communication architecture entirely. It is a persistent bidirectional WebSocket that stays open for the duration of a session, streams audio and video in both directions, and produces responses in under a second. It does not make the standard API faster. It replaces the request-response pattern with a continuous stream.

What the Live API Actually Provides

A Live API session is a WebSocket connection with several built-in capabilities that would require significant external infrastructure to replicate:

Native audio streaming. You can send raw audio bytes directly to the model and receive audio back. No separate speech-to-text service, no text-to-speech service, no intermediate conversion layer. The model handles the full audio pipeline internally.

Sub-second response latency. Typical end-to-end latency in a well-configured Live session is 300-600ms. This compares to 1,500-3,000ms for typical standard API calls. For voice interactions, the difference is perceptible at the human level — under 500ms feels conversational, over 1,500ms feels like you are talking to a slow service.

30 HD voices, 24 languages. Audio output quality is configurable, with a native voice library that covers the primary global languages and multiple speaking styles per language.

Proactive audio mode. In standard question-answer patterns, the model speaks only after you speak. Proactive audio mode allows the model to initiate speech — useful for tutoring systems, agent-driven interviews, or any use case where the model should drive the interaction rather than always responding.

Video input. You can send video frames to the Live API — either as a stream or frame-by-frame. This enables real-time visual context: a model that can see what the user is looking at while the conversation happens.

Model Availability

The Live API requires dedicated live model IDs — the base text models (gemini-2.5-flash, gemini-2.5-pro) cannot open a Live session. Connecting client.aio.live.connect with a non-live model ID fails.

gemini-3.1-flash-live-preview is the current frontier Live API model. Preview lifecycle — 2-week deprecation window, restricted rate limits compared to GA. Use when you need the highest Live API capability and can accept Preview constraints.

gemini-live-2.5-flash is the stable dedicated Live model (with gemini-2.5-flash native-audio variants for the highest-quality speech). Lower peak capability than 3.1-flash-live, but longer deprecation windows and more predictable rate limits. For production systems that need stability, start here.

The Basic Connection Pattern

Live API sessions use the async context manager pattern:

import asyncio
from google import genai
from google.genai import types

client = genai.Client()

async def run_live_session():
    config = types.LiveConnectConfig(
        response_modalities=["AUDIO"],
        speech_config=types.SpeechConfig(
            voice_config=types.VoiceConfig(
                prebuilt_voice_config=types.PrebuiltVoiceConfig(
                    voice_name="Aoede"
                )
            )
        )
    )
    
    async with client.aio.live.connect(
        model='gemini-3.1-flash-live-preview',
        config=config
    ) as session:
        # Send text input, receive audio response
        await session.send_client_content(
            turns={"role": "user", "parts": [{"text": "Tell me the current time in Tokyo."}]},
            turn_complete=True
        )
        
        async for response in session.receive():
            if response.data:
                # Raw audio bytes — write to file or stream to speaker
                print(f"Received {len(response.data)} audio bytes")
            elif response.text:
                print(f"Text: {response.text}")

asyncio.run(run_live_session())

The turn_complete=True flag signals that you have finished your input and the model should respond. (Older tutorials show session.send(input=..., end_of_turn=True) — that surface was removed from the SDK; use send_client_content for turn-based input and send_realtime_input for streaming media.)

Sending Audio Input

The audio input path is symmetric with the audio output path:

async def voice_session(audio_stream):
    async with client.aio.live.connect(
        model='gemini-live-2.5-flash',
        config=types.LiveConnectConfig(
            response_modalities=["AUDIO"]
        )
    ) as session:
        async for audio_chunk in audio_stream:
            await session.send_realtime_input(
                audio=types.Blob(
                    data=audio_chunk,
                    mime_type="audio/pcm;rate=16000"
                )
            )
        
        # Signal end of the audio stream
        await session.send_realtime_input(audio_stream_end=True)
        
        async for response in session.receive():
            if response.data:
                yield response.data

Audio format: the Live API expects PCM audio at 16kHz, 16-bit. Most modern audio capture libraries can output in this format directly. If you are working with a different format, convert before sending.

Session Management and Reconnection

Live sessions have a maximum duration — check the current limits in the API docs, as they are subject to change. For long-running applications (customer service agents, ongoing tutors), you need explicit reconnection logic.

async def reconnecting_session(get_input, send_output, max_retries=3):
    retries = 0
    while retries < max_retries:
        try:
            async with client.aio.live.connect(
                model='gemini-live-2.5-flash',
                config=types.LiveConnectConfig(
                    response_modalities=["AUDIO"]
                )
            ) as session:
                retries = 0  # Reset on successful connection
                async for input_data in get_input():
                    await session.send_realtime_input(audio=input_data)
                    async for response in session.receive():
                        if response.data:
                            await send_output(response.data)
        except Exception as e:
            retries += 1
            if retries < max_retries:
                await asyncio.sleep(1)
            else:
                raise

Do not assume a session persists indefinitely. Build reconnection into the architecture from the start, not as an afterthought.

When to Use the Live API

The decision boundary is latency requirement. If your use case can tolerate 1-3 second response times, use the standard API — it is more cost-efficient, has larger context windows, and is operationally simpler to manage.

Use the Live API when:

Voice is the interface. Any system where a human is speaking to the model in real time. Customer service agents, voice-controlled tools, conversational tutors, accessibility applications. The standard API creates a conversation that feels like calling a slow-loading website.

Real-time translation is needed. The Live API's streaming architecture allows near-simultaneous speech translation — the output starts before the input finishes. Standard API requires complete input before generating output.

Interactive agents need to drive conversation. Proactive audio mode enables the model to initiate dialogue. This is architecturally impossible in a request-response system where the model only speaks when spoken to.

Video is part of the context. If your application involves the model seeing and responding to what the user is looking at in real time, the Live API's video streaming handles this natively.

Do not use the Live API for:

Batch audio processing. Transcribing a large archive of recorded meetings or calls. The standard API with audio input is cheaper, more scalable, and simpler to manage for batch workloads.

High-context tasks. The Live API has a smaller effective context window per session than the standard API. Tasks that require deep context — analyzing a long document, working through a complex codebase — belong in the standard API.

Cost-sensitive high-volume workloads. Live sessions are priced differently from standard calls. Run cost modeling before committing to Live API for high-volume pipelines.

Production Checklist

Before shipping a Live API integration:

Pin to a specific model string — never use an alias
Implement reconnection logic with exponential backoff
Test your audio format pipeline (PCM 16kHz 16-bit) before wiring to the model
Set up session duration monitoring and graceful termination before the limit
Have a fallback to the standard API for cases where the Live session fails
Load test the rate limits for your chosen model tier before production traffic