> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modellix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gemini 3.8 Live

> Quick start, request parameters, messages, formats, limits, and FAQ for google/gemini-3.8-live: real-time voice and vision sessions over WebSocket.

<Card title="WSS" icon="radio">
  `wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live`
</Card>

| Item | Value |
| - | - |
| Model ID | `google/gemini-3.8-live` |
| Input | Text, audio (16 kHz PCM), video frames (JPEG) |
| Output | Audio (24 kHz PCM), with optional text transcription |
| Context | 131,072 input tokens, 65,536 output tokens |
| Auth | `Authorization: Bearer <Modellix API Key>` on the handshake |

For the session flow, billing, close codes, and handshake errors shared by all Live models, see the [Live Overview](https://docs.modellix.ai/live/overview).

## Quick Start

<CodeGroup>
  ```python Python theme={null}
  # pip install "websockets>=14"
  import asyncio
  import json
  import os

  import websockets

  URL = "wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live"


  async def main() -> None:
      headers = {"Authorization": f"Bearer {os.environ['MODELLIX_API_KEY']}"}
      async with websockets.connect(URL, additional_headers=headers, max_size=16 * 1024 * 1024) as ws:
          # 1. The first frame must be setup.
          await ws.send(json.dumps({
              "setup": {
                  "generationConfig": {"responseModalities": ["AUDIO"]},
                  "outputAudioTranscription": {},
              }
          }))
          reply = json.loads(await ws.recv())
          assert "setupComplete" in reply, reply

          # 2. Send a text turn.
          await ws.send(json.dumps({"realtimeInput": {"text": "Introduce yourself in one sentence."}}))

          # 3. Read frames until the turn completes.
          async for raw in ws:
              message = json.loads(raw)
              server = message.get("serverContent") or {}
              text = (server.get("outputTranscription") or {}).get("text")
              if text:
                  print(text, end="", flush=True)
              if message.get("usageMetadata"):
                  print("\nUsage:", message["usageMetadata"])
              if server.get("turnComplete"):
                  break


  asyncio.run(main())
  ```

  ```javascript Node.js theme={null}
  // npm install ws
  import WebSocket from "ws";

  const url = "wss://api.modellix.ai/api/v1/live/google/gemini-3.8-live";
  const ws = new WebSocket(url, {
    headers: { Authorization: `Bearer ${process.env.MODELLIX_API_KEY}` },
  });

  ws.on("open", () => {
    // 1. The first frame must be setup.
    ws.send(JSON.stringify({
      setup: {
        generationConfig: { responseModalities: ["AUDIO"] },
        outputAudioTranscription: {},
      },
    }));
  });

  ws.on("message", (data) => {
    const message = JSON.parse(data.toString());
    if (message.setupComplete) {
      // 2. Send a text turn.
      ws.send(JSON.stringify({ realtimeInput: { text: "Introduce yourself in one sentence." } }));
      return;
    }
    // 3. Read frames until the turn completes.
    const text = message.serverContent?.outputTranscription?.text;
    if (text) process.stdout.write(text);
    if (message.usageMetadata) console.log("\nUsage:", message.usageMetadata);
    if (message.serverContent?.turnComplete) ws.close();
  });

  ws.on("close", (code, reason) => console.log(`\nClosed: ${code} ${reason}`));
  ```
</CodeGroup>

Set `MODELLIX_API_KEY` first. Both examples send one text turn and print the spoken reply as text through output transcription.

## Configuration Examples

Each example is a complete `setup` frame. Pick one and adjust.

<Tabs>
  <Tab title="Audio Only">
    ```json theme={null}
    {
      "setup": {
        "generationConfig": { "responseModalities": ["AUDIO"] }
      }
    }
    ```
  </Tab>

  <Tab title="Transcription">
    Recommended. You get the spoken reply as audio and as text.

    ```json theme={null}
    {
      "setup": {
        "generationConfig": { "responseModalities": ["AUDIO"] },
        "systemInstruction": {
          "parts": [{ "text": "You are a friendly assistant. Answer naturally and briefly." }]
        },
        "inputAudioTranscription": {},
        "outputAudioTranscription": {}
      }
    }
    ```
  </Tab>

  <Tab title="Voice">
    ```json theme={null}
    {
      "setup": {
        "generationConfig": {
          "responseModalities": ["AUDIO"],
          "speechConfig": {
            "voiceConfig": { "prebuiltVoiceConfig": { "voiceName": "Kore" } }
          }
        },
        "systemInstruction": {
          "parts": [{ "text": "You are a concise voice assistant. Respond only in Japanese." }]
        },
        "inputAudioTranscription": {},
        "outputAudioTranscription": {}
      }
    }
    ```
  </Tab>

  <Tab title="Long Session">
    Enable compression and resumption in `setup`.

    ```json theme={null}
    {
      "setup": {
        "generationConfig": { "responseModalities": ["AUDIO"] },
        "contextWindowCompression": { "slidingWindow": {} },
        "sessionResumption": {}
      }
    }
    ```

    After `goAway` or a disconnect, reconnect and pass the latest handle.

    ```json theme={null}
    {
      "setup": {
        "generationConfig": { "responseModalities": ["AUDIO"] },
        "contextWindowCompression": { "slidingWindow": {} },
        "sessionResumption": { "handle": "<latest handle>" }
      }
    }
    ```
  </Tab>

  <Tab title="Manual VAD">
    Turn VAD off and mark speech yourself.

    ```text theme={null}
    → {"setup": {"generationConfig": {"responseModalities": ["AUDIO"]}, "realtimeInputConfig": {"automaticActivityDetection": {"disabled": true}}}}
    ← {"setupComplete": {}}
    → {"realtimeInput": {"activityStart": {}}}
    → {"realtimeInput": {"audio": {"mimeType": "audio/pcm;rate=16000", "data": "..."}}}
    → {"realtimeInput": {"activityEnd": {}}}
    ← {"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "..."}}]}}}
    ← {"serverContent": {"generationComplete": true, "turnComplete": true}}
    ```
  </Tab>

  <Tab title="Context">
    Append earlier turns without asking for a reply, then continue.

    ```json theme={null}
    {
      "clientContent": {
        "turns": [
          { "role": "user", "parts": [{ "text": "My name is Alex." }] },
          { "role": "model", "parts": [{ "text": "Nice to meet you, Alex." }] }
        ],
        "turnComplete": false
      }
    }
    ```
  </Tab>
</Tabs>

## Sending Audio and Video

| Input | Message | Format |
| - | - | - |
| Text | `realtimeInput.text` | UTF-8 string. Triggers a model reply. |
| Audio in | `realtimeInput.audio` | Raw 16-bit PCM, 16 kHz, little-endian, mono, base64. Send in chunks of about 100 ms. |
| Video in | `realtimeInput.video` | Discrete JPEG frames, base64. At most 1 frame per second. |
| Audio out | `serverContent.modelTurn.parts[].inlineData` | Raw 16-bit PCM, 24 kHz, little-endian, mono, base64. |

<Tip>
  **Send video as a stream, not a single frame.** A single JPEG followed
  immediately by a question may be ignored by the model, and no image tokens are
  counted. Send several frames at up to 1 frame per second, then ask your
  question. Video frames are reported and billed as image input tokens.
</Tip>

<Tabs>
  <Tab title="Audio Input">
    Stream a 16 kHz PCM recording in 100 ms chunks, followed by silence so the server detects the end of speech.

    ```python theme={null}
    import asyncio
    import base64
    import json

    CHUNK_BYTES = 3200  # 100 ms of 16 kHz, 16-bit mono PCM


    async def stream_audio(ws, pcm: bytes) -> None:
        pcm += bytes(16000 * 2 * 2)  # 2 seconds of silence
        for start in range(0, len(pcm), CHUNK_BYTES):
            chunk = base64.b64encode(pcm[start:start + CHUNK_BYTES]).decode()
            await ws.send(json.dumps({
                "realtimeInput": {"audio": {"mimeType": "audio/pcm;rate=16000", "data": chunk}}
            }))
            await asyncio.sleep(0.1)  # real-time pace
    ```
  </Tab>

  <Tab title="Video Input">
    Send several frames at 1 frame per second, then ask the question.

    ```python theme={null}
    import asyncio
    import base64
    import json


    async def ask_about_frames(ws, jpeg_frames: list[bytes], question: str) -> None:
        for frame in jpeg_frames:
            await ws.send(json.dumps({
                "realtimeInput": {"video": {"mimeType": "image/jpeg", "data": base64.b64encode(frame).decode()}}
            }))
            await asyncio.sleep(1)
        await ws.send(json.dumps({"realtimeInput": {"text": question}}))
    ```
  </Tab>
</Tabs>

## Client Messages

Every frame is a JSON text message with exactly one top-level field. Send `setup` first and wait for `setupComplete`. Parameters are optional unless marked **required**. Where a default is not listed, the upstream default applies.

### setup

First frame. It must arrive within 30 seconds and cannot be changed afterwards. Every field is optional. Other upstream fields, such as `tools`, are forwarded unchanged. See the [Gemini Live API reference](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-live).

<ParamField body="model" type="string">
  Ignored. The model is selected by the URL, and Modellix replaces this field.
</ParamField>

<ParamField body="generationConfig" type="object">
  Generation settings.

  <Expandable title="properties" defaultOpen>
    <ParamField body="responseModalities" type="string[]">
      Output modality. Allowed values: `["AUDIO"]`. Use `outputAudioTranscription` to also receive the reply as text.
    </ParamField>

    <ParamField body="speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName" type="string">
      Voice for audio output. Allowed values: one of the 30 [voices](#voices), for example `Puck`.
    </ParamField>

    <ParamField body="maxOutputTokens" type="integer">
      Maximum output tokens per turn. Allowed values: `1` to `65,536`.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="systemInstruction" type="object">
  Instruction that stays in effect for the whole session. Allowed values: `{"parts": [{"text": "..."}]}`, text parts only. It can only be set here; to add context later, use `clientContent`.
</ParamField>

<ParamField body="inputAudioTranscription" type="object" default="off">
  Allowed values: `{}`. Enables transcription of the audio you send, returned in `serverContent.inputTranscription`.
</ParamField>

<ParamField body="outputAudioTranscription" type="object" default="off">
  Allowed values: `{}`. Enables the text of the spoken reply, returned in `serverContent.outputTranscription`.
</ParamField>

<ParamField body="realtimeInputConfig.automaticActivityDetection" type="object">
  Server-side voice activity detection (VAD). While it is on, the user can interrupt the model by speaking.

  <Expandable title="properties" defaultOpen>
    <ParamField body="disabled" type="boolean" default="false">
      Allowed values: `true`, `false`. Set to `true` to turn VAD off. You then mark speech yourself with `activityStart` and `activityEnd`.
    </ParamField>

    <ParamField body="prefixPaddingMs" type="integer">
      Audio, in milliseconds, kept from before speech is detected.
    </ParamField>

    <ParamField body="silenceDurationMs" type="integer">
      Silence, in milliseconds, that ends a speech turn.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="contextWindowCompression" type="object" default="off">
  Compresses the context so a session can run past the audio (15 minutes) and audio plus video (2 minutes) limits, and keeps per-turn cost bounded.

  <Expandable title="properties" defaultOpen>
    <ParamField body="slidingWindow" type="object">
      Allowed values: `{}`. Enables compression with default behavior.
    </ParamField>

    <ParamField body="triggerTokens" type="integer">
      Context size, in tokens, that triggers compression.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="sessionResumption" type="object" default="off">
  Allowed values: `{}`. Enables `sessionResumptionUpdate` messages.

  <Expandable title="properties" defaultOpen>
    <ParamField body="handle" type="string">
      Continues an earlier session on a new connection. Allowed values: a handle from a previous `sessionResumptionUpdate`. Handles stay valid for 2 hours after the session ends.
    </ParamField>
  </Expandable>
</ParamField>

<Note>
  Do not set `generationConfig.thinkingConfig.thinkingLevel`; it is not supported for this model. Sampling parameters (`temperature`, `topP`, `topK`) and penalties (`presencePenalty`, `frequencyPenalty`) are not documented here because recent Gemini models ignore or reject them.
</Note>

### realtimeInput

Streams user input. Send exactly one of the following fields per message. Real-time input can be sent continuously without interrupting the model.

<ParamField body="text" type="string">
  Text input. Triggers a model reply. Allowed values: non-empty text.
</ParamField>

<ParamField body="audio" type="object">
  Raw 16-bit PCM, 16 kHz, little-endian, mono. Send in small chunks, for example 100 ms.

  <Expandable title="properties" defaultOpen>
    <ParamField body="mimeType" type="string" required>
      Allowed values: `audio/pcm;rate=16000`.
    </ParamField>

    <ParamField body="data" type="string" required>
      Base64-encoded PCM bytes.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="video" type="object">
  One JPEG frame. Send at most 1 frame per second, and keep sending while the user talks about the scene. A single frame followed immediately by a question may be ignored.

  <Expandable title="properties" defaultOpen>
    <ParamField body="mimeType" type="string" required>
      Allowed values: `image/jpeg`.
    </ParamField>

    <ParamField body="data" type="string" required>
      Base64-encoded JPEG bytes.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="audioStreamEnd" type="boolean">
  Allowed values: `true`. The audio stream ended, for example the microphone was turned off. Send audio again to resume. Use only while VAD is on.
</ParamField>

<ParamField body="activityStart" type="object">
  Allowed values: `{}`. Marks the start of user speech. Use only while VAD is off.
</ParamField>

<ParamField body="activityEnd" type="object">
  Allowed values: `{}`. Marks the end of user speech. Use only while VAD is off.
</ParamField>

### clientContent

Appends turns to the conversation history. Use it to restore earlier context or to send text turns with explicit roles.

<ParamField body="turns" type="object[]">
  Turns appended in order.

  <Expandable title="properties" defaultOpen>
    <ParamField body="role" type="string">
      Author of the turn. Allowed values: `user`, `model`.
    </ParamField>

    <ParamField body="parts" type="object[]" required>
      Text of the turn. Allowed values: `[{"text": "..."}]`.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="turnComplete" type="boolean" default="false">
  Allowed values: `true`, `false`. If `true`, the model starts replying now and interrupts any generation in progress. Otherwise the server waits for more input.
</ParamField>

## Server Messages

<ResponseField name="setupComplete" type="object">
  The session is ready for input. Always the first message after `setup`.
</ResponseField>

<ResponseField name="serverContent" type="object">
  Model output, transcriptions, and turn signals. Output streams across many frames and is generated faster than real time; buffer audio and play it back at 24 kHz.

  <Expandable title="properties">
    <ResponseField name="modelTurn.parts[].inlineData" type="object">
      Audio chunk. `mimeType` is `audio/pcm;rate=24000`; `data` is base64 16-bit PCM.
    </ResponseField>

    <ResponseField name="inputTranscription.text" type="string">
      Transcription of the audio you sent, when `inputAudioTranscription` is set.
    </ResponseField>

    <ResponseField name="outputTranscription.text" type="string">
      Text of the spoken reply, when `outputAudioTranscription` is set. Transcription is independent of `modelTurn`, so their order is not guaranteed.
    </ResponseField>

    <ResponseField name="generationComplete" type="boolean">
      The model finished generating. Playback may still be in progress.
    </ResponseField>

    <ResponseField name="turnComplete" type="boolean">
      The turn is complete. This ends a billed turn.
    </ResponseField>

    <ResponseField name="interrupted" type="boolean">
      The user interrupted the model. Stop playback and discard buffered audio.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usageMetadata" type="object">
  Token usage for the turn, sent at the end of each turn. Billing uses it, split by modality. Video frames are reported as `IMAGE` input.

  <Expandable title="properties">
    <ResponseField name="promptTokenCount" type="integer">
      Input tokens, including the accumulated context.
    </ResponseField>

    <ResponseField name="candidatesTokenCount" type="integer">
      Output tokens.
    </ResponseField>

    <ResponseField name="thoughtsTokenCount" type="integer">
      Thinking tokens, billed at the output text rate.
    </ResponseField>

    <ResponseField name="totalTokenCount" type="integer">
      Total tokens.
    </ResponseField>

    <ResponseField name="promptTokensDetails" type="object[]">
      Input tokens by modality: `{"modality": "TEXT" | "IMAGE" | "AUDIO", "tokenCount": 0}`.
    </ResponseField>

    <ResponseField name="candidatesTokensDetails" type="object[]">
      Output tokens by modality, in the same shape.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sessionResumptionUpdate" type="object">
  Sent periodically when `sessionResumption` is set. Treat the handle as a credential and do not log it.

  <Expandable title="properties">
    <ResponseField name="newHandle" type="string">
      Pass as `setup.sessionResumption.handle` on a new connection.
    </ResponseField>

    <ResponseField name="resumable" type="boolean">
      Whether the session can be resumed now. Only use handles where this is `true`.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="goAway" type="object">
  Sent about 60 seconds before the connection ends.

  <Expandable title="properties">
    <ResponseField name="timeLeft" type="string">
      Time before the connection ends, for example `50s`.
    </ResponseField>
  </Expandable>
</ResponseField>

Server frames can arrive as text or binary WebSocket messages. Both carry UTF-8 JSON.

### Example Frame Sequence

A text question with a spoken reply and transcription. Audio data is shortened.

```text theme={null}
→ {"setup": {"generationConfig": {"responseModalities": ["AUDIO"]}, "outputAudioTranscription": {}}}
← {"setupComplete": {}}
→ {"realtimeInput": {"text": "What is the capital of France?"}}
← {"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "..."}}]}}}
← {"serverContent": {"outputTranscription": {"text": "The capital of France is Paris."}}}
← {"serverContent": {"generationComplete": true, "turnComplete": true}}
← {"usageMetadata": {"promptTokenCount": 20, "candidatesTokenCount": 46, "totalTokenCount": 66, "promptTokensDetails": [{"modality": "TEXT", "tokenCount": 20}], "candidatesTokensDetails": [{"modality": "AUDIO", "tokenCount": 46}]}}
```

## Voices

Set `voiceName` to one of these 30 voices:

| Voice | Voice | Voice |
| - | - | - |
| `Zephyr` (bright) | `Puck` (upbeat) | `Charon` (informative) |
| `Kore` (firm) | `Fenrir` (excitable) | `Leda` (youthful) |
| `Orus` (firm) | `Aoede` (breezy) | `Callirrhoe` (easy-going) |
| `Autonoe` (bright) | `Enceladus` (breathy) | `Iapetus` (clear) |
| `Umbriel` (easy-going) | `Algieba` (smooth) | `Despina` (smooth) |
| `Erinome` (clear) | `Algenib` (gravelly) | `Rasalgethi` (informative) |
| `Laomedeia` (upbeat) | `Achernar` (soft) | `Alnilam` (firm) |
| `Schedar` (even) | `Gacrux` (mature) | `Pulcherrima` (forward) |
| `Achird` (friendly) | `Zubenelgenubi` (casual) | `Vindemiatrix` (gentle) |
| `Sadachbia` (lively) | `Sadaltager` (knowledgeable) | `Sulafat` (warm) |

The model switches language with the conversation. To keep replies in one language, say so in `systemInstruction`, for example `"Respond only in Japanese."`

## Sessions and Limits

Upstream limits apply to this model, independent of Modellix:

| Limit | Value |
| - | - |
| Single connection lifetime | About 10 minutes. A `goAway` message arrives about 60 seconds before the end. |
| Audio-only session context | About 15 minutes without compression |
| Audio plus video session context | About 2 minutes without compression |

To go beyond these limits, configure the following in your `setup` message. Modellix forwards them untouched:

* `contextWindowCompression`: keeps the context bounded with a sliding window, which also controls cost.
* `sessionResumption`: the server sends `sessionResumptionUpdate` messages containing a handle. After `goAway` or a disconnect, open a new connection and pass the latest handle in `setup.sessionResumption.handle` to continue with the previous context.

Each new connection is a new Modellix session. Turn numbering and billing records are independent per connection. Modellix also closes any connection at its maximum connection duration with code `4408`.

## FAQ

<AccordionGroup>
  <Accordion title="How do I choose a voice?">
    Set `setup.generationConfig.speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName` to one of the 30 [voices](#voices). If you omit it, the upstream default voice is used.
  </Accordion>

  <Accordion title="How do I get the reply as text?">
    The model replies in audio only. Add `outputAudioTranscription: {}` to `setup` and read `serverContent.outputTranscription.text`.
  </Accordion>

  <Accordion title="How do I get a transcript of what the user said?">
    Add `inputAudioTranscription: {}` to `setup` and read `serverContent.inputTranscription.text`.
  </Accordion>

  <Accordion title="How do I make the model reply in a specific language?">
    The model follows the conversation language. To fix it, say so in `systemInstruction`, for example `"Respond only in Japanese."`
  </Accordion>

  <Accordion title="How do I interrupt the model?">
    While voice activity detection is on, the user can interrupt by speaking. You can also send a `clientContent` message with `turnComplete` set to `true`. The server sends `serverContent.interrupted`; stop playback and discard buffered audio.
  </Accordion>

  <Accordion title="Does it support video?">
    Yes. Send JPEG frames through `realtimeInput.video` at up to 1 frame per second. See [Sending Audio and Video](#sending-audio-and-video).
  </Accordion>

  <Accordion title="How do I keep a session running longer than 10 minutes?">
    Enable `contextWindowCompression` and `sessionResumption`, then reconnect with the latest handle after `goAway`. See [Sessions and Limits](#sessions-and-limits).
  </Accordion>

  <Accordion title="How do I see token usage?">
    Read `usageMetadata` at the end of each turn. Billing uses it, split by modality.
  </Accordion>

  <Accordion title="Can I use tools or function calling?">
    `setup.tools` and the related messages are forwarded unchanged. See the upstream [Gemini Live API reference](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-live) for their format.
  </Accordion>
</AccordionGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.