> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modellix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GPT Realtime 2.1

> Quick start, session parameters, messages, billing, and limits for openai/gpt-realtime-2.1: real-time voice conversations with text, audio, and image input over WebSocket.

GPT Realtime 2.1 is OpenAI's real-time conversation model. It takes text, audio, and image input and replies with text or speech. Use it for voice assistants and other low-latency conversations.

> [card] WSS
>
> `wss://api.modellix.ai/api/v1/live/openai/gpt-realtime-2.1`

| Item | Value |
| - | - |
| Model ID | `openai/gpt-realtime-2.1` |
| Input | Text, audio, images |
| Output | Text or audio for each response |
| Audio Formats | `audio/pcm` at 24 kHz mono, `audio/pcmu`, `audio/pcma` |
| Context | 128,000 token context window, up to 32,000 output tokens |
| Features | Function calling, prompt caching |
| Auth | `Authorization: Bearer <Modellix API Key>` on the handshake |

For authentication, handshake errors, and close codes shared by all Live models, see the [Live Overview](https://docs.modellix.ai/live/overview.md).

## Quick Start

Set `MODELLIX_API_KEY`, then run the script. It sends one text message, prints the spoken reply's transcript, and saves the reply as raw 24 kHz PCM16 in `reply.pcm`.

```python Python theme={null}
# pip install "websockets>=14"
import asyncio
import base64
import json
import os

import websockets

URL = "wss://api.modellix.ai/api/v1/live/openai/gpt-realtime-2.1"


async def main():
    headers = {"Authorization": f"Bearer {os.environ['MODELLIX_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers, max_size=16 * 1024 * 1024) as ws:
        # 1. The server speaks first. Wait for session.created.
        created = json.loads(await ws.recv())
        if created["type"] != "session.created":
            raise RuntimeError(created)

        # 2. Configure the session.
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "type": "realtime",
                "instructions": "Answer briefly.",
                "output_modalities": ["audio"],
                "audio": {
                    "output": {
                        "format": {"type": "audio/pcm", "rate": 24000},
                        "voice": "marin",
                    }
                },
            },
        }))

        # 3. Add a user message and request a response.
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "Introduce yourself in one sentence."}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        # 4. Read events until the response is done.
        with open("reply.pcm", "wb") as output:
            async for raw in ws:
                event = json.loads(raw)
                if event["type"] == "response.output_audio.delta":
                    output.write(base64.b64decode(event["delta"]))
                elif event["type"] == "response.output_audio_transcript.delta":
                    print(event["delta"], end="", flush=True)
                elif event["type"] == "error":
                    raise RuntimeError(event["error"])
                elif event["type"] == "response.done":
                    print("\nUsage:", event["response"]["usage"])
                    return


asyncio.run(main())
```

## Sending Audio

Send base64-encoded audio in the format set by `audio.input.format`. With voice activity detection (VAD) enabled, the model detects speech boundaries and responds automatically.

```json theme={null}
{ "type": "input_audio_buffer.append", "audio": "<base64 audio>" }
```

To control turns yourself, set `audio.input.turn_detection` to `null`, then end each turn with these two messages:

```json theme={null}
{ "type": "input_audio_buffer.commit" }
```

```json theme={null}
{ "type": "response.create" }
```

## Client Messages

Messages use the upstream event format. See the [OpenAI client events reference](https://developers.openai.com/api/reference/resources/realtime/client-events) for all fields.

### session.update

Send `session.update` after `session.created`. The following fields belong inside `session`.

**type** `string` *required*

Allowed value: `realtime`.

**model** `string`

Optional. The URL selects the model, and Modellix replaces this field with the upstream model identifier.

**instructions** `string`

System instructions. An empty string clears them.

**output\_modalities** `string[]`

Allowed values: `["audio"]`, `["text"]`. A single response cannot request both. Audio responses include a transcript.

**max\_output\_tokens** `integer | string` (default: inf)

Allowed values: an integer from 1 to 4096, or `inf`.

**audio.input.format** `object`

Allowed values: `{"type": "audio/pcm", "rate": 24000}`, `{"type": "audio/pcmu"}`, `{"type": "audio/pcma"}`.

**audio.input.noise\_reduction** `object | null`

Noise reduction with `type` (`near_field` or `far_field`) and `strength` (0 to 1, default 1). Use `null` to turn it off.

**audio.input.transcription** `null`

Attached input transcription is not available through Modellix. Only `null` is accepted; any other value closes the connection with `1008`. To transcribe speech, use [GPT Live Transcribe](https://docs.modellix.ai/live/models/gpt-live-transcribe.md) or [GPT Realtime Whisper](https://docs.modellix.ai/live/models/gpt-realtime-whisper.md) on their own endpoints.

**audio.input.turn\_detection** `object | null`

Use `{"type": "semantic_vad"}`, `{"type": "server_vad"}`, or `null` for manual turns. Inside the object, `create_response` and `interrupt_response` both default to `true`. Set them to `false` to keep VAD without automatic responses or interruptions.

`semantic_vad` also accepts `eagerness`: `low`, `medium`, `high`, or `auto` (default). `server_vad` also accepts `threshold` (default 0.5), `prefix_padding_ms` (default 300), `silence_duration_ms` (default 500), and `idle_timeout_ms` (5000 to 30000, or `null`).

**audio.output.format** `object`

The same allowed values as `audio.input.format`.

**audio.output.speed** `number` (default: 1)

Allowed range: 0.25 to 1.5.

**audio.output.voice** `string`

Allowed values: `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin`, `cedar`. OpenAI recommends `marin` or `cedar`. The voice cannot change after the model has replied with audio in the session.

**tools** `object[]`

Function tool definitions. Your application runs each function and returns its result in the conversation. An empty array clears the tools.

**tool\_choice** `string | object` (default: auto)

Allowed strings: `none`, `auto`, `required`. You can also pass an object that selects one function.

**reasoning.effort** `string` (default: low)

Allowed values: `minimal`, `low`, `medium`, `high`, `xhigh`. Higher effort can increase latency and output token usage.

### Other Client Events

**input\_audio\_buffer.append** `event`

Appends base64-encoded audio in the `audio` field.

**input\_audio\_buffer.commit** `event`

Commits the buffered audio as a user turn. Use it when VAD is disabled.

**input\_audio\_buffer.clear** `event`

Discards the buffered audio.

**conversation.item.create** `event`

Adds an item to the conversation. A user message can contain `input_text`, `input_audio`, and `input_image` parts. An `input_image` part takes `image_url` as a base64 PNG or JPEG data URI, and `detail` as `auto` (default), `low`, or `high`.

**response.create** `event`

Starts a response. Use it for text input and when VAD does not create responses automatically.

**response.cancel** `event`

Cancels the response in progress. See [Billing](#billing) for how cancelled responses are charged.

## Server Messages

Server events are forwarded unchanged. See the [OpenAI server events reference](https://developers.openai.com/api/reference/resources/realtime/server-events) for all fields.

**session.created** `event`

The first event on a connection. Wait for it before sending `session.update`.

**session.updated** `event`

The session configuration after a `session.update`.

**response.output\_audio.delta** `event`

A base64-encoded audio chunk in `delta`. This is the only event that carries audio bytes.

**response.output\_audio\_transcript.delta** `event`

A fragment of the spoken reply's transcript in `delta`.

**response.done** `event`

The final event of a response. `response.status` is one of `completed`, `cancelled`, `failed`, `incomplete`, or `in_progress`. `response.usage` contains the token counts used for billing.

**response.usage**

**input\_tokens** `integer`

Total input tokens, including cached tokens.

**output\_tokens** `integer`

Total output tokens.

**total\_tokens** `integer`

Total input and output tokens.

**input\_token\_details** `object`

`text_tokens`, `audio_tokens`, `image_tokens`, `cached_tokens`, and `cached_tokens_details` (`text_tokens`, `audio_tokens`, `image_tokens`).

**output\_token\_details** `object`

`text_tokens` and `audio_tokens`.

**error** `event`

An upstream error. `error.message` describes it. An error event does not always end the session.

## Billing

Each response produces one request record, based on the final `response.usage` for that response ID. A repeated final event is not charged twice.

* **Token types.** Text, audio, and image input, cached input, and text and audio output are priced separately. Cached tokens are part of the input count, and are charged at the cached rate rather than twice. Reasoning tokens are already part of output and are not added again. Unit prices and invoices follow the console.
* **Cancelled and interrupted responses.** Charges follow the usage the upstream reports, which can be zero or positive. Disconnecting does not guarantee a response is free: if you disconnect mid-response, Modellix cancels it and waits up to 2 seconds for the final usage.
* **Missing usage.** If the connection ends before final usage arrives, only usage already received is billed. Modellix does not estimate missing usage.

You can inspect usage and the charged amount in the console's Live request records.

## Sessions and Limits

* WebSocket only. WebRTC, browser client secrets, and other upstream HTTP endpoints are not available through Modellix.
* Attached input transcription is not available. Use [GPT Live Transcribe](https://docs.modellix.ai/live/models/gpt-live-transcribe.md) or [GPT Realtime Whisper](https://docs.modellix.ai/live/models/gpt-realtime-whisper.md) on their own endpoints.
* Modellix closes any connection at its maximum connection duration with code `4408`.

See the [OpenAI model page](https://developers.openai.com/api/docs/models/gpt-realtime-2.1) and the [Realtime conversations guide](https://developers.openai.com/api/docs/guides/realtime-conversations) for model capabilities and conversation design.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.