realtime.md

Realtime API

The Realtime API enables you to build low-latency, multi-modal conversational experiences. It currently supports text and audio as both input and output, as well as function calling through a WebSocket connection.

The Realtime API works through a combination of client-sent events and server-sent events. Clients can send events to do things like update session configuration or send text and audio inputs. Server events confirm when audio responses have completed, or when a text response from the model has been received. A full event reference can be found here and a guide can be found here.

This SDK supports accessing the Realtime API through the WebSocket API or with ws.

Basic text based example with ws:

// requires `yarn add ws @types/ws`
import { OpenAIRealtimeWS } from 'openai/realtime/ws';

const rt = new OpenAIRealtimeWS({ model: 'gpt-realtime' });
let responseDone = false;

// access the underlying `ws.WebSocket` instance
rt.socket.on('open', () => {
  console.log('Connection opened!');
  rt.send({
    type: 'session.update',
    session: {
      output_modalities: ['text'],
      type: 'realtime',
    },
  });

  rt.send({
    type: 'conversation.item.create',
    item: {
      type: 'message',
      role: 'user',
      content: [{ type: 'input_text', text: 'Say a couple paragraphs!' }],
    },
  });

  rt.send({ type: 'response.create' });
});

rt.on('error', (err) => {
  console.error('Realtime error:', err);
});

rt.on('session.created', (event) => {
  console.log('session created!', event.session);
  console.log();
});

rt.on('response.output_text.delta', (event) => process.stdout.write(event.delta));
rt.on('response.output_text.done', () => console.log());

// response.done also covers failed, cancelled, and incomplete responses.
rt.on('response.done', (event) => {
  responseDone = true;
  if (event.response.status !== 'completed') {
    console.error('Response did not complete successfully.');
    process.exitCode = 1;
  }
  rt.close();
});

rt.socket.on('close', () => {
  if (!responseDone) {
    console.error('WebSocket closed before the response completed.');
    process.exitCode = 1;
  }
  console.log('\nConnection closed!');
});

response.done indicates a terminal response, not necessarily a successful one. Check event.response.status for completed; a socket close before response.done also leaves the response unfinished.

To use the web API WebSocket implementation, replace OpenAIRealtimeWS with OpenAIRealtimeWebSocket and adjust any rt.socket access:

import { OpenAIRealtimeWebSocket } from 'openai/realtime/websocket';

const rt = new OpenAIRealtimeWebSocket({ model: 'gpt-realtime' });
// ...
rt.socket.addEventListener('open', () => {
  // ...
});

To attach to an in-progress WebRTC or SIP call over a sideband control connection, pass callID instead of model:

import { OpenAIRealtimeWS } from 'openai/realtime/ws';

const rt = new OpenAIRealtimeWS({ callID: 'rtc_123456' });

To start a transcription-only session, pass intent: 'transcription' instead of a model:

const rt = new OpenAIRealtimeWS({ intent: 'transcription' });

Azure transcription sessions also use transcription intent. Do not pass a deployment in the connection options; configure the transcription deployment in a session.update event after the socket opens:

const rt = await OpenAIRealtimeWS.azure(azureClient, { intent: 'transcription' });

rt.socket.on('open', () => {
  rt.send({
    type: 'session.update',
    session: {
      type: 'transcription',
      audio: {
        input: {
          transcription: { model: 'your-transcription-deployment' },
        },
      },
    },
  });
});

model, callID, and transcription intent are mutually exclusive. Azure transcription sessions must not include a deploymentName. The web WebSocket helper supports the same connection options.

For an Azure Realtime GA call, pass callID to the Azure factory instead:

const rt = await OpenAIRealtimeWS.azure(azureClient, { callID: 'rtc_123456' });

To connect to a deployment that requires an exact WebSocket URL, such as SAP AI Core, provide a buildRealtimeURL callback. Both OpenAIRealtimeWS and OpenAIRealtimeWebSocket accept this option in their constructors, create() factories, and Azure factories:

import OpenAI from 'openai';
import { OpenAIRealtimeWS } from 'openai/realtime/ws';

const client = new OpenAI();
const rt = await OpenAIRealtimeWS.create(client, {
  model: 'gpt-realtime',
  buildRealtimeURL: () =>
    new URL('wss://sap-ai-core.example.com/v2/inference/deployments/my-deployment/realtime'),
});

The callback receives the client and validated connection target and returns the final wss: URL. Exactly one of model, callID, or transcription intent is still required, but the SDK does not add routing parameters such as model to the returned URL. Use only a trusted endpoint because connection credentials are sent to it. Native Azure WebSocket connections send API keys and bearer tokens as handshake request headers, never in the socket URL. Browsers cannot set WebSocket handshake headers, so browser-based Azure clients must use a server-side authentication proxy; server applications can use either OpenAIRealtimeWebSocket.azure() on a header-capable runtime or OpenAIRealtimeWS.azure().

A full example can be found in examples/realtime/websocket.ts.

Create a response from selected items

The GA helpers OpenAIRealtimeWS and OpenAIRealtimeWebSocket provide sendResponseCreate() for typed response inputs that refer to items already in the session. For example, call this function with an open connection and the ID of an item received in a conversation.item.created event on that connection:

import type { OpenAIRealtimeWS } from 'openai/realtime/ws';
import type { OpenAIRealtimeWebSocket } from 'openai/realtime/websocket';

function summarizeItem(rt: OpenAIRealtimeWS | OpenAIRealtimeWebSocket, itemId: string) {
  rt.sendResponseCreate({
    type: 'response.create',
    event_id: 'selected-items-response',
    response: {
      input: [
        { type: 'item_reference', id: itemId },
        { type: 'message', role: 'user', content: [{ type: 'input_text', text: 'Summarize this.' }] },
      ],
      instructions: 'Use the selected items.',
      metadata: { request: 'summary' },
    },
  });
}

The method preserves event_id and the other response parameters. References belong in response input; they cannot create a conversation item. Continue to use send() for its existing event types.

If model-session recovery is enabled, every unsent event submitted with sendResponseCreate() appears as { type: 'raw', data: string } in takePending() and the final close event, even when its input has no references. The string contains the complete JSON event captured when queued. An explicit flushPending() sends these frames in the same order as other retained events.

Realtime error handling

When an error is encountered, either on the client side or returned from the server through the error event, the error event listener will be fired. However, if you haven't registered an error event listener then an unhandled Promise rejection error will be thrown.

It is highly recommended that you register an error event listener and handle errors appropriately as typically the underlying connection is still usable.

const rt = new OpenAIRealtimeWS({ model: 'gpt-realtime' });
rt.on('error', (err) => {
  console.error('Realtime error:', err);
});

Disconnection and restoring a session

By default, Realtime does not retry or reconnect. A protocol error can leave the same socket usable, while a socket close ends that connection. Do not keep retrying permission or authentication failures. Check response.done to learn whether a response completed; a close alone cannot tell you its outcome.

To create a new connection yourself, call OpenAIRealtimeWS.create(client, options) or OpenAIRealtimeWebSocket.create(client, options). Each call resolves the current credential again. For Azure, call the appropriate .azure(client, options) factory. Register listeners on the new connection and wait for open before sending. Remove unwanted listeners with rt.off(event, originalListener).

The helpers imported from openai/realtime/ws and openai/realtime/websocket also support reconnect: true for model sessions. This option does not support transcription sessions or connections to an existing call. Helpers imported from openai/beta/realtime/ws and openai/beta/realtime/websocket keep their existing behavior and do not reconnect.

With reconnection enabled, the SDK keeps the same connection object but replaces its .socket. Each replacement starts a new server session and refreshes credentials. Listeners attached directly to the old socket do not move to the new one. Use the connection's reconnecting, reconnected, and close events.

The SDK allows five reconnect attempts in total by default. Set maxReconnectAttempts to change the limit. Delays start at 500 ms and grow to 8 seconds; use reconnectInitialDelay and reconnectMaxDelay to change them. Calling close() stops retries and prevents a pending credential refresh from opening another socket.

Wait for the initial open before sending. During reconnection, the SDK rejects sends by default. Set maxQueueBytes to retain unsent commands up to that byte limit. After reconnected, restore any session settings and conversation state your application needs, then call flushPending() to send the retained commands in order. Pause other application sends during restoration. To remove retained commands without sending them, call takePending(). The final close event includes remaining unsent commands as its third argument.

send() returns void; handle send and serialization errors with an error listener. The SDK never resends a command it already tried to send. A successful send does not confirm receipt or processing by the server, and a disconnect can leave that outcome unknown. The pending queue stays in memory; it is not durable storage. The SDK does not restore server-side state automatically.

Realtime translation

For Node.js translation sessions and draining final audio and transcripts, see Realtime translation.