OpenAI Realtime API

Overview

Best for: Production voice agents and phone bots that need the strongest tool calling and reasoning with a mature SDK, WebRTC in the browser and native SIP.

At a glance

Prices are gpt-realtime-2.1 (mini: $10/$20). SOC 2 and BAA are account-level OpenAI programs; confirm realtime coverage. EU residency needs approved abuse-monitoring controls. Languages: multilingual, no published list. Latency: only a relative p95 claim.

Features

• function calling (parallel tool calls)
• remote MCP servers as tools
• semantic_vad and server_vad turn detection, or manual push-to-talk (turn_detection null)
• barge-in with conversation.item.truncate to sync what the user heard
• image input (input_image) on gpt-realtime-2.x and gpt-realtime
• configurable reasoning effort on 2.x models
• input audio transcription in parallel
• out-of-band responses (response.conversation = none)
• automatic prompt caching
• truncation controls (retention_ratio, token_limits.post_instructions)
• SIP telephony with webhooks (realtime.call.incoming)
• sideband server WebSocket for a WebRTC or SIP call (?call_id=)
• realtime transcription sessions and live translation sessions
• Agents SDK (@openai/agents/realtime) for browser and server

Pricing

Free tier: None. Free tier is not supported for realtime models; usage tiers are now named Build, Launch, Grow.

Source: developers.openai.com

Setup

1. Create an OpenAI Platform account and a project at platform.openai.com; add a payment method (free tier cannot use realtime).
2. Create a project API key and store it only on your server.
3. Browser: your server calls POST https://api.openai.com/v1/realtime/client_secrets with the session config and returns the ek_ value; the browser POSTs its SDP offer to https://api.openai.com/v1/realtime/calls with that key and opens the oai-events data channel.
4. Server or telephony bridge: open the WebSocket below with the API key and send session.update with session.type = realtime.
5. Phone: point your SIP trunk at sip:<project_id>@sip.api.openai.com;transport=tls, add a realtime.call.incoming webhook, then POST /v1/realtime/calls/{call_id}/accept.
6. Log response.done usage on every turn so you can see context growth and cache hit rates.

Endpoint

wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1 (WebSocket); https://api.openai.com/v1/realtime/calls (WebRTC SDP); sip:$PROJECT_ID@sip.api.openai.com;transport=tls (SIP)

Authentication

Authorization: Bearer <API key> for server WebSocket and SIP control. For browsers and mobile, mint an ephemeral key (ek_...) server side via POST /v1/realtime/client_secrets (expires_after.seconds 10 to 7200, default 600) and lock instructions/tools in that session config. Optional OpenAI-Safety-Identifier header on the server request is bound to the token.

import WebSocket from "ws"; // server side only: never ship the API key to a browser
const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1", {
  headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
});
ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      type: "realtime",
      instructions: "You are a friendly support agent. Keep answers short.",
      audio: {
        input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: { type: "semantic_vad" } },
        output: { format: { type: "audio/pcm", rate: 24000 }, voice: "marin" },
      },
    },
  }));
});
// Call for every ~100 ms chunk of 24 kHz mono PCM16 from your mic or phone bridge
export function sendAudio(pcm16) {
  ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: pcm16.toString("base64") }));
}
ws.on("message", (raw) => {
  const ev = JSON.parse(raw.toString());
  if (ev.type === "response.output_audio.delta") playPcm16(Buffer.from(ev.delta, "base64"));
  if (ev.type === "input_audio_buffer.speech_started") stopPlayback(); // barge-in: flush local audio
  if (ev.type === "response.done") console.log(ev.response.usage); // log tokens per turn
  if (ev.type === "error") console.error(ev.error);
});

Written from the current docs. Check the vendor's SDK version before you ship.

Warnings

Plus 14 warnings that apply to all voice-to-voice APIs. See category warnings.

Limits

• Max session length 60 minutes, no warning event before cutoff (third-party SDKs reconnect at ~50 min)
• Context window 128,000 tokens; max output 32,000 (2.1 and 2.1-mini)
• Default rate limits gpt-realtime-2.1: Build 400 RPM / 200,000 TPM, Launch 10,000 RPM / 4,000,000 TPM, Grow 20,000 RPM / 15,000,000 TPM
• gpt-realtime-translate: Build 200, Launch 650, Grow 850 minutes of audio per minute
• Ephemeral client secret lifetime 10 s to 7,200 s, default 600 s
• Audio append chunk max 15 MB

Models and products

Docs and sources

Docs

• Realtime guide
• Realtime conversations (events, voices, VAD)
• Managing costs
• WebRTC guide
• SIP guide
• Realtime transcription
• Deprecations

Sources used

• developers.openai.com/api/docs/pricing
• developers.openai.com/api/docs/models/gpt-realtime-2.1
• developers.openai.com/api/docs/models/gpt-realtime-2.1-mini
• developers.openai.com/api/docs/models/gpt-realtime-translate
• developers.openai.com/api/docs/guides/realtime-costs
• developers.openai.com/api/docs/guides/realtime-conversations.md
• developers.openai.com/api/docs/guides/voice-websockets?api=realtime
• developers.openai.com/api/docs/guides/voice-webrtc?api=realtime
• developers.openai.com/api/docs/guides/voice-sip?api=realtime
• developers.openai.com/api/reference/resources/realtime/subresources/client_secr...
• developers.openai.com/api/docs/deprecations
• developers.openai.com/api/docs/guides/your-data
• datanorth.ai/news/openai-releases-gpt-realtime-2-1-voice-models