Doubao Realtime Voice Model (end-to-end realtime speech)

Overview

Best for: Chinese-language consumer voice products, role-play/companion apps and in-car or device assistants targeting mainland China.

At a glance

Prices converted from 80 / 300 CNY per 1M tokens at 7.1 CNY per USD. Free quota exists but size not documented. Mainland China only with real-name verification. Default 60 session starts per minute and 100k TPM per AppID. Connection released after 10 min idle. Older 2.0 models have 12K context.

Features

• full-duplex (3.0)
• barge-in
• server VAD, push-to-talk, text input and audio-file input modes
• function calling with parallel calls (3.0)
• system prompt / persona fields
• voice cloning (SC2.0, cloning 2.0 product)
• singing
• web search via extension
• hot words
• conversation history injection and resume (keeps last 20 rounds)

Pricing

Free tier: Free quota exists and can offset cached, uncached and output tokens, but the size was not found in the docs read.

Source: docs.volcengine.com

Setup

1. Create a Volcengine account and complete real-name verification (Chinese ID or business licence expected).
2. In the Doubao Speech console, enable the end-to-end realtime voice model and create an API key (new console).
3. Read the 'access must-read' page: 3.0 uses a new JSON event protocol that differs from the older binary protocol.
4. Open the WebSocket with X-Api-Key, send session.create with session.model = 1.2.6.1, then stream 20 ms PCM frames.
5. Handle response.output_audio.delta (Ogg Opus by default) and close with session.close.

Endpoint

wss://openspeech.bytedance.com/api/v3/duplex/realtime/dialogue (3.0 full duplex); legacy: wss://openspeech.bytedance.com/api/v3/realtime/dialogue

Authentication

3.0: X-Api-Key header (new console). Legacy: X-Api-App-ID, X-Api-Access-Key, X-Api-Resource-Id: volc.speech.dialog, X-Api-App-Key: PlgvMymc7f3tQnJ6 (fixed value)

# pip install websockets   (Doubao Realtime 3.0 "Seeduplex", full-duplex JSON protocol)
import asyncio, base64, json, os, websockets

URL = "wss://openspeech.bytedance.com/api/v3/duplex/realtime/dialogue"
HEADERS = {"X-Api-Key": os.environ["VOLC_SPEECH_API_KEY"]}  # new console > API Key management

async def main(pcm_chunks):  # 16 kHz mono int16 LE, 20 ms (640-byte) chunks, sent in real time
    async with websockets.connect(URL, additional_headers=HEADERS) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "session": {"model": "1.2.6.1",  # fixed value for the full-duplex version
                        "instructions": "You are a friendly assistant."},
        }))
        async def uplink():
            for chunk in pcm_chunks:
                await ws.send(json.dumps({"type": "input_audio_buffer.append",
                                          "audio": base64.b64encode(chunk).decode()}))
                await asyncio.sleep(0.02)  # real-time pace: too fast or too slow is an error
        async def downlink():
            async for msg in ws:
                ev = json.loads(msg)
                if ev["type"] == "response.output_audio.delta":
                    pass  # base64 Ogg Opus by default; decode and play
                elif ev["type"] == "error":
                    print(ev)
        await asyncio.gather(uplink(), downlink())
        await ws.send(json.dumps({"type": "session.close"}))  # wait for reply before closing

# asyncio.run(main(your_chunks))

Written from the current docs. Check the vendor's SDK version before you ship.

Warnings

Plus 14 warnings that apply to all voice-to-voice APIs. See category warnings.

Limits

• Default QPM 60 (StartSession / session.create per minute per AppID) and TPM 100,000; raise via sales
• Server releases the connection after 10 minutes with no interaction (error 45000003)
• Uplink audio must keep real-time pace; send input_audio_mute.commit when the mic is muted or the session times out
• Close with session.close and wait for the reply, otherwise error 55000001 ContextCanceled
• O2.0/SC2.0 max context 12K

Models and products

Docs and sources

Docs

• Realtime 3.0 full-duplex API
• Access must-read (3.0 protocol)
• Legacy end-to-end API
• Billing

Sources used

• docs.volcengine.com/docs/6561/1594356
• docs.volcengine.com/docs/DoubaoVoice/access-mustread?lang=zh
• docs.volcengine.com/docs/DoubaoVoice/endtoend-realtime-voice-full-duplex-versio...
• docs.volcengine.com/docs/6561/1359370?lang=zh