Azure Speech Voice Live API

Overview

Best for: Contact centers and branded assistants on Azure that want noise suppression, server echo cancellation, multilingual semantic VAD, hundreds of voices, custom voice or a talking avatar without stitching services together.

At a glance

Supports both native speech-to-speech models and a cascaded STT+LLM+TTS mode. 146 input / 151 output locales. 600+ Azure neural voices; custom voice is limited access. Prices are Pro tier with gpt-realtime-2.1 (Lite from $4/$12). HIPAA via Microsoft BAA for in-scope services; confirm Voice Live scope. Telephony via Azure Communication Services, not direct SIP. BYO model is preview.

Features

• azure_semantic_vad and azure_semantic_vad_multilingual turn detection (with filler-word removal), plus server_vad and semantic_vad
• azure_deep_noise_suppression
• server_echo_cancellation, plus Live-Reference AEC (client-supplied playback reference, API 2026-07-15+)
• function calling incl. asynchronous calls; MCP in model mode (not with phi models)
• Foundry Agent Service integration (agent_id + project_id)
• phrase lists and custom speech models for recognition
• custom voice, HD voice temperature, speaking rate, custom lexicon
• word-level audio timestamps and visemes
• text-to-speech avatars, photo avatars (VASA-1, VASA-2 preview), custom avatars
• transcription models: azure-speech, mai-transcribe-2 (preview), whisper-1, gpt-4o-transcribe(-diarize)
• telephony through Azure Communication Services

Pricing

Free tier: No dedicated free tier found for Voice Live (Azure Speech F0 is limited to one concurrent request).

Source: learn.microsoft.com

Setup

1. Create an Azure subscription and a Microsoft Foundry resource (recommended over a plain Speech resource; Speech resources lack Agent Service and BYOM).
2. Assign Cognitive Services User and Foundry User roles to your identity, or copy the resource key.
3. Pick a model by name (it sets the Pro / Standard / Lite price); no deployment is needed for natively supported models.
4. Connect a server WebSocket to the voice-live/realtime endpoint with api-version and model query parameters.
5. Send session.update with turn detection, noise suppression, echo cancellation and voice, then stream input_audio_buffer.append.
6. For browsers use the Voice Live WebRTC flow or relay through your backend; for avatars complete the session.avatar.connect SDP exchange.

Endpoint

wss://<foundry-resource>.services.ai.azure.com/voice-live/realtime?api-version=2026-04-10&model=gpt-realtime-2.1 (older resources: <name>.cognitiveservices.azure.com)

Authentication

Recommended Microsoft Entra token (scope https://ai.azure.com/.default) as Authorization: Bearer. API key works as an api-key header (server only) or an api-key query parameter; never use the query-string key in a browser.

import WebSocket from "ws"; // server side (API key in header is not possible from a browser)
const RES = process.env.FOUNDRY_RESOURCE; // Foundry resource name
const url = `wss://${RES}.services.ai.azure.com/voice-live/realtime?api-version=2026-04-10&model=gpt-realtime-2.1`;
const ws = new WebSocket(url, { headers: { "api-key": process.env.FOUNDRY_API_KEY } }); // or Authorization: Bearer <Entra token>
ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      instructions: "You are a friendly support agent. Keep answers short.",
      input_audio_sampling_rate: 24000,
      turn_detection: { type: "azure_semantic_vad", silence_duration_ms: 500, remove_filler_words: true },
      input_audio_noise_reduction: { type: "azure_deep_noise_suppression" },
      input_audio_echo_cancellation: { type: "server_echo_cancellation" },
      voice: { name: "en-US-Ava:DragonHDLatestNeural", type: "azure-standard" },
    },
  }));
});
export function sendAudio(pcm16) { // mono PCM16 at 24 kHz (or 16 kHz if configured)
  ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: pcm16.toString("base64") }));
}
ws.on("message", (raw) => {
  const ev = JSON.parse(raw.toString());
  // Voice Live follows the Azure OpenAI realtime event set; accept both naming styles
  if (ev.type === "response.audio.delta" || ev.type === "response.output_audio.delta")
    playPcm16(Buffer.from(ev.delta, "base64"));
  if (ev.type === "input_audio_buffer.speech_started") stopPlayback();
  if (ev.type === "error") console.error(ev.error);
});

Written from the current docs. Check the vendor's SDK version before you ship.

Warnings

Plus 14 warnings that apply to all voice-to-voice APIs. See category warnings.

Limits

• 100,000 tokens per minute per resource by default (increase on request)
• Max session duration and concurrency are not stated in Voice Live docs
• Phrase list under 500 words/phrases
• SIP is not supported directly; use Azure Communication Services for telephony
• HD voices only in southeastasia, centralindia, swedencentral, westeurope, eastus, eastus2, westus2 (Voice Live routes synthesis to a supported region)
• Content filtering cannot be modified or disabled (use BYOM for custom filtering)

Models and products

Docs and sources

Docs

• Voice Live overview and pricing
• How to use Voice Live
• FAQ
• Samples

Sources used

• learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live
• learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-how-to
• learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-faq
• prices.azure.com/api/retail/prices