AssemblyAI Voice Agent API
A single WebSocket voice agent (STT, LLM via AssemblyAI's LLM Gateway, TTS, turn detection, tools) at an all-inclusive $4.50 per hour. Attractive when you want one flat price that already includes the LLM.
Overview
Best for: Teams wanting a simple all-in price with LLM included and strong STT accuracy.
At a glance
Flat $4.50/hour includes STT, LLM, TTS and tools. Sessions have an expiry (expires_at) whose length is not documented. Twilio SIP was listed as coming Q2 2026.
PCM16 mono 24 kHz (audio/pcm default)
PCM16 24 kHz (reply.audio)
Declared via input.language_codes or auto-detected.
Preset voices (e.g. alba).
Not stated on the pages read.
Not stated.
Vendor describes the API as PCI-certified; other certifications not re-verified.
Features
- tool calling
- turn detection (VAD thresholds, min/max silence)
- interruptions
- greeting
- stored agents bound by agent_id
- mid-session prompt updates
- session resume via session_id
Pricing
| What | Price | Unit |
|---|---|---|
| Voice Agent session | $4.50 | per hour ($0.075/min) |
Flat all-inclusive rate; telephony not included.
Not token-billed.
Free tier: $50 free credits at signup, no card.
Source: assemblyai.com
Setup
- Create an AssemblyAI account and API key.
- Open wss://agents.assemblyai.com/v1/ws with Authorization: <key>.
- Send session.update (either an agent_id OR inline system_prompt/greeting/voice, not both).
- Wait for session.ready, then stream input.audio and play reply.audio.
Endpoint
wss://agents.assemblyai.com/v1/wsAuthentication
Authorization header with the raw API key (Bearer prefix also accepted)
Quick start javascript
// npm i ws
import WebSocket from "ws";
const ws = new WebSocket("wss://agents.assemblyai.com/v1/ws", {
headers: { Authorization: process.env.ASSEMBLYAI_API_KEY }, // raw key; "Bearer <key>" also accepted
});
let ready = false;
ws.on("open", () => {
ws.send(JSON.stringify({
type: "session.update",
session: {
system_prompt: "You are a friendly support agent. Keep responses under 2 sentences.",
greeting: "Hi! How can I help you today?",
output: { voice: "alba" },
},
}));
});
ws.on("message", (raw) => {
const ev = JSON.parse(raw.toString());
if (ev.type === "session.ready") ready = true; // only send audio after this
if (ev.type === "reply.audio") play(Buffer.from(ev.data, "base64")); // PCM16
if (ev.type === "error") console.error(ev);
});
// PCM16 mono 24 kHz
export function sendPcm(buf) {
if (ready) ws.send(JSON.stringify({ type: "input.audio", audio: buf.toString("base64") }));
}
function play(pcm) {}Written from the current docs. Check the vendor's SDK version before you ship.
Warnings
Sessions expire without warning
session.ready includes expires_at; when the TTL is reached you get session_expired with no advance notice. Track the timestamp and reconnect with the session_id.
Docs disagree on the URL
The quickstart uses /v1/realtime while deploy and config docs use /v1/ws; use /v1/ws and confirm.
Telephony still maturing
The pricing page described Twilio SIP as 'coming Q2 2026' with price TBA; confirm current status before planning phone deployments.
agent_id and inline config are exclusive
Sending system_prompt or tools alongside agent_id is rejected.
Cascaded under the hood
Despite the old 'Speech-to-Speech API' name it chains STT, LLM and TTS.
Plus 14 warnings that apply to all voice-to-voice APIs. See category warnings.
Limits
- Session has a maximum duration (expires_at in session.ready); session_expired arrives with no advance warning
- Streaming concurrency on the free tier: 5 new streams per minute (general streaming figure)
Models and products
| Name | Status |
|---|---|
| Universal-3.6 Pro realtime STT + LLM Gateway + TTS | GA |
Docs and sources
Docs
Sources used
- assemblyai.com/pricing.md
- assemblyai.com/docs/voice-agents/voice-agent-api/session-configuration
- assemblyai.com/docs/voice-agents/voice-agent-api/events-reference
- assemblyai.com/docs/voice-agents/voice-agent-api/deploy
Which LLMs are included at the flat rate, max session length, concurrency, language list, current SIP status.