Building AI Voice Agents with Amazon Nova Sonic and LiveKit: Patterns, Architecture, and a RAG Latency Surprise

I spent a few hours building a series of voice AI agents on Amazon Nova 2 Sonic and LiveKit’s Agents framework — starting from a basic voice round-trip and working up to a phone-answering restaurant concierge with real persistence, and a separate agent grounded in a real knowledge base. This is what actually held up: the patterns worth reusing, the architecture that survived a real phone call, and one latency finding that contradicts the standard assumption about voice RAG.

The stack

Amazon Nova 2 Sonic is a speech-to-speech model — it runs speech understanding, reasoning, tool use, and speech generation as one bidirectional stream, not a chained ASR → LLM → TTS pipeline. That matters for more than latency: acoustic context — tone, pace, hesitation — never gets flattened into a text transcript in between, so the model can respond to how something was said, not just what was said.

LiveKit’s Agents framework handles the part that has nothing to do with conversational AI and everything to do with real-time media: audio capture and streaming, voice activity detection, turn-taking, WebRTC transport, session lifecycle. LiveKit’s own plugin for Nova Sonic means you write an AgentSession and a handful of tools; you don’t write a media pipeline.

Together, that’s a small, honest surface area: the model owns the conversation, LiveKit owns the wire.

Patterns that held up

A few things worked consistently well enough, across enough different agents, that I’d call them defaults rather than one-off wins.

Interruption and multi-turn memory work out of the box. Barge-in — talking over the agent mid-response — and remembering earlier turns in the same conversation both worked cleanly on the first real test, with no special handling required on my end. This is the thing a cascaded STT/LLM/TTS pipeline has to engineer explicitly; a native speech-to-speech model gets it from the architecture.

Tool calls stay responsive during slow operations, if you let them. A simple tool call (a time lookup) completed in about a second. A deliberately slow tool (a 5-second simulated lookup) didn’t block the conversation — the model replied to a second, unrelated request while the slow tool was still in flight, then picked back up within roughly 0.01 seconds of the tool actually completing. It’s watching for completion, not blocking on it. That’s a real design affordance: you can let a slow backend call run without the caller sitting in dead air the whole time, as long as your prompt gives the model permission to keep talking.

Small toolsets, with a confirmation gate before every write. The restaurant concierge agent runs on 9 tools — inside LiveKit’s own guidance of 5 to 10 tools per agent, since tool-selection accuracy degrades past that. Every write (booking a reservation, confirming an order) takes a read_back parameter the model can only set to true after it has actually spoken the details back to the caller and gotten confirmation — call it with read_back=false and the tool hands back the exact wording the model should use to read the details back first. Writes also run with interruptions disabled, so a caller talking over the agent mid-confirmation can’t leave a booking half-committed. That’s the difference between a demo and something you’d trust with a real reservation.

Multilingual support is broad, and the one documented limitation is narrower than it reads. Nova 2 Sonic supports 16 voices across 8 languages, including mid-conversation language switching. AWS’s own AI Service Card states plainly that the model “does not support real-time speech-to-speech translation,” and that’s true if you ask it to act as a continuous interpreter — across repeated testing, it occasionally self-limited on that framing, in-language, and then still attempted a partial translation anyway. For straightforward one-off sentence translation, though, it was accurate and consistent across dozens of test runs in two languages. Worth knowing the difference before you rule the capability out entirely.

Telephony deployment looks nothing like a call center you’d provision yourself. A phone number tied to a LiveKit SIP dispatch rule routes inbound calls to an explicitly named agent — different from the automatic dispatch used for browser/local testing, and LiveKit’s documented recommendation for telephony specifically. The worker that actually answers calls runs as a container on LiveKit Cloud’s managed compute, deployed the same way you’d deploy any containerized service — and it scales to zero replicas when nobody’s calling, spinning one up per inbound call. There’s no EC2 instance sitting idle, no process to keep running by hand. The AWS side of the bill is similarly usage-shaped: DynamoDB on on-demand billing, Bedrock metered per invocation. This is a serverless cost model end to end, not a traditional always-on stack.

Reference architecture: telephony + persistence

This is the shape that actually took a real inbound phone call, in production:

No knowledge base in this path — the menu and hours are a static fixture, deliberately kept simple. That’s a separate pattern, tested as its own agent.

Grounding a voice agent in a knowledge base — and a real latency surprise

Adding Retrieval-Augmented Generation to a voice agent is straightforward: register a tool backed by Amazon Bedrock Knowledge Bases, let the model call it, let it speak the retrieved content back. I tested this as its own standalone agent, against a real technical knowledge base — not a toy dataset — and it worked correctly: asked about a specific technical topic, the reply cited real, specific details that were actually present in the source content, not fabricated.

But the response took noticeably longer than every other tool I’d tested — 46 to 100-plus seconds end to end, against roughly a second for a simple lookup. The instinct, and the standard advice, is to blame the vector database. A Salesforce AI Research paper on voice RAG frames the problem exactly that way: “A typical vector database query adds 50–300ms to the response pipeline, which… pushes total latency well beyond the 200ms budget required for natural conversational flow.” That’s a reasonable place to start, and for a lot of RAG pipelines it’s exactly right.

I wanted real numbers instead of a guess, so I isolated each leg of the pipeline and timed it separately, across three identical runs:

Retrieval and tool-decision latency were both fast and consistent — under 1.3 seconds combined, matching an isolated direct-call benchmark almost exactly. The summarization step — Nova Sonic turning roughly 1,500 characters of dense retrieved content into a spoken answer — accounted for over 99% of total latency, every single time, ranging 55 to 90 seconds by itself. It wasn’t a bug or a fluke: I initially suspected a known class of Nova Sonic tool-calling issue where the model appears to hang after a tool response when multiple tools are registered, but isolated single-tool testing with a longer timeout ruled that out cleanly. The model wasn’t hanging — it was genuinely taking that long to compose a spoken answer from that much dense source material.

That’s the opposite of where the conventional wisdom points. If you’re building a voice agent grounded in a knowledge base and you’re worried about latency, a faster vector database or a semantic cache in front of it is an answer to a question you probably don’t have. The lever that actually moves is what you hand the model to summarize: fewer chunks, shorter chunks, or a retrieval step that returns something closer to a direct answer than a stack of dense paragraphs for the model to condense on the fly.

It also changes what you tell the caller. A 60-to-90 second silence on a phone call reads as broken, even when it’s working correctly. The fix that mattered most wasn’t architectural — it was conversational: have the agent say “let me check that” before it goes quiet, the same thing a real support rep says when they’re actually looking something up. That one line of UX design did more for the felt experience than any latency optimization would have.

Where this fits in the industry

Voice AI has moved past proof-of-concept into a real, fast-growing category — one market estimate puts voice AI agents at $2.4B in 2024, growing to $47.5B by 2034 at a 34.8% CAGR, with BFSI and large enterprises as the earliest heavy adopters. In the restaurant vertical specifically — the use case I built the telephony pattern around — the pattern is consistent across every case study I found: Slang.ai reports 10x ROI and 200+ reservations booked monthly per location on average across its customer base, with 25M+ calls processed across 2,000+ locations. The category is crowded now — Loman, Hostie, Revmo, and others are all building variations of the same core loop: answer the phone, understand intent, take the action.

What’s notable is that the reference pattern for this use case is converging across vendors, independent of which model or framework you use. LiveKit’s own official example for its Agents framework is, not coincidentally, a restaurant reservation and ordering system with a small set of well-scoped tools — the same shape I converged on independently.

The other clear industry signal: real telephony integration is now a first-class feature, not a bolt-on. Nova 2 Sonic launched with direct integration into Amazon Connect, Vonage, Twilio, Audiocodes, and media platforms like LiveKit and Pipecat — the industry has moved past “can this work on a phone call” to “which integration path fits your existing infrastructure.”

What’s still open

I haven’t yet combined the two patterns — a phone agent that both takes orders and answers open-ended knowledge questions in the same session. If I did, I’d keep the grounded-lookup tool on its own latency budget and give the model explicit permission to acknowledge the wait for that specific tool, rather than treating every tool call as equally fast. I also don’t yet know where the summarization-latency curve bends — at what chunk size does 60-second summarization become 15 seconds, and is that relationship linear or does it fall off a cliff past some token threshold? That’s the next thing I want real numbers on, not a guess.