Building Voice Agents with LiveKit, Cartesia, and AWS

Building a useful voice agent connects media, speech, reasoning, business actions, security controls, data retention, and teardown.

I tested this with a fictional restaurant-ordering agent. A real phone call entered through LiveKit SIP, reached a LiveKit worker, used Cartesia for speech recognition and synthesis, and asked a text model to manage the conversation. After the caller confirmed the order, the worker sent a signed request to an API. A small AWS Lambda function validated it, encrypted the contact details, and wrote the simulated order to DynamoDB.

The call produced an API 201 response and a new encrypted test record. That was the first point at which I considered the agent meaningful. The result demonstrated a completed action, a durable record, and the privacy and security boundaries I needed to examine around that action.

The working unit is a chain of boundaries. The model is one component in that chain.

The System Behind the Voice

The working shape has a live conversation path and a separate durable-write path:

LIVE CONVERSATION

Caller speaks
  -> LiveKit SIP and room
  -> Cartesia STT: audio to text
  -> LiveKit worker + OpenRouter LLM: Gemini 2.5 Flash Lite
       - menu tools
       - per-call cart: items, name, pickup time, contact choice, order ID
  -> Cartesia TTS: response text to audio
  -> LiveKit SIP and room
  -> Caller hears the response

DURABLE WRITE, AFTER READ-BACK CONFIRMATION

LiveKit worker cart
  -> signed HTTPS order request
  -> API Gateway
  -> Lambda: validate catalog, timestamp, nonce, and signature
  -> KMS: encrypt contact details
  -> DynamoDB: simulated order record

The cart stays in the worker’s memory during the call. The worker builds it through validated order tools and reads it back to the caller. Cartesia converts caller audio to text before the model acts and converts the model’s response text back to streaming audio.

The text-model layer is separate from the speech layer. The phone test used Gemini 2.5 Flash Lite through OpenRouter for the ordering dialogue and tool selection. OpenRouter lists the model at $0.10 per million input tokens and $0.40 per million output tokens, which made it a low-cost choice for this controlled text workload. Cartesia handled speech recognition and synthesis, so the LLM did not process the live audio stream.

An OpenRouter-hosted Qwen3-8B remains a concrete open-weight, Apache-2.0 alternative. I did not test that substitution in this phone flow. Changing models requires a fresh conversation-quality and tool-reliability evaluation.

Each arrow is a different contract. The phone layer determines how a caller is routed. The media layer determines when the agent hears and speaks. The model decides whether the conversation has enough information to act. The tool boundary determines whether a spoken request becomes a durable write. The data boundary determines what survives after the call.

The visible conversation relies on deliberate design for identity, confirmation, validation, storage, observability, and deletion.

Restaurant ordering is a credible voice-agent workflow because it has a bounded catalog, a spoken interaction, a clear confirmation point, and an action a backend can validate. LiveKit publishes a restaurant-agent recipe and a drive-through example with order state and database-backed order management.

My test used an external HTTPS gateway so the worker could invoke one narrow write operation without cloud credentials. The request carried an HMAC signature over the exact body, a timestamp, a single-use nonce, and a random order ID. The API was Internet-reachable and accepted any request with that shared secret. This was a request-integrity pattern for one controlled test, not per-worker identity, user authorization, or a complete production security design.

The Lambda validated catalog values and request integrity, encrypted the contact details, and wrote the simulated order. Its DynamoDB policy allowed writes to the dedicated tables and excluded scan, query, delete, and unrelated data access. The function also needed narrowly scoped KMS permissions to encrypt the contact data.

Confirmation Is Part of the Write Path

The conversational path needed an explicit finalization rule.

The agent collected menu items, a pickup time, and a name. The worker’s final tool required a read-back flag and complete fields before it submitted the order. I also separated saving order details from finalizing an order after the model retried a finalization attempt with missing fields.

This is a useful pattern for any voice action with consequences:

Collect -> summarize -> caller confirms -> validate -> persist -> return confirmation

In this test, the read-back flag came from the model’s tool call. It was a conversational guard, not independently verifiable evidence of caller consent. The server validated the catalog, quantities, and other business rules again. A production write path needs a durable confirmation event that the server can verify independently.

LiveKit’s guidance on agent tools captures one part of the pattern: tools need narrow, clear purposes. The larger lesson is that a tool is a policy boundary. Its input schema, validation, idempotency behavior, and audit trail are product decisions.

Privacy Across the System

The test identified a privacy limit.

The full caller-number attribute entered the worker process for inbound SIP calls. The agent spoke only the last four digits, then asked whether that number should be recorded or whether the caller wanted to provide another one. When selected, the full number went to the order gateway. Caller ID is routing metadata and personal data, not identity proof or authority to disclose account information, take payment, or change an account.

In the August 6 test, application API and Lambda logs were configured not to contain the caller’s name or phone number. The encrypted DynamoDB record did not expose either field in plaintext. I also observed a LiveKit framework lifecycle log with a SIP participant identity containing the caller number. That observation applies to this test configuration. Hosted-runtime metadata and retention settings remain part of the privacy model.

That observation expands the architecture discussion. A privacy review tracks where data appears at every stage:

Caller speech can reach Cartesia before the model acts. Names, order details, and an alternate contact number can then reach the configured text-model provider. This happens before the AWS encryption boundary. Every row can retain data. Every row needs a retention decision.

LiveKit documents that HidePhoneNumber suppresses the sip.phoneNumber participant attribute. In that configuration, the agent asks the caller to provide a contact number. The setting applies to that documented attribute and does not establish the absence of all caller-related provider metadata. LiveKit Cloud’s agent observability documentation treats transcripts, recordings, traces, and runtime logs as session data, with a documented 30-day retention window. Its Build plan can retain anonymized session data longer for model improvement. Cartesia offers zero data retention for its STT and TTS inference APIs on eligible enterprise plans, while retaining operational metadata. This test did not establish that entitlement. Those controls are part of the architecture decision, not deployment settings to address later.

Teardown Is Also an Architecture Boundary

The system also needs a defined deletion path.

This test required a dedicated phone route, worker, API, Lambda function, DynamoDB tables, secret, encryption key, and logs. The teardown plan removes the route before the number, removes the API before its execution path becomes orphaned, deletes the data and secrets, and schedules the encryption key for deletion after its recovery period. It also records what was removed and what will disappear later.

That work belongs in the design from the first call. It limits cost, reduces the chance that test data persists unseen, and gives the architecture a defined end state.

So What

The operating question is whether the system can complete a task with bounded authority and an understood data trail.

LiveKit supplied the phone and real-time media layer. Cartesia supplied speech components. AWS supplied the validation, encryption, and durable-write boundary. The layers sit behind defined interfaces, but changing any one of them requires a fresh integration, privacy, and quality evaluation.

This was a controlled technical pattern, not a restaurant deployment or a compliance determination. A production team needs channel- and jurisdiction-specific review of recording, privacy, caller-ID, disclosure, retention, access, deletion, and escalation requirements. It also needs a written retention and access review for the telephony provider, hosted agent runtime, speech provider, text-model provider, and its own observability systems.

That review, alongside confirmation and durable-write controls, determines whether a voice agent is ready to support a real workflow.

I have not resolved the right way to prove caller confirmation independently without making the conversation less natural. That boundary remains open.