Voice Agents and the 500ms Window: An Inference Architecture Guide

Markdown body

Building a voice agent that feels natural in conversation is a harder engineering problem than it looks. The gap between a demo that works and a product that people actually want to use comes down to one number: 500 milliseconds.

That number is not arbitrary. Human conversation turn-taking research (Levinson, 2016; Stivers et al., 2009) consistently finds that response delays above 500ms start to feel awkward, and anything above 1000ms breaks the social contract of conversation. People start to wonder if they were heard, if the call dropped, or if they should speak again. These patterns are consistent across languages and cultures. A voice agent that misses the 500ms window will feel wrong to users even if they can't articulate why.

This post is a technical walkthrough of the entire inference architecture required to hit that target reliably. The short version: the LLM is almost always the bottleneck, streaming is mandatory, and model selection matters more than most teams realize.

## The Voice Pipeline

A voice agent has three distinct processing stages between the user finishing a sentence and audio playing back:

1. **Automatic speech recognition (ASR)** -- converts the user's audio to text
2. **LLM inference** -- generates the response text
3. **Text-to-speech (TTS)** -- converts the response text to audio

Each stage has its own latency profile, and they chain together. The total round-trip time is roughly:

```
total_latency = asr_latency + network_hops + llm_ttft + tts_first_chunk
```

Notice "llm_ttft" specifically, not total LLM generation time. This is important, and we will return to it.

### Realistic Latency Budgets

Here is what each stage actually costs in a well-optimized production system:

| Stage | Typical Range | Notes |
|---|---|---|
| ASR (cloud) | 80--200ms | Real-time streaming ASR can be faster |
| ASR (local/edge) | 30--80ms | Faster-Whisper on GPU |
| Network (ASR + LLM) | 20--60ms | Depends on geographic co-location |
| LLM time-to-first-token | 80--400ms | The main variable |
| TTS first audio chunk | 60--120ms | Streaming TTS systems |

Add these up at the optimistic end: 30 + 20 + 80 + 60 = 190ms. That leaves comfortable headroom under 500ms.

At the pessimistic end: 200 + 60 + 400 + 120 = 780ms. That is well over the threshold, and users will notice.

The LLM TTFT range (80--400ms) is the largest source of variance. Everything else in the pipeline is relatively predictable once you pick your ASR and TTS providers. LLM inference latency is where architecture decisions have the most impact.

## Why TTFT Is the Metric That Matters

Most discussions of LLM performance focus on tokens per second (throughput). For voice, that is the wrong metric. What matters is how quickly the model produces its first output token -- time-to-first-token -- because that is what determines when TTS can begin generating audio.

Once you have the first token, you can start TTS and pipeline the rest. The user does not need to wait for the full LLM response before hearing anything. But until that first token arrives, everything is blocked.

A model generating 100 tokens/second with a 400ms TTFT will sound worse in a voice application than a model generating 60 tokens/second with an 80ms TTFT. The throughput difference barely matters because most voice responses are short (50--150 tokens), and the latency difference means 320ms less dead air at the start.

This is why inference provider selection matters so much for voice specifically. A provider optimized for throughput (maximizing aggregate tokens/second across all users) may batch requests in ways that add 100--200ms to individual TTFT. A provider optimized for TTFT will process your request immediately rather than waiting for a batch to fill.

## Streaming: The Architecture That Makes It Work

Waiting for the full LLM response before starting TTS is a mistake that adds hundreds of milliseconds to every turn. The correct architecture pipelines LLM output directly into TTS in real time.

Here is the basic pattern:

```python
async def voice_response(user_text: str):
    llm_stream = await llm_client.chat.completions.create(
        model="qwen3-coder-7b",
        messages=[{"role": "user", "content": user_text}],
        stream=True,
    )
    
    tts_client = TTSClient()
    text_buffer = ""
    
    async for chunk in llm_stream:
        token = chunk.choices[0].delta.content or ""
        text_buffer += token
        
        # Send to TTS when we have a natural sentence boundary
        if ends_with_sentence_boundary(text_buffer):
            await tts_client.synthesize_stream(text_buffer)
            text_buffer = ""
    
    # Flush remaining text
    if text_buffer:
        await tts_client.synthesize_stream(text_buffer)
```

The key is chunking at sentence boundaries. Sending individual tokens to TTS produces choppy, unnatural audio. Sending complete sentences produces natural prosody. The trade-off is that you wait slightly longer per chunk, but users hear audio within 2--3 sentences of the LLM starting to generate.

In practice, many TTS systems expose their own streaming API that accepts a text stream and manages chunking internally. Systems like ElevenLabs, Deepgram Aura, and Cartesia handle this buffering for you. If you are building on Faster-Whisper or a self-hosted TTS system, you need to manage sentence boundaries yourself.

### The Sentence Boundary Problem

Detecting sentence boundaries in streaming text is trickier than it sounds. Simple period detection fails on abbreviations ("Dr.", "U.S.", "3.5"), mid-sentence pauses, and bullet points. A practical approach:

```python
import re

SENTENCE_END = re.compile(r'(?<=[.!?])\s+(?=[A-Z])|(?<=[.!?])$')

def ends_with_sentence_boundary(text: str) -> bool:
    # Minimum length to avoid flushing on short abbreviations
    if len(text) < 20:
        return False
    return bool(SENTENCE_END.search(text.strip()))
```

This is imperfect. For production, consider a lightweight ML-based sentence splitter, or tune heuristics to your specific domain (technical support conversations have different patterns than casual chat).

## Model Selection for Voice

Most voice conversations do not require a 70B parameter model. Users ask questions, make requests, and have short exchanges. A well-prompted 7B--14B model handles the vast majority of voice interactions with acceptable quality, and it does so with significantly lower TTFT.

The performance difference is real. On identical hardware, a 7B model might return a first token in 60--100ms. A 70B model on the same hardware might take 200--350ms. That is 200ms of difference applied to every single turn in every conversation.

The right model selection process for voice:

1. Define your domain and the hardest 10% of queries you expect
2. Test a 7B and 14B model against those queries with your actual system prompt
3. Only go larger if quality genuinely requires it for that query distribution

For many customer support, appointment scheduling, and information retrieval use cases, smaller models are sufficient. For complex reasoning or code generation over voice, you may need larger models and will need to accept higher latency or invest in faster infrastructure.

### Speculative Decoding

If you need a larger model but want better TTFT, speculative decoding can help. A small draft model (1B--3B parameters) generates candidate tokens that a larger verifier model checks in parallel. Accepted tokens are kept; rejected tokens are regenerated. The result is 2--3x throughput improvement with the same output quality as the large model.

The TTFT benefit for speculative decoding is more modest (it reduces time per generated token, not prefill time), but the reduced time-to-nth-token means audio starts playing sooner in practice.

## Infrastructure Patterns

### Co-location

Network latency is often overlooked. If your ASR provider is on the East Coast, your LLM inference is in the Midwest, and your TTS is on the West Coast, you are adding 40--80ms of pure network latency per hop. For a 500ms budget, that is significant.

Ideal setup: ASR, LLM inference, and TTS are in the same cloud region, or at least the same geographic area. For global products, consider regional deployments with traffic routing based on user location.

### Cold Starts

Serverless GPU inference introduces cold start latency that can be 3--10 seconds for model loading. For voice, that means the first request to a cold worker will fail the 500ms target by a wide margin. This is not usually a problem for high-traffic applications (workers stay warm), but it is a problem for low-traffic applications or during traffic spikes.

Options:
- Keep minimum instances running at all times (adds cost, eliminates cold starts)
- Use a dedicated inference provider with always-warm models
- Build a fallback path that gracefully handles the occasional slow response

### Latency Percentiles

p50 latency (median) looks great in benchmarks. Voice applications live and die by p95 and p99. If 5% of requests take 1200ms, users will notice those pauses. Monitor all percentiles, not just averages.

A good inference provider should offer:
- p50 TTFT: under 100ms
- p95 TTFT: under 250ms
- p99 TTFT: under 400ms

If a provider cannot give you these numbers, test them yourself with a production-like load pattern.

## Practical Architecture Checklist

Before deploying a voice agent, verify each of these:

**ASR**
- [ ] Using streaming ASR rather than batch mode where possible
- [ ] Endpoint detection tuned to your use case (not too aggressive, not too loose)
- [ ] ASR and LLM in the same region

**LLM Inference**
- [ ] Using TTFT-optimized inference (not throughput-maximized)
- [ ] Model size right-sized for quality requirements (7B first, go larger only if needed)
- [ ] Streaming enabled
- [ ] System prompt is fixed and short (longer prompts increase prefill time)

**TTS**
- [ ] Streaming TTS, not batch synthesis
- [ ] Sentence-boundary chunking implemented
- [ ] TTS latency profiled under realistic load

**Infrastructure**
- [ ] p95/p99 TTFT benchmarked, not just p50
- [ ] Cold start strategy defined
- [ ] Geographic co-location verified

## Measuring What Matters

Set up instrumentation that tracks the full round-trip from end of user speech to start of agent audio. Many teams track individual component latencies but miss the end-to-end number.

A minimal instrumentation setup:

```python
import time

class VoiceTurnMetrics:
    def __init__(self):
        self.user_speech_end = None
        self.asr_complete = None
        self.llm_first_token = None
        self.tts_first_audio = None
    
    def record(self, event: str):
        setattr(self, event, time.monotonic())
    
    def to_dict(self):
        if not all([self.user_speech_end, self.tts_first_audio]):
            return {}
        return {
            "asr_latency": self.asr_complete - self.user_speech_end,
            "llm_ttft": self.llm_first_token - self.asr_complete,
            "tts_first_chunk": self.tts_first_audio - self.llm_first_token,
            "total_round_trip": self.tts_first_audio - self.user_speech_end,
        }
```

Log these per turn, aggregate by percentile, and set alerts when p95 total round-trip exceeds 450ms. That gives you 50ms of headroom before users start noticing.

## The TTFT Problem in Practice

Here is what this looks like with real numbers from a production voice agent deployment. Assume:
- ASR (streaming Deepgram): ~90ms
- Network: ~25ms (same region)
- TTS first chunk (Cartesia): ~70ms

That accounts for 185ms. The remaining 315ms is the budget for LLM TTFT to hit a 500ms total. If the inference provider averages 200ms TTFT, you are fine at p50. But if p95 TTFT is 400ms, 5% of turns will take ~685ms -- 200ms over threshold.

Switching to an inference provider with p95 TTFT of 200ms drops that 5% of turns to ~385ms, which is comfortably under threshold. The difference is entirely in the inference layer.

## Where to Go From Here

If you are building voice agents, test your inference provider's TTFT under realistic load early in development, not at launch. The difference between providers is large enough to determine whether your product feels natural or frustrating.

General Compute's inference API is designed for exactly this use case -- low TTFT at p95 and p99, not just p50. The same API endpoint works for both voice and text applications, so the migration path from an existing OpenAI-compatible integration is a single base URL change.

Start with a 7B or 14B model and a streaming pipeline. Measure end-to-end round-trip latency at p50, p95, and p99. If you are hitting the 500ms target at p95, you are in good shape. If not, the bottleneck is almost always LLM TTFT -- and that is a solvable infrastructure problem.