An AI voice agent sounds robotic almost never because of the voice itself. Modern text-to-speech is indistinguishable from a person in isolation. What gives an agent away is timing: how long it takes to respond, whether it interrupts, whether it waits too long, and whether its intonation matches what it is saying. Fix the timing and the same voice suddenly sounds human.
Here is where the delay comes from, and what actually removes it.
The two-second rule#
In natural English conversation, the gap between one person finishing and the next starting is around 200 milliseconds. Sometimes it is negative — we start before the other person has finished, because we predicted the ending.
Callers tolerate more from a phone system, but not much. The thresholds we work to:
| Response delay | How it reads |
|---|---|
| Under 500ms | Natural |
| 500ms – 1s | Slightly considered, fine |
| 1s – 2s | Noticeably slow, still acceptable |
| 2s – 3s | "Is it broken?" |
| Over 3s | People start talking over it or hang up |
Everything below is about staying under two seconds, ideally under one.
Where the milliseconds actually go#
A naive pipeline, where each stage waits for the one before it to finish completely:
Caller stops speaking
↓ 800ms silence detection ("are they done?")
↓ 300ms final transcription
↓ 900ms LLM generates the full response
↓ 400ms TTS synthesises the full response
↓ 100ms network + telephony
= 2,500ms before the caller hears a single word
Two and a half seconds. That is the "is it broken?" band, and it is what most cheap agents do.
The fix is not a faster model. It is refusing to wait.
Fix 1: Stream everything#
This is the single largest win, and it is architectural rather than a matter of vendor choice.
Stream the transcription. Do not wait for the caller to stop before transcribing. The transcript should build word by word as they speak, so it is essentially complete the moment they finish.
Stream the model output. Do not wait for the full response before starting synthesis. Send the first clause to TTS as soon as it exists.
Stream the audio. Do not wait for the full audio file. Play the first chunk while the rest is still being generated.
Streamed, the same pipeline looks like this:
Caller stops speaking
↓ 300ms turn detection (tuned, see below)
↓ 80ms transcript already complete — just finalising
↓ 250ms first clause of the response generated
↓ 120ms first audio chunk synthesised
↓ 100ms network + telephony
= ~850ms to first word
Same models. Three times faster. Everything after the first chunk is generated while the caller is already listening.
Fix 2: Tune turn detection properly#
Turn detection is the decision "has the caller finished, or are they thinking?" Get it wrong in either direction and the agent feels wrong.
Too eager and it interrupts. Someone says "I'd like to book for... " while looking at their diary, and the agent starts talking over them. This is the more damaging error — being interrupted is genuinely irritating in a way that waiting is not.
Too patient and every response has an awkward pause hanging in front of it.
A fixed silence threshold cannot serve both. What works is making it adaptive:
- Short (~350ms) after a complete-sounding sentence. "I'd like to book an appointment" is grammatically finished.
- Long (~900ms) after a trailing conjunction or filler. "I'd like to book for... um..." is obviously mid-thought.
- Long after a number is spoken. People pause inside phone numbers constantly.
- Very short during confirmations. After "shall I book that?", "yes" is complete instantly.
Semantic turn detection — using the partial transcript to judge whether the utterance is finished, not just whether the audio is silent — is the difference between an agent that feels attentive and one that feels either rude or slow.
Fix 3: Let it be interrupted#
Humans interrupt each other constantly. An agent that keeps talking through an interruption is unmistakably a machine.
Barge-in means: when caller audio is detected while the agent is speaking, stop immediately, discard the queued audio, and process what they said. Not finish the sentence. Stop.
Two things people get wrong here. First, discard the queued audio, not just the currently playing chunk — otherwise the agent stops, then says three more words. Second, remember what it had already said, because the caller heard that much, and the conversation state has to reflect what was actually communicated rather than what was generated.
Fix 4: Fill the gap when there genuinely is one#
Some delays are unavoidable — a CRM lookup that takes 1.5 seconds is going to take 1.5 seconds.
Humans fill these naturally: "let me just check that for you". Agents should too. The trick is that the filler must be triggered by the delay, not scripted before it. Fire it only when a tool call exceeds ~600ms, otherwise you get an agent that says "let me check" and then answers instantly, which is its own kind of uncanny.
Vary the phrasing. The same filler three times in one call is worse than silence.
Fix 5: Prosody, not voice quality#
This is the last 10%, and it is where most of the remaining "something's off" lives.
Punctuate for speech, not for print. TTS models take pacing cues from punctuation. Commas and full stops in the generated text control the rhythm, so the model's output style is worth constraining explicitly.
Format for the ear. "$1,500" should be sent to TTS as "fifteen hundred dollars". "3pm" as "three PM". "Dr." as "Doctor". Phone numbers as grouped digits with pauses. A model left to guess will get some of these wrong on some calls, and a mispronounced price is memorable in the worst way.
Keep sentences short. Long subordinate clauses are hard to follow on the phone even from a human. Two short sentences beat one long one.
Match energy to content. A greeting and an apology should not be delivered identically. Most TTS providers support per-utterance style or emotion controls; using them is a small change with a disproportionate effect.
What "good" measures like#
Targets we build to:
| Metric | Target |
|---|---|
| Time to first audio byte | < 900ms |
| Turn detection accuracy | > 95% |
| Barge-in response | < 150ms |
| Failed transcription rate | < 3% |
| Escalation rate | 15–30% (by design) |
That last one surprises people. An agent that never escalates is not a good agent — it is an agent that is failing silently. A healthy escalation rate means it recognises its own limits, which is exactly what you want it doing.
Diagnosing your own agent#
If yours sounds robotic, work through this in order:
- Time it. Record a call and measure the gap from your last word to its first. Under a second? Timing is not your problem. Over two? Nothing else matters until this is fixed.
- Interrupt it. If it talks over you, barge-in is not implemented.
- Pause mid-sentence. If it jumps in, turn detection is too eager.
- Give it a phone number with a pause. If it responds halfway through, turn detection is not adaptive.
- Read the transcript. If the text is fine but the delivery is not, it is a prosody problem — formatting and punctuation, not the voice.
Nine times in ten, the answer is a pipeline that waits for each stage to finish instead of streaming. That is an architecture problem, not a budget problem, and swapping to a more expensive voice will not touch it.
If you would rather have this built properly the first time, that is what our voice agent service covers — or read the complete guide for the wider picture.