Building an AI Voice Agent That Books Appointments

Building an AI voice agent that books appointments means wiring four things together: a telephony layer that streams audio, a speech pipeline, a language model with tool access, and a live calendar integration that can write. The hard part is not the conversation — it is making the booking correct, atomic and recoverable when something fails mid-call.

This is how we build them, including the parts that only become obvious after the first one goes live.

The architecture#

Caller
  ↓ (SIP / WebRTC)
Telephony layer  ── streams audio in, plays audio out
  ↓
Speech-to-text   ── streaming transcription + turn detection
  ↓
Orchestrator     ── conversation state, tool routing
  ↓         ↘
LLM          Tool calls ── check_availability()
  ↓                       create_booking()
Text-to-speech            lookup_customer()
  ↓                       transfer_to_human()
Caller

The orchestrator is the piece people underestimate. It owns conversation state, decides when a tool result should be spoken versus acted on silently, and handles the case where the model wants to call a tool that is currently failing.

Step 1: Define the booking, precisely#

Before any code, write down what a valid booking is for your business. Not roughly — exactly.

• What information is mandatory to create one? (Usually: name, contact number, service type, slot.)
• What is optional but useful? (Existing customer reference, notes, preferred practitioner.)
• What makes a slot eligible? (Duration by service type, buffer either side, which staff can perform it, how far ahead someone may book, cut-off before start.)
• What is forbidden? (Double-booking, booking outside opening hours, booking a service the caller is not eligible for.)

This document is the specification. Almost every production bug traces back to a rule that lived in someone's head instead of on this list.

Step 2: Choose the voice stack#

Two viable routes.

Speech-to-speech models handle audio in and audio out in one model. Latency is excellent and the prosody is very natural. Control is weaker — steering exactly what gets said is harder, and tool-calling reliability varies.

Composed pipelines chain separate STT, LLM and TTS components. More moving parts and slightly more latency, but you can swap any component, log every stage, and constrain the model's output precisely.

For booking agents we default to a composed pipeline. When an agent is writing to a real calendar, the ability to inspect and constrain each stage is worth the extra 100–200ms — and latency is manageable if you engineer for it.

Step 3: Build the calendar integration first#

Not the conversation. The integration.

The reason is simple: conversation quality is tunable at any point, but if your calendar layer is wrong, everything built on top of it is wrong. You need four operations:

check_availability(service_type, date_range) → [slots]
hold_slot(slot_id, ttl)                      → hold_token
create_booking(hold_token, customer_details) → booking_id
cancel_booking(booking_id)                   → ok

That hold_slot step is the one people skip, and it is the one that matters.

The double-booking problem#

Here is the failure, and it is guaranteed to happen eventually:

1. The agent checks availability at 14:32:10 and finds Thursday 3pm free.
2. It offers Thursday 3pm to the caller.
3. The caller thinks about it, checks with their partner, agrees at 14:32:48.
4. Somebody else booked Thursday 3pm at 14:32:31.
5. The agent writes the booking anyway.

Thirty-eight seconds is a very long time in a shared calendar. The fix is to hold the slot the moment you offer it, with a short TTL — 90 seconds is usually right — and release it if the caller declines or the call drops. Then create_booking converts the hold rather than racing for the slot.

If your calendar system genuinely cannot hold, the fallback is to re-verify immediately before writing and have a scripted recovery: "I'm sorry — that slot went while we were talking. I have 3:30 or Friday at 10." Ugly, but far better than two people arriving at once.

Time zones#

Store and transmit UTC everywhere. Convert only at the edges: when speaking a time to the caller, and when displaying it. Every time-zone bug we have seen came from a system that did the conversion somewhere in the middle.

Step 4: Design the conversation around the data you need#

A booking conversation is a slot-filling exercise wearing a friendly hat. You need four or five facts. Design the flow to collect them naturally, and let the caller give you several at once.

A caller who opens with "Hi, I'd like to book a cleaning for next Tuesday afternoon if you have anything" has just supplied service type, date and time preference in one sentence. An agent that responds "Certainly! What service would you like?" has failed at the first turn. Extract everything present, then ask only for what is missing.

Three rules that carry most of the quality:

Confirm before writing, once. Read back the full booking — service, day, date, time, name — and get a yes. Once, not after every field.

Offer alternatives, do not report failure. Never say "that's not available" alone. Say "that one's gone — I've got 4pm the same day, or 10am Friday."

Spell back contact details. Phone numbers and email addresses are where transcription errors hurt most, because the error is invisible until the confirmation does not arrive.

Step 5: Write the escalation rules#

Decide upfront, in the spec, what triggers a hand-off:

• The caller explicitly asks for a person.
• The caller repeats themselves twice — a reliable signal the agent is not understanding.
• Frustration or distress in the transcript.
• Anything about a complaint, a refund, or a medical or legal question.
• Any tool failing twice in a row.
• Anything genuinely outside the defined scope.

When it hands off, it must pass the full transcript and any collected details. A caller who has just spent ninety seconds giving their information and is then asked for it again has had a worse experience than if you had never deployed the agent.

Out of hours, hand-off means taking the booking and flagging it for morning — not "please call back".

Step 6: Test like a caller, not like a developer#

Scripted tests pass. Real calls do not. Before launch, make at least thirty calls covering:

• Background noise — a car, a café, a TV.
• Accents unlike your own.
• Interrupting the agent mid-sentence.
• Changing your mind halfway through.
• Asking something completely out of scope.
• Giving a phone number with a pause in the middle.
• Two requests in one sentence.
• Silence after the agent's greeting.
• Hanging up mid-booking, then calling straight back.

That last one matters: it tells you whether your held slots release properly.

Step 7: Instrument everything#

From day one, log per call: full transcript, every tool call and its result, latency per turn, whether a booking was created, whether it escalated and why.

Then read the first two weeks of transcripts properly. Every booking agent we have deployed had at least one significant improvement that no test script predicted — a phrase people use that we had not anticipated, a question we assumed nobody would ask, a rule we did not know the business had.

What this looks like in practice#

For a typical single-service booking agent — one calendar, one CRM, one escalation path — this is two to three weeks: three days scoping and writing the spec above, a week on the integration and orchestration, four or five days on conversation design and testing, then a tuning window after launch.

If you would rather not build it yourself, that is exactly what we do — or start a scope and we will send back the spec and a fixed quote.