Text to speech for phone calls introduces constraints that are less prominent in a typical browser demo. PSTN codecs, media gateways, packet timing, endpointing, and playback queues can all affect whether a response feels immediate and natural -- or delayed and fragmented.

This production-focused walkthrough is for teams building AI voice agents for conversational phone calls with SIP, Twilio-style media streams, WebSockets, real-time voice AI over WebRTC, or contact center infrastructure. It follows the signal path in order, from captured speech through TTS playback and interruption.

Table of contents

• The production voice pipeline: Where TTS fits in a complete call.

• Three meanings of latency: Model execution, first audio, and conversational delay.

• Telephony audio constraints: Sample rates, G.711, resampling, and quality.

• Streaming and delivery: Chunking, buffering, networking, and playback.

• Turn-taking and barge-in: Endpointing, pauses, interruption, and cancellation.

• Measurement and latency budgets: Instrumentation and practical targets.

• Frequently asked questions: Common implementation and selection questions.

• Production deployment checklist: Final validation before real callers.

The Production Voice Pipeline

A conventional voice bot with TTS and STT pipeline looks like this:

Caller audio -> PSTN or SIP -> media gateway -> VAD and endpointing -> streaming STT -> turn manager -> LLM -> streaming TTS -> audio buffer -> telephony playback

Every arrow in that chain can add delay. The media gateway packetizes audio. Voice activity detection decides whether speech is present. Endpointing decides whether the turn is complete. The LLM must produce speakable text, the TTS service must return audio, and the playback system must convert that audio into whatever format the carrier expects.

What most teams get wrong: They optimize TTS inference while leaving a 700 ms endpointing timeout and a 400 ms playback buffer untouched. The call still feels slow because perceived conversational latency belongs to the entire pipeline.

For a broader system view, see this real-time voice AI architecture for telephony.

Model Latency, First Audio, and Conversational Latency

Latency terms that should never be reported interchangeably

Low latency TTS marketing typically emphasizes time to first audio. That number excludes endpointing, LLM work, network transit, transcoding, and queued playback. A provider can return the first chunk quickly while later chunks arrive too slowly to sustain uninterrupted speech -- and the caller will notice.

A practical voice assistant latency budget might allocate 200 to 500 ms for endpointing, 100 to 400 ms for initial language generation, 100 to 300 ms for TTS first audio, and 40 to 200 ms for transport, conversion, and buffering. These are engineering ranges, not universal thresholds. Geography, prompt complexity, carrier routing, and conversational context all shift the result.

Optimize the longest and most variable stage first. Guidance on how to build faster AI voice agents covers neural synthesis latency in more detail.

Telephony Audio Is Its Own Delivery Environment

Traditional narrowband telephony commonly uses an 8 kHz sampling rate. In the telephone narrowband, the nominal passband is roughly 300 to 3400 Hz. ITU-T G.711 specifies 8000 samples per second, eight bits per sample, and A-law or μ-law encoding, corresponding to a 64 kbit/s PCM stream.

That constraint does not make 8 kHz telephony audio inherently bad -- it just means high-frequency detail is unavailable. Voices must remain intelligible after band-limiting, companding, packet loss, and handset playback. Wideband VoIP or WebRTC paths can preserve more detail, but only when the complete route supports the wider format end to end.

Resample once, at a controlled boundary

For reliable telephony audio quality:

• Generate directly in the required format when the TTS API and call provider support it.

• Otherwise, use a production resampler with anti-alias filtering before encoding to mu-law or A-law.

• Avoid repeated conversions such as 24 kHz PCM to 16 kHz, then 8 kHz, then back to 16 kHz.

• Test cloned voices, numbers, names, and sibilants through the actual carrier path.

Studio-quality TTS benchmarks evaluate the clean model output. They do not capture transcoding artifacts, packet timing, acoustic echo, handset speakers, or PSTN routing. Evaluate telephony TTS by recording completed calls, not only API output files.

Streaming Generation, Buffering, and Network Delivery

Start with incremental text and audio

Non-streaming generation waits for a complete utterance, then synthesizes and transfers the file. That is simple and works well for fixed IVR prompts. Real-time text to speech should accept partial text and return playable audio frames as they become available, not after the full response is ready.

Keep the startup buffer small but observable

A smaller startup buffer can reduce playback delay but increases underrun risk, while a larger buffer can absorb more jitter at the cost of added latency. There is no universal buffer target: instrument buffer depth and underruns, then tune against real network conditions.

Treat geography as part of the architecture

Place telephony ingress, orchestration, and speech services in compatible regions. Network latency includes propagation, routing, congestion, and protocol overhead, as summarized by MDN's network latency explainer. Persistent connections remove repeated setup costs, but they do not eliminate distance.

Smallest.ai's Lightning TTS through the Waves API is one option for teams evaluating a TTS API for phone calls. Validate current streaming behavior, output formats, regional availability, and rate limits against official documentation during procurement rather than assuming a demo configuration matches production.

Turn-Taking, Endpointing, and Barge-In

Correct interruption requires coordinated detection, generation cancellation, and playback cancellation.

Voice activity detection answers, "Is someone speaking?" Endpointing answers, "Has this turn ended?" Turn detection adds lexical, semantic, and prosodic evidence to decide whether a pause signals completion or hesitation. That distinction matters because conversational fillers, account numbers, and self-corrections all contain natural pauses that a naive timeout will misread.

A fixed silence timeout is easy to implement but performs poorly across languages, speaking rates, and noisy calls. A robust turn-taking voice AI system combines VAD state, transcript stability, punctuation likelihood, semantic completeness, and configurable timeout limits. For an implementation-focused example, see how AI voice agents handle turn-taking and interruptions. The complexity of end-of-turn coordination and overlap is also documented in Gabriel Skantze's 2021 review of turn-taking in conversational systems and human-robot interaction.

Barge-in must cancel the whole outbound path

When reliable caller speech is detected during agent playback:

• Cancel the interrupted LLM generation, or stop consuming its remaining output.

• Cancel the active TTS request or discard all subsequent chunks.

• Clear application and carrier playback queues.

• Reset the conversation state so the model knows what the caller actually heard.

• Apply echo suppression and a short confirmation window to avoid self-interruption.

Stopping TTS generation is not playback cancellation. Audio already queued in your process, media gateway, or carrier can continue playing after cancellation unless every buffer is explicitly cleared.

Measuring Voice Agent Latency in Production

Instrument timestamps at every boundary you control: last detected caller speech, endpoint decision, transcript finalization, LLM request and first token, TTS request and first audio, first frame queued, and first frame sent to the carrier. Where possible, run a loopback or recorded-call test to estimate when audio actually became audible.

A practical production scorecard

Do not approve TTS for voice agents from a single average. Compare distributions under concurrency and include long numbers, abbreviations, emotional speech, noisy callers, and tool-call delays. A realistic latency budget is a negotiated system target -- not a claim that one headline number determines suitability.

Text to Speech for Phone Calls: Production Checklist

Before routing live traffic:

• Measure end-to-end caller-heard delay, not only model latency.

• Record time to first audio and chunk continuity at p50, p95, and p99.

• Confirm sample rate, frame size, channel count, and G.711 mu-law or A-law expectations.

• Resample once and test the final carrier-delivered signal.

• Tune VAD, endpointing, and natural pause handling by language and use case.

• Cancel LLM output, active TTS generation, and every playback buffer on barge-in.

• Test network regions, reconnect behavior, packet loss, and concurrency.

• Use cached audio for stable prompts and streaming synthesis for dynamic responses.

• Review transcripts and recordings for false endpoints, overlap, and repetitions.

• Load-test the complete phone call voice AI stack before deployment.

Production readiness comes from coordinated engineering across speech, networking, telephony, and dialogue control. Select a provider using complete-call measurements, then keep those measurements in the release process as models, prompts, carriers, and traffic patterns change.