Building a real-time voice agent has moved well past the "fun prototype" phase. In 2025 and 2026, these systems are handling real production traffic in customer support, healthcare intake, e-commerce, and logistics. The tooling has caught up, the APIs are quick, and "good enough" now means more than simply producing a reply: it needs to respond quickly enough to maintain a natural conversational flow, with many production systems targeting sub-second end-to-end response times and cope with interruptions without melting down. This tutorial is for developers aiming to ship a working real-time voice agent, not a demo that only behaves in ideal conditions.

Five steps gets you to a full, functioning loop: microphone audio streams into speech-to-text, finalized transcripts go to a language model, and the response comes back as synthesized speech with minimal delay. Here is the exact build sequence:

• Set up your project environment

• Capture and stream audio input with Pulse (STT)

• Connect the transcript to a language model

• Convert the LLM response to speech with Lightning (TTS)

• Close the loop with interrupt handling and turn-taking

What You Need Before You Start

Before you touch code, get the basics squared away. Tooling: Python 3.9+, a code editor, and a test setup with a working microphone. You will also need API access to Smallest.ai's Waves platform for Lightning (TTS), Pulse (STT), and Electron, plus any compatible LLM endpoint if you prefer using an external language model. Assumed knowledge: async Python, making REST calls, and a general comfort level with audio streams. You do not need DSP chops, but it helps to internalize one practical detail: audio shows up as a continuous stream of bytes, not neat little files. Treat it like a stream early and you will avoid a lot of frustrating "why is this laggy?" debugging later.

How a Real-Time Voice Agent Actually Works

The loop is always the same: speech-to-text (STT) turns the user's voice into text, a language model generates a response from that text, and text-to-speech (TTS) turns the response back into audio. The hard part is that latency stacks up fast. If STT needs 200ms, the LLM takes 400ms to produce a first token, and TTS takes 300ms to produce first audio, you are already flirting with a one-second time-to-first-audio (TTFA) before you even count network round trips.

The practical fix is to make everything streaming-first. Stream audio into STT instead of waiting for a finished recording. Stream tokens out of the LLM as it generates them instead of waiting for the full completion. Then feed those tokens into TTS in sentence-sized chunks, rather than holding the entire answer until the end.That is the architecture described in streaming-first voice agent design. Smallest.ai’s components map cleanly onto it: Pulse handles streaming STT, Electron covers the LLM layer, and Lightning (via the Waves API) handles low-latency TTS.

Step 1: Set Up Your Project Environment

Start with a clean virtual environment and install the dependencies needed for audio input, WebSockets, HTTP streaming, and environment configuration. From your terminal, run:

python -m venv venv

source venv/bin/activate

# Windows: venv\Scripts\activate

pip install pyaudio websockets httpx python-dotenv smallest-sdk

Next, create a .env file in your project root and add your Waves API key:

WAVES_API_KEY=your_key_here

Do not hardcode API keys directly inside your source files. Load them at runtime using python-dotenv. This keeps credentials out of version control and makes key rotation much easier later.

Step 2: Capture and Stream Audio Input with Pulse

20ms audio frames over WebSocket: the right chunking size keeps buffering delay minimal

Use PyAudio to open a microphone stream, then slice the incoming audio into 20ms frames. At 16kHz, 16-bit mono, that works out to 640 bytes per frame. The size is intentional: go smaller and you pay more WebSocket overhead; go larger and you start adding buffering delay before STT can do anything useful. Send each frame to Smallest.ai’s Pulse STT endpoint over a single persistent WebSocket so transcription can keep up with live speech.

import pyaudio

import asyncio

import websockets

import json

import os

from dotenv import load_dotenv

load_dotenv()

API_KEY = os.getenv('WAVES_API_KEY')

CHUNK = 640 # 20ms at 16kHz

FORMAT = pyaudio.paInt16

CHANNELS = 1

RATE = 16000

async def stream_audio_to_pulse(on_final_transcript):

    p = pyaudio.PyAudio()

    stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK)

    uri = 'wss://waves.smallest.ai/v1/stt/stream'

    async with websockets.connect(

        uri, extra_headers={'Authorization': f'Bearer {API_KEY}'}

    ) as ws:

        async def send_audio():

            while True:

                data = stream.read(CHUNK, exception_on_overflow=False)

                await ws.send(data)

        async def receive_transcripts():

            async for message in ws:

                result = json.loads(message)

                if result.get('is_final'):

                    await on_final_transcript(result['transcript'])

        await asyncio.gather(send_audio(), receive_transcripts())

Watch the `is_final` flag closely. Pulse will stream partial transcripts while the user is talking, then mark a final transcript once it detects end-of-utterance. Trigger the LLM only on the final transcript. If you fire on partials, you will spam the LLM with overlapping requests and end up with an agent that talks over itself and loses the thread.

Step 3: Connect the Transcript to a Language Model

Once you have a finalized transcript, send it to Electron (or any compatible LLM endpoint) along with a system prompt that pins down the agent’s role and scope. The biggest latency win here is simple: stream the response token-by-token instead of waiting for a full completion. With streaming turned on, you can start pushing words toward TTS almost immediately after the first token lands.

import httpx

SYSTEM_PROMPT = "You are a concise support agent. Answer in 1-2 sentences."

async def stream_llm_response(transcript, on_token):

    async with httpx.AsyncClient() as client:

        async with client.stream(

            "POST",

            "https://waves.smallest.ai/v1/chat/completions",

            headers={"Authorization": f"Bearer {API_KEY}"},

            json={

                "model": "electron",

                "messages": [

                    {"role": "system", "content": SYSTEM_PROMPT},

                    {"role": "user", "content": transcript}

                ],

                "stream": True

            }

        ) as response:

            async for line in response.aiter_lines():

                if line.startswith("data: "):

                    chunk = json.loads(line[6:])

                    token = chunk["choices"][0]["delta"].get("content", "")

                    if token:

                        await on_token(token)

Keep the system prompt lean. Every extra token is paid for on every turn. Moving from a 500-token prompt to a 50-token prompt can easily shave 100ms+ off first-token latency at scale.

Step 4: Convert the LLM Response to Speech with Lightning

Sentence-complete chunks balance naturalness and latency better than word-by-word or full-response approaches

Send the LLM’s streamed tokens to Smallest.ai’s Lightning TTS (via the Waves API) in sentence-complete chunks, not one word at a time. Word-by-word synthesis tends to sound brittle because the model does not have enough context to shape prosody. Waiting for the full response goes to the other extreme and makes the agent feel sluggish even if the final audio is high quality. Punctuation gives you a clean, low-effort chunking signal: periods, question marks, and exclamation points.

async def tokens_to_speech(token_queue, audio_output_stream):

    buffer = ''

    async for token in token_queue:

        buffer += token

        if buffer.endswith(('.', '?', '!')):

            await synthesize_and_play(buffer.strip(), audio_output_stream)

            buffer = ''

    if buffer.strip(): # flush any remaining tokens

        await synthesize_and_play(buffer.strip(), audio_output_stream)

async def synthesize_and_play(text, audio_output_stream):

    async with httpx.AsyncClient() as client:

        async with client.stream(

            'POST',

            'https://waves.smallest.ai/v1/tts/stream',

            headers={'Authorization': f'Bearer {API_KEY}'},

            json={'text': text, 'voice': 'default', 'format': 'pcm_16000'}

        ) as response:

            async for chunk in response.aiter_bytes(4096):

                audio_output_stream.write(chunk)

Step 5: Close the Loop with Interrupt Handling and Turn-Taking

Barge-in (the ability for a user to interrupt mid-sentence) is the difference between a production voice agent and a polished demo. Without it, the agent talks over people, misses corrections, and comes off robotic even when your latency numbers look great. The simplest pattern is to keep monitoring the microphone while the agent is speaking. When you detect new speech, stop TTS playback and route the fresh audio straight back into Pulse.

For many products, a lightweight energy-threshold VAD is enough. webrtcvad is a common choice because it provides frame-level speech detection without heavy CPU cost. Set agent_speaking = True when TTS playback begins and flip it back to False when playback ends. If VAD fires while agent_speaking is True, raise an interrupt_event that your TTS coroutine checks before writing each audio chunk. For more on production tuning (false triggers versus responsiveness) see the voice activity detection production guide. Smallest.ai’s components map cleanly onto it: Pulse handles streaming STT, Electron covers the LLM layer, and Lightning (via the Waves API) handles low-latency TTS.

Latency Benchmarks: What to Expect at Each Stage

For a production-grade agent, an end-to-end TTFA under 600ms is a solid target. Conversational turn-taking research suggests responses feel natural when they land in roughly the 200–500ms range (Levinson & Torreira, Speech Communication, 2015). You only get close to that budget by streaming every handoff. The non-streaming baseline in the table makes the penalty obvious: batch STT, full-completion LLM, and batch TTS do not just add delay, they stack it until the experience feels unavoidably slow.

Common Mistakes and How to Avoid Them

Each of these mistakes compounds latency or breaks the conversation loop

The five mistakes that consistently break real-time voice agents in production:

• Waiting for the full LLM response before calling TTS. The most common latency trap. Stream tokens and chunk on sentence boundaries instead.

• Sending audio chunks that are too large to Pulse. Once you push past ~100ms per chunk, you are baking buffering delay into the pipeline. Stick to 20ms frames.

• No interrupt handling. If users cannot cut in, the agent will feel broken even when it is fast. Build VAD-based barge-in early.

• Hardcoding API keys or not handling WebSocket reconnects. Use `.env` files and add exponential backoff reconnect logic. WebSockets drop; your agent should recover quietly.

• Overly long system prompts. Prompt tokens are processed on every turn. A 500-token prompt versus a 50-token prompt can add 100ms+ to first-token latency at scale.

The Problem This Solves and Where Smallest.ai Fits

Fragmented stacks compound latency; a purpose-built unified pipeline eliminates the gaps

The hard part of a real-time voice agent is rarely STT, the LLM, or TTS in isolation. It is the glue. When you stitch together multiple latency-sensitive services, you end up chasing down mismatched audio formats, juggling separate auth patterns, eating the routing overhead between vendors, and then discovering (late) that one provider’s “streaming” does not play nicely with another’s. Most multi-provider stacks were never designed to behave like one coherent real-time system.

Smallest.ai’s Waves API, Pulse, Lightning, and Electron are built to run as that single pipeline: one auth layer, consistent streaming behavior, and less inter-service latency when you keep the loop inside the same stack. If you would rather skip the plumbing, Atoms is Smallest.ai’s voice and text agent platform that adds a managed agent layer on top of the same infrastructure you just wired up manually. Pick the control of a custom build or the speed of a managed deployment, either way, the underlying loop stays the same. Start with the Waves API and the developer documentation to get your first agent running.

What to Build Next

At this point you have the full real-time loop working: microphone audio streams to Pulse, final transcripts go to Electron, tokens stream back and get chunked into sentences for Lightning, and VAD-based barge-in keeps the conversation from turning into a one-way lecture. From here, the next iterations are the ones that turn a functional agent into a shippable one. Swap the default voice for a branded voice with Smallest.ai's voice cloning API. Move the same pipeline onto a phone number via WebRTC for inbound or outbound calling. If you have a speech-to-speech use case that does not need text in the middle, Hydra offers a lower-latency path for specific applications. And if you are building toward a particular workflow, the multi-agent voice dashboard tutorial and AI voice agent for e-commerce guide show how teams extend this foundation into production deployments. If you're ready to move from a working prototype to a production voice agent, book a demo to see how Smallest.ai fits your architecture and deployment requirements.