Voice agents shouldn't be locked into whichever STT and TTS provider a framework ships with by default.

TEN Framework is built around that idea, letting the orchestration layer handle transport, state, and interruptions while STT, LLM, and TTS stay swappable pieces of the graph. That's why we're excited to announce that Smallest AI is now available as an STT and TTS provider on TEN Framework, so if you're building on TEN, our speech models are now a config change away.

TEN agents are declarative graphs. A voice agent is just a pipeline of nodes, RTC transport, STT, LLM, and TTS, defined in a property.json file. Switching to Smallest AI isn't a rewrite. It's a two-line config change.

What is TEN Framework?

TEN (short for Transformative Extensions Network) is an open-source framework for building real-time, multimodal conversational voice agents. Instead of hardcoding a pipeline, TEN represents an agent as a graph of extensions, RTC transport, speech-to-text, an LLM, text-to-speech, that can be swapped independently.

That extension-based architecture is the whole point. Need to change your STT provider? Edit one addon field. Want to test a new voice mid-project? Same story. TEN even ships a visual editor, TMAN Designer, so you can rewire the graph without touching JSON at all.

With this integration, developers building on TEN can now drop in:

• Pulse for real-time, streaming speech-to-text

• Lightning v3.1 and Lightning v3.1 Pro for low-latency, streaming text-to-speech

Why This Integration Matters

A typical TEN voice pipeline looks like this:

Microphone → RTC Transport → STT → LLM → TTS → Speakers

Every hop adds latency, but the speech layer is usually where conversations start feeling slow or robotic. That's the part we've spent the most time optimizing.

• Pulse (STT) connects over a real-time WebSocket endpoint with a 64ms time-to-first-token and support for 38 languages.

• Lightning (TTS) streams audio back with ~150ms to the first audio chunk, so the agent starts speaking almost as soon as the LLM starts generating.

Because both are exposed as TEN extensions (smallest_asr_python and smallest_tts_python), you're not stitching together a custom integration. You're pointing two addon fields at Smallest AI and letting TEN's graph runtime handle the rest, including barge-in: when a user interrupts the agent mid-sentence, TEN flushes the TTS extension and cancels the in-flight Lightning stream automatically. No custom interruption logic required.

Getting Started

Setting up Smallest AI inside TEN Framework takes a few minutes end to end.

Step 1: Clone TEN Framework

git clone https://github.com/TEN-framework/ten-framework.git
cd

git clone https://github.com/TEN-framework/ten-framework.git
cd

git clone https://github.com/TEN-framework/ten-framework.git
cd

Step 2: Configure Environment Variables

Copy .env.example to .env and add your keys, including an Agora App ID for RTC transport, your LLM provider key, and your Smallest AI API key from the Smallest AI dashboard:

AGORA_APP_ID=...
OPENAI_API_KEY=...
SMALLEST_API_KEY

AGORA_APP_ID=...
OPENAI_API_KEY=...
SMALLEST_API_KEY

AGORA_APP_ID=...
OPENAI_API_KEY=...
SMALLEST_API_KEY

Step 3: Point the Agent Graph at Smallest AI

Open agents/examples/voice-assistant/tenapp/property.json and update the addon field on the stt and tts nodes:

{
  "type": "extension",
  "name": "stt",
  "addon": "smallest_asr_python",
  "property": {
    "params": {
      "language": "en",
      "sample_rate": 16000
    }
  }
},

{
  "type": "extension",
  "name": "tts",
  "addon": "smallest_tts_python",
  "property": {
    "params": {
      "model": "lightning_v3.1",
      "voice_id": "magnus",
      "sample_rate": 24000
    }
  }
}

{
  "type": "extension",
  "name": "stt",
  "addon": "smallest_asr_python",
  "property": {
    "params": {
      "language": "en",
      "sample_rate": 16000
    }
  }
},

{
  "type": "extension",
  "name": "tts",
  "addon": "smallest_tts_python",
  "property": {
    "params": {
      "model": "lightning_v3.1",
      "voice_id": "magnus",
      "sample_rate": 24000
    }
  }
}

{
  "type": "extension",
  "name": "stt",
  "addon": "smallest_asr_python",
  "property": {
    "params": {
      "language": "en",
      "sample_rate": 16000
    }
  }
},

{
  "type": "extension",
  "name": "tts",
  "addon": "smallest_tts_python",
  "property": {
    "params": {
      "model": "lightning_v3.1",
      "voice_id": "magnus",
      "sample_rate": 24000
    }
  }
}

Your API key doesn't need to live in the graph itself. Both extensions read SMALLEST_API_KEY from the environment by default.

Step 4: Build and Run

docker compose up -d
docker exec ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task install"
docker exec -d ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task run"

docker compose up -d
docker exec ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task install"
docker exec -d ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task run"

docker compose up -d
docker exec ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task install"
docker exec -d ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task run"

Step 5: Talk to Your Agent

Open the playground at http://localhost:3000, join a channel, and start talking. Audio now flows through Smallest AI at both ends of the pipeline: microphone → Agora RTC → Pulse (STT) → your LLM → Lightning (TTS) → speakers.

Prefer a visual workflow over hand-editing JSON? TMAN Designer starts alongside task run and lets you swap providers by dragging and dropping nodes instead.

Configuration Reference

Both extensions ship with sensible defaults, but everything is overridable at the graph level, so different graphs in the same app can run different voices or languages.

smallest_asr_python (STT) defaults to the pulse model, English, and a 16kHz input sample rate. Any extra key you add under params, things like word_timestamps, eou_timeout, or punctuate, gets forwarded straight through as a query parameter to the live endpoint.

smallest_tts_python (TTS) defaults to lightning_v3.1 with the magnus voice at a 24kHz output sample rate, and supports speed control from 0.5x to 2x. Switch the model to lightning_v3.1_pro to unlock premium English and Hindi voices like meher. Extra params keys such as language or pronunciation_dicts are forwarded verbatim in the request body.

Interim STT transcripts stream in with final=false for live captions, and final results carry word-level timing, useful if you're building captions or analytics on top of the transcript stream.

What Can You Build?

TEN Framework plus Smallest AI is a fit for anything that needs a responsive, real-time voice loop:

• Multimodal conversational assistants

• Customer support and IVR replacements

• Real-time voice agents with live captioning

• Multilingual voice interfaces (Pulse covers 38 languages)

• Voice-first prototypes that need to swap providers without a rewrite

Building the Future of Composable Voice AI

TEN Framework was built extension-first so developers can pick the STT, LLM, and TTS that fit their use case, and Smallest AI's Pulse and Lightning models are now a config change away inside that graph.

Get started today, point your graph at Smallest AI, and start talking to your agent.

Link to Docs: https://docs.smallest.ai/models/integrations/agent-framework/ten-framework