VoiceThere

Spoken language identification

VoiceThere can identify the language the caller is speaking and deliver a user_language speech event to your agent. Use it to switch prompts, pick a TTS voice, or keep session state in the caller’s language. Detection uses on-cluster Sherpa Whisper tiny (ISO 639-1 codes such as en, de, es).

How it works

  • At the start of each new user utterance, the voice pipeline buffers inbound speech and runs spoken-language identification once after enough audio. On VoiceThere Cloud the default window is 2500 ms (2.5 seconds) before the first identify; the SDK and self-hosted runners default to 1000 ms when unset. Identification then idles until the next turn.
  • The confident ISO code is emitted as user_language. The event is re-emitted only if the detected language changes later in the session.
  • Both event.language and event.text carry the ISO code.
  • Language events are not stored as conversation-history turns. Finals still persist when conversation history is enabled.

Agent handler

Handle every speech event with onSpeechEvent, or use the convenience hook onUserLanguage in @voicethere/agent:

import { defineAgent, agentLog } from "@voicethere/agent";

export default defineAgent({
  onUserLanguage({ sessionId, language }) {
    agentLog("info", `detected language ${language} for ${sessionId}`);
    // Route system prompts or TTS voice from the ISO 639-1 code.
  },
});

The voice-starter template logs user_language. See Agent templates.

Continuous identification (optional)

By default, identification runs once per user utterance. For self-hosted runners or SDK apps you can opt in to continuous checks with languageId.continuous: true in VoiceAgentConfig, or SHERPA_LID_CONTINUOUS=1 on a local runner.

Warning: continuous mode runs Whisper more often and uses extra CPU. It can starve Piper TTS, stretching the gap between spoken sentences so listeners may miss short first TTS bursts (for example a brief echo. prefix). VoiceThere Cloud Voice and Voice+Data use the once-per-utterance default — do not enable continuous on cloud deploys.

Cloud runners

Voice and Voice+Data cloud deploys enable spoken-language identification automatically using on-cluster Sherpa Whisper tiny with the once-per-utterance default.

Tune identification from the dashboard or CLI under Advanced voice pipeline:

  • languageId.enabled — turn off to skip LID while keeping the model on the runner (SHERPA_LID_ENABLED=0).
  • languageId.minSpeechMs — integer 1000–5000, cloud default 2500. Lower values emit user_language sooner but Whisper tiny may mis-guess on short or ambiguous audio. Higher values wait for more inbound PCM for a more confident code; if the utterance is shorter than the window, identify still runs once at hang-up on the full buffer.

See also Advanced voice settings.

Whisper tiny is not an STT picker option. Choose a Zipformer language in STT & TTS vendors as usual; language ID runs alongside that model.

Local development

From a node-webrtc-rust checkout:

npm run download-lid --workspace=@node-webrtc-rust/example-voice-agent-local-sherpa
export SHERPA_LID_MODEL_PATH=.models/sherpa-onnx-whisper-tiny
npm run start:roundtrip-language-id --workspace=@node-webrtc-rust/example-voice-agent-local-sherpa

Local runner: set SHERPA_LID_MODEL_PATH in .env.local (see the runner .env.example).

← All documentation