Speech control

Overview

Speech control tells your agent what happened to each thing it said. You get a result for every speak() call, you can cancel speech that is no longer wanted, you can mark speech the caller must hear in full, and when the caller talks over the agent you learn which words they actually heard. It works for every agent, with or without an LLM.

You need @voicethere/agent0.11.0 or newer in your bundle. To show the agent's state in a browser, use @voicethere/client 0.13.0 or newer. Check ctx.speakControlAvailable (or isSpeakControlAvailable(ctx)) on session start to see whether the runner supports it.

A turnId groups several speak() calls into one reply, so a single interruption report covers the whole turn.

Speak results

speak(sessionId, text, options) returns { requestId, done, cancel() }. Options are interruptible, turnId and inReplyTo. The done promise never rejects. It resolves with { requestId, turnId?, status, playedMs, synthesizedMs?, errorCode? }.

import { defineAgent, speak } from "@voicethere/agent";

defineAgent({
  async onUserSpeechFinal({ sessionId }) {
    const handle = speak(sessionId, "Your table is booked for seven.", {
      turnId: "booking-1",
    });
    const result = await handle.done;
    if (result.status === "interrupted") {
      console.log("caller cut in after", result.playedMs, "ms");
    }
  },
});

Barge-in and queued speech

When the caller talks over the agent, the reply that is playing stops and every queued interruptible speech of that session is dropped with status cancelled. Speech queued with interruptible: false, and anything sent with announce(), always plays. Barge-in is on by default; the project settings vad.bargeIn.enabled and vad.bargeIn.flushTts control it. See Advanced voice settings.

Announcements

announce(sessionId, text, { turnId?, inReplyTo? }) is a speak()that the caller can't interrupt and that is never dropped from the queue. Use it for things the caller has to hear, such as a legal notice or an order summary.

import { announce, defineAgent } from "@voicethere/agent";

defineAgent({
  async onSessionStart({ sessionId }) {
    const handle = announce(
      sessionId,
      "This call may be recorded for quality purposes.",
    );
    await handle.done;
  },
});

Turning barge-in off for a block

withoutBargeIn(sessionId, fn) turns barge-in off while fn runs and turns it back on afterwards, even when fn throws. For manual control, setBargeIn(sessionId, enabled) resolves with { ok, reason? }.

import { defineAgent, speak, withoutBargeIn } from "@voicethere/agent";

defineAgent({
  async onUserSpeechFinal({ sessionId }) {
    await withoutBargeIn(sessionId, async () => {
      const handle = speak(sessionId, "Please listen to the four options.");
      await handle.done;
    });
  },
});

Cancelling speech

cancelSpeech(sessionId, { requestId?, turnId? }) cancels one request or a whole turn. With no target it cancels everything pending for the session. Queued speech ends as cancelled, the speech that is playing stops and ends as interrupted.

import { cancelSpeech, defineAgent, speak } from "@voicethere/agent";

const pending = new Map<string, string>();

defineAgent({
  async onUserSpeechFinal({ sessionId, text }) {
    const older = pending.get(sessionId);
    if (older) await cancelSpeech(sessionId, { turnId: older });

    const turnId = "turn-" + Date.now();
    pending.set(sessionId, turnId);
    const handle = speak(sessionId, "You said: " + text, { turnId });
    await handle.done;
  },
});

Interruption report

When a barge-in cuts the agent, the onAgentInterruptedhandler runs once, before the caller's next final transcript. playedText is what the caller heard: the sentences that finished, plus an estimate for the sentence that was cut, based on how much of its audio played and trimmed back to a whole word. playedTextExact is false when that last part is an estimate.

Put playedText into your LLM history instead of the full reply, and tell the model what the caller did not hear.

import { defineAgent } from "@voicethere/agent";

const history = new Map<string, { role: string; content: string }[]>();

defineAgent({
  async onAgentInterrupted({ sessionId, playedText, unplayedText }) {
    const turns = history.get(sessionId) ?? [];
    const last = turns[turns.length - 1];
    if (last?.role === "assistant") {
      last.content = playedText + " [interrupted; the caller did not hear: " + unplayedText + "]";
    }
  },
});

Agent state in the client

setAgentState(sessionId, state, { detail? }) sets the state to listening, thinking, tool_call or speaking and forwards it to the client. detail is at most 64 characters and must not contain customer content. thinking and tool_callcount as activity for the idle timeout, so a long tool call doesn't end the session.

In the browser, pass onAgentState and onAgentInterrupted to connectBrowserSession from @voicethere/client/browser. The interruption callback receives { turnId?, replyPlayedMs, playedTextExact }. No text reaches the browser.

// agent
import { defineAgent, setAgentState } from "@voicethere/agent";

defineAgent({
  async onUserSpeechFinal({ sessionId }) {
    await setAgentState(sessionId, "thinking", { detail: "looking up order" });
    // ... call your model or tool, then speak the answer
  },
});

// browser
import { connectBrowserSession } from "@voicethere/client/browser";

const thinking = document.getElementById("thinking")!;

// credentials come from your server's call to the VoiceThere sessions API
const session = await connectBrowserSession({
  credentials,
  onAgentState: ({ state, detail }) => {
    thinking.hidden = state !== "thinking" && state !== "tool_call";
  },
  onAgentInterrupted: ({ replyPlayedMs, playedTextExact }) => {
    console.log("caller cut in after", replyPlayedMs, "ms", playedTextExact ? "" : "(estimated)");
  },
});

Older runners and agents

On a runner that predates speech control, done resolves with unsupported. The speech still plays, but there is no cancel, no queue rule and no interruption report. The agent logs one warning per session when this happens. Redeploy your project to pick up a current runner.

@voicethere/agent/host exports the parent-side rules of speech control. The VoiceThere runner and local tooling use it; most agents never import it.