VoiceThere agents on Voice or Voice+Data sessions can pan TTS and listener poses in stereo. On Voice+Data you also get mix groups — who hears whom inside one voice session — plus per-listener mute, mix status, and STT overrides. Panning uses equal-power stereo with inverse-square distance — Y-up coordinates, default forward is −Z. There is no HRTF.

APIs live in @voicethere/agent: createMixGroup, addClientToMix, removeClientFromMix, setGlobalMute, setListenerMute, getClientMixStatus, setClientPose, setPositionalMixing, setTtsPose, clearTtsPose, setTtsMixPlacement, setDefaultMixPlacement, setSttEnabled, isMixAvailable, and isTtsPoseAvailable.

What works on which runner mode

Mix groups (createMixGroup, addClientToMix, removeClientFromMix, setDefaultMixPlacement) — Voice+Data only. Check isMixAvailable(ctx) before calling.
TTS pose / listener pose / positional panning (setTtsPose, setPositionalMixing, setClientPose, setTtsMixPlacement) — Voice or Voice+Data. Check isTtsPoseAvailable(ctx) (or ctx.ttsPoseAvailable). Data-only sessions throw TTS_POSE_REQUIRES_VOICE.
Mute / mix status (setGlobalMute, setListenerMute, getClientMixStatus) — Voice+Data only. Check isMixAvailable(ctx) before calling. STT overrides (setSttEnabled) are available on voice sessions and are not mix-gated.

Configure runner mode in the dashboard or CLI — see Session & runner settings.

Play audio clips

To play decoded sound files (not TTS) to one or all listeners, use play, getPlay, and stopPlay. Clips share the same outbound mix as microphones and TTS. See Clip playback.

Mix groups (Voice+Data only)

A mix group is a subgraph of who hears whom inside one voice session. Use:

createMixGroup() — returns a new group id.
addClientToMix(groupId, clientId) — add a client; moving is exclusive (a client leaves its previous group).
removeClientFromMix(groupId, clientId) — remove from the group.

Reassign clients mid-call for team channels, breakout voice, or “hear only your squad” rules. Each connected browser tab has its own orchestrator session id — use that id as clientId / sessionId in mix calls (the same id you pass to speak(sessionId, text)).

import {
  createMixGroup,
  defineAgent,
  isMixAvailable,
} from "@voicethere/agent";

defineAgent({
  onClientJoin(ctx) {
    if (!isMixAvailable(ctx)) return;
    const { sessionId } = ctx;
    // Collect other connected session ids as they join, then include them here.
    void createMixGroup({ id: "all", clientIds: [sessionId] });
  },
});

Mute (Voice+Data only)

Mute APIs apply to mix groups only — check isMixAvailable(ctx) before calling.clientId, listenerId, and targetId are orchestrator session ids (the same ids you pass to speak(sessionId, text)).

setGlobalMute({ clientId, muted, sttEnabled? }) — mute a client in the mix for every listener. By default, global mute also disables STT for that client; pass optional sttEnabled: true to keep speech recognition on while they are muted in the mix.
setListenerMute({ listenerId, targetId, muted }) — one listener stops hearing one target; other listeners still hear that target.

import {
  defineAgent,
  isMixAvailable,
  setGlobalMute,
  setListenerMute,
} from "@voicethere/agent";

defineAgent({
  onClientJoin(ctx) {
    if (!isMixAvailable(ctx)) return;
    const { sessionId } = ctx;

    void setGlobalMute({ clientId: sessionId, muted: true });
    void setListenerMute({
      listenerId: sessionId,
      targetId: "other-session-id",
      muted: true,
    });
    // Optional: keep STT on while globally muted
    void setGlobalMute({ clientId: sessionId, muted: true, sttEnabled: true });
  },
});

Mix status (Voice+Data only)

getClientMixStatus(clientId?) returns { ok, statuses? }. Each ClientMixStatus includes:

clientId — orchestrator session id
globallyMuted — muted for all listeners
sttEnabled — whether STT is on for this client
pose — listener pose when positional mixing is on
ttsPose — per-client TTS speaker pose, if set
mutedBy — listener ids that have muted this client via setListenerMute
groupId — current mix group, if any

Omit clientId to list status for every connected client.

import { defineAgent, getClientMixStatus, isMixAvailable } from "@voicethere/agent";

defineAgent({
  async onClientJoin(ctx) {
    if (!isMixAvailable(ctx)) return;

    const result = await getClientMixStatus();
    if (result.ok && result.statuses) {
      for (const status of result.statuses) {
        console.log(status.clientId, status.globallyMuted, status.groupId);
      }
    }
  },
});

Client poses (Voice or Voice+Data)

setClientPose(clientId, pose) sets a client’s listener position and orientation for positional panning. Toggle live pose math with setPositionalMixing(true | false). On Voice+Data, when positional mixing is off, client mic named placements apply via setDefaultMixPlacement(placement) — e.g. center, left, etc.

Named placements: center, left, right, front, behind, below, above.

import {
  defineAgent,
  isTtsPoseAvailable,
  setClientPose,
  setPositionalMixing,
} from "@voicethere/agent";

defineAgent({
  onSessionStart(ctx) {
    if (!isTtsPoseAvailable(ctx)) return;
    const { sessionId } = ctx;

    void setPositionalMixing(true);
    void setClientPose(sessionId, {
      position: { x: 3, y: 0, z: 0 },
      orientation: { x: 0, y: 0, z: 0, w: 1 },
    });
  },
});

Per-client TTS pose (Voice or Voice+Data)

TTS is mixed per client, matching speak(sessionId, text):

setTtsPose(sessionId, pose) — place that client’s TTS speaker in world space when positional mixing is on. Pan is computed against that client’s listener pose.
clearTtsPose(sessionId) — drop the live pose; named setTtsMixPlacement applies again.
setTtsMixPlacement(placement) — global named placement when no per-client TTS pose is set (or positional mixing is off).

Two clients in the same session can hear the same TTS text from different directions — each orbit is independent. With positional mixing off or no TTS pose, named placement is used for everyone.

Orbit around a listener

To spin a TTS speaker around a listener at the origin, update setTtsPose on a short interval and clear on session end:

import {
  clearTtsPose,
  defineAgent,
  isTtsPoseAvailable,
  setPositionalMixing,
  setTtsPose,
  speak,
} from "@voicethere/agent";

function orbitTtsPose(elapsedSec: number, radius = 2) {
  const x = Math.cos(elapsedSec) * radius;
  const z = Math.sin(elapsedSec) * radius;
  return {
    position: { x, y: 0, z },
    orientation: { x: 0, y: 0, z: 0, w: 1 },
  };
}

const orbitTimers = new Map<string, ReturnType<typeof setInterval>>();
const sessionStartTimes = new Map<string, number>();

defineAgent({
  onSessionStart(ctx) {
    const { sessionId } = ctx;
    if (!isTtsPoseAvailable(ctx)) return;

    void setPositionalMixing(true);
    const startMs = Date.now();
    sessionStartTimes.set(sessionId, startMs);

    const timer = setInterval(() => {
      const started = sessionStartTimes.get(sessionId);
      if (!started) return;
      const elapsedSec = (Date.now() - started) / 1000;
      void setTtsPose(sessionId, orbitTtsPose(elapsedSec));
    }, 50);

    orbitTimers.set(sessionId, timer);
    speak(sessionId, "I'll circle around you.");
  },

  onUserSpeechFinal({ sessionId, text }) {
    speak(sessionId, "You said: " + text);
  },

  onSessionEnd({ sessionId }) {
    const timer = orbitTimers.get(sessionId);
    if (timer) clearInterval(timer);
    orbitTimers.delete(sessionId);
    sessionStartTimes.delete(sessionId);
    void clearTtsPose(sessionId);
  },
});

To orbit a player who is not at the origin, add that client’s listener position from setClientPose to the circle offset before calling setTtsPose.

Conversation recording

Session recordings are a stereo WAV per listener, but they do not preserve live positional panning from the mix graph. Each leg is downmixed to mono first, then written as left = outbound (what that listener hears from the agent mix, including TTS) and right = inbound(that listener's own mic). Do not assume mic on the left and agent on the right — positional stereo from setClientPose is lost in the file.

Live showcases

Hear positional mix in the browser — use stereo headphones:

Orbiting voice — per-listener setTtsPose
Proximity room — mix groups, setClientPose, setListenerMute

Full protocol and template setup: Spatial audio showcases.

Template

The dashboard Template dropdown includes Orbiting TTS (positional-tts) — a seeded starter that implements the loop above on voice-only or Voice+Data projects. For all three live spatial demos in one bundle, use Spatial audio showcase (spatial-showcase). Source lives in the public agent repo:

templates/positional-tts/

Build locally:

npx @voicethere/agent build --entry templates/positional-tts/agent.ts

See also Agent templates for the full catalog.

Speech-to-text (STT) overrides

Project default STT comes from dashboard or CLI session.stt_enabled (maps to RUNNER_STT_ENABLED, default on). Turning STT off at the project level does not block per-session or per-client overrides.

Runtime from your agent: setSttEnabled({ enabled, clientId?, sessionId? }). When clientId is omitted, the change applies to every connected client. Use either clientId or sessionId to scope to one client.
One session at join: pass customerContext: { stt_enabled: true } on the browser client session hello.
Resolution order: client override → session override → project default (session.stt_enabled / RUNNER_STT_ENABLED).
Global mute: muted clients drop STT unless you pass sttEnabled: true to setGlobalMute or set an explicit setSttEnabled override.

import { defineAgent, setSttEnabled } from "@voicethere/agent";

defineAgent({
  onSessionStart(ctx) {
    const { sessionId } = ctx;
    void setSttEnabled({ enabled: true, clientId: sessionId });
  },
});

More runner settings: Session & runner settings.

Positional mix & orbiting TTS