Voice agents on Voice or Voice+Data sessions can play short audio clips to connected browsers — sound effects, hold music, notifications — without text-to-speech. Clips mix with live microphones and TTS in the outbound audio path. They do not emit agent_speaking_* speech events.

Where to try it

Live demo: Spatial soundboard on the marketing site — pads send clipId and volume (0–1); the agent resolves HTTPS URLs. See Spatial audio showcases.
Dashboard: Open your project → Code tab and call play() from your agent handler (any voice template works — there is no separate clip-playback setting). See Dashboard code editor.
CLI: Build, upload, and deploy as usual — voicethere build validate, voicethere build upload, voicethere deploy --wait. No new project settings key is required.
Runner mode: Clip playback requires Voice or Voice+Data mode. Data-only rejects play(). Configure mode in the dashboard or CLI — see Session & runner settings.

Agent API

APIs live in @voicethere/agent: play, getPlay, and stopPlay. Each call returns a result object with ok and requestId — check ok before using playId; failures include a reason string instead of throwing only.

play starts a clip and returns playId when successful. getPlay(playId) reports lifecycle status. stopPlay(playId) ends a clip early.

buffering — not enough decoded audio yet (common for progressive downloads)
playing — audible output is active
stopped — ended early via stopPlay
ended — clip finished naturally
error — fetch, decode, or routing failure

import { defineAgent, getPlay, play, stopPlay } from "@voicethere/agent";

defineAgent({
  async onUserSpeechFinal({ sessionId, text }) {
    if (!text.toLowerCase().includes("doorbell")) return;

    const result = await play({
      url: "https://cdn.example.com/sfx/doorbell.mp3",
      sessionIds: [sessionId],
      volume: 0.9,
    });

    if (!result.ok || !result.playId) {
      console.warn("play failed:", result.reason);
      return;
    }

    const status = await getPlay(result.playId);
    if (status.ok && status.status === "playing") {
      // clip is audible
    }

    // Optional early stop:
    // await stopPlay(result.playId);
  },
});

Target one client or all

Pass sessionIds with the same session ids you use for speak(sessionId, text). Omit sessionIds or pass an empty array to play to all connected voice clients on the session.

If any requested id is unknown or disconnected, play fails with ok: false and no partial play — no client hears the clip until every target is valid.

Play to all listeners

Omit sessionIds (or pass an empty array) to play the clip to all connected voice clients on the session. Keep the same ok / playId handling as a targeted play.

import { play, getPlay } from "@voicethere/agent";

const result = await play({
  url: "https://cdn.example.com/sfx/chime.mp3",
  volume: 0.8,
});

if (!result.ok || !result.playId) {
  console.warn("play failed:", result.reason);
  return;
}

const status = await getPlay(result.playId);
if (status.ok && status.status === "playing") {
  // every connected voice client hears the chime
}

Placement and pose

Optional placement or pose on play() pan the clip in the outbound mix — same coordinate system as positional TTS (Y-up, look −Z). Pass either a named placement or a world pose, never both (matches AudioPosition / PlayOptions in @voicethere/agent). Clips are still not TTS — see Positional mix & orbiting TTS for mix groups and orbiting speech.

Named placement — valid values:

center, left, right, front, behind, below, above

await play({
  url: "https://cdn.example.com/sfx/chime.mp3",
  sessionIds: [sessionId],
  placement: "left",
});

World pose — position and orientation quaternion:

await play({
  url: "https://cdn.example.com/sfx/chime.mp3",
  sessionIds: [sessionId],
  pose: {
    position: { x: 2, y: 0, z: -1 },
    orientation: { x: 0, y: 0, z: 0, w: 1 },
  },
});

HTTPS clips and small bytes

The primary source is a public HTTPS URL. Remote clips must use HTTPS — file:// and other schemes are not supported. The platform fetches and caches completed downloads; a second play of the same URL can reuse the cache.

For clips under 64 KiB decoded, you may pass optional bytes as base64 alongside the URL. In this release, play() still requires url even when bytes is set — use the real clip URL. Oversized inline payloads are rejected before playback starts.

Progressive formats (MP3, AAC fast-start, and similar) can start playback before the full file is downloaded when enough audio is decoded. Slow downloads stay in buffering until preroll is ready.

Supported encodings

WAV (PCM)
MP3
OGG Vorbis
OGG/Opus
FLAC
AAC-LC in ADTS .aac or MP4/M4A containers

Audio is resampled to the voice mix sample rate before routing. Multiple concurrent plays are allowed and sum in the outbound mix. Live radio, HLS, and other unbounded streams are not supported.

AAC progressive playback

AAC behavior depends on how the file exposes its index:

ADTS .aac and fast-start MP4/M4A (movie atom moov near the start) — short preroll (~200–400 ms of decoded audio), then playing while the rest of the file downloads.
Normal M4A/MP4 with moov at the end — stays in buffering until the index is available (often until the download completes). Playback does not fail; it may look idle until the file is whole.

Volume (0–1)

play({ volume }) accepts volume from 0 (silent) to 1 (full level in the mix). Values outside that range are clamped. Per-clip volume is independent of the browser master slider on showcase pages — visitors can lower overall agent audio locally while pads still send their own volume to the agent.

Where to get clips

Host short WAV, MP3, or AAC files on public HTTPS URLs your runner can fetch. Royalty-free packs (Kenney UI Audio, Interface, Impact) and CC0 sounds on Freesound are common sources — document license and author in your repo. The spatial soundboard showcase uses procedural and catalogued clips under /showcase/sounds/; your agent should resolve URLs server-side, not trust browser-supplied links. See Spatial audio showcases for sourcing guidance.

Mixing with speech

Clips share the outbound mix with TTS and client microphones. On Voice+Data they follow the same positional panning and mix groups as orbiting TTS — see Positional mix & orbiting TTS. Clips are not TTS: they do not trigger STT hold or agent_speaking_* speech events.

Clip playback