@voicethere/agent is the TypeScript SDK for the sandboxed child that VoiceThere runs as agent.js. Current npm cut: 0.8.1. Source: voicethere/agent.

Install it in the agent repo (not in the web app). The CLI uploads the compiled bundle; the client is a separate package for browsers.

Install

npm install @voicethere/agent
npx @voicethere/agent build
npx @voicethere/agent verify

Defaults: entry agent.ts, output dist/agent.js. Override with --entry / --outfile. Cloud compile uses the same esbuild settings.

Child vs parent

Layer	Runs	Owns
Parent (runner)	Trusted Node + WebRTC + speech	Mic, STT, TTS, mix, recording files
Child (this package)	Sandboxed `agent.js`	Your handlers. Call `speak`, `play`, `sendToClient` over IPC.

Speech events arrive as the same shapes as @node-webrtc-rust/sdk/voice. Types are re-exported from this package. Runtime constants such as SPEECH_EVENT_TYPE are not bundled into the child — import those only in trusted parent code, or compare speech.type string literals in the agent.

defineAgent handlers

import {
  defineAgent,
  speak,
  agentLog,
  type SpeechEvent,
} from "@voicethere/agent";

export default defineAgent({
  async onAgentStart({ env }) {
    // Project Redis injects AGENT_REDIS_URL when the plan includes it.
    const redisUrl = env.AGENT_REDIS_URL;
    if (redisUrl) {
      agentLog("info", "redis url present");
    }
  },
  onSessionStart({ sessionId }) {
    speak(sessionId, "Hello!");
  },
  onUserSpeechFinal({ sessionId, text }) {
    speak(sessionId, `You said: ${text}`);
  },
  onSpeechEvent({ sessionId }, speech: SpeechEvent) {
    if (speech.type === "barge_in") {
      agentLog("info", `interrupted on ${sessionId}`);
    }
  },
});

Handler	When
`onAgentStart`	Once per child, before session IPC. Connect Redis here (`AGENT_REDIS_URL`).
`onWebhook`	Process-wide inbound HTTP. Verify HMAC on `ctx.body` yourself — see inbound webhooks.
`onSessionStart` / `onClientJoin`	Peer joined. Same handler; `onClientJoin` is an alias.
`onSpeechEvent`	Every speech lifecycle event from the parent pipeline.
`onUserSpeechFinal`	Convenience for `user_speech_final` — the usual LLM turn boundary.
`onUserLanguage`	Detected ISO 639-1 code. Spoken language.
`onDataChannelMessage`	JSON / string on the sync channel.
`onDataChannelBinary`	`ArrayBuffer` / `Buffer` poses. Data-only world sync.
`onIdleTimeout`	Cleanup before the runner disconnects an idle peer. Must not throw.
`onSessionEnd` / `onClientLeave`	Peer left.
`errorHook`	Handler threw, before crash policy. Must not throw.

Verify requires at least one of onSpeechEvent, onUserSpeechFinal, onUserLanguage, onDataChannelMessage, or onDataChannelBinary.

Speak, chat, and media

Export	Purpose
`speak(sessionId, text)`	Request parent TTS to that peer.
`speakAndChat`	TTS plus a chat line. Pass `stream: true` to typewrite captions — spoken chat stream.
`play` / `getPlay` / `stopPlay` / `setPlayPose`	HTTPS or inline clips. Clip playback.
`startRecording` / pause / resume / stop	Conversation recording control. Recording.
`agentLog`	Structured logs to the parent. Agent logs.
`disconnectClient`	Kick a peer from agent code (stale multiplayer state, consent).

Data channels and mix

sendToClient / broadcastToClients send JSON. sendBinaryToClient / broadCastBinaryToClientssend a packed buffer. Keep IPC payloads around 64 kB or less (about 16,000 float32 values).

Mix and pose helpers (createMixGroup, setTtsPose, setClientPose, setGlobalMute, …) need Voice+Data and a runner that supports positional mix. isMixAvailable / isTtsPoseAvailable tell you if this child can call them. Guides: positional mix, spatial showcases.

With shared agent child enabled, every session on the pod shares one process. Handlers still receive a per-peer sessionId — pass that into speak and sendToClient. Isolated voice agents leave shared child off so each session gets its own process. Runner settings.

Build and verify

Command	When
`npx @voicethere/agent build`	Bundle `agent.ts` and npm deps (for example `ioredis`) into `dist/agent.js`.
`npx @voicethere/agent verify`	Default: build, then static checks (Node version, `defineAgent`, a supported callback).
`verify --no-build --bundle …`	Re-check an existing file.
`verify-start --no-build --bundle …`	Launch under production `--permission` flags, send `session_start`, require `session_start_ack`. Catches dynamic-require failures.

This is not a voice roundtrip. Deploy to VoiceThere (or run the agent live-test stack) for mic/WebRTC. voicethere build validate wraps the same verify path before upload.

Sandbox

The child is a forked process with Node's permission model (deny-by-default). Runners grant filesystem read of the bundle directory, outbound --allow-net for LLM/tool fetches, and a scoped Redis host when AGENT_REDIS_URL is set. They do not grant child_process, filesystem write, native addons, or worker threads. Environment is stripped to allowlisted keys (SESSION_ID, PROJECT_ID, BUILD_ID, AGENT_* you configured). Environment and secrets.

Private cluster addresses stay unreachable from the child. Use IPC for voice, not in-cluster HTTP. Crash policy decides what happens if the child exits.

Templates

Product templates ship inside this package: echo, voice-starter, world-sync, world-sync-binary, game-sync, webhooks, positional-tts, spatial-showcase, and others. voicethere init --template echo and the dashboard Code tab seed from the same registry. Agent templates. Folder on GitHub: templates/.