VoiceThere

Data-only world sync

A VoiceThere session can be data-only: no microphone, no TTS, just a WebRTC data channel between browsers and your TypeScript agent. Use it to sync positions, inventory, or any small world state. The same worker can later add voice without changing how you send poses.

There are two inbound APIs. onDataChannelMessage receives JSON (or a string) on the sync channel. onDataChannelBinary receives an ArrayBuffer / Buffer — pack float32 positions and reply with sendBinaryToClient.

Starters ship in @voicethere/agent templates 0.8.0+: world-sync (JSON), world-sync-binary (ArrayBuffer), and game-sync (Redis + binary snapshots).

Choose a path

WhenAPITemplate
One agent, a few peers, human-readable stateonDataChannelMessage + sendToClientworld-sync
One agent, tight pose packets, no RedisonDataChannelBinary + sendBinaryToClientworld-sync-binary
Many workers, persistent world, physicsBinary snapshots + Project Redisgame-sync

Simple JSON poses

The smallest useful agent: store the last message per session and broadcast a world object. Clients send { x, y, z } or { type: "pose", x, y, z } on the sync data channel. This is the example on the landing page.

import { defineAgent, sendToClient } from "@voicethere/agent";

const poses = new Map();

export default defineAgent({
  async onDataChannelMessage({ sessionId, message }) {
    poses.set(sessionId, message);
    const world = { type: "world", poses: Object.fromEntries(poses) };
    for (const peerId of poses.keys()) {
      sendToClient(peerId, world);
    }
  },
});

Add onClientJoin / onClientLeave when you need a default spawn pose and cleanup. The world-sync agent.ts template does that and validates finite numbers.

Binary ArrayBuffer poses

JSON is easy to debug. Binary is smaller and cheaper at 20–60 Hz. Pack three little-endian float32 values as a 12-byte ArrayBuffer in the browser:

const pose = new Float32Array(3);

function sendPose(dataChannel, x, y, z) {
  pose[0] = x;
  pose[1] = y;
  pose[2] = z;
  dataChannel.send(pose.buffer);
}

Handle the frame in onDataChannelBinary and fan it out with broadCastBinaryToClients (or sendBinaryToClient for one peer; channel defaults to sync). Both accept a Uint8Array view and forward it without copying the bytes:

import { broadCastBinaryToClients, defineAgent } from "@voicethere/agent";
import { decodePoseInto, WorldSnapshotBuffer } from "./protocol.js";

const snapshot = new WorldSnapshotBuffer(); // one growable buffer
const sessionIds = [];
const xyz = new Float32Array(24); // 8 peers × xyz, grown in place
const indexById = new Map();

export default defineAgent({
  onDataChannelBinary({ sessionId, rawBinary }) {
    let index = indexById.get(sessionId);
    if (index === undefined) {
      index = sessionIds.length;
      sessionIds.push(sessionId);
      indexById.set(sessionId, index);
    }
    if (!decodePoseInto(rawBinary, xyz, index * 3)) return;
    // Same backing buffer every tick; one Buffer view shared by all sends.
    const payload = snapshot.encodePacked(sessionIds, xyz, sessionIds.length);
    broadCastBinaryToClients(payload, sessionIds, "sync");
  },
});

The world-sync-binary template encodes a full snapshot: uint32le peer count, then per peer uint16le session-id length, UTF-8 id, and float32le x, y, z. Poses live in one Float32Array; WorldSnapshotBuffer rewrites a single growable snapshot and hands back the same view while its length is unchanged (no per-tick or per-peer copies). Reuse protocol.ts for the encode/decode helpers.

Redis plus binary world snapshots

One in-memory agent is enough while every client lands on the same runner pod. Across workers, store the world blob in Project Redis (AGENT_REDIS_URL). game-sync does that: JSON register / unregister on the data channel, a 60 Hz sim, and a binary Float32Array snapshot via sendBinaryToClient. The e2e redis-sync template is a smaller Redis world-buffer cousin.

Client notes

  • Use a data-only project so the browser does not wait on microphone permission. See Browser client.
  • Send JSON with the client data-channel helper, or RTCDataChannel.send(arrayBuffer) for binary. The runner delivers JSON to onDataChannelMessage and binary to onDataChannelBinary.
  • Idle timeouts still apply — data-channel sends can reset the timer. See Session idle timeout.

← All documentation