@three-ws/voice

Give your avatar a voice, speech in, speech out, and lips that move. ASR, TTS, and Audio2Face lipsync in one import.

Install · Quick start · API · How it works · three.ws

@three-ws/voice is the official client for the three.ws voice loop, the three endpoints that let an avatar hear, speak, and move its face in sync. transcribe() turns spoken audio into text (NVIDIA Riva ASR), speak() turns text into a voiced clip (NVIDIA Magpie TTS, ElevenLabs for cloned voices), and lipsync() turns that audio into a per-frame ARKit blendshape track (NVIDIA Audio2Face-3D) that drives the avatar's mouth and face. It wraps the live, auth-free /api/asr, /api/tts/speak, and /api/a2f endpoints. The visemes it produces are exactly what @three-ws/avatar plays back on a loaded GLB: this is the voice half of a talking avatar.

Why

A talking avatar needs three separate systems wired together: a recognizer to hear the user, a synthesizer to give the avatar a voice, and a facial-animation model so the mouth matches the words. Each is a different provider, a different wire format, and a different failure mode. Hand-rolling that means juggling gRPC audio encodings, WAV header parsing, sample-rate resampling, ARKit blendshape ordering, and three sets of rate limits.

@three-ws/voice is that loop, done once:

• One import, the whole loop. transcribe(audio), speak(text), lipsync(audio): hear, speak, and animate the face from three plain calls.
• Free first. All three lanes lead with NVIDIA NIM (Riva, Magpie, Audio2Face-3D): no key, no wallet, no card on the free path.
• Cross-browser by default. Server-side Riva recognition replaces the Chrome/Edge-only window.webkitSpeechRecognition, so voice input works in Firefox and inside embeds too.
• Lips that match the exact bytes. lipsync() returns time-coded ARKit weights for the precise clip you'll play: drop them straight into @three-ws/avatar.

Every lane is purely additive on the platform: when a provider isn't configured, the endpoint returns a clean not_configured state instead of crashing, and the avatar falls back to in-browser recognition / amplitude lipsync.

Install

npm install @three-ws/voice

Zero runtime dependencies. Works in Node 18+ and the browser (uses fetch). To render the visemes on a GLB, add @three-ws/avatar.

Quick start

Text to a voiced clip, no key:

import { speak } from '@three-ws/voice';

const clip = await speak('Hi, I am your three.ws avatar.', { voice: 'nova' });
new Audio(clip.url).play(); // clip.url is an object URL for the synthesized audio

The full loop: hear the user, answer, animate the face:

import { transcribe, speak, lipsync } from '@three-ws/voice';

// 1. Speech → text (record audio in the browser, send the bytes)
const { text } = await transcribe(audioBlob); // → "what's the weather like"

// 2. Text → speech (your agent decides the reply)
const reply = await speak(`You said: ${text}`, { voice: 'nova' });

// 3. Speech → ARKit visemes for the exact clip you'll play
const face = await lipsync(reply.blob);
//   face.blendShapeNames → ["eyeBlinkLeft", "jawOpen", …] (ARKit-52)
//   face.frames → [{ t: 0.0, w: [...] }, { t: 0.033, w: [...] }, …] at 30 fps

One-shot: synthesize and animate in a single call (server speaks the text with Magpie, then animates that exact audio):

const { audio, animation } = await say('Welcome back.', { voice: 'nova' });
// audio.url → play it · animation.frames → drive the face, perfectly aligned

API

transcribe(audio, options?) → Promise<Transcript>

Speech → text on the free NVIDIA Riva ASR lane. Wraps POST /api/asr. audio is a Blob, ArrayBuffer, or Uint8Array; the SDK reads the encoding from the blob's MIME type (or options.format).

Options

Returns Transcript

speak(text, options?) → Promise<Clip>

Text → a voiced audio clip. Wraps POST /api/tts/speak (NVIDIA Magpie free lane; OpenAI is the paid backstop). Returns the complete audio as a Blob plus a ready-to-play object url.

Options

Returns Clip: { blob, url, contentType, voice, format, model }. The x-tts-voice / x-tts-model / x-tts-format response headers always describe the bytes actually sent.

text is capped at 4096 characters per call.

lipsync(audio, options?) → Promise<FaceTrack>

Speech → a per-frame ARKit blendshape track. Wraps POST /api/a2f with audio (NVIDIA Audio2Face-3D). audio is a Blob / ArrayBuffer of wav or pcm. The track's time codes are in seconds from clip start, so play the original audio and sample the track by the audio element's currentTime.

Returns FaceTrack

say(text, options?) → Promise<{ audio, animation }>

One-shot text → speech → face. Wraps POST /api/a2f with { text, voice, language }: the server synthesizes with Magpie, animates that exact clip, and returns both. audio is { url, blob, contentType, format, voiceName }; animation is the same shape as FaceTrack. Use this when latency matters more than picking your own TTS: one round trip instead of two.

voices() → Promise<VoiceCatalog>

Fetch the live voice catalog (GET /api/tts/voices), ids, names, descriptions, which synthesis lanes are configured. Render a picker before the user commits.

Capability probes: asrInfo() / lipsyncInfo()

Each lane answers a GET probe so a UI can decide whether to use the server lane or its in-browser fallback without sniffing the browser:

import { asrInfo, lipsyncInfo } from '@three-ws/voice';

const asr = await asrInfo();      // GET /api/asr
// → { configured, encodings: ['wav','pcm','flac','ogg'], sampleRate, raw }

const face = await lipsyncInfo(); // GET /api/a2f
// → { configured, canSynthesize, model, fps, blendshapeFormat: 'arkit', sampleRate, accepts, raw }

configured: false means that lane's provider key isn't set on the deployment, so the call would raise not_configured: render the in-browser fallback instead. raw carries the untouched endpoint response for anything not shaped above.

createVoice(options?) → client

Bind every call above to one configuration (a custom baseUrl, a bearer token, a payment-aware fetch for the paid backstops) and reuse it:

import { createVoice } from '@three-ws/voice';

const voice = createVoice({ baseUrl: 'https://three.ws', apiKey: process.env.THREE_WS_API_KEY });
const clip = await voice.speak('Bound to one config.');

Voices

speak() and say() accept these ids (full catalog is live at voices()):

The same id renders on either lane: the free Magpie lane maps each to a persona, the paid backstop uses the id directly.

How it works

Three NVIDIA NIM models on one face, closing the loop user → avatar → user:

  user speaks                                 avatar speaks
       │                                            ▲
       ▼                                            │
  ┌──────────┐   POST /api/asr            POST /api/tts/speak ┌──────────┐
  │ microphone│ ───────────────▶  text  ──────────────────▶  │  speaker │
  └──────────┘   Riva ASR (free)   │     Magpie TTS (free)    └──────────┘
                                   │     ElevenLabs (cloned)        ▲
                                   ▼                                │
                         agent / app logic        audio ───────────┘
                                                    │
                                          POST /api/a2f (Audio2Face-3D, free)
                                                    │
                                                    ▼
                                ARKit-52 blendshape track  ──▶  @three-ws/avatar
                                  { fps, blendShapeNames, frames:[{t,w}] }   (lips move)

• ASR: Riva offline Recognize over NVCF gRPC. Browsers can't all produce the same codec, so the cross-browser default is raw 16-bit PCM (or WAV, whose header is parsed to LINEAR_PCM). Needs NVIDIA_API_KEY + NVIDIA_ASR_FUNCTION_ID.
• TTS: Magpie leads (free, gRPC); OpenAI /v1/audio/speech is the paid last-resort backstop. The clip is fully buffered before a byte ships, so a Magpie failure fails over cleanly.
• Lipsync: Audio2Face-3D bidirectional streaming. Audio is downmixed to mono and resampled to 16 kHz, then streamed; the server streams back the ordered ARKit names followed by per-frame weights at 30 fps. Works out of the box with just NVIDIA_API_KEY (a stable published model id; override with NVIDIA_A2F_FUNCTION_ID).

A2F emits ARKit-52 names; @three-ws/avatar maps them onto whatever convention the loaded GLB exposes (ARKit / RPM / VRM / Oculus), so the same track drives any avatar.

Pricing

The three NVIDIA NIM lanes are free (credit-metered, no payment). The lanes are metered to bound abuse: signed-in callers get a per-user budget, anonymous callers a tighter per-IP one: exceeding it returns 429. The OpenAI TTS backstop and ElevenLabs cloned voices are paid provider lanes used only when configured; the free Magpie lane covers the default voices end to end.

Errors & edge cases

Each call rejects with a typed ThreeWsError carrying a code that mirrors the endpoint's response:

Every state is designed: an unconfigured lane returns not_configured (not a crash), so the client keeps its existing browser path. A2F's text path additionally needs Magpie TTS: without it, say() returns not_configured and you pass pre-synthesized audio to lipsync() instead.

Examples

A runnable script that exercises the whole loop against the live endpoints lives in examples/:

node examples/voice-loop.mjs   # probe → speak → lipsync → transcribe → say

Browser voice loop → animated avatar. Record, transcribe, answer, and drive the face on a loaded GLB:

import { transcribe, say } from '@three-ws/voice';

const heard = await transcribe(recordedBlob);            // user speech → text
const { audio, animation } = await say(reply(heard.text)); // reply → voice + face

const el = document.createElement('audio');
el.src = audio.url;
el.play();
// sample `animation.frames` by el.currentTime and apply to the avatar's
// morph targets: @three-ws/avatar does this mapping for you.

Node: synthesize a narration clip to a file:

import { writeFile } from 'node:fs/promises';
import { speak } from '@three-ws/voice';

const clip = await speak('The only coin is $THREE.', { voice: 'onyx', format: 'wav' });
await writeFile('line.wav', Buffer.from(await clip.blob.arrayBuffer()));

Agent: caption an audio clip (word-level timestamps):

const { text, words } = await transcribe(clip, { words: true });
for (const w of words) console.log(`${(w.startMs / 1000).toFixed(2)}s  ${w.word}`);

Related

• @three-ws/avatar, render a GLB and play these visemes as real lipsync.
• @three-ws/forge, generate and auto-rig the GLB this voice loop speaks through.
• @three-ws/react, React bindings for the three.ws avatar runtime.

Built by three.ws · The only coin is $THREE

Security and contributing

See SECURITY.md, CONTRIBUTING.md and AGENTS.md (for AI agents). Part of three.ws.