Speech synthesis and recognition

From Robowaifu Institute of Technology
Jump to navigation Jump to search

Speech synthesis and recognition — usually shortened to speech I/O or STT/TTS (Speech-To-Text/Text-To-Speech) — is the part of a robowaifu build that lets her hear you, understand you, answer out loud, and move her mouth while she does it. It is the cheapest and most immediately rewarding subsystem to build: a complete, low-latency, fully offline voice loop runs on a Raspberry Pi 5 or any spare laptop and can be assembled in a weekend.

This page is a build guide for hobbyists, not a research survey. It covers the pipeline stage by stage, the hardware each stage needs, the current open-source options and their licenses, how to build a voice of your own, how to make the mouth move, and the legal questions that appear as soon as a synthetic voice starts to sound like a real person.

The page grew out of the archived /robowaifu/ thread "Speech Synthesis/Recognition general", which remains worth reading for its primary sources and for how much of its 2019 reasoning turned out to be right. Where its recommendations are now obsolete, that section records what replaced them.

At a glance

Stage Job Common 2026 choice Runs on
Wake word Decide that speech is addressed to her, not to the TV openWakeWord, microWakeWord Any CPU, ESP32-S3
Voice activity detection Find where an utterance starts and ends Built into Moonshine; Silero VAD elsewhere Any CPU
Speech recognition (ASR) Turn audio into text Moonshine, sherpa-onnx, Parakeet TDT, whisper.cpp for batch CPU or GPU
Dialogue Produce a reply in character A local 3B–8B instruct model via llama.cpp or Ollama CPU, GPU, or a home server
Speech synthesis (TTS) Turn text into audio KittenTTS, Piper, Kokoro, Chatterbox, GPT-SoVITS CPU (KittenTTS, Piper, Kokoro) or GPU (the rest)
Visemes and motion Move the mouth, jaw, head and antennas in time uLipSync, wLipSync, Rhubarb, servo jaw Host, MCU, or Unity

The pipeline

Speech I/O is a chain of small, replaceable stages. Build them one at a time and keep each one independently testable; a robot that can only be debugged end to end is a robot that never gets debugged.

  mic array ──► wake word ──► VAD / endpointing ──► ASR ──► dialogue model ──► TTS ──► visemes ──► mouth, jaw, head
                 (always on)                          │         │                          │
                                                transcript   memory,                  blend shapes
                                                + emotion    persona                  or servo PWM

Two rules save the most time:

  • Half-duplex first. Let her finish speaking before you let the microphone back in. Barge-in (interrupting her mid-sentence) is a nice feature and a hard one; add it only after the simple loop is solid.
  • Measure the whole turn, not the parts. The number that matters is the gap between you finishing a sentence and her starting to answer. Individual model speeds are misleading on their own.

Latency budget

Human conversational turn-taking gaps are around 200 ms, but a companion robot does not need to hit that. A gap of roughly 1.0–1.5 s reads as "she is thinking", which is usually charming. Past about 2.5 s it reads as "she is broken".

Stage Target Notes
Wake word detection 50–150 ms Frame size dominates this; 80 ms frames are a good compromise
Endpointing 300–700 ms Trailing silence before the engine decides you stopped; Moonshine handles this itself
ASR 150–600 ms Streaming models beat chunked Whisper here
Dialogue, first token 200–800 ms Keep the prompt short and the context small
TTS, first audio 100–400 ms Sentence-level streaming hides most of the total
Perceived total 1.0–2.0 s Measure with a timestamped log, not by ear

Design principles

  • Offline by default. A companion that stops working when the Wi-Fi drops, or when a vendor changes a price, is not a companion. Every stage on this page has a local option.
  • Prefer boring technology for the parts you do not care about. Piper and Moonshine are unglamorous and they will still work in five years.
  • Degrade gracefully. If the GPU is busy, fall back to the CPU voice. If ASR confidence is low, ask her to repeat rather than inventing a reply.
  • Cache aggressively. Greetings, acknowledgements and filler ("mm?", "one moment") should be pre-rendered WAV files, not generated. They are also where most of the perceived snappiness comes from.
  • Keep provenance. Log where every voice model and every training clip came from. You will not remember in a year, and it matters.
  • Do not chase the leaderboard. TTS quality is judged by ear on your hardware with your text. Install two candidates, generate the same twenty sentences, and listen.

Hardware tiers

Tier Example hardware What it can run comfortably Realistic turn time
0 ESP32-S3 satellite + a host on the network Wake word on device (microWakeWord); everything else remote Network bound
1 Raspberry Pi 5, 8 GB KittenTTS nano or Piper, Moonshine small/medium, a 1B–4B model at short context 1.5–3 s
2 Mini-PC (N100/i5) or Pi 5 + Coral/Hailo accelerator KittenTTS mini or Kokoro, Moonshine medium, 4B model 1–2 s
3 Desktop with 8–16 GB GPU (RTX 3060/4060 class) KittenTTS 2 or Chatterbox or GPT-SoVITS, Parakeet or Whisper, 8B model, vision 0.6–1.2 s
4 24 GB GPU Everything above at once, plus training and fine-tuning 0.5–1 s

Practical notes that do not show up in spec sheets:

  • Microphone quality buys more accuracy than model size. A single decent USB mic with a foam windscreen outperforms a cheap bare capsule array in a real room.
  • Active cooling is not optional on a Pi 5 running sustained ASR; thermal throttling looks exactly like a slow model.
  • Audio output through HDMI often resamples and adds latency. Use a USB DAC or I2S amp and check the actual sample rate.
  • Keep the GPU out of the display's compositor if you can. On Windows, exclusive fullscreen games will evict your TTS model from VRAM.
  • 25 MB of KittenTTS nano int8 will fit on hardware that cannot hold a Kokoro checkpoint in RAM at all. Footprint, not quality, is often the binding constraint on a small board.

Wake word and voice activity detection

Wake word detection is a small always-on classifier. Voice activity detection (VAD) is a smaller one that answers "is this speech at all". You need both: the wake word decides *who* is being addressed, the VAD decides *when* to stop recording.

Project License Where it runs Notes
openWakeWord Apache-2.0 Linux, macOS, Windows, Pi Pre-trained phrases plus a documented path to train your own; pairs with Silero VAD
microWakeWord Apache-2.0 ESP32-S3, Android Small enough for a $13 satellite board; used by Home Assistant
Silero VAD MIT Any CPU The de-facto standalone VAD; about 1 ms per frame
Porcupine Commercial, free tier Many platforms Good accuracy, per-platform binaries, vendor dependency

Two hardware habits that prevent most problems:

  • Put the mic away from the speaker and enable echo cancellation, or she will hear herself and answer herself. Half-duplex operation plus a 150–250 ms mute after playback is a crude but effective fix.
  • Give her ears that know direction. A four-mic array with direction-of-arrival (DoA) lets her turn toward the speaker. The Reachy Mini exposes DoA over its local API as a worked example.

If you use Moonshine you can skip Silero entirely: it segments on voice activity internally and hands you completed lines, so there is no separate VAD stage to wire up or tune. Keep Silero in the toolbox for anything that is not going into Moonshine.

Training your own wake phrase with openWakeWord is mostly a data problem: it synthesizes thousands of clips of the phrase with a TTS engine, augments them with room impulse responses and noise, and trains a small classifier on top of a shared audio embedding. A phrase of three or more syllables with unusual phonemes has far fewer false positives than a two-syllable name.

Speech recognition

Moonshine is the default choice on this page because it was designed for exactly this use case: live, on-device transcription where the audio arrives in pieces. It runs a streaming encoder over audio as it arrives, so it starts emitting text while you are still talking, and it segments on voice activity, so it hands you finished lines rather than one 30-second window to post-process.

Whisper-family models remain excellent, but they process fixed 30-second windows, which is wasteful for a five-second question and useless for a partial result. Use them for batch work — dataset transcription, logs, subtitle timing — and Moonshine for the live loop.

Engine Model family Streaming License (code / weights) Best for
Moonshine Moonshine Native MIT / MIT for English; Moonshine Community License for other languages Live voice interfaces; VAD and segmentation built in
whisper.cpp Whisper Chunked MIT / MIT Portable batch transcription on CPU and Apple Silicon
faster-whisper Whisper Chunked MIT / MIT Python batch jobs; CTranslate2 backend
WhisperX Whisper + alignment No BSD-2 / MIT Word timestamps and speaker labels, useful for dataset work
sherpa-onnx Zipformer, SenseVoice, Parakeet, Whisper Native Apache-2.0 / per model One ONNX runtime across Pi, Jetson, Android, WASM, RISC-V
NeMo (Parakeet TDT) Parakeet, Canary Native Apache-2.0 / CC-BY-4.0 Highest accuracy per unit of compute on NVIDIA hardware
Vosk Kaldi Native Apache-2.0 / Apache-2.0 Very small models, older hardware, many languages

Rules of thumb, all of which should be re-measured on your own board:

  • Moonshine ships eight languages: English, Spanish, Mandarin, Japanese, Korean, Vietnamese, Arabic and Ukrainian. If you are building a Japanese-speaking character, that list matters more than a benchmark score.
  • English models and the inference code are MIT. Every other language is under the separate Moonshine Community License — read it before you ship a non-English build.
  • On a Pi 5 the recommended English model runs faster than real time. On weaker hardware, override the architecture with model_arch() and drop to a smaller size rather than switching engines.
  • Use transcribe_without_streaming() for files and the stream API for live audio. They share one loaded model, so the same object can back both your test harness and the robot.
  • If you do use Whisper, always put a VAD in front of it: Whisper-family models hallucinate confident text on silence, and a robot that answers nothing is unsettling.
  • Constrain the output. Moonshine accepts key terms in its options map, and ships an embedding-based IntentRecognizer for fixed commands ("stop", "come here", "good night"). Either removes most absurd mishears without hand-writing a grammar.

Older tutorials import useful-moonshine and call moonshine.transcribe(path, "moonshine/tiny"). That package still exists and still works, but the maintained path is pip install moonshine-voice, which is what the examples here use. On Raspberry Pi OS the documented install is sudo pip install --break-system-packages moonshine-voice.

Minimum working example

Moonshine loads from a model directory, not a model name, so the first call to get_model_for_language() downloads and caches the recommended model and every later run is offline:

"""transcribe.py - transcribe a WAV file offline with Moonshine."""

from moonshine_voice import Transcriber, get_model_for_language, load_wav_file

MODEL_LANGUAGE = "en"


def load_transcriber():
    """Download the recommended model on first run, then reuse the cache."""
    model_path, model_arch = get_model_for_language(MODEL_LANGUAGE)
    return Transcriber(model_path=model_path, model_arch=model_arch)


def transcribe_wav(transcriber, wav_path):
    audio_samples, sample_rate = load_wav_file(str(wav_path))
    transcript = transcriber.transcribe_without_streaming(audio_samples, sample_rate)
    return " ".join(line.text.strip() for line in transcript.lines).strip()
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Dialogue

The "brain" is the stage this wiki covers least elsewhere, so keep it simple here. A companion robot needs three things from a language model: a stable persona, a short memory, and low first-token latency. Everything else is optional.

  • Model size: 3B–4B parameters in a 4-bit GGUF is the sweet spot for CPU or 8 GB GPU; 8B is a clear quality step if you have the VRAM. Specific best-in-class models change monthly — check a current leaderboard rather than trusting a page written earlier.
  • Runner: llama.cpp for control, Ollama for convenience and a local HTTP API.
  • Persona: a character card plus a short system prompt. Test it in a frontend such as SillyTavern before wiring it to the robot; the frontend is a free debugging harness for prompt and memory design.
  • Memory: a rolling transcript summary plus a small key-value store ("your name is", "you dislike"). Retrieval is a later problem.
  • Guardrails: constrain the reply length in the prompt (one to three sentences) and post-process: strip stage directions, emoji, and anything that will sound absurd read aloud.
  • Latency: the first token arrives long before the last. Start TTS on the first sentence rather than waiting for the whole reply.

moonshine-voice ships an ollama_voice.py example that pipes finished transcript lines into a local Ollama model and streams the reply, which is the shortest working version of this whole section.

A useful trick from the original thread, still true: if the model is asked to speak a language it handles imperfectly, mixing in a few words of another language and allowing imperfect delivery is more charming than fluent nonsense. Deliberate, confident mistakes read as character; confident gibberish reads as a bug.

Speech synthesis

Speech synthesis is the artificial production of human speech. It is used to create computer-generated voices that can be used for conversational AIs, voice cloning, voice acting, singing, audiobooks and more. Speech synthesis is an important tool for robowaifus, as it allows them to interact with humans in a more natural manner, allowing them to engage in natural spoken conversations with people and provide quick understood responses. Additionally, speech synthesis can be used to produce sound effects, such as crying or laughter, to give interactions with AI agents a more well-rounded experience.

The TTS field moves faster than any other part of this stack. Treat the table as a starting point and verify licenses on the model card, not the repository.

Engine Size Voice cloning Weights license Best for
Piper ~15M (VITS, ONNX) No — large preset voice library Engine GPL-3.0; per-voice terms Pi, embedded, always-available fallback
Kokoro-82M 82M No — 50+ preset voices Apache-2.0 Best quality-per-cycle; runs on CPU
KittenTTS 2 1.7B (1-bit), 506–947 MiB Yes, 5–30 s zero-shot Stellon Labs Community License (code is Apache-2.0) Cloning plus emotion tags, still realtime on CPU
KittenTTS v0.8 15M–80M ONNX, 25–80 MB No — 8 preset voices Apache-2.0 Smallest usable CPU voice; 25 MB int8 fits anywhere
Chatterbox 110M–500M Yes, zero-shot from ~5–10 s MIT Cloning you can ship; watermarks output
GPT-SoVITS ~167M + GPT Yes, 5 s zero-shot, 1 min fine-tune MIT Character voices, CJK languages, huge community
F5-TTS ~336M Yes, zero-shot with reference transcript Code MIT / weights CC-BY-NC Research-grade cloning and fine-tuning
XTTS v2 ~0.5B Yes, ~6 s, 17 languages Coqui Public Model License (non-commercial) Legacy multilingual pipelines only
Orpheus 3B Yes, plus emotion tags Apache-2.0 Expressive delivery, if you have the VRAM
Bark ~1B Limited and non-deterministic MIT Laughter, sighs, non-speech sounds
Dia 1.6B Scripted dialogue with speaker tags Apache-2.0 Two-speaker scenes and audio drama
eSpeak NG Under 10 MB No GPL-3.0 100+ languages, last-resort fallback, G2P tool
Festival + CMU Arctic Small No, but you can record your own Permissive, per-voice terms Retro sound, teaching, custom diphone voices

Choosing in sixty seconds

  • Need her to talk at all, on a Pi, today: Piper.
  • Best sound on modest hardware, permissive license, no cloning needed: Kokoro.
  • Smallest possible footprint — a Pi Zero, an old phone, an ESP32 with a DAC: KittenTTS nano (25 MB int8, Apache-2.0).
  • Cloning and emotion tags with no GPU at all: KittenTTS 2 (but check its weights license first).
  • A specific voice you have the rights to, in something you might distribute: Chatterbox.
  • An anime or game character, especially in Japanese or Chinese: GPT-SoVITS.
  • Emotion tags and expressive acting with a bigger model: Orpheus.
  • Laughter, gasps, weird noises: Bark.
  • Multilingual and you are not shipping it: XTTS v2.
  • Tiny footprint, obscure language, or phonemes you want to control by hand: eSpeak NG.

Piper and Kokoro

Piper, as a pipe — this is the shape most robot integrations want:

# One-off install: grab a release binary and a voice from the project's releases.
echo "Good evening. I missed you." \
  | piper --model voices/en_US-amy-medium.onnx --output_file greeting.wav

# Or stream raw PCM straight to the sound card, which avoids writing a file:
echo "Good evening. I missed you." \
  | piper --model voices/en_US-amy-medium.onnx --output-raw \
  | aplay -q -r 22050 -f S16_LE -t raw -
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Kokoro, from Python:

"""say.py - generate a WAV with Kokoro-82M. Requires espeak-ng on the system."""

import soundfile as sf
from kokoro import KPipeline

AMERICAN_ENGLISH = "a"
DEFAULT_VOICE = "af_heart"
SAMPLE_RATE = 24_000


def synthesize(text, voice=DEFAULT_VOICE, output_path="line.wav"):
    pipeline = KPipeline(lang_code=AMERICAN_ENGLISH)
    chunks = []
    for _graphemes, _phonemes, audio in pipeline(text, voice=voice, speed=1.0):
        chunks.append(audio)
    sf.write(output_path, chunks[0] if len(chunks) == 1 else _concat(chunks), SAMPLE_RATE)
    return output_path


def _concat(chunks):
    import numpy as np

    return np.concatenate([chunk.numpy() for chunk in chunks])
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Note that Kokoro splits long input into chunks. For a robot, chunking is a feature: send the first chunk to the speaker while the second is still being generated.

KittenTTS

KittenTTS is two product lines with very different footprints and, importantly, very different licenses:

  • v0.8 ONNX models — nano 15M, micro 40M, mini 80M. Apache-2.0 weights, 25–80 MB on disk, eight preset voices, no cloning. This is the one to put on a Pi.
  • KittenTTS 2 — 1.7B parameters quantised to 1 bit, 506 MiB or 947 MiB of weights, 47 voices, 5–30 s zero-shot cloning, 20 languages, emotion tags. Runs realtime on a CPU. Its weights are under the Stellon Labs Community License, not Apache.
"""say.py - generate a WAV with KittenTTS on CPU."""

import soundfile as sf
from kittenml import KittenTTS

# v0.8 ONNX models: smallest footprint, eight preset voices, no cloning.
SMALL_MODEL = "KittenML/kitten-tts-mini-0.8"
# Flagship: cloning and expression control, still realtime on CPU.
FLAGSHIP_MODEL = "KittenML/kitten-tts-2"
DEFAULT_VOICE = "expr-voice-2-f"


def synthesize(text, output_path="line.wav", voice=DEFAULT_VOICE, model_name=SMALL_MODEL):
    model = KittenTTS(model_name)
    audio = model.generate(text, voice=voice)
    sf.write(output_path, audio, model.sample_rate)
    return output_path


def speak_expressively(text, output_path="line.wav"):
    """Flagship only: [emotion] tags, <vocal events> and (((emphasis)))."""
    model = KittenTTS(FLAGSHIP_MODEL)
    audio = model.generate(text, voice="Kiki", preset="expressive")
    sf.write(output_path, audio, model.sample_rate)
    return output_path
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Four things worth knowing before you build on it:

  • The preset voice IDs differ between the two lines. The v0.8 models take embedding IDs such as expr-voice-2-f; KittenTTS 2 takes display names such as Kiki. The eight voices (Bella, Jasper, Luna, Bruno, Rosie, Hugo, Kiki, Leo) are the same speakers in both, so switching generations does not change her identity. Check model.available_voices rather than guessing.
  • pip install kittenml pulls in PyTorch, which is heavy on a Pi. If you only want the small ONNX models, the older kittentts package depends on onnxruntime and phonemizer only and exposes the same generate() call. Tutorials written in late 2025 import from kittentts; both shapes work.
  • Streaming is chunk-level, not token-level. model.generate_stream(text, voice=...) yields numpy arrays you can play as they arrive, but each chunk is fully generated and vocoded first, so the first chunk still costs its own generation time. It buys you a shorter gap to first audio on long replies, not on one-liners.
  • The expression markup is genuine markup. A leading [joyful], inline <laugh>, and (((emphasis))) reach the model as control tokens rather than being spoken aloud. Pair it with the dialogue model's emotion tag and you get delivery that matches the reply.

Making a voice sound like a character

A clone of a clean speaking voice often sounds too ordinary for a robot companion. Two cheap tools:

Orthography as a pronunciation control. Most open models have no SSML and no phoneme editor, but they do respond to spelling. Maintain a per-voice substitution table and grow it as you hear problems:

# word -> respelling, applied before synthesis
good   -> ghood
to     -> to.
obey   -> obey..
your   -> your
owner  -> ohner

The doubled full stops insert a small pause. This is crude, and it is exactly what the Festival users in the original thread were doing in 2019; it works on Piper and Kokoro for the same reason it worked then.

An effect chain as a shortcut. If you have a few minutes of a voice rather than a few hours, do not train on the distorted result. Clone the clean voice, then apply the character treatment afterwards so you can tune it without retraining:

# Slight pitch drop, band limiting, short slap-back echo.
# asetrate + atempo together change pitch without changing duration.
ffmpeg -i dry.wav -af "\
asetrate=22050*0.94,aresample=22050,atempo=1.064,\
highpass=f=120,lowpass=f=7500,\
aecho=0.8:0.6:18:0.25" robot.wav
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Restraint matters. One or two effects read as "android"; five read as "broken radio". Test on the actual speaker, at the actual volume, in the actual room.

Building a voice of your own

Everything in this section assumes you have the right to use the audio. See Legal and ethical considerations before you start, not after.

How much audio do you actually need

Goal Data Method
Try a voice out 5–30 s Zero-shot cloning (Chatterbox, GPT-SoVITS, F5-TTS, KittenTTS 2)
A usable character voice 1–10 min Fine-tune or few-shot (GPT-SoVITS is built for exactly this)
A clean, controllable voice 1–5 h Fine-tune a mid-size model; enough to cover prosody, not just timbre
Train from scratch 10 h+ Do not. Fine-tuning wins at hobby scale

The 2019 thread's most useful argument was about this exact question: an anon pointed out that a few hours were needed for training, another replied that the SV2TTS speaker-encoder result needed only about five seconds, and both were right — they were describing training and adaptation. In 2026 the second answer has won for hobbyists.

The dataset pipeline

  1. Pick a source you have rights to. Your own recordings, a hired voice actor with a signed release, a public-domain audiobook, or a permissively licensed corpus.
  2. Cut it into utterance-length clips. 2–15 seconds, one speaker, one sentence.
  3. Transcribe it. WhisperX if you want word timings, Moonshine if you just want text. Proofread either way — errors here become mispronunciations later.
  4. Clean it. Remove music and noise, normalize loudness, resample to the rate the trainer expects.
  5. Align it. Forced alignment gives word or phoneme boundaries; some trainers need them, all of them benefit.
  6. Split it. Hold out 2–5% for validation, and keep speakers and recording sessions from leaking across the split.
  7. Augment it if you are short: split long clips, add random leading silence, nudge volume and tempo slightly.
  8. Train, then listen to the same twenty sentences every time so you can hear regressions.

Extracting clips from subtitles

This is the modern equivalent of the thread's clipchan script, and the single most useful piece of code on this page if your source is a video with subtitles.

"""clip_extractor.py - cut one audio clip per subtitle line from a video.

Writes <out>/wavs/*.wav plus <out>/metadata.csv in LJSpeech style
(path|text), which is the format most fine-tuning recipes expect.
"""

import csv
import pathlib
import subprocess

import pysubs2

SAMPLE_RATE = 48_000
MIN_CLIP_MS = 700       # shorter lines are usually grunts or edit noise
MAX_CLIP_MS = 15_000    # longer lines are hard to align and rarely useful
MIN_CHARS = 3


def export_clip(video_path, start_ms, end_ms, out_path):
    subprocess.run(
        [
            "ffmpeg", "-nostdin", "-y", "-loglevel", "error",
            "-ss", f"{start_ms / 1000:.3f}",
            "-to", f"{end_ms / 1000:.3f}",
            "-i", str(video_path),
            "-map", "0:a:0", "-ac", "1", "-ar", str(SAMPLE_RATE),
            str(out_path),
        ],
        check=True,
    )


def is_usable(line):
    duration_ms = line.end - line.start
    text = line.plaintext.strip()
    return MIN_CLIP_MS <= duration_ms <= MAX_CLIP_MS and len(text) >= MIN_CHARS


def extract_all(video_path, subtitle_path, output_dir):
    subtitles = pysubs2.load(str(subtitle_path))
    wav_dir = output_dir / "wavs"
    wav_dir.mkdir(parents=True, exist_ok=True)
    rows = []
    for index, line in enumerate(subtitles):
        if not is_usable(line):
            continue
        wav_path = wav_dir / f"{index:05d}.wav"
        export_clip(video_path, line.start, line.end, wav_path)
        rows.append((wav_path.name, line.plaintext.strip()))
    return rows


def write_metadata(rows, output_dir):
    with open(output_dir / "metadata.csv", "w", newline="", encoding="utf-8") as handle:
        csv.writer(handle, delimiter="|").writerows(rows)


def main():
    video_path = pathlib.Path("episode01.mkv")
    subtitle_path = pathlib.Path("episode01.ass")
    output_dir = pathlib.Path("dataset")
    rows = extract_all(video_path, subtitle_path, output_dir)
    write_metadata(rows, output_dir)
    print(f"wrote {len(rows)} clips")


if __name__ == "__main__":
    main()
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Subtitle timing is often 100–300 ms late relative to the audio, and opening or ending themes usually carry no useful dialogue. Review a random sample of clips by ear before training; ten minutes of checking saves an hour of retraining.

Cleaning

  • Music and effects removal: Demucs (MIT) with --two-stems=vocals, or Spleeter (MIT). Both leave artefacts; a lighter denoiser often sounds better on already-clean dialogue.
  • Noise suppression: DTLN (MIT), DeepFilterNet, or RNNoise. Aggressive settings strip breath and room tone and make the clone sound synthetic, so keep a copy of the unprocessed audio.
  • Loudness: normalize to a consistent target (for example -23 LUFS) rather than peak-normalizing, which leaves quiet clips quiet.
  • Resampling: match the trainer exactly. A 44.1 kHz dataset and a 48 kHz expectation produce a subtle, hard-to-diagnose muffling.

Case study: the Pony Preservation Project

The best-documented hobbyist voice dataset effort is the Pony Preservation Project, a /mlp/ community project to build TTS voices for a large cast of characters. It is worth studying even if the subject matter is not yours, because the pipeline is the same one you will build:

  • A per-line labelling tool with human review of three audio sources per line, so the cleanest version wins.
  • Metadata in LJSpeech format plus per-line emotion tags.
  • Published tooling and a ~15 GB dataset on Hugging Face, released under a permissive license.
  • Several generations of backends: Tacotron 2, then Controllable TalkNet, so-vits-svc and RVC, now exporting directly to GPT-SoVITS training lists.
  • A unified web UI over Docker containers, built specifically because Python dependency hell was the main reason contributors gave up.

Two lessons transfer directly. First, human review of a few hundred lines beats automated cleaning of a few thousand. Second, the tooling matters as much as the model: projects die of installation friction far more often than of bad data.

Making the mouth move

Lip sync is a separate, much easier problem than speech, and it is the single biggest contributor to perceived life. You need a mapping from sound to mouth shape, and a source of timing.

System Mouth shapes Timing source License Notes
Hanna-Barbera / Preston Blair set A–F, plus G, H, X Pre-baked Public domain convention Six shapes is genuinely enough for 2D
VRM visemes aa, ih, ou, ee, oh Real time VRM specification What most avatar tooling expects
uLipSync Configurable Real time (MFCC) MIT Unity; analyses the audio buffer, no ASR needed
wLipSync Configurable Real time (MFCC) MIT Browser/WebAudio port of uLipSync; works with three-vrm
Rhubarb Lip Sync A–F, G, H, X Pre-baked, from a WAV MIT Command-line, ideal for pre-rendered lines and cutscenes
ChatdollKit VRM expressions Real time Apache-2.0 Unity SDK that ties ASR, LLM, TTS and lip sync together

For a physical jaw, the cheapest convincing approach is a single servo driven by an amplitude envelope with a smoothing filter, plus a small time offset so the mouth opens slightly *before* the sound. For an animatronic face, map viseme classes to servo positions or blend shapes rather than trying to drive individual muscles.

Practical details that matter more than the model:

  • Anticipate by 30–80 ms. Mouths lead sound in real speech. With pre-baked data this is a slider; with real-time analysis you need a small lookahead buffer.
  • Idle motion is not optional. A perfectly still face with a moving mouth is uncanny. Add breathing, blinks, and slow head drift.
  • Close the mouth on silence rather than holding the last shape; a stuck open mouth is the classic tell.
  • Do not sync to phonemes you did not generate. If you use a cloned voice with unusual prosody, amplitude-driven jaw motion is more robust than phoneme-driven.

Integration patterns

Pattern Shape Good for Watch out for
Single Python process One script, models loaded at start Prototypes, one robot One crash kills the whole robot
Local services over HTTP/JSON Each stage its own process, one host Most builds; restart a stage without rebooting her Serialization overhead is small, startup order is not
Satellite + server Microcontroller streams audio, host does the work Whole-home presence, battery devices Network latency and dropouts
Wyoming protocol Home Assistant's audio service protocol Interop with Home Assistant voice satellites You inherit its assumptions about turns
ROS 2 nodes One node per stage, DDS transport Robots with real motion planning Heavy for a desktop companion
Unity / VRM app C# host, models over sockets Virtual avatars and Vtuber-style presentation GPU contention between renderer and models

Whatever you choose, define the message format once and log it. A minimal turn record is enough to debug almost everything:

{
  "turn_id": "turn-0007",
  "speaker": "user",
  "text": "good morning",
  "started_ms": 4210,
  "ended_ms": 4980,
  "asr_confidence": 0.91,
  "reply_text": "Morning. You are up early.",
  "emotion": "sleepy",
  "tts_first_audio_ms": 5330,
  "wake_word": "hey_jarvis"
}

With timestamps in the log, "she feels slow" becomes a measurement instead of an argument.

Minimal working loop

A complete, offline, CPU-only loop: wake word, Moonshine endpointing and ASR, a placeholder reply, and Piper out to the speakers.

Because Moonshine segments on voice activity and reports completed lines, there is no silence threshold to tune here — that whole stage disappears.

python -m venv .venv && . .venv/bin/activate
pip install openwakeword moonshine-voice sounddevice numpy
python -c "import openwakeword.utils; openwakeword.utils.download_models()"
# Plus a Piper binary and a voice, and alsa-utils for aplay.
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.
"""voice_loop.py - minimal half-duplex voice loop for a desktop companion.

Pipeline: wake word -> Moonshine streaming ASR -> reply -> Piper TTS.
Everything runs locally on CPU. Replace answer() with your dialogue model.
"""

import queue
import subprocess

import numpy as np
import sounddevice as sd
from moonshine_voice import (
    Transcriber,
    TranscriptEventListener,
    get_model_for_language,
)
from openwakeword.model import Model

SAMPLE_RATE = 16_000          # both openWakeWord and Moonshine work natively here
FRAME_SAMPLES = 1_280         # 80 ms frames, openWakeWord's preferred size
WAKE_THRESHOLD = 0.5          # raise toward 0.7 in a noisy room
MAX_TURN_FRAMES = 250         # ~20 s cap so a stuck mic cannot hang her
INT16_PEAK = 32_768.0         # divisor that maps int16 onto -1.0..1.0
PIPER_VOICE_PATH = "voices/en_US-amy-medium.onnx"
PIPER_SAMPLE_RATE = "22050"


class LineCollector(TranscriptEventListener):
    """Pushes each finished speech segment onto a queue."""

    def __init__(self, finished_lines):
        self._finished_lines = finished_lines

    def on_line_completed(self, event):
        text = event.line.text.strip()
        if text:
            self._finished_lines.put(text)


def open_microphone():
    return sd.RawInputStream(
        samplerate=SAMPLE_RATE,
        channels=1,
        dtype="int16",
        blocksize=FRAME_SAMPLES,
    )


def read_frame(stream):
    raw_bytes, _overflowed = stream.read(FRAME_SAMPLES)
    return np.frombuffer(raw_bytes, dtype=np.int16)


def to_float_samples(frame):
    """Moonshine wants mono float32 in -1.0..1.0, not int16."""
    return frame.astype(np.float32) / INT16_PEAK


def wait_for_wake_word(stream, detector):
    """Block until the wake word fires, then reset for the next turn."""
    while True:
        scores = detector.predict(read_frame(stream))
        if max(scores.values()) >= WAKE_THRESHOLD:
            detector.reset()
            return


def capture_turn(stream, transcriber):
    """Feed mic audio to Moonshine until it closes a speech segment."""
    finished_lines = queue.Queue()
    session = transcriber.create_stream()
    session.add_listener(LineCollector(finished_lines))
    session.start()
    try:
        frames_seen = 0
        while frames_seen < MAX_TURN_FRAMES and finished_lines.empty():
            session.add_audio(to_float_samples(read_frame(stream)), SAMPLE_RATE)
            frames_seen += 1
    finally:
        session.stop()
        session.close()
    return "" if finished_lines.empty() else finished_lines.get()


def answer(text):
    """Stand-in for the dialogue model. Swap in a local LLM call."""
    return f"You said: {text}"


def speak(text):
    """Stream text through Piper straight into the speakers."""
    piper = subprocess.Popen(
        ["piper", "--model", PIPER_VOICE_PATH, "--output-raw"],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
    )
    player = subprocess.Popen(
        ["aplay", "-q", "-r", PIPER_SAMPLE_RATE, "-f", "S16_LE", "-t", "raw", "-"],
        stdin=piper.stdout,
    )
    piper.stdin.write(text.encode("utf-8"))
    piper.stdin.close()
    piper.stdout.close()
    player.wait()
    piper.wait()


def reply_and_speak(stream, heard):
    """Half-duplex: stop the mic while she talks, so she cannot hear herself."""
    reply = answer(heard)
    stream.stop()
    try:
        speak(reply)
    finally:
        stream.start()


def main():
    model_path, model_arch = get_model_for_language("en")
    transcriber = Transcriber(model_path=model_path, model_arch=model_arch)
    detector = Model(wakeword_models=["hey_jarvis"], vad_threshold=0.5)
    with open_microphone() as stream:
        while True:
            wait_for_wake_word(stream, detector)
            heard = capture_turn(stream, transcriber)
            if heard:
                reply_and_speak(stream, heard)


if __name__ == "__main__":
    main()
Code not tested
Not tested by a human yet. Treat it as a sketch, not as a recipe.
Generated on 2026-10-24 by 大肥鱼.
Language models write plausible code that is sometimes subtly wrong — invented functions, renamed parameters, confident misuse of a real API.
Check anything you rely on against the project's own documentation.

Three details in that script are load-bearing:

  • stream.stop() around playback is the half-duplex enforcement, and it also discards whatever the microphone buffered while she was talking. Without it, the next turn starts with several seconds of stale audio and she transcribes your own voice.
  • session.stop() in the finally block matters: it flushes any speech segment still open, so a user who pauses right at the twenty-second cap still gets their sentence transcribed rather than silently dropped.
  • The default English model is the medium streaming architecture. On a Pi 5 that is comfortable; on anything weaker, swap in a smaller model_arch before you consider changing engines.

To use Kokoro or KittenTTS instead of Piper, replace the body of speak() with a call to synthesize() from the examples above and play the resulting file. Expect roughly 1–2 s per turn on a modern desktop CPU.

Troubleshooting

Symptom Likely cause Fix
Wake word fires constantly Threshold too low, or the mic is picking up the speaker Raise threshold, enable VAD gating, add echo cancellation, stop the mic during playback
Wake word never fires Wrong sample rate, wrong channel, or int16 vs float input Confirm 16 kHz mono int16 frames of 1280 samples before the detector
Moonshine returns nothing, or errors on add_audio Feeding int16 where it expects float32 in -1.0..1.0 Divide by 32768 and cast to float32; check with load_wav_file() on a known-good clip
Transcription is fine from a file but wrong from the mic The test file and the live path use different sample rates or channel counts Resample once, at the source; never let two rates into the same pipeline
Using Whisper and it answers silence with confident nonsense No VAD in front of the model Enable vad_filter, raise no_speech_threshold, or switch to Moonshine
Replies are correct but arrive late Waiting for the full LLM reply before starting TTS Stream by sentence; pre-render filler phrases
TTS sounds muffled or aliased Sample-rate mismatch between model output and the sound card Match rates explicitly; avoid HDMI audio; check aplay -D settings
KittenTTS sounds thin or robotic You are on the nano int8 checkpoint Try kitten-tts-mini-0.8, or the flagship if you have the RAM
Voice clone sounds like a stranger Reference audio has music, room tone, or more than one speaker Re-cut clean single-speaker clips; try a different checkpoint; fine-tune instead of zero-shot
Mouth lags the voice Audio buffer latency, or no lookahead Shrink the buffer; add a 30–80 ms viseme lead; use pre-baked lip sync for fixed lines
Everything is slow after ten minutes Thermal throttling, or VRAM evicted by another process Active cooling; keep the GPU off the display; pin models in memory
She answers herself Half-duplex not enforced, no echo cancellation Stop the input stream during playback and for 200 ms after

Legal considerations

This section is general information, not legal advice. Voice, likeness and synthetic-media law differs by country and is changing quickly; get advice for anything you intend to publish or sell.

Software licenses

Read the weights license, not just the repository license. TTS projects frequently pair permissive code with restricted models, and KittenTTS is the cleanest current example: the GitHub repository is Apache-2.0, the v0.8 ONNX weights are Apache-2.0, and the flagship KittenTTS 2 weights are under the Stellon Labs Community License. Someone who reads only the README concludes "KittenTTS is Apache-2.0" and is wrong about the model they are actually using.

  • Permissive: Kokoro (Apache-2.0), Chatterbox (MIT), GPT-SoVITS (MIT), Orpheus (Apache-2.0), Bark (MIT), Dia (Apache-2.0), KittenTTS v0.8 (Apache-2.0), Moonshine code and English models (MIT). Safe for personal and commercial use, subject to attribution.
  • Weights with their own terms: KittenTTS 2 (Stellon Labs Community License), XTTS v2 (Coqui Public Model License, non-commercial), F5-TTS (CC-BY-NC), Moonshine's non-English models (Moonshine Community License), several Fish Audio releases. Read each one; the terms are not interchangeable even within a single project.
  • Copyleft engines: Piper's active engine is GPL-3.0, and the older MIT repository is archived. You can still use it commercially, but if you link it into a closed-source binary you are in copyleft territory. Talk to it over its CLI or an HTTP server instead.
  • Watermarking: Chatterbox embeds a neural watermark in every clip. For most hobbyists this is irrelevant or a bonus; if you need unmarked audio, find out now rather than after building on it.
  • Datasets: your own recordings are cleanest; VCTK and LibriTTS are permissively licensed; LJSpeech is public domain; some large modern corpora are non-commercial, which propagates to models trained on them.

Voice and likeness laws

Cloning a real person's voice can raise legal issues that have nothing to do with the software license:

  • United States: no federal statute yet. Tennessee's ELVIS Act (effective July 2024) treats a voice as property, covers simulations as well as recordings, and reaches tools whose primary purpose is producing a specific person's voice. California and other states have similar or overlapping publicity and digital-replica rules. The NO FAKES Act has been introduced and reported out of committee but is not law; if it passes, it would create a federal digital-replica right and grandfather some earlier state laws.
  • European Union: Article 50 of the AI Act has applied since 2 August 2026. It is a transparency regime: deployers must disclose deepfake audio and video, and providers must mark synthetic output machine-readably. Purely personal, non-professional use is outside the deployer obligations. Separate rules on intimate synthetic material of identifiable people apply from December 2026.

Guideline to keep your project out of trouble:

  1. Use a voice actor with a written release, or a voice design model to create an original voice.
  2. Keep a provenance log: source, date, permission, license, and what you did to it.
  3. Do not publish audio that could be mistaken for a real person saying something they did not say without clearly labelling it as AI.
  4. Ripping audio from games, anime or films to train a character voice is widespread in fan communities and remains legally grey at best. Keep it private, keep it unmonetized, and expect that it may not be distributable for long without companies getting butthurt and sending a cease and desist.
  5. If you put her on a stream or in a video, you may need to disclose that the voice is synthetic.

History and the original thread

The starting point for this page was thread No. 199, "Speech Synthesis/Recognition general", which started on /robowaifu/ in September 2019. Reading it now is instructive: the participants identified the right problems, linked the right papers, and were wrong mainly about timescales.

What still holds up

  • Sequence-to-sequence TTS is the right frame. The Tacotron 2 paper and its audio samples remain the clearest explanation of text-to-mel-to-waveform synthesis, even though the architecture is superseded.
  • Five seconds is enough to characterize a speaker. The SV2TTS result, raised in the thread, is the direct ancestor of every zero-shot cloning model in use today.
  • Subtitle-driven clip extraction works. The thread's clipchan tool cut hundreds of clips per episode from subtitle timing. That is still the standard approach; see the extractor above.
  • Denoise, then train. Spleeter and DTLN were the 2019 answers; Demucs and DeepFilterNet are the 2026 answers. The principle did not change.
  • A single voice actor is not enough data to train from scratch, but is plenty to adapt. The thread argued both sides and both were correct.
  • Expectation management is a design tool. One poster's observation that imperfect speech reads as endearing effort rather than failure — if the character is framed as trying — is still the cheapest quality improvement available to a hobbyist.

What was replaced

Then (2019–2020) Now Why
Tacotron 2 + WaveGlow VITS (Piper), flow matching (F5-TTS), LLM-backbone TTS (Orpheus) Faster, simpler, better prosody
Real-Time-Voice-Cloning / SV2TTS Chatterbox, GPT-SoVITS, F5-TTS, KittenTTS 2 The reference implementation is unmaintained and its author recommends alternatives
Hours of audio required 5–30 s zero-shot, 1–10 min fine-tune Speaker embeddings and better pretraining
Multi-gigabyte models as the floor A 15M-parameter model in 25 MB Aggressive quantisation and 1-bit weights moved the floor by two orders of magnitude
Chunked Whisper everywhere (2023–2025) Moonshine for the live loop Streaming encoders beat a 30-second window when the user is still talking
aeneas forced alignment Montreal Forced Aligner, WhisperX, torchaudio Faster and far more accurate
Spleeter and DTLN for cleaning Demucs, DeepFilterNet, UVR Better separation with fewer artefacts
Festival and eSpeak as the default Still useful, now a fallback Neural TTS won on quality; eSpeak NG remains unbeaten on size and language count
Commercial cloning APIs (Lyrebird and similar) Local open-weight models The API never became a general service; the open models caught up
Waiting for Google to release Tacotron code The code was released, and then two generations of models replaced it Do not wait for a vendor to open-source the thing you need

Sources from the thread still worth opening

  • Lemmetty, "Review of Speech Synthesis Technology" — the best free introduction to how synthesis actually works, and the reason the thread's participants understood formants and concatenation before they understood neural nets.
  • Festival and CMU Arctic — still installable, still the fastest way to hear the difference between unit selection and neural synthesis.
  • Tacotron 2 and its samples — listen to them; they are a useful reminder of how fast this field moved.
  • Tacotron repository — historical, but the demos directory is a good audio archive.
  • InMoov — the open 3D-printed humanoid the thread recommended for the body; still one of the few complete open humanoid designs.

Starter projects

Project 1: desk companion, no GPU (about $100–200)

Raspberry Pi 5 (8 GB), USB microphone, powered speaker, a 3D-printed head with one jaw servo.

  1. Install Piper and a voice; get echo "hello" | piper ... | aplay working first.
  2. Install moonshine-voice and confirm transcribe_without_streaming() on a recorded WAV file.
  3. Wire up openWakeWord, then the loop above.
  4. Add a 4B instruct model via llama.cpp with a 4K context and a 200-token reply cap.
  5. Swap Piper for KittenTTS nano int8 and compare; keep whichever sounds better on your speaker.
  6. Add the jaw servo on a PWM pin, driven by the audio envelope with a 50 ms lead.

Project 2: expressive avatar (about $0 plus a GPU you already own)

Any desktop with a GPU; a VRM model; a chat frontend for prompt design.

  1. Generate a voice with Kokoro and pick a preset you like.
  2. Render the avatar in Unity with UniVRM and uLipSync, or in the browser with three-vrm and wLipSync.
  3. Add blinks, breathing and idle head motion before you add anything else.
  4. Move to KittenTTS 2, GPT-SoVITS or Chatterbox once you want a specific voice or emotion tags.
  5. Keep a character card and a rolling summary so she stays in character past the first hour.

Project 3: physical companion with a body (about $500–1,500)

Reachy Mini (open hardware, onboard Pi 4, mic array, speaker, six-DoF head) or a printed InMoov torso.

  1. Use the platform's SDK to get speech out and DoA in.
  2. Replace the stock voice with KittenTTS, Piper or Kokoro over the same audio path.
  3. Map emotion tags from the dialogue model to the platform's recorded moves.
  4. Add lip sync to the mouth or display, then tune the lead time.
  5. Only then add vision; it doubles the compute budget and the debugging surface.

See also

References

  1. /robowaifu/, "Speech Synthesis/Recognition general", thread No. 199 (September 2019 – October 2020). Retrieved 24 October 2026. The seed for this page; posts are anonymous and their claims should be verified against primary sources.
  2. Moonshine — streaming on-device ASR; MIT code and English models, Moonshine Community License for other languages.
  3. Moonshine Voice documentation, "Transcription" — Transcriber, MicTranscriber, event listeners, transcribe_without_streaming().
  4. Moonshine Voice quickstart — including the Raspberry Pi install line.
  5. KittenTTS — Apache-2.0 code; v0.8 ONNX weights Apache-2.0, KittenTTS 2 weights under the Stellon Labs Community License.
  6. KittenML docs, "Models and voices" — checkpoint sizes, the eight shared preset voices, and cache handling.
  7. Kokoro-82M — model and inference library; Apache-2.0 weights.
  8. Piper (OHF-Voice) — maintained GPL-3.0 engine; the older rhasspy/piper repository is archived.
  9. Chatterbox — Resemble AI; MIT weights, PerTh watermark.
  10. Coqui TTS and the Coqui Public Model License — XTTS v2 weights are non-commercial.
  11. F5-TTS — MIT code, CC-BY-NC pretrained weights.
  12. GPT-SoVITS — MIT; zero-shot and one-minute fine-tuning.
  13. Orpheus-TTS — Apache-2.0, emotion tags.
  14. whisper.cpp, faster-whisper, WhisperX — the batch-transcription alternatives.
  15. sherpa-onnx — portable ONNX speech runtime; Apache-2.0.
  16. NVIDIA NeMo — Parakeet and Canary checkpoints; Apache-2.0 code, CC-BY-4.0 weights.
  17. openWakeWord, microWakeWord, Silero VAD — wake word and voice activity detection.
  18. Reachy Mini and documentation — open companion robot with mic array, speaker and DoA.
  19. uLipSync, wLipSync, Rhubarb Lip Sync — viseme generation.
  20. ChatdollKit — Apache-2.0 Unity SDK combining ASR, LLM, TTS and lip sync.
  21. Pony Preservation Project dataset, PonySorter, Hay Say — worked example of a community voice dataset and its tooling.
  22. Tacotron 2 and SV2TTS — the two papers that defined the 2019 thread's approach.
  23. European Commission, transparency obligations under Article 50 of the AI Act — applies from 2 August 2026.
  24. AI voice cloning laws: the ELVIS Act and the states that followed — state right-of-publicity summary and NO FAKES Act status.
  25. Moonshine vs Whisper benchmarks — edge-device latency figures including Raspberry Pi 5.

External links