AudiJRahardja/voice-chess-whisper-base

Voice Chess — Whisper-base for spoken chess commands

A Whisper-base (74M) fine-tuned to transcribe what people say to a chess app — "knight to f3", "e2 to e4", "castle kingside", "where is my queen", "no, bishop c4" — plus a 21k-parameter wake-word model trained from scratch for "make move" / "hey chess". Both run in the browser (8-bit ONNX, transformers.js / ONNX Runtime Web) in Voice Chess, an accessible app for playing Stockfish by voice.

▶ Try it: speech-chess-audi.vercel.app — the model downloads once (~98 MB) and then runs on your device.

Results

Scored by what the app does with each transcript (the app's own conversation logic decides whether the intended move, question or answer was understood), plus word error rate.

Real speech — 150 recorded commands, one speaker:

System	Commands understood	Wrong move	WER
Chrome Web Speech API	88.7%	0.7%	26.2%
openai/whisper-base	81.3%	2.0%	20.4%
openai/whisper-small (3× larger)	90.7%	1.3%	15.0%
this model (PyTorch, 5-best beam)	97.3%	1.3%	4.2%
this model (8-bit ONNX, greedy — the browser build)	96.0%	1.3%	5.4%

Held-out synthetic voices (never trained on): 99.0–99.7% of commands understood (stock whisper-base: 71–87%). LibriSpeech dev-clean: 7.5% WER (stock: 8.2%) — specialised without forgetting ordinary English.

One real speaker and 150 clips is a small test set (≈±3 points); treat the real-speech numbers as an estimate.

Training data

No public dataset of spoken chess commands exists, so the core data is synthetic:

• Chess commands: 40k generated utterances (random legal positions; moves in many phrasings, questions, yes/no, corrections, wake phrases), every label validated by the app's parser/matcher, voiced by Piper en_US-libritts_r-medium (904 speakers, split by speaker), with varied rate and prosody.
• Accent fix (round 2): 8k rank-3 commands voiced with "tree"/"tri" for "three" — every real-speech error of the first model was a rank-3 square said that way.
• Real speech where it overlaps: LibriSpeech (train-clean-100 slice, CC BY 4.0), Google Speech Commands v2 ("yes", "no", "one"–"eight"; CC BY 4.0), CMU AN4 (spelled letters and digits).
• Augmentation: OpenSLR 28 room impulse responses and noises (Apache 2.0), band-limiting, gain.

Training: 4,000 steps, batch 16, lr 1e-5, fp16, on one RTX 2060 (~45 min). Pipeline: ml/.

Files

Path	What
model.safetensors, config.json, tokenizer files	PyTorch weights for 🤗 Transformers
onnx/encoder_model_quantized.onnx, onnx/decoder_model_merged_quantized.onnx	8-bit ONNX for transformers.js (dtype q8), 98 MB
wakeword/wakeword.onnx, wakeword/wakeword.json	Wake-word detector: 1.5 s of 16 kHz audio in, P(wake phrase) out; log-mel is computed inside the network. Threshold 0.8.

Usage

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="AudiJRahardja/voice-chess-whisper-base")
asr("move.wav", generate_kwargs={"language": "en", "task": "transcribe"})  # -> "knight to f3"

import { pipeline } from '@huggingface/transformers';
const asr = await pipeline('automatic-speech-recognition', 'AudiJRahardja/voice-chess-whisper-base', { dtype: 'q8' });
const { text } = await asr(float32Audio16k, { language: 'english', task: 'transcribe' });

Output is lowercase without punctuation, with squares written as f3 — the form the Voice Chess parser reads.

Wake word

At threshold 0.8: ~1% missed wake phrases from held-out training-voice speakers, ~15% from unseen voices, ~2 false triggers per hour of LibriSpeech. In the app a false trigger only starts listening — nothing is played without a spoken "yes".

Limitations

• English only, chess vocabulary only: it is meant to be bad at other domains' jargon.
• The real-speech test set is a single speaker; accents unlike the training voices may do worse.
• Designed to be used with a confirmation step: the app reads every move back and plays it only after "yes".