Multy123/kitsune-transcribe-p0.05b-ja

Kitsune-Transcribe P-0.05B (Japanese speech-to-text, CTC)

A 52M-parameter Japanese speech recognition model: a CTC model, distilled from NVIDIA's Parakeet TDT-CTC 0.6B Japanese model on 11,533 hours of Japanese speech by Kitsune-Transcribe.

Model	Type	Params	M4 CER	Speed (RTFx, batched)	Repository
T-0.6B	encoder-decoder, from Cohere Transcribe	617M	9.42 %	763x	Multy123/kitsune-transcribe-t0.6b-ja
P-0.3B	CTC, from Parakeet	309M	9.73 %	5,848x	Multy123/kitsune-transcribe-p0.3b-ja
P-0.1B	CTC, from Parakeet	98M	11.45 %	9,544x	Multy123/kitsune-transcribe-p0.1b-ja
P-0.05B (this model)	CTC, from Parakeet	52M	16.35 %	11,800x	this repository

Results

Character error rate (CER, %), greedy decoding on the full test sets, lower is better. M4 is the mean of JSUT, CV8, ReazonSpeech and Galgame (neutral).

Model	Params	JSUT	CV8	ReazonSpeech	Galgame (neutral)	Galgame (all)	Emilia	M4
Kitsune P-0.05B (this model)	52M	13.24	16.59	20.19	15.38	21.61	47.73	16.35
Parakeet TDT-CTC 0.6B ja, CTC head (the teacher)	611M	6.71	7.58	9.71	13.32	19.64	40.61	9.33
Whisper large-v3	1.54B	7.19	8.51	14.50	12.41	21.90	30.65	10.65
Whisper large-v3-turbo	809M	7.30	9.30	10.70	13.07	21.52	39.34	10.09
Kotoba-Whisper v2.0	756M	8.39	9.23	11.66	13.17	20.02	39.18	10.61

Speed: 11,800x real time (RTFx: seconds of audio per second of processing). One run on one RTX 5090 (bf16, PyTorch eager, greedy): 200 test utterances (5.6 s on average) in batches of up to 400 s of audio, timed on the 1,000-hour study checkpoint of this architecture (a CTC model's speed does not depend on its weights). Speeds differ between machines: a second RTX 5090 host ran it at 15,362x. Not comparable with the Open ASR Leaderboard's RTFx.

Usage

Tested with transformers 5.13.1 (ParakeetForCTC). Input: 16 kHz mono audio.

import librosa
import torch
from transformers import AutoProcessor, ParakeetForCTC

repo = "Multy123/kitsune-transcribe-p0.05b-ja"
processor = AutoProcessor.from_pretrained(repo)
model = ParakeetForCTC.from_pretrained(repo, dtype=torch.bfloat16).to("cuda").eval()  # or dtype=torch.float32 on CPU

audio, _ = librosa.load("speech.wav", sr=16000, mono=True)  # resample to 16 kHz
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(model.device, model.dtype)
with torch.no_grad():
    ids = model.generate(**inputs)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Decoding is greedy CTC (argmax per 80 ms frame, repeats merged, blank id 3072 dropped). The model was trained on utterances of up to 30 s. Long-form tests used chunks of 8 s with 1.5 s overlap.

Quantised versions

The root holds the weights in bfloat16. Each folder holds the same model in another format, with its own config and tokenizer files and a quantization.json that records the recipe. M4 CER as in Results.

Folder	Weights	Activations	Size	M4 CER	Change vs bfloat16
(root)	bfloat16	bfloat16	104 MB	16.35 %
fp16/	float16 (encoder self-attention and norms kept in bfloat16)	float16 (encoder self-attention in float32)	104 MB	not measured
fp8-w8a8/	FP8 (E4M3)	FP8 (E4M3)	56 MB	16.29 %	-0.4 %
int8-w8a16/	int8	bfloat16	56 MB	16.34 %	-0.1 %
int8-w8a8/	int8	int8	56 MB	16.55 %	+1.2 %
nvfp4-w4a16/	NVFP4	bfloat16	34 MB	17.18 %	+5.1 %
nvfp4-w4a4/	NVFP4	NVFP4	34 MB	19.51 %	+19.4 %
mxfp4-w4a4/	MXFP4	MXFP4	33 MB	26.24 %	+60.5 %

fp16/ was exported again with its encoder self-attention in float32 (the format's current recipe), which has not been scored on this checkpoint. The earlier export ran every layer in float16 and decoded nothing (the self-attention scores overflow float16 in this model): use only this folder's recipe for float16.

The 4-bit formats cost this small model a lot (NVFP4 weights and activations +19 %, MXFP4 +61 %): prefer int8 or fp8.

GPU support (tested on an RTX 5090 with torchao 0.18): fp8-w8a8/ needs compute capability 8.9 or newer (RTX 40xx / 50xx, L4, H100) and nvfp4-w4a4/ 10.0 or newer (Blackwell). On older GPUs torchao stops with an AssertionError.

# git clone https://github.com/Multysquid/Kitsune-Transcribe && git -C Kitsune-Transcribe checkout 76b28a1
# pip install torchao==0.18.0  # the CUDA kernels; not needed on CPU (tested with torch 2.14.0, transformers 5.13.1)
import sys
sys.path.insert(0, "Kitsune-Transcribe")

import librosa
import torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor
from kitsune.quant import load_quantized

repo, folder, device = "Multy123/kitsune-transcribe-p0.05b-ja", "int8-w8a16", "cuda"  # any folder (GPU support: see above); or device = "cpu"
path = f"{snapshot_download(repo, allow_patterns=[f'{folder}/*'])}/{folder}"
processor = AutoProcessor.from_pretrained(repo)
model, recipe = load_quantized(path, device)
# each format runs under the autocast its recipe was scored with: float16 for fp16/, bfloat16 for the others
amp = torch.float16 if recipe["format"] == "fp16" else torch.bfloat16

audio, _ = librosa.load("speech.wav", sr=16000, mono=True)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device)
with torch.no_grad(), torch.autocast(device, dtype=amp):
    ids = model.generate(**inputs)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Training

• Teacher: the CTC head of nvidia/parakeet-tdt_ctc-0.6b-ja. Per-frame soft targets (its top-k CTC posteriors), computed on the clean audio. The loss is KL to them plus 0.8 x CTC on the teacher's greedy path.
• Initialisation: Parakeet itself, pruned (4 of 24 encoder layers, FFN 4096 -> 768; see Attribution).
• Data: 11,533 h (7.03 M utterances) of Japanese speech from Galgame_Speech_ASR, ReazonSpeech (large and small), Emilia-YODAS ja and Emolia ja. Rows were kept where Cohere Transcribe's transcript agreed well enough with a second model's.
• Schedule: planned 10 epochs at peak LR 1e-3 (1,000 warm-up steps, then constant). Early stopping on held-out dev loss (patience 6) triggered at step 67,449 (2.5 epochs) and started the decay to zero, which ended at step 80,939 (3.0 epochs).
• Augmentation: SpecAugment, and: 20 % of the utterances of 3 s or more are cut inside a sentence (before a word, half of the cuts in a pause), so an input that ends mid-sentence gets no 。. Half of the micro-batches join 2-4 utterances into one input of up to 28 s, so sentence marks also occur inside long inputs. Label-preserving acoustics (the targets stay those of the clean audio): background noise, music and songs (30 %, 0-20 dB SNR), background speech of 1-4 other talkers (15 %, 10-25 dB), room reverberation (20 %), gain -20 to +10 dB (30 %), MP3 / GSM / 8 kHz mu-law codecs (10 %), and trailing silence trimmed to a short random tail (30 %).

Attribution (CC BY 4.0)

This model is derived from parakeet-tdt_ctc-0.6b-ja by NVIDIA (nvidia/parakeet-tdt_ctc-0.6b-ja, revision 44edb27eea9317daf89333e75eb830db4b1cc298), licensed under the Creative Commons Attribution 4.0 International licence. NVIDIA does not endorse this derivative.

Changes made:

• converted to transformers ParakeetForCTC: the TDT prediction network and joint are dropped; the CTC head is kept;
• pruned: 4 of 24 encoder layers [0, 8, 15, 23], FFN 4096 -> 768 (the exact layers and neurons are in student_meta.json); dropout and layer drop set to 0;
• then trained by distillation from the teacher's outputs on the data above (all weights updated);
• root weights stored in bfloat16 (BatchNorm statistics in float32); the folders fp16/, fp8-w8a8/, int8-w8a16/, int8-w8a8/, nvfp4-w4a16/, nvfp4-w4a4/ and mxfp4-w4a4/ hold the same weights converted to float16 or quantised to FP8, int8, NVFP4 or MXFP4 (see "Quantised versions"; each folder's quantization.json records the recipe);
• the tokenizer and processor files were generated from the .nemo's SentencePiece tokenizer and preprocessor settings with transformers' convert_nemo_to_hf.py (vocabulary unchanged; <blank> serves as the pad token, as the vocabulary has no <pad>); processor_config.json then has decoder_type set to ctc.

Licence

CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International; legal code: LICENSE-CC-BY-NC-4.0.txt). You may use, share and adapt this model, its quantised folders included, for non-commercial purposes, if you credit "Kitsune-Transcribe P-0.05B by Multy123", link the licence and indicate your changes. Commercial use is not allowed: part of the training data, Galgame_Speech_ASR (GPL-3.0 with conditions that carry over to models trained on it), forbids it, and part of Emolia's source (Amphion's Emilia) is CC BY-NC 4.0.

The base model's own material keeps its licence: NVIDIA's tokenizer and processor files (tokenizer.json, tokenizer_config.json, processor_config.json) stay under CC BY 4.0; see Attribution. Full terms, the third-party notices and the training data's licences: LICENSE.md.