Multy123/kitsune-transcribe-p0.3b-ja

Kitsune-Transcribe P-0.3B (Japanese speech-to-text, CTC)

A 309M-parameter Japanese speech recognition model: a CTC model, distilled from NVIDIA's Parakeet TDT-CTC 0.6B Japanese model on 11,533 hours of Japanese speech by Kitsune-Transcribe.

Model	Type	Params	M4 CER	Speed (RTFx, batched)	Repository
T-0.6B	encoder-decoder, from Cohere Transcribe	617M	9.42 %	763x	Multy123/kitsune-transcribe-t0.6b-ja
P-0.3B (this model)	CTC, from Parakeet	309M	9.73 %	5,848x	this repository
P-0.1B	CTC, from Parakeet	98M	11.45 %	9,544x	Multy123/kitsune-transcribe-p0.1b-ja
P-0.05B	CTC, from Parakeet	52M	16.35 %	11,800x	Multy123/kitsune-transcribe-p0.05b-ja

Results

Character error rate (CER, %), greedy decoding on the full test sets, lower is better. M4 is the mean of JSUT, CV8, ReazonSpeech and Galgame (neutral).

Model	Params	JSUT	CV8	ReazonSpeech	Galgame (neutral)	Galgame (all)	Emilia	M4
Kitsune P-0.3B (this model)	309M	7.19	8.62	10.12	12.99	19.29	40.32	9.73
Parakeet TDT-CTC 0.6B ja, CTC head (the teacher)	611M	6.71	7.58	9.71	13.32	19.64	40.61	9.33
Whisper large-v3	1.54B	7.19	8.51	14.50	12.41	21.90	30.65	10.65
Whisper large-v3-turbo	809M	7.30	9.30	10.70	13.07	21.52	39.34	10.09
Kotoba-Whisper v2.0	756M	8.39	9.23	11.66	13.17	20.02	39.18	10.61

Speed: 5,848x real time (RTFx: seconds of audio per second of processing). One run on one RTX 5090 (bf16, PyTorch eager, greedy): 200 test utterances (5.6 s on average) in batches of up to 400 s of audio, timed on the 1,000-hour study checkpoint of this architecture (a CTC model's speed does not depend on its weights). Speeds differ between machines: a second RTX 5090 host ran it at 6,654x. Not comparable with the Open ASR Leaderboard's RTFx.

Usage

Tested with transformers 5.13.1 (ParakeetForCTC). Input: 16 kHz mono audio.

import librosa
import torch
from transformers import AutoProcessor, ParakeetForCTC

repo = "Multy123/kitsune-transcribe-p0.3b-ja"
processor = AutoProcessor.from_pretrained(repo)
model = ParakeetForCTC.from_pretrained(repo, dtype=torch.bfloat16).to("cuda").eval()  # or dtype=torch.float32 on CPU

audio, _ = librosa.load("speech.wav", sr=16000, mono=True)  # resample to 16 kHz
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(model.device, model.dtype)
with torch.no_grad():
    ids = model.generate(**inputs)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Decoding is greedy CTC (argmax per 80 ms frame, repeats merged, blank id 3072 dropped). The model was trained on utterances of up to 30 s. Long-form tests used chunks of 8 s with 1.5 s overlap.

Quantised versions

The root holds the weights in bfloat16. Each folder holds the same model in another format, with its own config and tokenizer files and a quantization.json that records the recipe. M4 CER as in Results.

Folder	Weights	Activations	Size	M4 CER	Change vs bfloat16
(root)	bfloat16	bfloat16	617 MB	9.73 %
fp16/	float16 (encoder self-attention and norms kept in bfloat16)	float16 (encoder self-attention in float32)	617 MB	not measured
fp8-w8a8/	FP8 (E4M3)	FP8 (E4M3)	314 MB	9.70 %	-0.3 %
int8-w8a16/	int8	bfloat16	314 MB	9.70 %	-0.3 %
int8-w8a8/	int8	int8	314 MB	9.73 %	±0.0 %
nvfp4-w4a16/	NVFP4	bfloat16	179 MB	9.74 %	+0.1 %
nvfp4-w4a4/	NVFP4	NVFP4	179 MB	9.85 %	+1.3 %
mxfp4-w4a4/	MXFP4	MXFP4	170 MB	10.90 %	+12.0 %

fp16/ was exported again with its encoder self-attention in float32 (the format's current recipe); that export has not been scored. The earlier fp16 export, all in float16, scored 9.71 % (-0.2 %).

MXFP4 costs this model +12 %; the other scored formats stay within +1.3 %.

GPU support (tested on an RTX 5090 with torchao 0.18): fp8-w8a8/ needs compute capability 8.9 or newer (RTX 40xx / 50xx, L4, H100) and nvfp4-w4a4/ 10.0 or newer (Blackwell). On older GPUs torchao stops with an AssertionError.

# git clone https://github.com/Multysquid/Kitsune-Transcribe && git -C Kitsune-Transcribe checkout 76b28a1
# pip install torchao==0.18.0  # the CUDA kernels; not needed on CPU (tested with torch 2.14.0, transformers 5.13.1)
import sys
sys.path.insert(0, "Kitsune-Transcribe")

import librosa
import torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor
from kitsune.quant import load_quantized

repo, folder, device = "Multy123/kitsune-transcribe-p0.3b-ja", "int8-w8a16", "cuda"  # any folder (GPU support: see above); or device = "cpu"
path = f"{snapshot_download(repo, allow_patterns=[f'{folder}/*'])}/{folder}"
processor = AutoProcessor.from_pretrained(repo)
model, recipe = load_quantized(path, device)
# each format runs under the autocast its recipe was scored with: float16 for fp16/, bfloat16 for the others
amp = torch.float16 if recipe["format"] == "fp16" else torch.bfloat16

audio, _ = librosa.load("speech.wav", sr=16000, mono=True)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device)
with torch.no_grad(), torch.autocast(device, dtype=amp):
    ids = model.generate(**inputs)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Training

• Teacher: the CTC head of nvidia/parakeet-tdt_ctc-0.6b-ja. Per-frame soft targets (its top-k CTC posteriors), computed on the clean audio. The loss is KL to them plus 0.8 x CTC on the teacher's greedy path.
• Initialisation: Parakeet itself, pruned (16 of 24 encoder layers, FFN 4096 -> 2560; see Attribution).
• Data: 11,533 h (7.03 M utterances) of Japanese speech from Galgame_Speech_ASR, ReazonSpeech (large and small), Emilia-YODAS ja and Emolia ja. Rows were kept where Cohere Transcribe's transcript agreed well enough with a second model's.
• Schedule: 6 epochs, 211,720 optimizer steps, peak LR 2e-4: 300 warm-up steps, constant, then decayed to zero over the last 20 % of the steps. Early stopping on held-out dev loss (patience 6) did not trigger.
• Augmentation: SpecAugment, and: 20 % of the utterances of 3 s or more are cut inside a sentence (before a word, half of the cuts in a pause), so an input that ends mid-sentence gets no 。. Half of the micro-batches join 2-4 utterances into one input of up to 28 s, so sentence marks also occur inside long inputs. No noise or speaker mixing.

Attribution (CC BY 4.0)

This model is derived from parakeet-tdt_ctc-0.6b-ja by NVIDIA (nvidia/parakeet-tdt_ctc-0.6b-ja, revision 44edb27eea9317daf89333e75eb830db4b1cc298), licensed under the Creative Commons Attribution 4.0 International licence. NVIDIA does not endorse this derivative.

Changes made:

• converted to transformers ParakeetForCTC: the TDT prediction network and joint are dropped; the CTC head is kept;
• pruned: 16 of 24 encoder layers [0, 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 17, 18, 20, 21, 23], FFN 4096 -> 2560 (the exact layers and neurons are in student_meta.json); dropout and layer drop set to 0;
• then trained by distillation from the teacher's outputs on the data above (all weights updated);
• root weights stored in bfloat16 (BatchNorm statistics in float32); the folders fp16/, fp8-w8a8/, int8-w8a16/, int8-w8a8/, nvfp4-w4a16/, nvfp4-w4a4/ and mxfp4-w4a4/ hold the same weights converted to float16 or quantised to FP8, int8, NVFP4 or MXFP4 (see "Quantised versions"; each folder's quantization.json records the recipe);
• the tokenizer and processor files were generated from the .nemo's SentencePiece tokenizer and preprocessor settings with transformers' convert_nemo_to_hf.py (vocabulary unchanged; <blank> serves as the pad token, as the vocabulary has no <pad>); processor_config.json then has decoder_type set to ctc.

Licence

CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International; legal code: LICENSE-CC-BY-NC-4.0.txt). You may use, share and adapt this model, its quantised folders included, for non-commercial purposes, if you credit "Kitsune-Transcribe P-0.3B by Multy123", link the licence and indicate your changes. Commercial use is not allowed: part of the training data, Galgame_Speech_ASR (GPL-3.0 with conditions that carry over to models trained on it), forbids it, and part of Emolia's source (Amphion's Emilia) is CC BY-NC 4.0.

The base model's own material keeps its licence: NVIDIA's tokenizer and processor files (tokenizer.json, tokenizer_config.json, processor_config.json) stay under CC BY 4.0; see Attribution. Full terms, the third-party notices and the training data's licences: LICENSE.md.