Transformers documentation

MOSS-Transcribe-Diarize

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Model ini disumbangkan ke Hugging Face Transformers pada 2026-10-14.

MOSS-Transcribe-Diarize

Ikhtisar

MOSS-Transcribe-Diarize 0.9B adalah model pemahaman audio end-to-end untuk transkripsi multi-pembicara format panjang, diarisasi pembicara, dan timestamp. Model ini menggabungkan encoder bergaya Whisper, langkah penggabungan frame 4×, projektor multi-modal, dan model bahasa Qwen3.

Checkpoint model tersedia di: itazap/MOSS-Transcribe-Diarize-HF

Kemampuan utama meliputi:

Transkripsi dan diarisasi gabungan dalam satu kali proses, dengan segmen yang diformat sebagai [start][S01]text[end].
Audio berdurasi panjang melalui chunking Whisper-window di hulu pada processor; model merakit ulang chunk per sampel.
Transkripsi yang dapat diprompt melalui apply_transcription_request atau template chat.
Dukungan multibahasa untuk lebih dari 50 bahasa.

Model ini disumbangkan oleh Ita Zaporozhets dari tim Hugging Face. Lihat model card dan OpenMOSS repository untuk detail lebih lanjut.

Penggunaan

Sebagian besar contoh di bawah menggunakan apply_transcription_request, sebuah fungsi praktis yang membangun chat template untuk Anda. Lihat Penggunaan lanjutan dengan chat template untuk pemanggilan apply_chat_template yang setara.

Transkripsi dan diarisasi dasar

Setiap giliran pembicara didekode sebagai segmen [start][S0N]text[end]. processor.decode dapat mengembalikan string hasil dekode mentah, segmen yang telah diurai sebagai daftar dict (return_format="parsed"), atau hanya transkripsi yang digabungkan (return_format="transcription_only").

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/podcast_sample.wav"
inputs = processor.apply_transcription_request(audio_url)
inputs = inputs.to(model.device, dtype=model.dtype)

outputs = model.generate(**inputs, do_sample=False, max_new_tokens=512)
generated_ids = outputs[:, inputs.input_ids.shape[1] :]

decoded = processor.decode(generated_ids, skip_special_tokens=True)
print("RAW OUTPUT:", decoded)

parsed = processor.decode(generated_ids, return_format="parsed")[0]
print("\nPARSED (list of dicts):")
for segment in parsed:
    print(segment)

transcription = processor.decode(generated_ids, return_format="transcription_only")
print("\nTRANSCRIPTION ONLY:", transcription[0])

"""
RAW OUTPUT: ["[0.48][S01] Hey everyone, welcome back to the show.[2.43][2.46][S01] Today, we're talking about the best way to learn a new skill.[5.22][5.43][S01] I'm here with Jordan to dig into it.[7.14][8.40][S02] Thanks for having me.[9.42][9.63][S02] My take is that consistency beats intensity every single time.[12.87][13.14][S01] That makes sense.[13.92][13.92][S01] Even 20 minutes a day adds up over a few months.[16.17][16.32][S02] Exactly, and writing down small wins keeps you motivated along the way.[19.62]"]

PARSED (list of dicts):
{'Start': 0.48, 'End': 2.43, 'Speaker': 1, 'Content': 'Hey everyone, welcome back to the show.'}
{'Start': 2.46, 'End': 5.22, 'Speaker': 1, 'Content': "Today, we're talking about the best way to learn a new skill."}
{'Start': 5.43, 'End': 7.14, 'Speaker': 1, 'Content': "I'm here with Jordan to dig into it."}
{'Start': 8.4, 'End': 9.42, 'Speaker': 2, 'Content': 'Thanks for having me.'}
{'Start': 9.63, 'End': 12.87, 'Speaker': 2, 'Content': 'My take is that consistency beats intensity every single time.'}
{'Start': 13.14, 'End': 13.92, 'Speaker': 1, 'Content': 'That makes sense.'}
{'Start': 13.92, 'End': 16.17, 'Speaker': 1, 'Content': 'Even 20 minutes a day adds up over a few months.'}
{'Start': 16.32, 'End': 19.62, 'Speaker': 2, 'Content': 'Exactly, and writing down small wins keeps you motivated along the way.'}

TRANSCRIPTION ONLY: Hey everyone, welcome back to the show. Today, we're talking about the best way to learn a new skill. I'm here with Jordan to dig into it. Thanks for having me. My take is that consistency beats intensity every single time. That makes sense. Even 20 minutes a day adds up over a few months. Exactly, and writing down small wins keeps you motivated along the way.
"""

Prompt dan hotwords kustom

apply_transcription_request menerima keywords untuk membiaskan transkripsi ke arah istilah tertentu (misalnya nama diri atau kosakata spesifik domain), yang dirender sebagai 热词列表： (“hotword list”) dalam template chat. prompt kustom dapat dilewatkan bersamanya untuk konteks tambahan.

Di bawah ini kami mentranskripsi klip audio yang menyebut “Yorùbá”, membandingkan dengan dan tanpa melewatkan sebagai kata kunci.

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/languages_sample.wav"

# Without keywords
inputs = processor.apply_transcription_request(audio_url)
inputs = inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
transcription = processor.decode(outputs[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(f"WITHOUT keywords: {transcription[0]}")

# With keywords
inputs = processor.apply_transcription_request(audio_url, keywords=["Mandarin", "Yorùbá"])
inputs = inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
transcription = processor.decode(outputs[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(f"WITH keywords   : {transcription[0]}")

"""
WITHOUT keywords: Let's talk about the most spoken languages in the world. Mandarin Chinese has the most native speakers, followed by Spanish, English, and Hindi. In West Africa, one of the most widely spoken languages is Yoruba, with over 50 million speakers.
WITH keywords   : Let's talk about the most spoken languages in the world. Mandarin Chinese has the most native speakers, followed by Spanish, English, and Hindi. In West Africa, one of the most widely spoken languages is Yorùbá, with over 50 million speakers.
"""

Penggunaan lanjutan dengan template chat

apply_transcription_request ekuivalen dengan chat template yang memiliki giliran user hanya berisi audio:

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "url": audio_url},
        ],
    },
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
).to(model.device, dtype=model.dtype)

outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
decoded = processor.decode(
    outputs[:, inputs.input_ids.shape[1] :],
    skip_special_tokens=True,
)
print(decoded)

Inferensi batch

Lewatkan daftar audio, dan secara opsional daftar prompt dan/atau kata kunci yang sesuai, untuk mentranskripsi batch dalam satu panggilan. Setel entri ke None untuk melewatinya pada sampel tersebut.

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
# `from_pretrained` is used here for convenience, the model can also be built from a config directly.
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio = [
    "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav",
    "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/languages_sample.wav",
]
keywords = [None, ["Mandarin", "Yorùbá"]]

inputs = processor.apply_transcription_request(audio, keywords=keywords)
inputs = inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
transcription = processor.decode(outputs[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(transcription)

Pelatihan

Lewatkan transkripsi target, dalam format output model yang diberi penanda waktu dan penanda pembicara, ke apply_transcription_request dengan transcription. labels yang dikembalikan digunakan untuk menghitung loss.

from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
model.train()

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
transcription = (
    "[0.00][S01] Welcome to this demonstration of automatic speech recognition and speaker diarization using an "
    "end-to-end transformer model.[7.47]"
)

inputs = processor.apply_transcription_request(audio_url, transcription=transcription)
inputs.to(model.device, dtype=model.dtype)

loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()

Torch compile

Model dapat dikompilasi untuk inferensi/pelatihan yang lebih cepat. device_map="auto" membungkus jalur maju dengan hook accelerate yang menyebabkan kompilasi ulang berulang, jadi muat model langsung pada satu perangkat saja.

Pada B200, kami mengamati percepatan sekitar 2,1x untuk ukuran batch 4.

import time

import torch

from transformers import AutoProcessor, AutoModelForCausalLM


model_id = "itazap/MOSS-Transcribe-Diarize-HF"

num_warmup = 5
num_runs = 20

# Load processor + model
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to("cuda")

# Prepare static inputs
audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
inputs = processor.apply_transcription_request([audio_url] * 4).to(model.device, torch.bfloat16)

# Benchmark without compile
print("Warming up without compile...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs)

torch.accelerator.synchronize()

print("\nBenchmarking without torch.compile...")
torch.accelerator.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs)
torch.accelerator.synchronize()
no_compile_time = (time.time() - start) / num_runs
print(f"Average time without compile: {no_compile_time:.4f}s")

# Benchmark with compile
print("\nCompiling model...")
model = torch.compile(model)

print("Warming up with compile (includes graph capture)...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs)

torch.accelerator.synchronize()

print("\nBenchmarking with torch.compile...")
torch.accelerator.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs)
torch.accelerator.synchronize()
compile_time = (time.time() - start) / num_runs
print(f"Average time with compile: {compile_time:.4f}s")

speedup = no_compile_time / compile_time
print(f"\nSpeedup: {speedup:.2f}x")

Untuk transkripsi autoregresif, torch.compile mempercepat langkah maju per token di dalam generate dengan menyediakan objek CompileConfig alih-alih mengompilasi model.forward secara langsung.

import torch
from transformers import AutoProcessor, AutoModelForCausalLM, CompileConfig

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
num_warmup = 3

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to("cuda")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
inputs = processor.apply_transcription_request([audio_url] * 4).to(model.device, torch.bfloat16)

compile_config = CompileConfig()

# Warmup
for _ in range(num_warmup):
    _ = model.generate(**inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config)

# Apply model
generated_ids = model.generate(
    **inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config
)
transcription = processor.decode(generated_ids[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(transcription)

Transformers

MOSS-Transcribe-Diarize

Ikhtisar

Penggunaan

Transkripsi dan diarisasi dasar

Prompt dan hotwords kustom

Penggunaan lanjutan dengan template chat

Inferensi batch

Pelatihan

Torch compile

MossTranscribeDiarizeConfig

kelas transformers.MossTranscribeDiarizeConfig

MossTranscribeDiarizeAudioConfig

kelas transformers.MossTranscribeDiarizeAudioConfig

MossTranscribeDiarizeAudioModel

kelas transformers.MossTranscribeDiarizeAudioModel

forward

MossTranscribeDiarizeFeatureExtractor

kelas transformers.MossTranscribeDiarizeFeatureExtractor

__call__

MossTranscribeDiarizeProcessor

kelas transformers.MossTranscribeDiarizeProcessor

__call__

apply_chat_template

apply_transcription_request

MossTranscribeDiarizeModel

kelas transformers.MossTranscribeDiarizeModel

forward

get_audio_features

MossTranscribeDiarizeForConditionalGeneration

kelas transformers.MossTranscribeDiarizeForConditionalGeneration

forward

get_audio_features

call

call