# Whisper model fine tuning

**URL:** <https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045>\
**Category:** Models\
**Created:** [November 14, 2022, 12:21pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045 "2022-11-14T12:21:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ankit-Kumar-Saini](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Ankit-Kumar-Saini](https://discuss.huggingface.co/u/Ankit-Kumar-Saini)\
**Post date:** [November 14, 2022, 12:21pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/1 "2022-11-14T12:21:41Z")

</div>

There is an excellent blog on fine-tuning whisper model on hindi (Devanagari) dataset by @sanchit-gandhi .  
I need to fine-tune whisper model on the hinglish dataset (mix of english and hindi). The supervised data contains hinglish annotations (not Devanagari).  
Is there a way to fine-tune whisper model on this dataset? Can I replace the model tokenizer with my custom tokenizer to proceed with fine-tuning process?  
Please suggest a way to proceed on the task.

---

<div class="post-metadata">

**Author:** ![sanchit-gandhi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sanchit-gandhi/32/21280_2.png) [@sanchit-gandhi](https://discuss.huggingface.co/u/sanchit-gandhi)\
**Post date:** [November 14, 2022, 1:48pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/2 "2022-11-14T13:48:03Z")

</div>

Hey @Ankit-Kumar-Saini!

To clarify, does your dataset contain Hindi characters **and** Roman ones (i.e. the letters a-z)? Just Hindi ones? Or just Roman ones (a-z)?

Likelihood is we won’t need a new tokenizer - the Whisper tokenizer has the Hindi alphabet and Roman alphabet among others. It’s just how we initialise the tokenizer that changes.

FYI this is the update blog post: [Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper)

---

<div class="post-metadata">

**Author:** ![Ankit-Kumar-Saini](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Ankit-Kumar-Saini](https://discuss.huggingface.co/u/Ankit-Kumar-Saini)\
**Post date:** [November 14, 2022, 3:13pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/3 "2022-11-14T15:13:44Z")

</div>

Dataset contains only roman ones (i.e. the letters a-z).  
For example: “aapka age kitna hai”.  
How should I tokenize this text?

---

<div class="post-metadata">

**Author:** ![sanchit-gandhi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sanchit-gandhi/32/21280_2.png) [@sanchit-gandhi](https://discuss.huggingface.co/u/sanchit-gandhi)\
**Post date:** [November 14, 2022, 5:34pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/4 "2022-11-14T17:34:19Z")

</div>

Okay! I probably wound’t build a new tokeniser - I would first try using the existing Whisper one as it contains the entire Roman alphabet and more words in word-piece form.

You need to change this part of the script:

> **[Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper#combine-to-create-a-whisperprocessor)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

I would first try omitting the language and task code:

```auto
processor = WhisperProcessor.from_pretrained("openai/whisper-small")

```

If you train for long enough, the model should learn the correct output alphabet.

---

<div class="post-metadata">

**Author:** ![Deveshp](https://avatars.discourse-cdn.com/v4/letter/d/e5b9ba/32.png) [@Deveshp](https://discuss.huggingface.co/u/Deveshp)\
**Post date:** [December 17, 2022, 6:51am UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/5 "2022-12-17T06:51:41Z")

</div>

Hi @sanchit-gandhi, Thanks for your blog post, it was excellent but i am facing issue in fine tuning the whisper model on my data and need your support. I have audio files and corresponding meta-data csv file with two columns, one is wav\_filename which contains the path of audios and corresponding text in second column as transcript and i am getting below issue while running the code: FileNotFoundError: [Errno 2] No such file or directory: ‘wav\_filename’. It will be great help. Thanks.

---

<div class="post-metadata">

**Author:** ![sanchit-gandhi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sanchit-gandhi/32/21280_2.png) [@sanchit-gandhi](https://discuss.huggingface.co/u/sanchit-gandhi)\
**Post date:** [December 19, 2022, 4:01pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/6 "2022-12-19T16:01:40Z")

</div>

Hey @Deveshp! Welcome to the HF forum and thanks for asking a great question! 🤗 Would you mind opening a new forum post for your question and we can shift the discussion there?

The reason being that it makes searching for issues much easier if we keep each forum post related to one question. This way, if someone has the same issue as you later down the line they can sift through all the previous issues and hopefully find our discussion! Thanks!

---

<div class="post-metadata">

**Author:** ![crossdelenna](https://avatars.discourse-cdn.com/v4/letter/c/e19adc/32.png) [@crossdelenna](https://discuss.huggingface.co/u/crossdelenna)\
**Post date:** [February 23, 2024, 9:50pm UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/7 "2024-02-23T21:50:29Z")

</div>

Hey Ankit, I know I am replying to a two-year-old post but this is exactly what I needed and Google gave me this link. Have you found the solution, what was the WER and how long did you train? Most research and ChatGPT suggest it’s better to train on Devnagri but the base model of even Whisper 3 for multi-lingual is so bad that I am not optimistic anymore. Preparing the data going to consume so much time that I need to be sure whether it will be worth it LOL.

---

<div class="post-metadata">

**Author:** ![shivamtiwari2112](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@shivamtiwari2112](https://discuss.huggingface.co/u/shivamtiwari2112)\
**Post date:** [June 8, 2024, 4:58am UTC](https://discuss.huggingface.co/t/whisper-model-fine-tuning/26045/8 "2024-06-08T04:58:35Z")

</div>

Do I also need to remove language, and task attribute for tokenizer , like you said to remove from the processor?
