# Pre-Train LayoutLM **URL:** \ **Category:** Beginners\ **Created:** [October 2, 2020, 11:47pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350 "2020-10-02T23:47:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sharathmk99](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sharathmk99](https://discuss.huggingface.co/u/sharathmk99)\ **Post date:** [October 2, 2020, 11:47pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/1 "2020-10-02T23:47:16Z")
Hello, We are using pretrained LayoutLM model which is working well with only English Language. We have many forms or invoices from different languages. How can i pre-train LayoutLM model with my own corpus? Thank you. ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [October 3, 2020, 11:40am UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/2 "2020-10-03T11:40:07Z")
Hi sharathmk99, LayoutLM model is not currently available in the huggingface transformers library. If you want to add it that should be possible (though not simple). Alternatively, you could put in a suggestion and hope that someone else will incorporate it. If you decide instead to pre-train a LayoutLM model using native Tensorflow or native PyTorch, the first question is whether you have enough data. How large is your corpus? If your corpus is not large enough, you might be better off using a different model that has been pre-trained for the language(s) you need. Do you definitely want to pre-train (from randomly-initialized) or would it work to fine-tune? I don’t know what results people get for fine-tuning with a new language. I expect it would not work at all if the alphabet is different, but it might be at least partly effective if the languages are quite similar (eg english + french which have almost the same alphabet and many of the same word-pieces). ---
**Author:** ![sharathmk99](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sharathmk99](https://discuss.huggingface.co/u/sharathmk99)\ **Post date:** [October 3, 2020, 1:51pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/3 "2020-10-03T13:51:08Z")
Hi @rgwatwormhill, LayoutLM is available in the huggingface transformers right? Link: [https://huggingface.co/transformers/model\_doc/layoutlm.html](https://huggingface.co/transformers/model_doc/layoutlm.html) ---
**Author:** ![sharathmk99](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sharathmk99](https://discuss.huggingface.co/u/sharathmk99)\ **Post date:** [October 3, 2020, 2:41pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/4 "2020-10-03T14:41:03Z")
Hi @rgwatwormhill, I’m planning to use [https://www.cs.cmu.edu/~aharley/rvl-cdip/](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset by adding my own domain data. Planning to train multilingual model with multiple languages. ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [October 4, 2020, 7:01pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/5 "2020-10-04T19:01:03Z")
hi @sharathmk99, Sorry, I must have been looking at an old version of the documentation, or something. You are correct, and it’s clearly present on the page you’ve linked. (It doesn’t seem to be on the model summary page [https://huggingface.co/transformers/model\_summary.html](https://huggingface.co/transformers/model_summary.html) , but there are two options on the pretrained-models page [https://huggingface.co/transformers/pretrained\_models.html](https://huggingface.co/transformers/pretrained_models.html) ). To train from scratch, you need to start by defining your model based on the LayoutLM config. Have you read the Training and Fine Tuning page [https://huggingface.co/transformers/training.html](https://huggingface.co/transformers/training.html) . The amount of data you have available will be important. ---
**Author:** ![sharathmk99](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sharathmk99](https://discuss.huggingface.co/u/sharathmk99)\ **Post date:** [October 4, 2020, 8:17pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/6 "2020-10-04T20:17:15Z")
Hi @rgwatwormhill, Thank you for your response. Training and Fine Tuning page doesn’t show how to pre train the model. It shows how to fine tune the model. Thank you! ---
**Author:** ![tuner007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tuner007/32/2168_2.png) [@tuner007](https://discuss.huggingface.co/u/tuner007)\ **Post date:** [November 30, 2020, 6:39pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/7 "2020-11-30T18:39:45Z")
Hi @sharathmk99 , By any chance you tried fine tuning layout\_lm ? If yes, did you use transformers library ? or official source ? thanks ! ---
**Author:** ![g3casey](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@g3casey](https://discuss.huggingface.co/u/g3casey)\ **Post date:** [November 5, 2021, 8:05pm UTC](https://discuss.huggingface.co/t/pre-train-layoutlm/1350/8 "2021-11-05T20:05:06Z")
@tuner007 we were able to fine tune the model using the sample notebook code in COLAB. We had some trouble getting it to run on other platforms though.