# Pretraining ALBERT

**URL:** <https://discuss.huggingface.co/t/pretraining-albert/5607>\
**Category:** Intermediate\
**Created:** [April 19, 2021, 6:55am UTC](https://discuss.huggingface.co/t/pretraining-albert/5607 "2021-04-19T06:55:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![stoddur](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@stoddur](https://discuss.huggingface.co/u/stoddur)\
**Post date:** [April 19, 2021, 6:55am UTC](https://discuss.huggingface.co/t/pretraining-albert/5607/1 "2021-04-19T06:55:54Z")

</div>

Hello!

I have a question regarding using the transformers library to pretrain ALBERT. I have been using RoBERTa for some while now which I have pretrained with custom data with run\_mlm.py from the examples directory which is fine since RoBERTa only uses MLM loss when pretraining. However, ALBERT adds sentence order prediction (SOP) which is not implemented in run\_mlm.py. Are there any examples on implementing SOP which I have overlooked in the transformers library? If not, anyone care to share a code example of how to implement this? If not I would have to dive a bit deeper to implement from scratch but I’m hoping I won’t have to 🙂

Thanks!

Edit:  
I found a DataCollatorForSOP which appears to solve this task: [transformers/data\_collator.py at d9c62047a8d75e18d2849d345ab3394875a712ef · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/src/transformers/data/data_collator.py#L515)

Would be great to have a seperate file in examples which implements the pretraining for ALBERT.

---

<div class="post-metadata">

**Author:** ![hdm](https://avatars.discourse-cdn.com/v4/letter/h/8797f3/32.png) [@hdm](https://discuss.huggingface.co/u/hdm)\
**Post date:** [February 6, 2022, 1:26pm UTC](https://discuss.huggingface.co/t/pretraining-albert/5607/2 "2022-02-06T13:26:53Z")

</div>

May I ask how you used that DataCollator? I’m also trying to pretrain ALBERT but I am also facing these difficulties.

---

<div class="post-metadata">

**Author:** ![ibraheemmoosa](https://avatars.discourse-cdn.com/v4/letter/i/a587f6/32.png) [@ibraheemmoosa](https://discuss.huggingface.co/u/ibraheemmoosa)\
**Post date:** [February 16, 2022, 3:00pm UTC](https://discuss.huggingface.co/t/pretraining-albert/5607/3 "2022-02-16T15:00:08Z")

</div>

I have pretrained an ALBERT model last year. You do not need a special DataCollator for SOP. Just use the DataCollatorForLanguageModeling.