# Albert pre-train convergence problem

**URL:** <https://discuss.huggingface.co/t/albert-pre-train-convergence-problem/436>\
**Category:** 🤗Transformers\
**Created:** [July 23, 2020, 4:36am UTC](https://discuss.huggingface.co/t/albert-pre-train-convergence-problem/436 "2020-07-23T04:36:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yl-to](https://avatars.discourse-cdn.com/v4/letter/y/7ba0ec/32.png) [@yl-to](https://discuss.huggingface.co/u/yl-to)\
**Post date:** [July 23, 2020, 4:36am UTC](https://discuss.huggingface.co/t/albert-pre-train-convergence-problem/436/1 "2020-07-23T04:36:28Z")

</div>

My abert pre-train from scratch model can’t converge to 0 in even using wiki-Text2.

- The model training loss converged at 6.6 when using AlbertForMaskedLM as model class
- negative training loss when using AlbertForPretrain as model class

notice: I was deliberately set the eval dataset the same as training set for checking training loss at last run.  
I also raised a issue here:

> <https://github.com/huggingface/transformers/issues/5984>
>
> \# 🐛 Bug
> Albert pre-train convergence problem
> \- The model training loss converg…ed at 6.6 when using AlbertForMaskedLM as model class
> \- negative training loss when using AlbertForPretrain as model class
> 
> notice: I was deliberately set the eval dataset the same as training set for checking training loss at last run.
> \## Information
> Using AlbertForMaskedLM as model class, figure showed below:
> !\[image\](https://user-images.githubusercontent.com/23205976/88234831-7ca56f00-cc2e-11ea-8ab7-75ce3e841b88.png)
> 
> 
> Using AlbertForPretrain as model class, figure showed below:
> !\[image\](https://user-images.githubusercontent.com/23205976/88230876-0867cd00-cc28-11ea-90aa-a7412dd9c0c6.png)
> 
> Besides, when I was using the official \`run\_lanugage\_modeling.py\`, the training loss on wikiText-2 is also not converge to 0, it converged at 6.6 for several epochs.
> 
> Model I am using (Bert, XLNet ...):
> Albert
> Language I am using the model on (English, Chinese ...):
> English
> The problem arises when using:
> \* \[x\] the official example scripts: (give details below)
> \* \[x\] my own modified scripts: (give details below)
> 
> The tasks I am working on is:
> \* \[x\] an official task: wikiText-2
> \* \[\] my own task or dataset: (give details below)
> 
> \## To reproduce
> \`\`\`
> from transformers import (
> AlbertConfig,
> AlbertTokenizer,
> BertTokenizer,
> AlbertForPreTraining,
> AlbertForMaskedLM,
> LineByLineTextDataset,
> TextDataset,
> DataCollatorForLanguageModeling,
> Trainer,
> TrainingArguments
> )
> import math
> 
> albert\_base\_configuration = AlbertConfig(
> hidden\_size=768,
> num\_attention\_heads=12,
> intermediate\_size=3072,
> )
> 
> tokenizer = AlbertTokenizer.from\_pretrained('albert-base-v2')
> \# model = AlbertForPreTraining(config=albert\_base\_configuration)
> model = AlbertForMaskedLM(config=albert\_base\_configuration)
> train\_dataset = LineByLineTextDataset(
> tokenizer=tokenizer,
> file\_path="/home/ubuntu/data\_local/wikitext-2-raw/wiki.train.raw",
> block\_size=512,
> )
> eval\_dataset = LineByLineTextDataset(
> tokenizer=tokenizer,
> file\_path="/home/ubuntu/data\_local/wikitext-2-raw/wiki.test.raw",
> block\_size=512,
> )
> 
> data\_collator = DataCollatorForLanguageModeling(
> tokenizer=tokenizer, mlm=True, mlm\_probability=0.15
> )
> 
> training\_args = TrainingArguments(
> output\_dir="./results/new",
> overwrite\_output\_dir=True,
> num\_train\_epochs=5,
> per\_gpu\_train\_batch\_size=5,
> save\_steps=10\_000,
> save\_total\_limit=1,
> logging\_steps=100,
> learning\_rate=1.76e-3
> )
> 
> trainer = Trainer(
> model=model,
> args=training\_args,
> data\_collator=data\_collator,
> train\_dataset=train\_dataset,
> eval\_dataset=eval\_dataset,
> prediction\_loss\_only=True,
> )
> 
> trainer.train()
> trainer.save\_model("./results/new")
> 
> eval\_output = trainer.evaluate()
> perplexity = math.exp(eval\_output\["eval\_loss"\])
> print({"loss": eval\_output\["eval\_loss"\]})
> result = {"perplexity": perplexity}
> print(result)
> \`\`\`
> Steps to reproduce the behavior:
> 
> 1.Download Albert here https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/
> 2.Run the script
> 
> \<!-- If you have code snippets, error messages, stack traces please provide them here as well.
> Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
> Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.--\>
> 
> \## Expected behavior
> \- The training loss should converge to 0 in tiny datasets.
> \- Cross entropy should be always positive and eventually converge to zero.
> 
> 
> \## Environment info
> \<!-- You can run the command \`transformers-cli env\` and copy-and-paste its output below.
> Don't forget to fill out the missing fields in that output! --\>
>      
> \- \`transformers\` version: v3.02
> \- Platform: AWS instance
> \- Python version: 3.7.7
> \- PyTorch version (GPU?): 1.5.1 with GPU
> \- Tensorflow version (GPU?): None
> \- Using GPU in script?: Yes, 8 GPU in total
> \- Using distributed or parallel set-up in script? Not sure.

---

<div class="post-metadata">

**Author:** ![sxdmit](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@sxdmit](https://discuss.huggingface.co/u/sxdmit)\
**Post date:** [December 6, 2024, 11:43am UTC](https://discuss.huggingface.co/t/albert-pre-train-convergence-problem/436/2 "2024-12-06T11:43:50Z")

</div>

i believe it might due to your inappropriate setting on lr or the size of your dataset, you should take a careful look into your training