# Continue pre-training BERT **URL:** \ **Category:** Intermediate\ **Created:** [November 13, 2023, 10:42am UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053 "2023-11-13T10:42:19Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![DHar2023](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dhar2023/32/21143_2.png) [@DHar2023](https://discuss.huggingface.co/u/DHar2023)\ **Post date:** [November 13, 2023, 10:42am UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053/1 "2023-11-13T10:42:19Z")
Hello, I have a small portion of label data, and a much bigger set of unlabeled observations. I want to use the unlabeled samples in order to continue the pre-training of BERT, and then built a classifier on top of it. Following this post > [@Continue pre-training of Greek BERT with domain specific dataset \[clarified\]](https://discuss.huggingface.co/t/continue-pre-training-of-greek-bert-with-domain-specific-dataset-clarified/4030): > > Hello, I want to further pre-train Greek BERT of the library on a domain specific dataset that is not annotated in MLM task. The downstream task of BERT is Name Entity Recognition where I have a second annotated dataset to subsequently fine tune the model. I found that the library provides [scripts](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling) for that also. In the example RoBERTa is further trained on wikitext-2-raw-v1. As I saw [here](https://huggingface.co/datasets/wikitext), the dataset is formatted as: { “text”: “” The gold dollar or gold one @-@ dollar … } although, I down… I tried to use BertModel.from\_pretrained(‘bert-base-uncased’), and specifically ```auto model = BertModel.from_pretrained(HF_BERT_MODEL) model.cuda() optimizer = AdamW(model.parameters(), lr = 2e-5, eps = 1e-8 ) model.train() # For each batch of training data... for step, batch in enumerate(train_dataloader): b_input_ids = batch[0].to(device) b_input_mask = batch[1].to(device) b_labels = batch[2].to(device) model.zero_grad() result = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, return_dict=True) loss = result.loss torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() scheduler.step() >>>'BaseModelOutputWithPoolingAndCrossAttentions' object has no attribute 'loss' ``` I get the above error. My question is, how to do the fine-tuning? ---
**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\ **Post date:** [November 13, 2023, 12:16pm UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053/2 "2023-11-13T12:16:52Z")
Loss is available only when labels are provided to the model. ---
**Author:** ![DHar2023](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dhar2023/32/21143_2.png) [@DHar2023](https://discuss.huggingface.co/u/DHar2023)\ **Post date:** [November 13, 2023, 1:24pm UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053/3 "2023-11-13T13:24:44Z")
Thanks for your answer. The question is, though, how can we train the embeddings (using gradient descent) if there is no loss? ---
**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\ **Post date:** [November 13, 2023, 2:13pm UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053/4 "2023-11-13T14:13:45Z")
I don’t understand the question. Don’t you need to provide labels in order to calculate a loss? You either provide labels to the model so it can calculate the loss or you calculate loss yourself using the output of the model. So in your code above, for training I would expect to pass in the labels to model, the. get the loss as you are expecting and then call `loss.backward` ---
**Author:** ![DHar2023](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dhar2023/32/21143_2.png) [@DHar2023](https://discuss.huggingface.co/u/DHar2023)\ **Post date:** [November 13, 2023, 2:42pm UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053/5 "2023-11-13T14:42:24Z")
The idea was to continue the pre-training, which, according to BERT is masking words (around 15% of the provided texts) and trying to predict the masked tokens. To the best of my knowledge, this process is considered to be “self-supervised” and therefore you don’t implicitly provide labels, but instead they are inferred from the data. In this case, you still have a loss (otherwise how could you learn the embeddings). From your question though, I understand that I might need to mask tokens myself and add the masked token as the label. Am I correct? Care you to refer me to a notebook that shows an example? ---
**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\ **Post date:** [November 13, 2023, 6:34pm UTC](https://discuss.huggingface.co/t/continue-pre-training-bert/62053/6 "2023-11-13T18:34:14Z")
> **[Masked-Language Modelling With BERT](https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c)** > > Transformer models like BERT are incredibly powerful when fine-tuned for specific tasks using masked-language modeling (MLM), we explore how here. Or you can use the trainer class from huggingface as described in their guide > **[Masked language modeling](https://huggingface.co/docs/transformers/tasks/masked_language_modeling#train)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science.