# BERT: AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

**URL:** <https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362>\
**Category:** Beginners\
**Created:** [July 14, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362 "2021-07-14T20:12:22Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 14, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/1 "2021-07-14T20:12:22Z")

</div>

I am trying to freeze some layers of my masked language model using the following code:

```auto
for param in model.bert.parameters():
    param.requires_grad = False

```

However, when I execute the code above, I get this error:

```auto
AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

```

In my code, I have the following imports for my masked language model, but I am unsure what is causing the error above:

```auto
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

```

So far, I have tried to replace `bert` with `model` in my code, but that did not work.

Any help would be good.

Thanks.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [July 15, 2021, 12:01pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/2 "2021-07-15T12:01:54Z")

</div>

The name of the body of the model `roberta` for Roberta models, not `bert`. So you should loop on `for param in model.roberta.parameters()`. In general, the attribute that is model agnostic is `base_model`, so `for param in model.base_model.parameters()` should work anywhere.

---

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 15, 2021, 2:20pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/3 "2021-07-15T14:20:58Z")

</div>

Okay, thanks for that. Now, is it possible to freeze just the top layer or bottom layer of the BERT model?

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 17, 2021, 6:45pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/4 "2021-07-17T18:45:04Z")

</div>

Yes, you can use the `names_parameters()` method for that.  
It gives you the names of the parmeters along with the paramters themselves,  
so you can filter only the paramters of the top / bottom layer based on their names and freeze them.

---

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 19, 2021, 1:52pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/5 "2021-07-19T13:52:02Z")

</div>

Hello, thanks for the reply. So, do I just simply add this code?

```auto
names_parameters()

```

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 19, 2021, 3:29pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/6 "2021-07-19T15:29:06Z")

</div>

No, you should iterate over them.  
Replcae the following line: `for param in model.roberta.parameters():`  
with: `for name, param in model.roberta.named_parameters():`.  
Then filter the parameters you want to freeze using the `name` variable.

You may print the names to see how they look like, and then come up with a condition that will filter the ones you need.

Hope it helps 🙂 .

---

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 19, 2021, 4:54pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/7 "2021-07-19T16:54:56Z")

</div>

Hello, thanks very much for that explanation and solution - it has really helped me quite a lot. In terms of filtering parameter, say my dataset has emojis, can I filter emojis? Like, for example, this 😀 emoji?

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 20, 2021, 10:11am UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/8 "2021-07-20T10:11:43Z")

</div>

Amm could you please elaborate more? I’m not sure I understood your question.  
BTW If it is an unrelated question to the original question of this post, you may ask it in a new one.

---

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 21, 2021, 3:58pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/9 "2021-07-21T15:58:35Z")

</div>

Apologies, basically I am trying to do masked language modelling using emojis, but when I deploy my model, the predicted tokens only show words, not emojis; hence, I think that the emojis are not frequent enough in the vocabulary, causing them to be less likely for the masked prediction. Therefore, I was wondering if I could freeze some layers of my BERT model to get the less frequent tokens, which are emojis, to be the top predictions when I deploy my masked language model.

---

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 25, 2021, 11:57am UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/10 "2021-07-25T11:57:05Z")

</div>

@Yuti - How would I go about filtering characters ?

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 25, 2021, 12:24pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/11 "2021-07-25T12:24:10Z")

</div>

Have you tried increasing the masking probabilty for emojis?  
If you have controll over the masking when training your model, you can increase the probability an emoji will be masked and then the model will output higher probabilities for emojis.

I’m not sure this is a good solution but you may try it.

---

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [July 25, 2021, 5:03pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/13 "2021-07-25T17:03:36Z")

</div>

@Yuti In my code, I have this line which allows me to adjust the masked language probability:

```auto
from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)

```

However, if I adjusted it, I am unsure how to do it for the emojis.

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 25, 2021, 6:12pm UTC](https://discuss.huggingface.co/t/bert-attributeerror-robertaformaskedlm-object-has-no-attribute-bert/8362/14 "2021-07-25T18:12:33Z")

</div>

Changing the `mlm_probability` argument wont give you the result you need,  
but I think you can create a sub class of `DataCollatorForLanguageModeling` that does the emoji masking.

You can find the source code for `DataCollatorForLanguageModeling` [here](https://huggingface.co/transformers/_modules/transformers/data/data_collator.html#DataCollatorForLanguageModeling).