# Bert model on Acceptability Judgement Task || Optimizer Grouped Parameters

**URL:** <https://discuss.huggingface.co/t/bert-model-on-acceptability-judgement-task-optimizer-grouped-parameters/9905>\
**Category:** Beginners\
**Created:** [September 11, 2021, 7:47pm UTC](https://discuss.huggingface.co/t/bert-model-on-acceptability-judgement-task-optimizer-grouped-parameters/9905 "2021-09-11T19:47:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JamesBond](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jamesbond/32/2688_2.png) [@JamesBond](https://discuss.huggingface.co/u/JamesBond)\
**Post date:** [September 11, 2021, 7:47pm UTC](https://discuss.huggingface.co/t/bert-model-on-acceptability-judgement-task-optimizer-grouped-parameters/9905/1 "2021-09-11T19:47:14Z")

</div>

I have just started learning about Transformers and the BERT model.  
I refered to this notebook : `https://github.com/PacktPublishing/Transformers-for-Natural-LanguageProcessing/blob/main/Chapter02/BERT_Fine_Tuning_Sentence_Classification_DR.ipynb` ,  
and I got stuck here :

```auto
In [41]:
##@title Optimizer Grouped Parameters
#This code is taken from:
# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L102

# Don't apply weight decay to any parameters whose names include these tokens.
# (Here, the BERT doesn't have `gamma` or `beta` parameters, only `bias` terms)
param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'LayerNorm.weight']
# Separate the `weight` parameters from the `bias` parameters. 
# - For the `weight` parameters, this specifies a 'weight_decay_rate' of 0.01. 
# - For the `bias` parameters, the 'weight_decay_rate' is 0.0. 
optimizer_grouped_parameters = [
    # Filter for all parameters which *don't* include 'bias', 'gamma', 'beta'.
    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.1},
    
    # Filter for parameters which *do* include those.
    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.0}
]
# Note - `optimizer_grouped_parameters` only includes the parameter values, not 
# the names.

```

Can anyone please explain about how this weight decay and parameters are affecting the model or maybe give a reference to learn about it.