# Parameter groups and GPT2 LayerNorm

**URL:** <https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239>\
**Category:** 🤗Transformers\
**Created:** [March 8, 2021, 1:36pm UTC](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239 "2021-03-08T13:36:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![salti](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/salti/32/46133_2.png) [@salti](https://discuss.huggingface.co/u/salti)\
**Post date:** [March 8, 2021, 1:36pm UTC](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239/1 "2021-03-08T13:36:45Z")

</div>

When creating a default optimizer the `Trainer` class creates two parameter groups based on whether weight decay should be applied or not and it does that based on the parameter name (does it contain “LayerNorm” or “bias” in it or not).  
Problem is, not all models’ parameters are named the same way; GPT2’s layer normalization layers for example are named `ln_` followed by a number or an `f`, hence weight decay will be applied to the weights of the `LayerNorm` layers in GPT2.

I don’t know if this is an “issue” per se, but it’s definitely something to be cautious about when training/fine-tuning GPT2.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 8, 2021, 2:17pm UTC](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239/2 "2021-03-08T14:17:47Z")

</div>

Oh indeed. A check on names only does not sound super smart. Will try to write something that checks the class of the modules instead.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 8, 2021, 9:35pm UTC](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239/3 "2021-03-08T21:35:45Z")

</div>

Done in [#10598](https://github.com/huggingface/transformers/pull/10598).

---

<div class="post-metadata">

**Author:** ![salti](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/salti/32/46133_2.png) [@salti](https://discuss.huggingface.co/u/salti)\
**Post date:** [March 9, 2021, 1:14pm UTC](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239/4 "2021-03-09T13:14:12Z")

</div>

Thanks!  
That was very quick ⚡