# Lazy model initialization

**URL:** <https://discuss.huggingface.co/t/lazy-model-initialization/20933>\
**Category:** 🤗Transformers\
**Created:** [July 27, 2022, 6:15pm UTC](https://discuss.huggingface.co/t/lazy-model-initialization/20933 "2022-07-27T18:15:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![arteagac](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arteagac/32/32065_2.png) [@arteagac](https://discuss.huggingface.co/u/arteagac)\
**Post date:** [July 27, 2022, 6:15pm UTC](https://discuss.huggingface.co/t/lazy-model-initialization/20933/1 "2022-07-27T18:15:44Z")

</div>

Hello. How can I create a model object and skip the random initialization of weights? The random initialization is time consuming and unnecessary for my case, as I want to load the weights using `torch.load_state_dict`. For instance, see the code below.

```python
config = BloomConfig.from_pretrained("bigscience/bloom")
block = BloomBlock(config) # initializes weights randomly, which is time consuming
block.load_state_dict(torch.load("path_to_pytorch_bin"))

```

---

<div class="post-metadata">

**Author:** ![Fazzie](https://avatars.discourse-cdn.com/v4/letter/f/9d8465/32.png) [@Fazzie](https://discuss.huggingface.co/u/Fazzie)\
**Post date:** [May 7, 2024, 7:56am UTC](https://discuss.huggingface.co/t/lazy-model-initialization/20933/2 "2024-05-07T07:56:26Z")

</div>

same question

---

<div class="post-metadata">

**Author:** ![ayushgoel](https://avatars.discourse-cdn.com/v4/letter/a/6f9a4e/32.png) [@ayushgoel](https://discuss.huggingface.co/u/ayushgoel)\
**Post date:** [May 8, 2024, 10:12pm UTC](https://discuss.huggingface.co/t/lazy-model-initialization/20933/3 "2024-05-08T22:12:46Z")

</div>

Use the `init_empty_weights` ContextManager from `accelerate`

```auto
from accelerate import init_empty_weights
with init_empty_weights():
   block = BloomBlock(config)
```

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 8, 2024, 10:27pm UTC](https://discuss.huggingface.co/t/lazy-model-initialization/20933/4 "2024-05-08T22:27:34Z")

</div>

You can set `low_cpu_mem_usage=True` which will skip that: [Models](https://huggingface.co/docs/transformers/en/main_classes/model#transformers.PreTrainedModel.from_pretrained.low_cpu_mem_usage)