# 'DistributedDataParallel' object has no attribute 'no\_sync'

**URL:** <https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469>\
**Category:** Amazon SageMaker\
**Created:** [April 13, 2021, 4:05pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469 "2021-04-13T16:05:58Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![efinkel88](https://avatars.discourse-cdn.com/v4/letter/e/9de053/32.png) [@efinkel88](https://discuss.huggingface.co/u/efinkel88)\
**Post date:** [April 13, 2021, 4:05pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/1 "2021-04-13T16:05:58Z")

</div>

Hi,  
I am trying to fine-tune layoutLM using with the following:

```
distribution = {'smdistributed':{'dataparallel':{ 'enabled': True }}}

estimator = HuggingFace(
    entry_point = 'train.py',
    py_version = 'py36',
    transformers_version='4.4.2',
    pytorch_version='1.6.0', 
    role = role,
    instance_type='ml.p3.16xlarge',
    instance_count=1,
    checkpoint_s3_uri=checkpoint_dir,
    checkpoint_local_path='/opt/ml/checkpoints',
    hyperparameters = {'epochs': 3, 
                       'batch-size': 16, 
                       'learning-rate': 5e-5, 
                       'use-cuda': True, 
                       'model-name':'microsoft/layoutlm-base-uncased'
                      },
    debugger_hook_config=False,
    volume_size = 40,
    distribution = distribution,
    source_dir = source_dir)

estimator.fit({'input_data_dir': data_uri}, wait = True)

```

Relevant code in train.py file:

```
model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased',num_labels = len(labels))

training_args = TrainingArguments(
    output_dir='./results',          
    num_train_epochs=4,              
    per_device_train_batch_size=16,  
    per_device_eval_batch_size=32,   
    warmup_ratio=0.1,               
    weight_decay=0.01,               
    report_to='wandb',
    run_name = 'test_run',
    logging_steps = 500,
    fp16 = True,
    load_best_model_at_end = True,
    evaluation_strategy = 'steps',
    gradient_accumulation_steps = 1,
    save_steps = 500,
    save_total_limit = 5,
)

trainer = Trainer(
    model=model,                         
    args=training_args,                  
    train_dataset=train_dataset,        
    eval_dataset=val_dataset,          
    data_collator = data_collator,
    compute_metrics=compute_metrics,
    callbacks = [EarlyStoppingCallback]
)

trainer.train()

```

Unfortunately I keep getting the following error. Tried tracking down the problem but cant seem to figure it out.

```
[1,7]<stdout>:Traceback (most recent call last):
[1,7]<stdout>: File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,7]<stdout>: " __main__", mod_spec)
[1,7]<stdout>: File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,7]<stdout>: exec(code, run_globals)
[1,7]<stdout>: File "/opt/conda/lib/python3.6/site-packages/mpi4py/ __main__.py", line 7, in <module>
[1,7]<stdout>: main()
[1,7]<stdout>: File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,7]<stdout>: run_command_line(args)
[1,7]<stdout>: File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,7]<stdout>: run_path(sys.argv[0], run_name=' __main__')
[1,7]<stdout>: File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,7]<stdout>: pkg_name=pkg_name, script_name=fname)
[1,7]<stdout>: File "/opt/conda/lib/python3.6/runpy.py", line 96, in _run_module_code
[1,7]<stdout>: mod_name, mod_spec, pkg_name, script_name)
[1,7]<stdout>: File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,7]<stdout>: exec(code, run_globals)
[1,7]<stdout>: File "train.py", line 619, in <module>
[1,7]<stdout>: trainer.train()
[1,7]<stdout>: File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1050, in train
[1,7]<stdout>: with model.no_sync():
[1,7]<stdout>: File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 800, in __getattr__
[1,7]<stdout>: type(self). __name__ , name))
[1,7]<stdout>:torch.nn.modules.module.ModuleAttributeError: 'DistributedDataParallel' object has no attribute 'no_sync'

```

Any help would be appreciated!

---

<div class="post-metadata">

**Author:** ![efinkel88](https://avatars.discourse-cdn.com/v4/letter/e/9de053/32.png) [@efinkel88](https://discuss.huggingface.co/u/efinkel88)\
**Post date:** [April 13, 2021, 4:08pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/2 "2021-04-13T16:08:48Z")

</div>

Oh and running the same code without the ddp and using a 1 GPU instance works just fine but obviously takes much longer to complete

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [April 13, 2021, 4:21pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/3 "2021-04-13T16:21:17Z")

</div>

Hey @efinkel88,

thanks for creating the topic. Could you upload your complete `train.py`? This would help to reproduce the error.

---

<div class="post-metadata">

**Author:** ![efinkel88](https://avatars.discourse-cdn.com/v4/letter/e/9de053/32.png) [@efinkel88](https://discuss.huggingface.co/u/efinkel88)\
**Post date:** [April 13, 2021, 5:53pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/4 "2021-04-13T17:53:14Z")

</div>

ugh it just started working with no changes to my code and I have no idea why. 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [April 13, 2021, 5:55pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/5 "2021-04-13T17:55:34Z")

</div>

Could it be possible that you had `gradient_accumulation_steps`\>1? Or are you installing `transformers` from git master branch?

---

<div class="post-metadata">

**Author:** ![Ishitori](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ishitori/32/1716_2.png) [@Ishitori](https://discuss.huggingface.co/u/Ishitori)\
**Post date:** [June 2, 2021, 9:47pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/6 "2021-06-02T21:47:27Z")

</div>

I have the same issue when I use multi-host training (2 multigpu instances) and set up `gradient_accumulation_steps` to 10.

I don’t install transformers separately, just use the one that goes with Sagemaker.

I wonder, if gradient\_accumulation\_steps is not compatible with multi-host training at all, or there are other parameters I need to tweak?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [June 4, 2021, 11:41am UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/7 "2021-06-04T11:41:52Z")

</div>

Hey @Ishitori,

which `transformers_version` are you using?

---

<div class="post-metadata">

**Author:** ![Ishitori](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ishitori/32/1716_2.png) [@Ishitori](https://discuss.huggingface.co/u/Ishitori)\
**Post date:** [June 7, 2021, 9:59pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/8 "2021-06-07T21:59:06Z")

</div>

Hi @philschmid,

I was using the default version published in AWS Sagemaker. I have switched to 4.6.1 version, and the problem is gone.

Thanks for a hint!

---

<div class="post-metadata">

**Author:** ![short-mamba](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@short-mamba](https://discuss.huggingface.co/u/short-mamba)\
**Post date:** [February 8, 2022, 5:31pm UTC](https://discuss.huggingface.co/t/distributeddataparallel-object-has-no-attribute-no-sync/5469/9 "2022-02-08T17:31:45Z")

</div>

Hey @efinkel88. I was wondering if you can share the train.py file. I am also using the LayoutLM for doc classification. I wanted to train it on multi gpus using the huggingface trainer API. I have all the features extracted and saved in the disk. But I am not quite sure on how to pass the train dataset to the trainer API.  
Thanks