# Using DistributedSampler with accelerate **URL:** \ **Category:** 🤗Transformers\ **Created:** [April 2, 2025, 2:12am UTC](https://discuss.huggingface.co/t/using-distributedsampler-with-accelerate/148474 "2025-04-02T02:12:22Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mesistla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mesistla/32/44593_2.png) [@mesistla](https://discuss.huggingface.co/u/mesistla)\ **Post date:** [April 2, 2025, 2:12am UTC](https://discuss.huggingface.co/t/using-distributedsampler-with-accelerate/148474/1 "2025-04-02T02:12:22Z")
I want to run CustomSFTTrainer (inherits [SFTTrainer](https://github.com/huggingface/trl/blob/main/trl/trainer/sft_trainer.py) which inturn inherits [Trainer](https://github.com/huggingface/transformers/blob/v4.50.0/src/transformers/trainer.py) class) on a multi-GPU setup using accelerate. I understand that the Trainer class already uses accelerate and hence appropriately creates a dataloader and calls accelerate.prepare(dataloader) in its train method. However, I fail to understand if it uses DistributedSampler. I noticed that it uses only RandomSampler and accelerate inturn calls SeedableRandomSampler and not a DistributedSampler. I want to run the model on different GPUs with exclusive unique chunks of data so that the training is faster. How do I use DistrubutedSampler with accelerate and the inbuilt Trainer class? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 2, 2025, 7:53am UTC](https://discuss.huggingface.co/t/using-distributedsampler-with-accelerate/148474/2 "2025-04-02T07:53:12Z")
There may be no advantage to explicitly using DistributedSampler… > [@Can accelerator handle the distributed sampler?](https://discuss.huggingface.co/t/can-accelerator-handle-the-distributed-sampler/12943): > > As far as I know, for Pytorch, RandomSampler can not be directly used in the distributed data parallel training since DistributedSampler is desired (this [link](https://discuss.pytorch.org/t/how-to-use-my-own-sampler-when-i-already-use-distributedsampler/62143) discusses the problem). I am wondering whether accelerator.prepare(dataloader) handles the data split for multiple GPUs if I use the RandomSampler, so that the sub-dataset on each device are exclusive. > You don’t have to worry about using a distributed sampler with Accelerate. Whatever your sampler is, Accelerate will automatically shard it for all processes. ---
**Author:** ![mesistla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mesistla/32/44593_2.png) [@mesistla](https://discuss.huggingface.co/u/mesistla)\ **Post date:** [April 2, 2025, 2:28pm UTC](https://discuss.huggingface.co/t/using-distributedsampler-with-accelerate/148474/3 "2025-04-02T14:28:01Z")
I see. So, just to be clear, Accelerate will ensure that, given any sampler, the data will be split exclusively for each GPU? Interesting, because I wasn’t able to find this functionality in the prepare\_dataloader method of the Accelerate function. Is it wrapped in any other Accelerate method? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 2, 2025, 2:54pm UTC](https://discuss.huggingface.co/t/using-distributedsampler-with-accelerate/148474/4 "2025-04-02T14:54:31Z")
It’s hard to tell what’s where in the code of the library in charge of optimization… There’s no example that directly mentions the mechanism. > **[Introducing 🤗 Accelerate](https://huggingface.co/blog/accelerate-library)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Accelerate’s internal mechanisms](https://huggingface.co/docs/accelerate/concept_guides/internal_mechanism)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > > > \### System Info > > \`\`\`Shell > accelerate: 0.12.0 > OS: Linux 5.4.188+ (Colab) > Pyt…hon: 3.7.13 > numpy: 1.21.6 > torch: 1.12.1+cu113 > config: 1 CPU > \`\`\` > > > \### Information > > \- \[\] The official example scripts > \- \[X\] My own modified scripts > > \### Tasks > > \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`) > \- \[X\] My own task or dataset (give details below) > > \### Reproduction > > MRE : https://colab.research.google.com/drive/17krCJCF\_nWtNFSiMBo3oz12l7eX1bBZ6 > > First of all, thanks for this library and the great docs and examples that comes with it 😄! > > I am using a custom torch Dataset that contains a Hugging Face Dataset (pyarrow) instance. Therefore, as indicated in the Datasets docs (https://huggingface.co/docs/datasets/v2.4.0/en/use\_with\_pytorch#use-a-batchsampler), I tried to use a BatchSampler to reduce the number of queries. However, I have not been able yet to make it work yet with accelerate. > > I tried many different possibilities, one of which works one CPU or one GPU, but gets stuck when using distributed training. > > Thanks for your help! > > > \### System Info > > \`\`\`Shell > accelerate 0.31.0 > Ubuntu 22.04 (WSL) > python=3.10.…14 > \`\`\` > > > \### Information > > \- \[\] The official example scripts > \- \[X\] My own modified scripts > > \### Tasks > > \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`) > \- \[X\] My own task or dataset (give details below) > > \### Reproduction > > I would like to combine distributed training and a weighted random sampler. In order to do that, i : > > 1. Create my Dataset inheriting from torch.utils.data.Dataset > 2. Compute weights specific to my classes and data > 3. Create my DataLoader with the random sampler > 4. Prepare my dataloader with accelerate > > But it seems that this is not working because we have data leaks between processes. > !\[image\](https://github.com/huggingface/accelerate/assets/26071804/a8cff431-29f1-414a-97c2-c89c0d453ef1) > > I would like to make sure, processes uses different data, like that : > \ > > I developped an example script in order to understand the process : > \`\`\`python > from accelerate import Accelerator > import argparse > import os > import torch.distributed as dist > import torch > from tqdm.auto import tqdm > from torch.utils.data import Dataset,DataLoader > from torch.utils.data import WeightedRandomSampler,BatchSampler > > WORLD\_SIZE = int(os.getenv('WORLD\_SIZE',1)) > MAIN\_PROCESS = not int(os.getenv('RANK',0)) > > parser = argparse.ArgumentParser() > parser.add\_argument('--dataset\_count',default=12800) > parser.add\_argument('--epochs',default=20) > parser.add\_argument('--batch\_size',default=64) > parser.add\_argument('--balance',action='store\_true',default=False) > > def is\_even(number): > return not number%2 # example 10 =\> 10%2 == 0 > > class DummyDataset(Dataset): > def \_\_init\_\_(self,dataset\_count:int): > self.data = range(dataset\_count) > > def \_\_len\_\_(self): > return len(self.data) > > def dataloader(self,batch\_size,balance:bool=False,seed=42,batch\_sampler=False,drop\_last:bool=False): > generator = torch.Generator().manual\_seed(seed) > def get\_weight(num): > if is\_even(num): > # even > return 1.0 > else: > # odd (impair) > return 0.1 > if balance: > weights = \[get\_weight(i) for i in self.data\] > sampler = WeightedRandomSampler(weights,len(self),replacement=True,generator=generator) > else: > sampler = None > > if batch\_sampler: > return DataLoader(self,batch\_sampler=BatchSampler(sampler,batch\_size,drop\_last)) > else: > return DataLoader(self,batch\_size,sampler=sampler,drop\_last=drop\_last) > > def \_\_getitem\_\_(self,idx): > row\_index = self.data\[idx\] > return row\_index > > def main( > dataset\_count:int, > epochs:int, > batch\_size:int, > balance:bool=True): > > if int(os.environ.get('WORLD\_SIZE',1))\>1: > dist.init\_process\_group(backend='gloo') > > accelerator = Accelerator(cpu=True) > # We mount the right storage... > # We get the path > dataset = DummyDataset(dataset\_count) > # Dataloader without Accelerate... > dataloader = dataset.dataloader(batch\_size,balance) > batched\_data = \[\] > if MAIN\_PROCESS: > print(f'Running {epochs\*len(dataloader)} iterations') > for epoch in range(epochs): > for batch in dataloader: > batch:torch.Tensor > batched\_data.extend(batch.tolist()) > > count\_even = len(\[v for v in batched\_data if is\_even(v)\]) > count\_odd = len(\[v for v in batched\_data if not is\_even(v)\]) > ratio\_odd = count\_odd/(count\_even+count\_odd) > if MAIN\_PROCESS: > print('Get proportion of Odd data without accelerate') > print(f'Ratio Odd = {ratio\_odd}') > # Dataloader with Accelerate... > dataloader = accelerator.prepare(dataloader) > # We increase learning rate when multiGPU > batched\_data = \[\] > if MAIN\_PROCESS: > print(f'Running {epochs\*len(dataloader)} iterations') > for epoch in range(epochs): > for batch in dataloader: > batch:torch.Tensor > batched\_data.extend(batch.tolist()) > count\_even = len(\[v for v in batched\_data if is\_even(v)\]) > count\_odd = len(\[v for v in batched\_data if not is\_even(v)\]) > ratio\_odd = count\_odd/(count\_even+count\_odd) > if MAIN\_PROCESS: > print('Get proportion of Odd data with accelerate') > print(f'Ratio Odd = {ratio\_odd}') > # We save to a file for further processing... > suffix = '\_balanced' if balance else '\_unbalanced' > rank = str(os.environ.get('RANK',0)) > with open(f'test\_{rank}{suffix}.json','w') as jsf: > import json > json.dump(sorted(batched\_data),jsf,indent=4) > > accelerator.wait\_for\_everyone() > > seen\_data = set(batched\_data) > if WORLD\_SIZE\>1: > # Now every one will open the other... > other\_rank = str(int(not int(os.environ.get('RANK',0)))) > with open(f'test\_{other\_rank}{suffix}.json','r') as jsf: > import json > other\_data = json.load(jsf) > > # We get unique ids in order to check that we don't have leaks... > other\_data = set(other\_data) > batched\_data = set(batched\_data) > unique\_in\_rank = batched\_data.difference(other\_data) > if MAIN\_PROCESS: > print('Verify the unicity of the data on each rank...\\n') > print(f'{len(unique\_in\_rank)}/{len(batched\_data)} data only are not leaking from rank {rank} to rank {other\_rank}') > seen\_data = unique\_in\_rank.union(other\_data) > # Unseen data > unseen\_data = set(dataset.data).difference(seen\_data) > if MAIN\_PROCESS: > print("Unseen Data") > print(f'{len(unseen\_data)}/{len(dataset)} have not been seen...') > if \_\_name\_\_=='\_\_main\_\_': > params = vars(parser.parse\_args()) > print('----------------------------------------') > \[print(f'{k}: {v}') for k,v in params.items()\] > print('----------------------------------------') > main(\*\*params) > \`\`\` > > You can try to run this script different ways : > \## Single node without "balance" > \`\`\` > \---------------------------------------- > dataset\_count: 12800 > epochs: 20 > batch\_size: 64 > balance: False > \---------------------------------------- > Running 4000 iterations > Get proportion of Odd data without accelerate > Ratio Odd = 0.5 > Running 4000 iterations > Get proportion of Odd data with accelerate > Ratio Odd = 0.5 > Unseen Data > 0/12800 have not been seen... > \`\`\` > \## Multiple node (2) without "balance" > \`\`\` > \---------------------------------------- > dataset\_count: 12800 > epochs: 20 > batch\_size: 64 > balance: False > \---------------------------------------- > Running 4000 iterations > Get proportion of Odd data without accelerate > Ratio Odd = 0.5 > Running 2000 iterations > Get proportion of Odd data with accelerate > Ratio Odd = 0.5 > Verify the unicity of the data on each rank... > > Verify the unicity of the data on each rank... > 6400/6400 data only are not leaking from rank 0 to rank 1 > > 6400/6400 data only are not leaking from rank 1 to rank 0 > Unseen Data > 0/12800 have not been seen... > \`\`\` > We see that we do not have any leak, all data are seen. > > \## Single node with "balance" > \`\`\` > \---------------------------------------- > dataset\_count: 12800 > epochs: 20 > batch\_size: 64 > balance: True > \---------------------------------------- > Running 4000 iterations > Get proportion of Odd data without accelerate > Ratio Odd = 0.09179296875 > Running 4000 iterations > Get proportion of Odd data with accelerate > Ratio Odd = 0.09139453125 > Unseen Data > 167/12800 have not been seen... > \`\`\` > We see that a few data has not been seen. It's normal because we have a very low rate of Odd data. > > \## Multiple node with "balance" > \`\`\` > \---------------------------------------- > dataset\_count: 12800 > epochs: 20 > batch\_size: 64 > balance: True > \---------------------------------------- > Running 4000 iterations > Get proportion of Odd data without accelerate > Ratio Odd = 0.09179296875 > Running 2000 iterations > Get proportion of Odd data with accelerate > Ratio Odd = 0.0917890625 > Verify the unicity of the data on each rank... > > 895/11760 data only are not leaking from rank 0 to rank 1 > 873/11738 data only are not leaking from rank 1 to rank 0 > > Unseen Data > 167/12800 have not been seen... > \`\`\` > > We see that data are leaking from one node to the other. Like if there was an issue with the distributed sampler. > How to fix it ? > > > \### Expected behavior > > I would like the weighted sampler to be used and i would like nothing to leak from node 1 to node 2 like in the case where we don't have weighted sampler. > > > \*\*Do you have any idea about how to get this result ?\*\* > > Thanks ! > ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [April 3, 2025, 2:55am UTC](https://discuss.huggingface.co/t/using-distributedsampler-with-accelerate/148474/5 "2025-04-03T02:55:27Z")
Closed on April 3, 2025, 2:55am UTC This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.