# Loading webdatasets across multiple nodes

**URL:** <https://discuss.huggingface.co/t/loading-webdatasets-across-multiple-nodes/88276>\
**Category:** 🤗Datasets\
**Created:** [May 24, 2024, 2:13pm UTC](https://discuss.huggingface.co/t/loading-webdatasets-across-multiple-nodes/88276 "2024-05-24T14:13:03Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 27, 2024, 12:36pm UTC](https://discuss.huggingface.co/t/loading-webdatasets-across-multiple-nodes/88276/2 "2024-05-27T12:36:13Z")

</div>

**In streaming mode** , only the requested samples are downloaded on-the-fly when iterating on the dataset

> - Does each node download the full dataset?

it depends how you separate your dataset by node. By default the Hugging Face Trainer will make every node stream the full dataset but skip samples to avoid duplicates. There is [split\_dataset\_by\_node()](https://huggingface.co/docs/datasets/v2.19.0/en/use_with_pytorch#distributed) that can assign shards to each node though so that each node streams only the sample it uses for training:

> - Are the webdataset files optionally cached as arrow/parquet, and from here onwards can be treated as a naive HF dataset?

there is no cache in streaming mode

> - How are the sequential read .tar shards compatible with the Dataset approach of [skipping individual indexes to rebalance across nodes](https://github.com/huggingface/datasets/pull/5369#issuecomment-1609736186)?

Yes webdataset or any sharded format is compatible with this. See [split\_dataset\_by\_node()](https://huggingface.co/docs/datasets/v2.19.0/en/use_with_pytorch#distributed) 😉

> If I instead upload HF native .parquet files:
> 
> - Will I see reduced I/O performance from losing sequential read access?

it’s a bit slower in single process, but if you use multiple DataLoader workers you should be able to saturate your bandwidth

> Lastly, I wanted to bump another user’s [thread](https://discuss.huggingface.co/t/mutligpu-training-using-split-dataset-per-node-with-pytorch-lightning/49402/2) who asked if (in general, i.e. assuming the typical arrow/parquet setup) the Datasets library is compatible with PyTorch distributed training via Lightning’s Trainer.

I’m not too familiar with Lightning’s Trainer, but calling `split_dataset_by_node` in a custom IterableDataset should work indeed. Maybe we should add a way to split by node without having to specify the rank and world\_size manually in `split_by_node`, this could solve the issue.

e.g. calling `split_dataset_by_node(dataset)` without rank and world\_size could default to using the values for torch.distributed, WDYT ? cc @rbrthogan as well

---

_[View the full topic](https://discuss.huggingface.co/t/loading-webdatasets-across-multiple-nodes/88276)._