# Download only requested split?

**URL:** <https://discuss.huggingface.co/t/download-only-requested-split/165968>\
**Category:** Beginners\
**Created:** [August 7, 2025, 12:34pm UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968 "2025-08-07T12:34:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ego-thales](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ego-thales/32/52182_2.png) [@ego-thales](https://discuss.huggingface.co/u/ego-thales)\
**Post date:** [August 7, 2025, 12:34pm UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/1 "2025-08-07T12:34:07Z")

</div>

Hi everyone,

I don’t understand how to download only the requested split. See the following for [this](https://huggingface.co/datasets/ego-thales/cifar10) [EDIT: current version uses `.zip`] dataset:

```python-repl
>>> ds = load_dataset("ego-thales/cifar10", name="no_airplane", split="left_out_calibration")
train/automobile/25706.png: 100%|██████████████████████████████████████████████████████| 2.27k/2.27k [00:01<00:00, 2.07kB/s]
train/automobile/25700.png: 100%|██████████████████████████████████████████████████████| 2.39k/2.39k [00:00<00:00, 3.64kB/s]
train/automobile/25701.png: 100%|██████████████████████████████████████████████████████| 2.28k/2.28k [00:00<00:00, 3.19kB/s]
...

```

It starts with `train/automobile`, probably because of the `train` split (I did not request!). Dataset config (only the relevant part):

```yaml
- config_name: no_airplane
    data_files:
      - split: train
        path:
          - train/automobile/*.png
          - train/bird/*.png
          - train/cat/*.png
          - train/deer/*.png
          - train/dog/*.png
          - train/frog/*.png
          - train/horse/*.png
          - train/ship/*.png
          - train/truck/*.png
      - split: calibration
        path:
          - calibration/automobile/*.png
          - calibration/bird/*.png
          - calibration/cat/*.png
          - calibration/deer/*.png
          - calibration/dog/*.png
          - calibration/frog/*.png
          - calibration/horse/*.png
          - calibration/ship/*.png
          - calibration/truck/*.png
      - split: test
        path:
          - test/automobile/*.png
          - test/bird/*.png
          - test/cat/*.png
          - test/deer/*.png
          - test/dog/*.png
          - test/frog/*.png
          - test/horse/*.png
          - test/ship/*.png
          - test/truck/*.png
      - split: left_out_train
        path: train/airplane/*.png
      - split: left_out_calibration
        path: calibration/airplane/*.png
      - split: left_out_test
        path: test/airplane/*.png
      - split: left_out
        path:
          - train/airplane/*.png
          - calibration/airplane/*.png
          - test/airplane/*.png

```

Is there an elegant solution? It’s for a tutorial, and user may only want to download very small splits…

Thanks in advance!

Élie

---

<div class="post-metadata">

**Author:** ![ego-thales](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ego-thales/32/52182_2.png) [@ego-thales](https://discuss.huggingface.co/u/ego-thales)\
**Post date:** [August 7, 2025, 12:48pm UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/2 "2025-08-07T12:48:29Z")

</div>

I know I can use the `streaming=True` option if I want this mode, but I’d hope that without it, HF would simply eagerly download the whole split, but only the requested split!

---

<div class="post-metadata">

**Author:** ![ego-thales](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ego-thales/32/52182_2.png) [@ego-thales](https://discuss.huggingface.co/u/ego-thales)\
**Post date:** [August 8, 2025, 8:15am UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/3 "2025-08-08T08:15:21Z")

</div>

> I know I can use the `streaming=True` option if I want this mode, but I’d hope that without it, HF would simply eagerly download the whole split, but only the requested split!

I realize now that even this **does not work**.

Indeed, my current version uses `.zip` archives since it’s impossible to load thousands of images individually, even if they are really small. But then, using the `streaming` option still triggers a metadata search in all config data\_files.

But the [documentation](https://huggingface.co/docs/datasets/en/image_dataset#create-an-image-dataset) says:

> You can also zip your images, and **in this case each zip should contain both the images and the metadata**

So hugginface goes through the trouble of **unpacking all zip files** to find the metadata….

##### Is there a way to ship metadata alongside the zip file or even at the root of the dataset folder so that huggingface finds it easily and does not unpack the entire dataset when I’m requesting only a very tiny split in `streaming` mode?

Thanks

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 8, 2025, 10:16am UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/4 "2025-08-08T10:16:03Z")

</div>

I can only think of [sharding with WebDataset](https://huggingface.co/docs/hub/datasets-webdataset)…

Edit:  
I found [this](https://discuss.huggingface.co/t/allow-streaming-of-large-datasets-with-image-audio/8062/3).

---

<div class="post-metadata">

**Author:** ![ego-thales](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ego-thales/32/52182_2.png) [@ego-thales](https://discuss.huggingface.co/u/ego-thales)\
**Post date:** [August 8, 2025, 10:31am UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/5 "2025-08-08T10:31:47Z")

</div>

Me too so far… 😕 I won’t resort to it though since my goal is to have clean few-liner downloads for tutorials.

But for the life of me I really feel like I’m missing something. I’ve heard so much talk about HuggingFace and it’s sooo popular. Yet, somehow, requesting a split fetches the entire dataset **by default**?? Even `streaming` mode unpacks every archive looking for metadata?? It makes absolutely no sense to me since the configuration explicitly says what to look for…

As a workaround, I defined essentially one dataset configuration per potential split (like `”no_bird_calibration"`,`”train”`, `”truck_test”`), so `load_dataset(path, name=…)` only fetches the requested part of the dataset. But it feels really monkeypatchy and gives huge 1000 lines `README.md` metadata lol 🤷

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 8, 2025, 10:44am UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/6 "2025-08-08T10:44:54Z")

</div>

Hmm… @lhoestq

---

<div class="post-metadata">

**Author:** ![ego-thales](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ego-thales/32/52182_2.png) [@ego-thales](https://discuss.huggingface.co/u/ego-thales)\
**Post date:** [August 8, 2025, 1:16pm UTC](https://discuss.huggingface.co/t/download-only-requested-split/165968/7 "2025-08-08T13:16:26Z")

</div>

Just so you can see what I came down to x)

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/1/a121b5d5fa23f6226ee02783f346a2ecd6cd248b.png)