# Loading multiple custom splits using \`load\_dataset('audiofolder', data\_dir=/some/path)\`

**URL:** <https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931>\
**Category:** Beginners\
**Created:** [November 11, 2023, 6:13pm UTC](https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931 "2023-11-11T18:13:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jaksen93](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@Jaksen93](https://discuss.huggingface.co/u/Jaksen93)\
**Post date:** [November 11, 2023, 6:13pm UTC](https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931/1 "2023-11-11T18:13:37Z")

</div>

When I load a folder structure containing multiple test sets (`test1`, `test2`) and a `train` set like the below, using `ds = load_dataset("audiofolder", data_dir="/path/to/directory")`, the resulting `DatasetDict` has a single test set that is the combined test1 and test2 sets.

How do I get both sets back separately?

The folder structure:

```bash
.
├── test1
| ├── 01.mp3
| ├── 02.mp3
| ├── ...
│ └── metadata.csv
├── test2
| ├── 01.mp3
| ├── 02.mp3
| ├── ...
│ └── metadata.csv
└── train
    ├── 01.mp3
    ├── 02.mp3
    ├── ...
    └── metadata.csv

```

The resulting `DatasetDict` with a single “merged” test set:

```python
>>> load_dataset("audiofolder", data_dir="/path/to/directory")`
DatasetDict({
    train: Dataset({
        features: ['audio', 'case_id', 'segment_id', 'created_at', 'submitted_at', 'text', 'start_audio_s', 'end_audio_s', 'speaker_label', 'location_label'],
        num_rows: 9984
    })
    test: Dataset({
        features: ['audio', 'case_id', 'segment_id', 'created_at', 'submitted_at', 'text', 'start_audio_s', 'end_audio_s', 'speaker_label', 'location_label'],
        num_rows: 742
    })
})

```

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\
**Post date:** [November 12, 2023, 8:22am UTC](https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931/2 "2023-11-12T08:22:45Z")

</div>

May not be possible. There is only a set number of keywords that are automatically detected by the load\_dataset method. It will merge partial names like test1 and test2 into a single test class.

[see](https://huggingface.co/docs/datasets/repository_structure#automatic-splits-detection)

if you only have 3 categories = consider renaming the folders as test, validate and train. Each is treated separately by dataloader. Otherwise fallback on something like

```auto
import os
path_to_dir='/path/to/directory'
load_dataset("audiofolder", data_paths=[{folder: f'{path_to_dir}/{folder}/metadata.csv'} for folder in os.listdir(path_to_dir)])

```

---

<div class="post-metadata">

**Author:** ![Jaksen93](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@Jaksen93](https://discuss.huggingface.co/u/Jaksen93)\
**Post date:** [November 12, 2023, 11:52am UTC](https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931/3 "2023-11-12T11:52:14Z")

</div>

Ah, beautiful, thank you!

Parenthetically, it seems pretty straight forward to generalise the “detection” of subfolder splits to arbitrary names, does it not? And the `Trainer` gets told explicitly which sets are for training and evaluation (and supports multiple evaluation sets via a `DatasetDict` for evaluation), so that seems compatible with such a change.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 13, 2023, 11:21am UTC](https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931/4 "2023-11-13T11:21:41Z")

</div>

@panigrah correct, and you can also pass glob patterns:

```python
load_dataset("audiofolder", data_files={
    "train": "path/to/train/*",
    "test1": "path/to/test1/*",
    "test2": "path/to/test2/*",
})

```

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\
**Post date:** [November 13, 2023, 12:01pm UTC](https://discuss.huggingface.co/t/loading-multiple-custom-splits-using-load-dataset-audiofolder-data-dir-some-path/61931/5 "2023-11-13T12:01:01Z")

</div>

Yes indeed. My guess is it was designed for specific purpose of only supporting the three “standard” splits as that may be how it’s used most frequently.