# Loader for dataset with multiple source files in one split

**URL:** <https://discuss.huggingface.co/t/loader-for-dataset-with-multiple-source-files-in-one-split/17617>\
**Category:** 🤗Datasets\
**Created:** [May 6, 2022, 3:21pm UTC](https://discuss.huggingface.co/t/loader-for-dataset-with-multiple-source-files-in-one-split/17617 "2022-05-06T15:21:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![leondz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/leondz/32/22922_2.png) [@leondz](https://discuss.huggingface.co/u/leondz)\
**Post date:** [May 6, 2022, 3:21pm UTC](https://discuss.huggingface.co/t/loader-for-dataset-with-multiple-source-files-in-one-split/17617/1 "2022-05-06T15:21:31Z")

</div>

Hi, I have a conll-format dataset where the training “split” comprises multiple files. I’m writing a loader based on the conll2003 loader but can’t figure out what the best way is to load in multiple files for a single split. SplitGenerator seems to pass just a single filepath on in gen\_kwargs param. What’s the best way to tackle this, or a place to look for a good example?

(The dataset comes with multiple sections, A B E F G and H, which I’d like to be surfaceable through the loader, but the main things I’m struggling with is the best way to combine four of these files’ instances to form the training partition)

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 9, 2022, 10:06am UTC](https://discuss.huggingface.co/t/loader-for-dataset-with-multiple-source-files-in-one-split/17617/2 "2022-05-09T10:06:15Z")

</div>

Hi ! You can just pass a list of files to `gen_kwargs`, and iterate over the files in `_generate_examples` 🙂