# Filtering Dataset

**URL:** <https://discuss.huggingface.co/t/filtering-dataset/10228>\
**Category:** Beginners\
**Created:** [September 22, 2021, 5:58pm UTC](https://discuss.huggingface.co/t/filtering-dataset/10228 "2021-09-22T17:58:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![baumstan](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@baumstan](https://discuss.huggingface.co/u/baumstan)\
**Post date:** [September 22, 2021, 5:58pm UTC](https://discuss.huggingface.co/t/filtering-dataset/10228/1 "2021-09-22T17:58:33Z")

</div>

I’m trying to filter a dataset based on the ids in a list. This approach is too slow. The dataset is an Arrow dataset.

```auto
responses = load_dataset('peixian/rtGender', 'responses', split = 'train')
# post_id_test_list contains list of ids
responses_test = responses.filter(lambda x: x['post_id'] in post_id_test_list)

```

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\
**Post date:** [September 24, 2021, 1:43pm UTC](https://discuss.huggingface.co/t/filtering-dataset/10228/2 "2021-09-24T13:43:02Z")

</div>

Hi baumstan.

I’m not sure I understand the question. Why does it matter if it is slow?

I would expect you to create and then save your train/test datasets only once, before you start using your model. If it takes a long time, just leave it running.

Are you trying to use a dynamic post\_id\_test\_list, or to train with transient data, or what?

I suspect you might find better answers on Stack Overflow, as this doesn’t look like a Huggingface-specific question.

---

<div class="post-metadata">

**Author:** ![baumstan](https://avatars.discourse-cdn.com/v4/letter/b/f05b48/32.png) [@baumstan](https://discuss.huggingface.co/u/baumstan)\
**Post date:** [September 26, 2021, 6:16pm UTC](https://discuss.huggingface.co/t/filtering-dataset/10228/3 "2021-09-26T18:16:59Z")

</div>

Have tried Stackoverflow. [python - pyarrow Table Filtering -- huggingface - Stack Overflow](https://stackoverflow.com/questions/69290604/pyarrow-table-filtering-huggingface/69296872#69296872)

---

<div class="post-metadata">

**Author:** ![louislu9911](https://avatars.discourse-cdn.com/v4/letter/l/779978/32.png) [@louislu9911](https://discuss.huggingface.co/u/louislu9911)\
**Post date:** [April 8, 2024, 8:36am UTC](https://discuss.huggingface.co/t/filtering-dataset/10228/4 "2024-04-08T08:36:39Z")

</div>

I think setting [`input_columns`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.Dataset.filter.input_columns) is what you want. See [my answer](https://stackoverflow.com/a/78291174/16374708) on Stack Overflow.