# Filtering performance

**URL:** <https://discuss.huggingface.co/t/filtering-performance/28305>\
**Category:** 🤗Datasets\
**Created:** [December 22, 2022, 6:23pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305 "2022-12-22T18:23:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![grakocevic](https://avatars.discourse-cdn.com/v4/letter/g/76d3ee/32.png) [@grakocevic](https://discuss.huggingface.co/u/grakocevic)\
**Post date:** [December 22, 2022, 6:23pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305/1 "2022-12-22T18:23:10Z")

</div>

Hi All,

I’m trying to filter a dataset that has about 1 million rows and about a dozen columns (some of those are Array2D, so the arrow files is ~100GB, but I don’t think that’s very relevant).  
The column I’m filtering on is made up of 100-300 character long strings, and I have a set of strings that I want to exclude.

Each of the follwing takes about 1min 30s to execute:

```auto
a = dataset.filter(lambda x: not x in exclude, input_columns=['column'])
a = dataset.filter(lambda x: not x in exclude, input_columns=['column'], batch_size=None)
a = dataset.filter(lambda xs: [x not in exclude for x in xs], input_columns=['column'], batched=True)
a = dataset.filter(lambda xs: [x not in exclude for x in xs], input_columns=['column'], batch_size=None, batched=True)

```

But this only takes 1.85s:

```auto
a = dataset['column']
b = [i for i, x in enumerate(a) if x not in exclude]
c = dataset.select(b)

```

Is this expected, or am I doing something wrong with the dataset?  
(The other option solves the problem for me, I’m more interested as to why such a difference)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [December 22, 2022, 7:08pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305/2 "2022-12-22T19:08:12Z")

</div>

Hi! Which version of `datasets` are you using? We’ve made some improvements in the latest release (`2.8.0`) to optimize decoding, so use this version for the best performance.

Also, unlike `select` (creates an indices mapping), `filter` writes a new dataset to disk/memory, which can take some time for larger datasets (some benefits are faster indexing, etc.)

---

<div class="post-metadata">

**Author:** ![grakocevic](https://avatars.discourse-cdn.com/v4/letter/g/76d3ee/32.png) [@grakocevic](https://discuss.huggingface.co/u/grakocevic)\
**Post date:** [December 22, 2022, 10:37pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305/3 "2022-12-22T22:37:11Z")

</div>

I’m on 2.7.1, I’ll try upgrading. But writing the new dataset to probably explains the difference, as there is close to 100GB to be written out, eve after the filtering

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 3, 2023, 2:09pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305/4 "2023-01-03T14:09:47Z")

</div>

> [@mariosasko](#):
>
> `filter` writes a new dataset to disk/memory, which can take some time for larger datasets

That’s not true - filter does add an indices mapping. We should update the docstring

---

<div class="post-metadata">

**Author:** ![abdulfatir](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@abdulfatir](https://discuss.huggingface.co/u/abdulfatir)\
**Post date:** [March 4, 2025, 4:06pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305/5 "2025-03-04T16:06:15Z")

</div>

@lhoestq can I confirm that only the valid (according to filter criteria) indices are cached by `filter()` and it does not actually create a cached copy of the dataset with valid entries?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [March 5, 2025, 2:33pm UTC](https://discuss.huggingface.co/t/filtering-performance/28305/6 "2025-03-05T14:33:18Z")

</div>

Yes correct, `filter()` only stores the indices to save disk space.

For people who want to rewrite the dataset completely (e.g. to end up with contiguous data and get faster reads), there is `ds.flatten_indices()` that rewrites the dataset and removes the indices mapping