# NotImplementedError when solidifying a streaming dataset **URL:** \ **Category:** 🤗Datasets\ **Created:** [January 24, 2023, 3:15pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314 "2023-01-24T15:15:23Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![howtoosee](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@howtoosee](https://discuss.huggingface.co/u/howtoosee)\ **Post date:** [January 24, 2023, 3:15pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/1 "2023-01-24T15:15:24Z")
Hello, I’m trying to take only the top 100 rows from a very large dataset, using the following method from the documentation [here](https://huggingface.co/docs/datasets/main/en/stream#split-dataset). ```python dataset = load_dataset("laion/laion-coco", split="train", streaming=True) list(dataset.take(100)) ``` However, this raises the error ```auto NotImplementedError: os.walk is not extended to support URLs in streaming mode ``` The code from the documentation worked fine, I’ve only changed the dataset URL below. Am I missing something? Any help would be great, thanks in advance. Environment: - MacOS Ventura 13.1 - Python 3.10.6 - Datasets 2.8.0 Also tried on Ubuntu 20.04.4 LTS with the same python and dataset versions. The full stacktrace is here: ```auto --------------------------------------------------------------------------- NotImplementedError Traceback (most recent call last) Cell In[29], line 2 1 dataset = load_dataset("laion/laion-coco", split="train", streaming=True) ----> 2 list(dataset.take(100)) File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/iterable_dataset.py:846, in IterableDataset. __iter__ (self) 843 yield from self._iter_pytorch(worker_info) 844 return --> 846 for key, example in self._iter(): 847 if self.features: 848 # `IterableDataset` automatically fills missing columns with None. 849 # This is done with `_apply_feature_types_on_example`. 850 yield _apply_feature_types_on_example( 851 example, self.features, token_per_repo_id=self._token_per_repo_id 852 ) File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/iterable_dataset.py:788, in IterableDataset._iter(self) 786 else: 787 ex_iterable = self._ex_iterable --> 788 yield from ex_iterable File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/iterable_dataset.py:628, in TakeExamplesIterable. __iter__ (self) 627 def __iter__ (self): --> 628 yield from islice(self.ex_iterable, self.n) File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/iterable_dataset.py:113, in ExamplesIterable. __iter__ (self) 112 def __iter__ (self): --> 113 yield from self.generate_examples_fn(**self.kwargs) File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/iterable_dataset.py:713, in _generate_examples_from_tables_wrapper..wrapper(**kwargs) 711 def wrapper(**kwargs): 712 python_formatter = PythonFormatter() --> 713 for key, table in generate_tables_fn(**kwargs): 714 batch = python_formatter.format_batch(table) 715 for i, example in enumerate(_batch_to_examples(batch)): File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py:65, in Parquet._generate_tables(self, files) 61 if sorted(field.name for field in schema) != sorted(self.config.columns): 62 raise ValueError( 63 f"Tried to load parquet data with columns '{self.config.columns}' with mismatching features '{self.config.features}'" 64 ) ---> 65 for file_idx, file in enumerate(itertools.chain.from_iterable(files)): 66 with open(file, "rb") as f: 67 parquet_file = pq.ParquetFile(f) File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:792, in _IterableFromGenerator. __iter__ (self) 791 def __iter__ (self): --> 792 yield from self.generator(*self.args, **self.kwargs) File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:847, in FilesIterable._iter_from_urlpaths(cls, urlpaths, use_auth_token) 845 yield urlpath 846 else: --> 847 for dirpath, dirnames, filenames in xwalk(urlpath, use_auth_token=use_auth_token): 848 # skipping hidden directories; prune the search 849 # [:] for the in-place list modification required by os.walk 850 # (only works for local paths as fsspec's walk doesn't support the in-place modification) 851 dirnames[:] = sorted([dirname for dirname in dirnames if not dirname.startswith((".", "__"))]) 852 if xbasename(dirpath).startswith((".", "__")): 853 # skipping hidden directories File /opt/homebrew/Caskroom/miniconda/base/envs/fyp/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:561, in xwalk(urlpath, use_auth_token, **kwargs) 558 else: 559 # walking inside a zip in a private repo requires authentication 560 if not rest_hops and (main_hop.startswith("http://") or main_hop.startswith("https://")): --> 561 raise NotImplementedError("os.walk is not extended to support URLs in streaming mode") 562 elif rest_hops and (rest_hops[0].startswith("http://") or rest_hops[0].startswith("https://")): 563 url = rest_hops[0] NotImplementedError: os.walk is not extended to support URLs in streaming mode ``` ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [January 24, 2023, 5:13pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/2 "2023-01-24T17:13:57Z")
Hi ! Sorry for the inconvenience, we are fixing this right now for datasets hosted on [huggingface.co](http://huggingface.co) - will keep you posted ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [January 24, 2023, 6:54pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/3 "2023-01-24T18:54:01Z")
It’s fixed now ! you can now stream data again from the HF Hub 🙂 ---
**Author:** ![lqtrung](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@lqtrung](https://discuss.huggingface.co/u/lqtrung)\ **Post date:** [February 7, 2023, 4:54am UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/4 "2023-02-07T04:54:23Z")
Hello, I’m also facing this issue but with “bigcode/the-stack-dedup” dataset. The error message is “os.walk is not extended to support URLs in streaming mode” and I’m using datasets==2.9.0. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [February 7, 2023, 7:05pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/5 "2023-02-07T19:05:02Z")
Hi ! Can you try again ? ---
**Author:** ![lqtrung](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@lqtrung](https://discuss.huggingface.co/u/lqtrung)\ **Post date:** [February 9, 2023, 2:17am UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/6 "2023-02-09T02:17:14Z")
Hi, it still shows the same error … Is clearing cache, upgrade datasets library required ? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [February 16, 2023, 12:03pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/7 "2023-02-16T12:03:45Z")
Does this happen right away when you start iterating on the dataset ? Can you try updating `huggingface-hub` and make sure you have a working connection to [huggingface.co](http://huggingface.co) from your python environment ? ---
**Author:** ![lqtrung](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@lqtrung](https://discuss.huggingface.co/u/lqtrung)\ **Post date:** [February 19, 2023, 7:36am UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/8 "2023-02-19T07:36:37Z")
This happen in the middle of the iterating process … I’ve updated huggingface-hub but the error is still there … The `c4` dataset can be load in the same way without error … so I guess it is something specific to `the-stack-dedup` ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [February 20, 2023, 1:29pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/9 "2023-02-20T13:29:15Z")
Ok. Not sure what’s causing this exactly. I see two possibilities - either one of the file is a ZIP file, and therefore it tries to `walk` inside it - OR there is an issue with `xisfile()` not returning `True` for one of the parquet files In any case we need to know which file is causing this. It should be possible by mocking `datasets.download.streaming_download_manager.xisfile` to add some logging every time it’s called until the error raises. Its first argument is the URL of the file we’re looking for. ---
**Author:** ![carlhyde](https://avatars.discourse-cdn.com/v4/letter/c/8c91f0/32.png) [@carlhyde](https://discuss.huggingface.co/u/carlhyde)\ **Post date:** [February 21, 2023, 5:56am UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/10 "2023-02-21T05:56:09Z")
In Python, NotImplementedError is an exception that is raised when a method or function is not implemented, or when a subclass does not implement an abstract method from its parent class. This exception can be raised intentionally to signal that the functionality is not yet implemented, or it can be raised accidentally due to a coding mistake. Check if the method or function is intentionally not implemented: If the NotImplementedError is raised intentionally, you can leave the method or function as is, or provide a placeholder implementation that raises the NotImplementedError. However, if the NotImplementedError is raised accidentally, you need to provide a proper implementation. If you are dealing with an [python](http://net-informations.com/python/iq/default.htm) abstract method, you can either provide a concrete implementation for the method in the subclass, or mark the subclass as abstract and force its children to implement the method. ---
**Author:** ![voidspace](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidspace/32/21424_2.png) [@voidspace](https://discuss.huggingface.co/u/voidspace)\ **Post date:** [November 23, 2023, 1:34am UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/11 "2023-11-23T01:34:54Z")
I have the same question， is there any update？ ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 23, 2023, 12:31pm UTC](https://discuss.huggingface.co/t/notimplementederror-when-solidifying-a-streaming-dataset/30314/12 "2023-11-23T12:31:01Z")
Is this happening at the beginning if streaming or while streaming after some time ?