Enable Automatic Parquet File Detection in StreamingLeRobotDataset (Removes Hard-Coded data/*/*.parquet Assumption) issue #2312 #2324
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### Summary
This PR resolves a streaming issue where StreamingLeRobotDataset
assumed all datasets stored parquet files under "data/*/*.parquet".
This caused load failures for valid HF datasets with different layouts.
### Changes
- Added new parameter `data_files=None` to StreamingLeRobotDataset.__init__()
- When `data_files=None`, the underlying `load_dataset()` automatically
detects parquet files in the repo.
- Users can still specify custom directory patterns if needed.
- Maintains full backward compatibility.
### Result
Now this works as expected:
from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset
dataset = StreamingLeRobotDataset("yaak-ai/L2D-v3")
### Motivation
This allows streaming from any Hub dataset that contains parquet data,
without requiring a specific folder hierarchy.
Coffeempty
changed the title
2 tasks
|
Hi, I tried running the streaming dataset feature with this PR, but got error from #2312 again. I changed to another dataset and successfully ran the repo_id = "lerobot/metaworld_mt50"
dataset = StreamingLeRobotDataset(repo_id) # streams directly from the Hub
print(dataset[0])I got the following error ---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) dataset[0]
File ~/sattwik/projects/robot-learning/jepavizor/.venv/lib/python3.12/site-packages/torch/utils/data/dataset.py:59, in Dataset.__getitem__(self, index)
58 def __getitem__(self, index) -> _T_co:
---> [59](https://file+.vscode-resource.vscode-cdn.net/home/moonlab/sattwik/projects/robot-learning/jepavizor/tests/notebooks/~/sattwik/projects/robot-learning/jepavizor/.venv/lib/python3.12/site-packages/torch/utils/data/dataset.py:59) raise NotImplementedError("Subclasses of Dataset should implement __getitem__.")
NotImplementedError: Subclasses of Dataset should implement __getitem__.It seems not all datasets on the hub follow some standard naming/directory conventions or API conventions, leading to these inconsistencies? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #2312
hey, i tried solving the issue. It was quite easy actually.
StreamingLeRobotDatasetcurrently assumes parquet files follow the patterndata/*/*.parquet.However, some valid Hub datasets (e.g.,
yaak-ai/L2D-v3) do not use this directory structure, causingload_datasetto fail even though the dataset is fully compatible.This PR removes that structural assumption and makes dataset loading more flexible.
here is what i did:
data_files=Noneparameter toStreamingLeRobotDataset.__init__().data_filesis not provided,load_dataset()will automatically detect parquet files in the repo.Looking forward to review + feedback!