Skip to content

[STORAGE] Rename bridges to storage and organize for various storage solutions #270

@casenave

Description

@casenave

Some storages are valid only on some cases. For instance, hf_datasets is very powerfull but not compatible with very large field features. Other services can motivate other formats: keep the hierarchical structure (zarr), optimize streaming for distributed learning (webdatasets)

storage
├── hf_datasets
├── zarr
├── webdataset
├── ...
└── common

We should limit the role of the storage classes to provide to_plaid_function (either complete or partial). Converting plaid samples for dataloader purposes are dedicated to plaid-bridges

Sub-issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions