Skip to content

Conversation

fjansson
Copy link

@fjansson fjansson commented Aug 5, 2025

Proposed as fix for #3201, as an alternative to the one in #3302.

Some filesystems need the scheme as part of the path, while others don't. FsspecStore.from_url() throws an exception if the scheme is left in the path, for any filesystem except http and https. However, the swift fs also needs the scheme in the path. This commit removes the path check, rather than adding more special cases which are difficult to unit test (discussed in #3302).

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Aug 5, 2025
Copy link

codecov bot commented Sep 18, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.24%. Comparing base (62551c7) to head (e3273e8).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3343      +/-   ##
==========================================
- Coverage   61.24%   61.24%   -0.01%     
==========================================
  Files          83       83              
  Lines        9907     9902       -5     
==========================================
- Hits         6068     6064       -4     
+ Misses       3839     3838       -1     
Files with missing lines Coverage Δ
src/zarr/storage/_fsspec.py 68.20% <ø> (-0.34%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Proposed as fix for zarr-developers#3201. Some filesystems need the scheme as part of the path,
while others don't. FsspecStore.from_url() throws an exception if the scheme is
left in the path, for any filesystem except http and https. However, the swift fs
also needs the scheme in the path. This commit removes the exception, rather than
adding more special cases.
@d-v-b
Copy link
Contributor

d-v-b commented Sep 18, 2025

@jhamman could you have a look

@d-v-b
Copy link
Contributor

d-v-b commented Sep 18, 2025

I would really like the path attribute of FSSpecStore to be consistent with the normal definition of a path in the context of a URL. This means, given the URL "https://mydata.zarr", the FSSpecStore.path should be "mydata.zarr", not "https://mydata.zarr", because that's not the path of the URL!

I understand that fsspec has some design choices that make this tricky, but we should see what we can do to hide that stuff instead of propagating forward

@d-v-b
Copy link
Contributor

d-v-b commented Sep 18, 2025

@fjansson take a look at https://github.com/d-v-b/zarr-python/tree/remove-fsspec-path-check, would that work for you? Basically I ensure that the Fsspecstore.path is always a valid URL path, but internally we give fsspec the full URL for operations that need it. We need to test this on lots of different fsspec backends though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs release notes Automatically applied to PRs which haven't added release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants