-
Notifications
You must be signed in to change notification settings - Fork 43
Open
Labels
bug 🐛Something isn't workingSomething isn't working
Description
I'm not sure if this is a upath or s3fs issue but when trying to pass S3Path to rioxarray I am encountering errors.
For local filesystem there are no issues, included cloudpathlib for comparison:
❯ uv run -s rasterio_upath_error.py test.tif
Open using cloudpathlib AnyPath for path: test.tif
Open using universal-pathlib UPath for path: tests.tif
For S3 paths opens fine with cloudpathlib but fails with universal-pathlib
uv run -s rasterio_upath_error.py s3://bucket/test.tif
Open using cloudpathlib AnyPath for path: s3://bucket/test.tif
Open using universal-pathlib UPath for path: s3://bucket/test.tif
Traceback (most recent call last):
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
file = self._cache[self._key]
~~~~~~~~~~~^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
value = self._cache[key]
~~~~~~~~~~~^^^^^
KeyError: [<function open at 0x76955c36b600>, (S3Path('s3://bucket/test.tif'),), 'r', (('sharing', False),), '1535b53f-793d-44de-ac73-393caa242dd5']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "rasterio/_vsiopener.pyx", line 381, in rasterio._vsiopener._opener_registration
File "rasterio/_vsiopener.pyx", line 594, in rasterio._vsiopener._FileContainer.size
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
f = self._open(
^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/s3fs/core.py", line 720, in _open
return S3File(
^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/s3fs/core.py", line 2232, in __init__
raise ValueError("Attempt to open non key-like path: %s" % path)
ValueError: Attempt to open non key-like path: test
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "rasterio_upath_error.py", line 30, in <module>
main()
File "rasterio_upath_error.py", line 25, in main
with rxr.open_rasterio(UPath(path)) as ds:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/rioxarray/_io.py", line 1135, in open_rasterio
riods = manager.acquire()
^^^^^^^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 193, in acquire
file, _ = self._acquire_with_cache_info(needs_lock)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/rasterio/env.py", line 463, in wrapper
return f(*args, **kwds)
^^^^^^^^^^^^^^^^
File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/rasterio/__init__.py", line 350, in open
registered_vsi_path = stack.enter_context(vsi_path_ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 526, in enter_context
result = _enter(cm)
^^^^^^^^^^
File ".local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "rasterio/_vsiopener.pyx", line 383, in _opener_registration
rasterio.errors.OpenerRegistrationError: Opener is invalid.
Script used to test
#!/usr/bin/env -S uv run -s
# /// script
# requires-python = ">=3.12"
# dependencies = [
# "cloudpathlib[s3]",
# "fsspec[s3]",
# "rioxarray",
# "universal-pathlib",
# ]
# ///
import sys
import rioxarray as rxr
from xarray import DataArray
from upath import UPath
from cloudpathlib import AnyPath
def main() -> None:
path = sys.argv[1]
print(f"Open using cloudpathlib AnyPath for path: {path}")
with rxr.open_rasterio(AnyPath(path)) as ds:
assert isinstance(ds, DataArray)
print(f"Open using universal-pathlib UPath for path: {path}")
with rxr.open_rasterio(UPath(path)) as ds:
assert isinstance(ds, DataArray)
if __name__ == "__main__":
main()
Metadata
Metadata
Assignees
Labels
bug 🐛Something isn't workingSomething isn't working