Skip to content

Errors opening S3Path with rasterio and rioxarray #377

@cas--

Description

@cas--

I'm not sure if this is a upath or s3fs issue but when trying to pass S3Path to rioxarray I am encountering errors.

For local filesystem there are no issues, included cloudpathlib for comparison:

❯ uv run -s rasterio_upath_error.py test.tif 
Open using cloudpathlib AnyPath for path: test.tif
Open using universal-pathlib UPath for path: tests.tif

For S3 paths opens fine with cloudpathlib but fails with universal-pathlib

uv run -s rasterio_upath_error.py s3://bucket/test.tif
Open using cloudpathlib AnyPath for path: s3://bucket/test.tif
Open using universal-pathlib UPath for path: s3://bucket/test.tif
Traceback (most recent call last):
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<function open at 0x76955c36b600>, (S3Path('s3://bucket/test.tif'),), 'r', (('sharing', False),), '1535b53f-793d-44de-ac73-393caa242dd5']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rasterio/_vsiopener.pyx", line 381, in rasterio._vsiopener._opener_registration
  File "rasterio/_vsiopener.pyx", line 594, in rasterio._vsiopener._FileContainer.size
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        ^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/s3fs/core.py", line 720, in _open
    return S3File(
           ^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/s3fs/core.py", line 2232, in __init__
    raise ValueError("Attempt to open non key-like path: %s" % path)
ValueError: Attempt to open non key-like path: test

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "rasterio_upath_error.py", line 30, in <module>
    main()
  File "rasterio_upath_error.py", line 25, in main
    with rxr.open_rasterio(UPath(path)) as ds:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/rioxarray/_io.py", line 1135, in open_rasterio
    riods = manager.acquire()
            ^^^^^^^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 193, in acquire
    file, _ = self._acquire_with_cache_info(needs_lock)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/rasterio/env.py", line 463, in wrapper
    return f(*args, **kwds)
           ^^^^^^^^^^^^^^^^
  File ".cache/uv/environments-v2/rasterio-upath-error-423a0df2cfd1344e/lib/python3.12/site-packages/rasterio/__init__.py", line 350, in open
    registered_vsi_path = stack.enter_context(vsi_path_ctx)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 526, in enter_context
    result = _enter(cm)
             ^^^^^^^^^^
  File ".local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "rasterio/_vsiopener.pyx", line 383, in _opener_registration
rasterio.errors.OpenerRegistrationError: Opener is invalid.

Script used to test

#!/usr/bin/env -S uv run -s
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "cloudpathlib[s3]",
#     "fsspec[s3]",
#     "rioxarray",
#     "universal-pathlib",
# ]
# ///
import sys
import rioxarray as rxr
from xarray import DataArray
from upath import UPath
from cloudpathlib import AnyPath


def main() -> None:
    path = sys.argv[1]
    print(f"Open using cloudpathlib AnyPath for path: {path}")
    with rxr.open_rasterio(AnyPath(path)) as ds:
        assert isinstance(ds, DataArray)

    print(f"Open using universal-pathlib UPath for path: {path}")
    with rxr.open_rasterio(UPath(path)) as ds:
        assert isinstance(ds, DataArray)


if __name__ == "__main__":
    main()

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug 🐛Something isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions