Cannot read datasets from EWC S3

[frazane]

I cannot read a dataset from EWC by just specifying its name. On top of the obvious unexpected argument (see traceback), there seems to be more general inconsistencies related to the change of S3 interface to obstore. Related to ecmwf/anemoi-utils#210 and #220.

Versions:

anemoi-datasets           0.5.28.dev14+g5cef7207b
anemoi-transform          0.1.17
anemoi-utils              0.4.37

After configuring access to S3 under your ~/.config, run

from anemoi.datasets import open_dataset

ds = open_dataset("aifs-ea-an-oper-0001-mars-n320-1979-2023-6h-v8")

results in

  File "/users/fzanetta/tmp/mre.py", line 13, in <module>
    ds = open_dataset("aifs-ea-an-oper-0001-mars-n320-1979-2023-6h-v8")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/__init__.py", line 92, in open_dataset
    ds = _open_dataset(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/misc.py", line 501, in _open_dataset
    sets.append(_open(a))
                ^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/misc.py", line 373, in _open
    return Zarr(zarr_lookup(a)).mutate()
                ^^^^^^^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/stores.py", line 539, in zarr_lookup
    z = open_zarr(full, dont_fail=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/stores.py", line 171, in open_zarr
    store = name_to_zarr_store(path)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/stores.py", line 139, in name_to_zarr_store
    return S3Store(store)
           ^^^^^^^^^^^^^^
  File "/scratch/mch/fzanetta/.cache/uv/environments-v2/mre-a06fe1ea3e376ac8/lib/python3.12/site-packages/anemoi/datasets/data/stores.py", line 93, in __init__
    self.s3 = s3_client(self.bucket, region=region)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: s3_client() got an unexpected keyword argument 'region'

[NatalieZelenka]

Hi @frazane, could you let me know what versions of anemoi packages you are using? I'd like to look into this for you :)

[frazane]

Hi @NatalieZelenka,

I am using

anemoi-datasets           0.5.28.dev14+g5cef7207b
anemoi-transform          0.1.17
anemoi-utils              0.4.37

[NatalieZelenka]

Hi @frazane. Small update to say I've had a quick look and I can confirm this is a definite bug. It looks like S3Store.__getitem__ is trying to use the old boto3 function to try to access the zarr rather than the obstore one. I'll try to fix it.

[b8raoult]

Solved with #459