Add xcube support

[bouweandela]

Description

Add support for loading data with xcube

Related to #2584

To use the data source, run

esmvaltool config copy data-xcube-ccizarr.yml

Example recipe, run it with --max-parallel-tasks 1 for now:

documentation:
  description: Example recipe that plots a map.

  title: Recipe that runs an example diagnostic written in Python.

  authors:
    - andela_bouwe

  maintainer:
    - andela_bouwe

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

datasets:
  - dataset: "ESACCI-WATERVAPOUR-L3C-TCWV-meris-005deg-2002-2017-fv3.2.zarr"

diagnostics:
  map:
    description: Global map of total column of water in January 2010.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tcwv:
        project: external
        mip: Amon
        frequency: mon
        timerange: 2010/P1M
        caption: Global map of {long_name} in January 2010 according to {dataset}.
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: pcolormesh
          cmap: Reds

This code currently skips all fixes and CMOR checks.

Link to documentation: https://esmvaltool--2917.org.readthedocs.build/projects/ESMValCore/en/2917/api/esmvalcore.io.xcube.html

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[valeriupredoi]

zarr3 is perfectly able to read zarr2 datasets, bud

[bouweandela]

xcube requires zarr==2

[valeriupredoi]

well that's a bummer - that means it can't read Zarr3 spec?

[valeriupredoi]

also Zarr2 is borderline archaic - good luck to us trying to maintain such an evironment

[valeriupredoi]

this is shaping up nicely - couple quick questions: why not use the standard xarray - to iris via ncdata path? Is XCube really needed to load the ESA-CCI Zarr files, and if not, do we know what database they are in so we can bolt on an eg intake-esm functionality?

[bouweandela]

why not use the standard xarray - to iris via ncdata path?

That is exactly what is used:

ESMValCore/esmvalcore/io/xcube.py

Line 104 in 428ae04

return dataset_to_iris(dataset)

and

ESMValCore/esmvalcore/iris_helpers.py

Line 617 in 428ae04

conversion_func = ncdata.iris_xarray.cubes_from_xarray

Is XCube really needed to load the ESA-CCI Zarr files

No, but it is convenient

do we know what database they are in

It looks like the data is available here: https://github.com/esa-cci/xcube-cci/blob/fe8ac26405bd36b0176e1a0cae30238f52009a10/xcube_cci/zarraccess.py#L47

so we can bolt on an eg intake-esm functionality?

That might be possible, but I'm not sure if that will be easier to maintain or less work. xcube supports an interesting range of data sources from ESA, Copernicus, Climate Data Store, and even Zenodo, so I think that having support for xcube will be an interesting feature for our users. Of course, that shouldn't stop us from adding support for intake-esm as well.