How to spatially concatenate (along latitude and longitude) multiple Kerchunk-referenced NetCDF files into a valid mosaic.json

[PouriaRezz]

Hi,

I’m working with a large dataset , each one covering a distinct spatial tile (no overlaps), but all sharing the same time axis. Each file has shape (281, 168, 168) corresponding to (time, latitude, longitude).
Each NetCDF file has been individually converted to a Kerchunk JSON reference using kerchunk.hdf.SingleHdf5ToZarr.

The structure is always the same across files. For example:

{
  "version": 1,
  "refs": {
    "bm_herb/0.0.0": ["http://example.com/file1.nc", 123456, 65536],
    "bm_herb/.zarray": "{\"shape\": [281, 168, 168], \"chunks\": [281, 168, 168], ...}",
    "latitude/.zarray": "{\"shape\": [168], ...}",
    "longitude/.zarray": "{\"shape\": [168], ...}",
    "crs/.zattrs": "{\"geotransform\": \"X0 DX 0 Y0 0 DY\", ...}"
  }
}

Note:

• The time coordinate is identical across all files.
• latitude and longitude ranges are disjoint across files (i.e., tiles in a spatial grid).
• All data chunks are in bm_herb/t.i.j form.
• No overlap across files, and spatial layout is known via geotransform in the CRS metadata.

---

Goal

I want to build a single valid mosaic.json that allows me to open the full spatial-temporal dataset.

The desired result is a dataset with shape:

(time=281, latitude=TOTAL_LAT, longitude=TOTAL_LON)
where TOTAL_LAT and TOTAL_LON come from the concatenation of all disjoint tiles.

---

What I’ve Tried

• kerchunk.combine.MultiZarrToZarr works well for time-based concatenation, but it assumes that the coordinate variables like ["latitude", "longitude"] are identical across files, which is not the case here.

Questions

1. What is the officially recommended approach to mosaic Kerchunk JSONs spatially, not temporally, when latitude and longitude vary across tiles?
2. Is there a supported tool or helper (or planned feature) to do this spatial concatenation correctly?
3. Is it expected that users build such mosaics manually, or is Kerchunk expected to support this pattern?

[NikosAlexandris]

Just an idea, not tested :

• open data via Xarray and get a list of all lats and another for all lons
• merge'm meaningfully via numpy.unique() in one total_longitude and one total_latitude arrays
• use/force these total coordinates in your input data
  • each of your files would be "total" in space yet really carry data only for its original "space" -- and maybe it should be fine to let lon and total_longitude (and same or latitudes) co-exist too, as coordinates !
• use the total coordinates in the list of concat_dims

source_directory = Path(source_directory)
reference_file_paths = list(source_directory.glob(pattern))
reference_file_paths = list(map(str, reference_file_paths))
from kerchunk.combine import MultiZarrToZarr
mzz = MultiZarrToZarr(
    reference_file_paths,
    concat_dims=["time"],
    identical_dims=["total_latitude", "total_longitude"],
)
multifile_kerchunk = mzz.translate()
combined_reference_filename = Path(combined_reference)
local_fs = fsspec.filesystem("file")
with local_fs.open(combined_reference_filename, "wb") as f:
    f.write(ujson.dumps(multifile_kerchunk).encode())

?

[martindurant]

To be sure: MZZ should cope with concatenation along multiple dimensions like this case of tiling, and not just along time. Perhaps you can run

# note that we concatenate on coords, but times are labeled identical
mzz = MultiZarrToZarr(mzz = MultiZarrToZarr(
    reference_file_paths,
    concat_dims=["latitude", "longitude"],
    identical_dims=["time"],
)
mzz.first_pass()  # populates mzz.coos withvalues inferred from all the inputs
mzz.store_coords()  # writes output arrays

and see what happens.

Also, the logger "kerchunk.combine" might tell you more.