Reference-FS's target filesystem must have same value of asynchronous

[robertbridgart]

Using an older version of fsspec from last year (v2024.10.0), I used to run code which could, in a Python Notebook, create a reference file system (which points to a kerchunk file system stored in parquet format which is hosted on a HTTPS server with basic authentication requirements) and then use its map with xarray to create and use a lazy-loaded Dataset object to read the underlying data files. Both the parquet index file system files and the data files (NetCDFs) to which they point, are both hosted on the same HTTPS server with its authorization requirement (a header x-api-key key/value being passed in to authenticate all requests). I created and set the target and remote of this reference file system to https with the header containing the auth password as in the code below:

fs = fsspec.filesystem(
    "reference",
    fo="https://path/to/parquet/index/filesystem/dir",
    target_protocol="https",
    target_options={
        "headers": {
            "x-api-key": API_KEY
        },
        "timeout": 30
    },
    remote_protocol="https",
    remote_options={
        "headers": {
            "x-api-key": API_KEY
        },
        "timeout": 30
    }
)

ds = xr.open_dataset(
    fs.get_mapper(""),
    engine="zarr",
    backend_kwargs=dict(consolidated=False),
    chunks={'time':12, 'latitude':50, 'longitude':50}
)

This used to work fine. I believe that at some point a check was added to fsspec which ensures that all connected file systems are either synchronous or asynchronous and refuses to proceed unless they all match. Fair enough.

My issue is that since the filesystem asynchronous check came in, my original code has stopped working. I cannot figure out how to create a reference file system with both target https and remote https header options at the same time as trying to force either synchronous or asynchronous behavior on the entire system. No matter what I do, or how I try to tell the reference filesystem or the target /remote systems to be sync or async (i.e. matching), it either results in an exception about the mismatch, or an authentication error when attempting the read from the server being raised (presumably because somewhere internally it has either minted a new remote and lost my header option or ignores it?).

Can someone please explain to me how I can get past this issue and or what I'm doing wrong here? Thanks in advance for any insights.

[martindurant]

Have you moved to using zarr 3? That is, essentially, the big change that the async checks were designed to cope with.