NetCDF: HDF error when saving big array to netcdf4 with dask.distributed #10734

pingponghero12 · 2025-09-11T14:42:03Z

pingponghero12
Sep 11, 2025

Hello. I have problem when saving data to NETCDF4 using both netcdf4 and h5netcdf engines. The data is handled by dask.distributed. What I don't understand is that when I work on smaller data(cube of (60 or 600, 4500, 2000) chunked by 120 in last dimention) it works just fine, but for bigger data(cube of (700, 9000, 1000), chunked by 4 in last dim) some of the breakpoints I have fail at saving the NETCDF4.

The error:

Traceback (most recent call last):
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
^^^^^^^^^^^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('<path>',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'eceb0bc6-d5c1-4ebb-a438-f3ed3647c00d']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

... internal traceback ...

  File "<path>", line 56, in save_nc
    ds.to_netcdf(file_path, engine='netcdf4', format='NETCDF4', compute=True)
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/core/dataset.py", line 2021, in to_netcdf
    return to_netcdf(  # type: ignore[return-value]  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/api.py", line 1937, in to_netcdf
    writes = writer.sync(compute=compute)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/common.py", line 358, in sync
    delayed_store = chunkmanager.store(
                    ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py", line 247, in store
    return store(
           ^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/dask/array/core.py", line 1262, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/dask/base.py", line 395, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/distributed/client.py", line 3505, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/distributed/client.py", line 2565, in gather
    return self.sync(
           ^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 81, in __setitem__
    data = self.get_array(needs_lock=False)
^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 94, in get_array
    ds = self.datastore._acquire(needs_lock)
  ^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 455, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  ^^^^^^^^^^^^^^^^^
  File "/nix/store/2i76abyx06asxr2k3rgsp8zx647qj38h-python3-3.12.11/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  ^^^^^^^^^^^^^^^^^
  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: '<path>'

And also at first at worker:

2025-09-11 16:17:11,097 - distributed.worker - ERROR - Compute Failed
...
Exception: "OSError(-101, 'NetCDF: HDF error')"
Traceback: '  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/dask/array/core.py", line 4634, in store_chunk\n    return load_store_chunk(x, out, index, lock, return_stored, False)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/dask/array/core.py", line 4616, in load_store_chunk\n    out[index] = x\n    ~~~^^^^^^^\n  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 81, in __setitem__\n    data = self.get_array(needs_lock=False)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 94, in get_array\n    ds = self.datastore._acquire(needs_lock)\n         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 455, in _acquire\n    with self._manager.acquire_context(needs_lock) as root:\n         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/nix/store/2i76abyx06asxr2k3rgsp8zx647qj38h-python3-3.12.11/lib/python3.12/contextlib.py", line 137, in __enter__\n    return next(self.gen)\n           ^^^^^^^^^^^^^^\n  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context\n    file, cached = self._acquire_with_cache_info(needs_lock)\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/nix/store/iqlxm816wbl1hfl1yw2vccsds9x7w01b-python3-3.12.11-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info\n    file = self._opener(*self._args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__\n  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success\n'

So from what I understand for some reason with this new data the engine backend fails to secure lock or sometimes file gets corrupted during writing.

Also when I skip save at this breakpoint and next one then there is no problem with writing. Those two that fail are relatively simple, rest is much longer and data is undersampled as well, so the size drop by around 3 times. The data that I work on takes approximately 32 GB of memmory, which is similar to what I have RAM.

I tried to save using this and other engine and also NETCDF4_CLASSIC but none helped.

ds.to_netcdf(file_path, engine='netcdf4', format='NETCDF4', compute=True)

I opend client in following way:

    client = Client(n_workers=16,
                    threads_per_worker=1,
                    memory_limit = 0,
                    death_timeout = 1000,
                    dashboard_address=':8787')

Where 16 is also number of cores that I have avaible at the machine.

Could anyone help me understand why this error is happening and is it just a problem between the NETCDF saving and dask.distributed or something that I could fix easily.

I've tried to look for similar issues but I could not find any. This one is similar, but I get different error message and now the error is consistent, because for smaller data the program sometimes crashes in similar fassion but sometimes works.

dask/dask#11960

SarahG-579462 · 2025-09-12T16:37:52Z

SarahG-579462
Sep 12, 2025

It might be related to #10725 or #10736 ? I've been having similar issues since 2025.8.0 with the asynchronous update, you might have luck downgrading to 2025.7.0 until these bugs are sorted out, though it might be a bit slower.

1 reply

pingponghero12 Sep 15, 2025
Author

Thank you very much for answer, now I've tested for 2025.9.0, 2025.7.0 and 2025.4.0 and I seem to have same problem everywhere. Example logs with xarray versions:

2025.9.0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
<...>
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/core/dataset.py", line 2102, in to_netcdf
    return to_netcdf(  # type: ignore[return-value]  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/api.py", line 2116, in to_netcdf
    writes = writer.sync(compute=compute)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/common.py", line 412, in sync
    delayed_store = chunkmanager.store(
                    ^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py", line 247, in store
    return store(
           ^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/dask/array/core.py", line 1221, in store
    dask.compute(arrays, **kwargs)
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/dask/base.py", line 681, in compute
    results = schedule(expr, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 86, in __setitem__
    data = self.get_array(needs_lock=False)
^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 99, in get_array
    ds = self.datastore._acquire(needs_lock)
  ^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 471, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  ^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: '<path>.nc'

real	3m48.154s
user	4m25.010s
sys	3m48.232s

It computes and actually saves few chunks then crashes.
This test was for

INSTALLED VERSIONS
------------------
commit: None
python: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.8.0-63-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development

xarray: 2025.9.0
pandas: 2.3.1
numpy: 2.3.1
scipy: 1.16.0
netCDF4: 1.7.2
pydap: None
h5netcdf: 1.6.4
h5py: 3.14.0
zarr: 3.1.0
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: 2025.9.0
distributed: 2025.9.0
matplotlib: 3.10.3
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.7.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: 24.0
conda: None
pytest: None
mypy: None
IPython: 9.4.0
sphinx: None

2025.7.0

Traceback (most recent call last):
<internal shit>
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/core/dataset.py", line 2029, in to_netcdf
    return to_netcdf(  # type: ignore[return-value]  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/api.py", line 1938, in to_netcdf
    writes = writer.sync(compute=compute)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/common.py", line 357, in sync
    delayed_store = chunkmanager.store(
                    ^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py", line 247, in store
    return store(
           ^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/dask/array/core.py", line 1227, in store
    dask.compute(arrays, **kwargs)
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/dask/base.py", line 681, in compute
    results = schedule(expr, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 86, in __setitem__
    data = self.get_array(needs_lock=False)
^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 99, in get_array
    ds = self.datastore._acquire(needs_lock)
  ^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  ^^^^^^^^^^^^^^^^^
  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: '<path>.nc'

INSTALLED VERSIONS
------------------
commit: None
python: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.8.0-63-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development

xarray: 2025.7.0
pandas: 2.3.1
numpy: 2.3.1
scipy: 1.16.0
netCDF4: 1.7.2
pydap: None
h5netcdf: 1.6.4
h5py: 3.14.0
zarr: 3.1.0
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: 2025.7.0
distributed: 2025.7.0
matplotlib: 3.10.3
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.7.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: 24.0
conda: None
pytest: None
mypy: None
IPython: 9.4.0
sphinx: None

Overall key thing is that dask workers fail on netCDF4._netCDF4._ensure_nc_success

2025-09-15 09:27:03,553 - distributed.worker - ERROR - Compute Failed
Key:       ('store-map-e15c6e7210f2ab0ffd6551d1793be3f6', 0, 0, 83)
State:     executing
Task:  <Task ('store-map-e15c6e7210f2ab0ffd6551d1793be3f6', 0, 0, 83) _execute_subgraph(...)>
Exception: "OSError(-101, 'NetCDF: HDF error')"
Traceback: '  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/dask/array/core.py", line 4622, in load_store_chunk\n    out[index] = x\n    ~~~^^^^^^^\n  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 86, in __setitem__\n    data = self.get_array(needs_lock=False)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 99, in get_array\n    ds = self.datastore._acquire(needs_lock)\n         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 460, in _acquire\n    with self._manager.acquire_context(needs_lock) as root:\n  File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__\n    return next(self.gen)\n           ^^^^^^^^^^^^^^\n  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context\n    file, cached = self._acquire_with_cache_info(needs_lock)\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/mnt/data/WORKSPACE/pyism/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info\n    file = self._opener(*self._args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__\n  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success\n'

If you could maybe let me know how I should proceed with this, should I make an issue for xarray or netCDF4 or dask maybe?

pingponghero12 · 2025-09-15T07:57:12Z

pingponghero12
Sep 15, 2025
Author

it seems to work with:

with dask.config.set(scheduler='synchronous'):
    ds.to_netcdf(file_path)

but I this makes it horrendously slow.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

NetCDF: HDF error when saving big array to netcdf4 with dask.distributed #10734

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

NetCDF: HDF error when saving big array to netcdf4 with dask.distributed #10734

Uh oh!

pingponghero12 Sep 11, 2025

Replies: 2 comments · 1 reply

Uh oh!

SarahG-579462 Sep 12, 2025

Uh oh!

pingponghero12 Sep 15, 2025 Author

2025.9.0

2025.7.0

Uh oh!

pingponghero12 Sep 15, 2025 Author

pingponghero12
Sep 11, 2025

Replies: 2 comments 1 reply

SarahG-579462
Sep 12, 2025

pingponghero12 Sep 15, 2025
Author

pingponghero12
Sep 15, 2025
Author