Help with error "RuntimeError: error during blosc decompression: 0"

[mike-lawrence]

I have a zipped zarr DirectoryStore (here) that throws the error RuntimeError: error during blosc decompression: 0 whenever I try to read the arrays inside. The same script I used for generating this zarr produced others that read just fine, so I'm hoping that the data in this one is recoverable somehow!

[scottstanie]

This would be nice to have any debugging guidance- I got this error (and RuntimeError: error during blosc decompression: -1, or Stored and computed checksum do not match. Stored: b'\xfb\xeb,\x80'. Computed: b'\xdd\xcf\x07\x93'.) on about ~10% of a batch script running 1000 times. It's an opaque enough error that I'm not really sure what could be a path to debug this, or even whether it would be in the new sharding mechanism, or in the upstream blosc library.

[d-v-b]

hi @scottstanie can you say more about the array you are creating? In broad strokes what you are describing sounds consistent with a race condition, but we would need to know more about the array and the execution context to know where / how that could be occurring

[scottstanie]

Sure, it's a workflow that is downloading a stack of images, fitting a simple model to the data, then writing the output to a local file. After, I'm uploading to S3. I wasn't trying to do any reading from or writing to S3 directly. I'm working with geospatial data, so i'm working with xarray and dask (because of the open_mfdataset step).

For the output zarr incoding, I'm using something close to this for the output encoding

print(encoding)
{'connected_component_labels': {'compressors': [BloscCodec(typesize=None, cname=<BloscCname.zstd: 'zstd'>, clevel=6, shuffle=None, blocksize=0)],
  'chunks': (1, 512, 512),
  'shards': (1, 2048, 2048)},
 'displacement': {'compressors': [BloscCodec(typesize=None, cname=<BloscCname.zstd: 'zstd'>, clevel=6, shuffle=None, blocksize=0)],
  'chunks': (1, 512, 512),
  'shards': (1, 2048, 2048)},
 'estimated_phase_quality': {'compressors': [BloscCodec(typesize=None, cname=<BloscCname.zstd: 'zstd'>, clevel=6, shuffle=None, blocksize=0)],
  'chunks': (1, 512, 512),
  'shards': (1, 2048, 2048)},
  ...
  
  ds_fit.to_zarr("local_output.zarr", mode="w", encoding=encoding)

the output, if I used to_netcdf instead, looks like

$ h5ls fit_disp_s1_F08882.nc
average_temporal_coherence Dataset {7733, 9464}
coefficient              Dataset {2}
constant                 Dataset {7733, 9464}
spatial_ref              Dataset {SCALAR}
stderr                   Dataset {2, 7733, 9464}
velocity                 Dataset {7733, 9464}
x                        Dataset {9464}
y                        Dataset {7733}

(Sorry i don't have an actual reproducible example at the moment, this seems to be very annoyingly intermittent)

If this sounds like a race condition, then I could believe that there's some bad interaction between the dask chunks and the output shards. I'm not actually using any LocalCluster with dask... but it's still unfortunate how hard it is to debug, or even reproduce in a simple way