poor hsload performance

[jonatantreijs]

Uploading a hdf file to hsds using hsload takes a fairly long time.
I have observed this on a small hdf file (<200MB) that contains a lot of groups and attributes.
This particular file takes just about 3 min to upload to a local HSDS instance running an S3 storage backend.
(It has 6 service nodes, but since hsload is sequential, this number should be indifferent).

Since the hsds api is very granular, the number of requests needed to upload my benchmarking shot is about 11000.

[jreadey]

Yes, I've seen this too with files that contain many small objects. Since there's a minimum per request latency, it can take a lot of time just doing the POST requests to create all the objects.

I'm currently working on changes to HSDS to enable more work do get down with each request (e.g. create a group and it's attributes in one request). At the same time, there will be some h5pyd updates coming to take advantage of this without requiring any client code alterations. Also, your PR (#257) will be interesting as well.

I'll keep this issue open and hopefully you'll be able to report back on how these changes (hopefully!) help performance for hsload.

If you have HDF5 filers of this type that are publicly available, please share a link and I will use that as part of my benchmarking.

[jonatantreijs]

Sounds great.
The changes in the upcoming HSDS/h5pyd releases sounds like it would greatly increase the ingestion speed in our case.
I have no publicly available hdf files to share at the moment I'm afraid.