Zarr / OME-Zarr support

[jcupitt]

Discussed in #3516

^{Originally posted by jstriebel May 30, 2023}
It would be great if libvips could support Zarr, and possibly also OME-Zarr (also called OME-NGFF) for metadata. Zarr is a chunked storage format and API specification for n-dimensional arrays, and a common format in the bio-imaging and geo-spatial domains, especially for large datasets. Metadata is written in json files, and chunks are stored as separate files by default, optionally compressed by different algorithms, such as gzip or blosc. It is often used in cloud contexts, but also with local files. For libvips, support of local files would be awesome, I'm not sure if and how support of remote storage such as GCP or AWS would fit in.

Zarr just approved the v3 specification, which probably should be considered for implementation instead of v2.

I'm aware of the following C / C++ libraries for Zarr:

• https://github.com/constantinpape/z5
  (A pure C API is discussed in C API wrapper for z5 constantinpape/z5#68)
• https://github.com/google/tensorstore
• https://github.com/xtensor-stack/xtensor-zarr
• https://github.com/Unidata/netcdf-c

OME-Zarr (or OME-NGFF) is the successor to OME-Tiff, which in either case adds a metadata specification and layout structure for microscopy images. Supporting this is not super important IMO, but might be useful for storing relevant metadata, such as resolutions.

It would be great and interesting to hear some thoughts of the maintainers about adding Zarr support. About me: I'm a co-author of the Zarr v3 spec, and happy user of libvips.

[jstriebel]

I just double-checked the different implementations for compatibility with Zarr v3:

• z5 is not adapting zarr v3 soon
• tensorstore is implementing v3 and sharding very soon
• xtensor currently has an outdated implementation of an earlier v3 version, which might get updated, but there have not been new commits since 1½ years.
• netcdf will implement v3

@jcupitt Do you think integrating tensorstore would be a viable option? Also ccing @jbms.