---
# System prepended metadata

title: Sharding Breakout

---

# Sharding Breakout

```
### Improving ease-of-use and performance of sharding
*7 adopter mentions*
- **Lead It**: Mark Kittisopikul (@markkitti)
- **Work On It**: @perlman, @jakirkham
- **Join If Space**: @LDeakin, @keewis, @malmans2, @rouault, @markkitti, @maxrjones
```

## Sub-topics for discussion
1. Who is using sharding and what is the target optimization?
    - What are we trying to optimize? Performance?
    - Choices?
    - Performance
        - writing
        - reading
        - concurrency
    - Discussion
        - Address the large number of small files
            - as an alternative to having large chunks
            - same number of read requests
            - overhead (extra inodes with / separator)
            - problems on (networked) file systems (on clusters)
            - metadata iops overhead
        - Precedence: Neuroglancer pre-computed
        - Why not default?
            - Precedence, no sharding
        - zarr-python
            - No coalescing
        - tensorstore
            - cache headers, transactions
        - Dask + Zarr
        - byte range coalescing
        - Z-order discussion / example
        - Zarr of Zarrs
            - Could this be a VirtualiZarr approach where the chunk is a file format?
        - Currently chunk indices are used. Could we use hashes for load balancing benefits?
        - How can we make writing better?
            - Could we pad chunks to all be the same size to allow jumping to an offset and writing it out?
            - There is interest in direct acquisition cases to write results directly out in Zarr. How can we write directly in sharded/chunked format?
            - Icechunk has compaction, which could allow writing normal chunks that are optimized in the backend. IOW should Zarr be a database?
        - Concurrency
            - Zarr Python v3 has thread and async, which can create contention between other libraries also doing threading (like Dask, Cubed/executor, etc.).
            - 
2. APIs (application programming interface)
    - Experience with existing implementations
        - zarr-python
        - tensorstore
    - Discussion
        - Zarr Python
            - https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#sharding
            - https://zarr.readthedocs.io/en/stable/api/zarr/codecs/index.html#zarr.codecs.ShardingCodec
                - encode and decode are async
            - Index codecs could be a useful extension point
                - https://zarr.readthedocs.io/en/stable/api/zarr/codecs/index.html#zarr.codecs.ShardingCodecIndexLocation
            - How does transpose and shard interact?
                - There might be ways of combining that would break (transposing shard index)
            - Eric: Convenience APIs would be useful
                - Might already have, but would be good to confirm these are working as expected.
                - Try the example in the zarr-python docs
        - TensorStore
            - https://google.github.io/tensorstore/python/tutorial.html
            - Everything works by passing a large `dict`
            - Zarr v3 Sharding
                - https://google.github.io/tensorstore/driver/zarr3/index.html#json-driver/zarr3/Codec/sharding_indexed
            - Need to call `.result()` or `await` to launch compute and get a result
            - Transactions
                - https://google.github.io/tensorstore/python/api/tensorstore.Transaction
                - Can be used to combine multiple changes and batching them
                - 
3. Implementation of sharding
    - It is challenging. How to make this easier?
    - GDAL
    - Basic versus advanced implementations
        - Support Zarr v3 without sharding (SHOULD VS MUST)
        - 1-level of sharding
        - n-levels
        - 
    - Discussion
        - Several compliance levels of implementing Zarr v3
        - Is there anything that needs to change with shards?
        - Is recursion useful? How would this be useful?
        - What spec changes would be worthwhile?
            - Maybe a standalone index?
                - Could help with concurrency (lock index)
                - Appending
            - Would padding be useful?
            - VirtualiZarr / TIFF (w/tiled JPEGs or pages)
            - Learning from COG
                - https://element84.com/software-engineering/is-zarr-the-new-cog/
            - Could Zarr be written in a way that is compatible with other file formats
                - Maintaining headers needed for compatibility with other file formats
                - https://github.com/mkitti/simple_image_formats/blob/main/header_formats.ipynb
4. Recursive (nested) sharding
5. As a variable length codec?
6. Where to put the variable length information?
    - External chunk index?
7. Specification