Zarr Bi-weekly ZEPs Meetings

--- tags: zarr, ZEPs, Meeting --- # Zarr Bi-weekly ZEPs Meetings ### **Check out the website: https://zarr.dev/zeps/meetings** Joining instructions: [https://openmicroscopy-org.zoom.us/j/82447735305?pwd=U3VXTnZBSk84T1BRNjZxaXFnZVQvZz09](https://zoom.us/j/300670033?pwd=OFhjV0FHQmhHK2FYbGFRVnBPMVNJdz09#success) GitHub repo: https://github.com/zarr-developers/zeps ## 2022-11-17 **Attending:** Sanket Verma (SV)... **TL;DR:** N.A. **Updates:** - ZEP1 Update, see [here](https://gitter.im/zarr-developers/community?at=6374fae6f9491f62c9b7ea61) - Check out the ZEP1 GH Project board [here](https://github.com/orgs/zarr-developers/projects/2/views/2); maintained by Jonathan Striebel **Meeting minutes:** No one joined! ## 2022-11-03 **Attending:** Josh Moore (JM), Jonathan Striebel (JS), Jeremy Maitin-Shepherd (JMS), Sanket Verma (SV) **TL;DR:** Discussions were held on how to move forward with ZEP1 quickly. The summary can be viewed [here](https://github.com/zarr-developers/zarr-specs/pull/149#issuecomment-1302440391). Then the attendees discussed extensions in V3, and JMS is considering trying with non-zero origin. SV joined the meeting after 30 mins. After that, JS mentioned some high-level issues looming around V3 spec. **Meeting Minutes:** * JMS: number of PRs that could be merged into the working draft - JS: don't want to just close it - JM: can we cross link e.g. JMS' PR? Yes. - ==> once all cross-linked close PR. * JS: when to merge? - JM: when it matches the consensus? - JS: ok, but don't have merge rights. - ==> Let's merge proactively. * see: https://github.com/zarr-developers/zarr-specs/pull/149#issuecomment-1302440391 * extensions - JMS: thinking of trying with non-0-origin - JM: think that's a general principle we should try for all issues/PR is "could it be an extension" - JMS: thinking of extensions as plugins? Not exactly. - JS: how to influence if an implenentation adopts an extension? if there's a concrete implementation / clear interface - JMS: agreed and some obvious ones (codecs) but not clear there will be a broader abstraction - JS: "index transformer" _perhaps_ - or as transformer _if_ multiple of chunking - JMS: unfortunate limitation - JMS: re: transformers - it doesn't make sense to compose a different storage transform _before_ sharding - JS: depends. cache of chunks or shards? also checksum - JM: codec is similar - JMS: caching enabled in code, but not in zarr metadata - JS: that's in spec, yes. "runtime-only" but still before or after - JMS: when implementing sharding, would check if it's first - want to be able to tell the user "this is the graularity to write" - JS: good that it's flexible. like c/f order. - JS: mention in implementation "sharding must be first" - JMS: composing makes for useful extension point - JS: most important point: are we sure enough about the extension points? - _Sanket joins_ 🧑🏻‍💻 - Jonathan: high-level issues looming - paths/URL discussion (needs an issue) - global transformers - variable chunk length (possibly origin offset) - indexing more abstract - upgrade path! (`{“extension”: [“@v2-layout”]}`) ## 2022-10-20 **Attending:** Ward Fisher (WF), John Kirkham (JK), Jeremy Maitin-Shepard (JMS) **TL;DR:** WF is working on the maintenance NetCDF release candidate (`v4.9.1-rc1`), and JMS added CMake support to TensorStore. After this, JMS initiated a discussion on Path structure and was stretched for the remaining meeting. **Updates:** - (WF) Working on maintenance netcdf release candidate (`v4.9.1-rc1`). No new features, just bug fixes and improvements. - (JMS) Added CMake support to TensorStore - Discussion about CMake, dependency management - https://cmake.org/cmake/help/book/mastering-cmake/chapter/CDash.html - https://github.com/cpm-cmake/CPM.cmake **Meeting Minutes:** * (JMS) Path structure * Require or encourage root directory to end in .zarr * How to name all the metadata files? * Root metadata could contain extension information * (JK) Mentioned `.zmeta` metadata file with paths to metadata file * (JMS) About listing * (WF) Possible issues with writing * (WF) Spec vs. library tension * (JK) Have file expire? * (JMS) Handle as read-only * (JK) Could also delete as part of writing? * (JMS) HDF5 has hierachary and Zarr replicates this * Have some array and non-array data next to each other * (JK) Examples? * (JMS) Segmentations & mesh representations * (JMS) Collection of volumes with annotations related to them * (WF) Have Zarr hierarchy with non-Zarr? * (JMS) Only have single individual arrays * (WF) Wouldn't have considered this structure * (WF) Does there need to be something in the spec about interleaving data? * (WF) Maybe interleaving poses some challenges * (JMS) Doesn't NetCDF have extra files as well? * (WF) Yes. Extra metadata used to map Zarr model to NetCDF model. * (JMS) Reason to use this structure as opposed to Zarr metadata files? * (WF) NetCDF supports different formats HDF5, Zarr, etc. * (JMS) Have user defined attributes. Types are stored in metadata file? Could those be in zattrs? * (WF) Yes. Not sure * (JMS) Hierarchy becomes more apparent with V3 as opposed to V2 * (WF) Groups were a new feature that users were slow to pick up on * (JK) Does adding more top-level metadata cause issues? * (JMS) Could it contain the metadata? * (WF) Maybe include subset of metadata * (WF/JMS) Perhaps special case single array use case * (JK) How does data relate in non-hierachical form * (JMS) Related, but not all Zarr data * (JK) Would other kinds of chunk formats (standardizing on kerchunk) be useful * (JMS) Meshes probably don't make sense in this way * (JMS) Neuroglancer meshes are a good example * (JMS) Sparse arrays seem similar in that they might be better handled by being their own file format * (WF) NetCDF users mention performance issues in moving to new version. Usually suggest using old NetCDF. Maybe same with V2/V3? * (JMS) Want to use V3 (sharding being of value). * (JK) Including unstructured binary blobs in Zarr? * (JMS) Has a group of files for mesh * (JK) Maybe ignore specific paths? * (WF) Having mixed media is valuable though can be logisticially tricky * (WF) What defines Zarr as a data model? At least need to say some behavior is undefined (mixed media). Ideally ignores mixed media files. * ## 2022-10-06 **Attending:** Ward Fisher (WF), Josh Moore (JM), Jeremy Maitin-Shepard (JMS), Greg Lee (GL), Jonathan Striebel (JS) **TL;DR:** JM shared that there were some good conversations around OME-Zarr yesterday. The summary is available [here](https://forum.image.sc/t/ome-ngff-community-call-transforms-and-tables/71792/10). WF shared that Kitware is looking for partners and a link to the sign-up form. GL shared that during the CZI Open-Science Summit 2022, he worked on writing tests for Xarray. After this, there was an extensive discussion on URL syntax initiated by JMS. **Updates:** * miscellaneous reading before the meeting (JM) - https://arrow.apache.org/blog/2022/10/05/arrow-parquet-encoding-part-1/ - https://github.com/kaitai-io/kaitai_struct/issues/125 * NGFF (JM) - https://forum.image.sc/t/ome-ngff-community-call-transforms-and-tables/71792/10 - Good conversations around OME-Zarr yesterday * Enthusiasm for Kitware (WF) - Looking for partners. [Have form on webpage](https://www.kitware.com/contact/project/). - Unidata an option. They've mentioned Zarr a couple of times (Kitware Blog). * xarray test (GL) - during czi conference. - release of 2.13 hopefully fixed it all :tada: **Meeting Minutes:** * URL syntax? (JMS) - helps to figure out the metadata location. - Josh: great idea. have several ongoing discussions at the NGFF level - current proposal would be to support URIs internally (relative, absolute, remote) - however, in V2 - JMS: in v3 the root exists - though not entirely clear that the new metadata organization is necessary - designed for S3 where there's no directory, but other problems exist - Josh: _summarized previous discussions for Greg_ - GL, thoughts on the V3 situation? - GL: at the moment, you need helper methods to do that. - JM: one proposal was to have the metadata be the main directory which lets you then bootstrap the chunk loading - JMS: support multiple? - JM: conceivably. as extension or configuration. - JM: downside for consolidated metadata is that nothing exists in the metadata hierarchy - workaround of having a thin-hierarchy only with references to where the metadata exists - JS: losing the ability to be able to next any hierarchy. (everything is a root) - JM: are we proposing rolling it back completely - JMS: problem is the URI+rootpath metadata - JS: walking up the hierarchy would be an option (URL doesn't actually point) - JMS: would be nicer if you don't have to perform a search - Use case - URL case - Desktop double click on something - Similar issue: **Zips** :warning: - JMS: have an additional level - JM: except ZipStore v2 assumes the whole zip is a zgroup - JS: propose zip is a special case which is _easier_ - JMS: unless you are mixing volumetric with a zarr then it wouldn't be at the top-level - Btree (JMS): need to be able to compose multiple layers (similar to fsspec and double colons) - Remote chunk store (or point to V2 chunks) - Renaming folders (keep data with arrays) - Options - Keep "/meta", clients must know - Drop "/meta", direct URLs - `?param` syntax - `#param` syntax - Separator syntax (e.g. "`//`") - root dir ends in .zarr - fsspec `::` separator - multiple protocols (git+ssh, zip+zarr) - further discussion - JS: without /meta and .zarr requirement, you still don't know where the root is - JS: if you drop "/meta" then you can't name anything "/data" - JMS: could use something more obfuscated - JS: why split? - JMS: if you are not using the filesystem (s3 or gcs) and you want to list all the metadata, it's not (as) efficient - JM: "data" could be registered in the metadata so it's a known (and configurable) thing - WF: NC anything with leading underscore is assumed reserved for the library - permitted to create them, but the spec says "please don't" - JM: `.z` prefix - WF: utilities and tools can scrape everything with that - WF: also don't have to put too much thought into new features - JMS: would prefer not a `.` prefix because of archiving tools, etc. - then `_z`? - JMS: root metadata file doesn't really do anything - JS: creates ambiguity - JM: think it was largely for bootstrapping global plugins (e.g. transformers) - JS: perhaps V2 compatibility - JMS: not clear you would nest sharding with other transformers. it would be the thing applied to the chunks. - JS: the metadata needs to be somewhere, and for that can be at the array level - brief summary - zarrs are essentially a metadata hierarchy - that configure (possibly remote) chunk stores - and the root is identified with .zarr ## 2022-09-22 **Attending:** Ward Fisher (WF), Josh Moore (JM), Ryan Abernathey (RA), Jeremy Maitin-Shepard (JMS), Dennis Heimbigner (DH) **TL;DR:** Consolidate metadata needs an extension for V3, which might result in a new ZEP. Next, JMS shared a document titled ‘Optionally-cooperative distributed b-tree for Tensorstore’. The participants discussed the document after that. After that, JM initiated the discussion on codecs-registry, which was built by one of the GSoC students this summer. The meeting ended with a discussion on the path to the metadata files. **Meeting Minutes:** - Java/NetCDF side: - JM: Sanket met people - WF: Unidata should be 3x the staff. - JM: perhaps starting with a kerchunk implementation? - WF: looking for more community involvement (like netcdf-c had) - JM: Greg mentioned consolidated metadata needs an extension for V3 - RA: Iceberg issue, also see JMS' proposal - https://github.com/zarr-developers/zarr-specs/issues/154 - JMS: touches on not needing a file per chunk (like discussed last night) - https://docs.google.com/document/d/1PLfyjtCnfJRr-zcWSxKy-gxgHJHSZvJ2y4C3JEHRwkQ/edit?resourcekey=0-o0JdDnC44cJ0FfT8K6U2pw#heading=h.8g8ih69qb0v - db format that stores a btree. - uniquely: designed to allow distributed writes (s3, etc.) *but* doesn't need a peristent database - can also read it in a non-distributed fashion - downside: adds quite a bit of added complexity (greatly for binary format) - also good where sharding isn't appropriate (e.g. pre-defined shard size which is required for write) - e.g. large number of small arrays (where sharding won't help) - RA: nice document. comments: - focused on big distributed writes, but with iceberg had a different main motivation: more flexibility in mapping keys to chunks. kerchunk-like. virtual concatenate . can you reference random chunks? yes. - JMS: btree nodes have references to files (like kerchunk). but datafiles are identified with 128-bit path (not an fsspec URL) - RA: different use case, so can have them be optional transformers/extensions - RA: really similar to tiledb! why not use it? - JMS: tiledb is organized by time not space. - JM: need a compaction - JMS: and even after that you still have a million files. - DH: HDF5? internally it's btrees. (which is responsible for most of its complexity). Are you sure this is the path? - JMS: not sure there's an alternative to btrees. used in databases, filesystems, etc. - DH: if you don't want some ordered searches, then linear hashes are an alternative - JMS: ordered is useful for a lot of use cases. but there wasn't an obvious solution for distributed writes - DH: [extendable hashing](https://en.wikipedia.org/wiki/Extendible_hashing) is an easier data structure (old paper) works well with disk storage. - JMS: think this is more a key-value store (like zip) - RA: agreed. Nice that it's possible to experiment like this. - RA: can the V3 spec support this experimentation? (right extension points?) - RA: trying to do that with Iceberg. Martin suggested "IceChunk". - See also: hooty and others. Lots of smart ideas that we can copy. - Goal is to provide some level of branching & transactions for/on a Zarr store - Allow you to work on your staged area which all get written at once. - Branch non-destructively (or rollback) - The key is having a "manifest" (they all have some concept of that, even kerchunk) - Don't depend on the object stores listing as the source of truth - Need storage transformers at the top level, not array. But for JMS' idea array-level might suffice. - JMS: wasn't planning on an extension. root metadata would be in the same data store. - JM: basically writing DB/filesystem :+1: ZarrFS ;) - JMS: planning on mongo? Yeah, or Dynamo. (They store JSON) - JSON in S3 isn't ideal. - metadata in document store and chunks on disk. Beyond just filesystem. It's a data lake. - "meta-store" - JMS: regarding versioning, how are you representing the delta? - The chunk is the minimal writable unit. (out-of-scope) - Every chunk write is a uniquely ID'd (e.g. content addressable). That gets a key. Write that to DB. - JMS: expecting the database to provide the versioning? - RA: no, just a place for documents. versioning (in iceberg) has a branch or a tag that points to a specific chunk manifest. you can create a new one and point your HEAD at that. only rely on database to atomically change the references. iceberg tracks a number for the transaction. - JMS: use kerchunk model? limitation on the number of chunks? - RA: chunks are likely in a separate manifest. discussed that another extension with Martin. - RA: but can just query a chunk from the database. - JMS: 1M chunks in v1. then update to v2. What's the diff? A copy. - RA: yeah need to play with it. - JMS: when you get to wanting to update just a portion of it, then you get to b-trees :smile: - RA: no db guys, trying to keep it hackable. - RA: but megabyte kerchunk is already getting :heart: since it's so easy. looking for incremental improvement on _that_. (NASA will be pumping out GRIB forever...) - JMS: looking forward to hearing more and exchanging info re: b-trees - JMS: see also https://github.com/janelia-flyem/dvid (backed by KV database) - JM: sharing layers with them? - JMS: complicated by other priorities of the EM team. invite Bill to the Zarr meetings? - RA: see https://lakefs.io/ - JM: API versus format - RA: thinking about it more like an API - JM: briefly codecs-registry - https://zarr.dev/codecs-registry/ - https://github.com/zarr-developers/codecs-registry - JMS: still want a schema per codec. JM: agreed! - JMS: talks about codecs having URLs. - would by an annoyance to have difference V2 and V3 identifiers. - e.g. just numeric constants in the JSON that are from the C API - e.g. shuffle parameter which would be nicer as a string. - support integer or string for a while (in order to deprecate) - JM: have plans to have code in each languages that checks for an id from the central registry - DH: approx. that with nczarr. ncdump lists the actual codecs in the file - would be good to have something more sophisticated - have the disadvantage of C code and interpreted files - 3 repositories on the C side. unidata + irvine + hdf5 - hdf5 only has names, hdf5-ids and a pointer (which is often out of date) - something universal would be nice - WF: roping in the HDF5 group would be a heavy lift - JMS: **URL interface** :rocket: - DH: :+1: for the REST API - WF: NSF/CSSI solicitation has opened - https://beta.nsf.gov/funding/opportunities/cyberinfrastructure-sustained-scientific-innovation-cssi - perhaps something here - WF: planning on getting to https://www.egu23.eu/ - tweet something from zarr_dev to see if there is interest :question: - could collaborate something re: nc/zarr - JMS: don't have clear resolution on the paths to the metadata files - JM: re-capped the previous discussion and think it's still good. - JMS: some details around the root array (the named files, etc.) - JMS: consolidated metadata? duplicated? - JM: would make it possible to have everything in the top-level - JMS: pointers in the subdirectories? bit annoying. - JMS: with iceberg & co. you likely don't need a consolidated metadata - JM: so you'd push it to the store level? - JMS: possibly, but not that simple - JMS: there are cases where you need path separation anyway (Zips) - JMS: so could see using a path separation strategy entirely - JMS: Davis did have a use case ... - (...details zip, consolidated brainstorming...) - JM: need both solutions... ## 2022-09-08 **Attending:** Sanket Verma (SV), Josh Moore (JM), Ward Fisher (WF), Jonathan Striebel (JS), Norman Rzepka (NR), Ryan Abernathey (RA), Dennis Heimbigner (DH), Jeremy Maitin-Shepard (JMS) **TL;DR:** This was the first ZEP meeting ever. Representatives from the [Zarr Implementations Council](https://github.com/zarr-developers/governance/blob/main/GOVERNANCE.md#zarr-implementation-council-zic) joined the meeting. Most discussions revolved around [ZEP1](https://zarr.dev/zeps/draft/ZEP0001.html). One of the critical decisions about ZEP1 was to accept it ‘[Provisionally](https://zarr.dev/zeps/active/ZEP0000.html#review-and-resolution)’ and move forward with the implementation in various programming languages. **Updates:** - SV: open ("draft") ZEPs: - https://zarr.dev/zeps/draft/ZEP0001.html - https://zarr.dev/zeps/draft/ZEP0002.html - SV: Author discussion on all the comments on ZEP0001 - Have proposed resolutions for a number of those - Meeting again tomorrow to try to finish the list - https://hackmd.io/sOos8rxrRvKCJPbbUKWtwA?view - SV: Critically propose marking ZEP0001 as "provisionally accepted" after the above are handled and passed by the ZIC - Implementations are free (and encouraged!) to start implementing. - Any blocking changes could still be handled. - Otherwise, "feature freeze". - Feedback? - RA: process to get to provisionally accepted - SV: draft == under review. - on vote, can move to provisionally or accepted state. - once implemented, moves to final. - could move to "deferred" state if the ZIC vetoes - WF: "ready to implement" jumped out (and caused anxiety but only since there's too much to do) - https://zarr.dev/zeps/active/ZEP0000.html#review-and-resolution - JMS: no substantial changes since early draft - JM: editors are preparing a rebuttal (Alistair's paper model) - JMS: not sure a paper model is best - RA: not in the sense that there's only one round and someone will decide. iterative - good to have authors who are organizing. - now in revision and we can continue until everyone is happy - gone slowly for various reasons (availability, summer, and it's our first time & massive) - would be useful to go through the outstanding issues - JS: in this cycle and not limited iterations is just the limited time. - but for now, trying to make batched changes **Agenda:** - JS: **review of memory order decision number-16 from list** - zarr's goal is interoperability. therefore propose to keep C & F (benefit for community) - could support read only, even with a transpose (if too slow, add a warning?) - JMS: agree. but would like an arbitrary permutation. - DH: good use case? - JMS: dimension that represents time. order you display to the user is logical for them but need not be logical for compression/access patterns. - JM/JS: core or extension? - RA: that's a key question - NR: re: backwards compatibility C/F is in V2 therefore that would need to be in core. but arbitrary could be an extension. - RA: but v3 is a chance to break backwards compatibility (explicitly not a goal) - NR: upgrade path? so be able to upgrade without re-writing the chunks. - RA: v2 will still be supported. - WF: that would be the hope, but worry about netcdf & archival -- assuming software will support it without it being expressed somewhere. aspirational sure but makes us nervous. - e.g. will future software implement the v2 standard? - RA: transform based solution? (but only if we support F) **if** we say the chunks should be backwards compatibility. - WF/DH: no one has ever asked for arbitrary. Someone at NOA asks for things that would help their lab. Technical debt. (Won't even request a pull request) See the trap that the HDF group fell into (single-writer-multiple-reader, several orders of magnitude that they are trying to recover from.) - JMS: arbitrary seems most natural. pass to `numpy.transpose()` - WF: shocked at the assertion that there _wouldn't_ be a migration path - JM: clarification -- were only differentiating if _binary_ transformation is needed - **can add _requirement_ to v3 that implementations read v2** - WF: requirement of netcdf. can decide if that's a requirement. - DH: depends if it's alot. operational definition - "too painful to copy v2 to v3" - (for RA): petabytes of data - JS: RA proposed transformer strategy - essentially rewriting metadata **formalize it?** - DH: how comfortable are you not supporting older version? - JM: for OME, got agreement but that's a layer higher - DH: will there be new implementations without V3 support? - NR: think there will be - JM: but it's so easy to implement - WF: people won't do that...what do we do if a popular implementation doesn't support v2? - other packages? - RA: recommend storage layer / translation? - JM: agreed but that's SHOULD (versus MUST) - JMS: only way to force it is a standardization - JM: agreed, but we can only do what the spec document allows us (i.e. labeling something as "compliant") - JS: it's a new major version and people know what we mean. (as a user, I wouldn't expect support for v2 if an implementation says "v3") - WF: convinced myself I'm worrying too much instead: - WF: in 18 months how do you know which Zarr is used to open it. - JM: metadata file is different (essentially the magic number). The proposal for `.zr3` was currently turned down. - SV: [data type naming](https://github.com/zarr-developers/zarr-specs/pull/149#discussion_r929140806) - JM: dropping the python-ness - JMS: helps provide a more nature scheme for some datatypes (and endianness as a codec) - no argument against (just "convenient in Python") - JM: will need names - JMS: in https://github.com/zarr-developers/zarr-specs/pull/155 - DH: netcdf ncchar type equivalent to 8-bit ascii, no equivalent in Zarr. Needed? NC uses it all the time. Why not in numpy? - JMS: thought numpy has char. - RA: revisit char question? JMS: different than varstring - RA: where does the encoding go? DH: in an attribute. "ascii" (or "utf-8") - RA: used for? DH: you see a lot of flags stored that way. - also historical: NC-3 didn't have strings of any type. (arrays of chars workaround) - JM: extension mechanism? - DH: where the wheel hits the road - JMS: just metadata? - RA: disagree, influences ... - JMS: agreed, but doesn't change how hard it is to implement - JM: but need to feel confident that they are low cost so we can change *when* we discuss these things - JMS: will changes start appearing? - JM:

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.