HackMD
    • Sharing Link copied
    • /edit
    • View mode
      • Edit mode
      • View mode
      • Book mode
      • Slide mode
      Edit mode View mode Book mode Slide mode
    • Customize slides
    • Note Permission
    • Read
      • Owners
      • Signed-in users
      • Everyone
      Owners Signed-in users Everyone
    • Write
      • Owners
      • Signed-in users
      • Everyone
      Owners Signed-in users Everyone
    • Commenting & Invitee
    • Publishing
      Please check the box to agree to the Community Guidelines.
      Everyone on the web can find and read all notes of this public team.
      After the note is published, everyone on the web can find and read this note.
      See all published notes on profile page.
    • Commenting Enable
      Disabled Forbidden Owners Signed-in users Everyone
    • Permission
      • Forbidden
      • Owners
      • Signed-in users
      • Everyone
    • Invitee
    • No invitee
    • Options
    • Versions and GitHub Sync
    • Transfer ownership
    • Delete this note
    • Note settings
    • Template
    • Insert from template
    • Export
    • Dropbox
    • Google Drive Export to Google Drive
    • Gist
    • Import
    • Dropbox
    • Google Drive Import from Google Drive
    • Gist
    • Clipboard
    • Download
    • Markdown
    • HTML
    • Raw HTML
Menu Note settings Sharing Help
Menu
Options
Versions and GitHub Sync Transfer ownership Delete this note
Export
Dropbox Google Drive Export to Google Drive Gist
Import
Dropbox Google Drive Import from Google Drive Gist Clipboard
Download
Markdown HTML Raw HTML
Back
Sharing
Sharing Link copied
/edit
View mode
  • Edit mode
  • View mode
  • Book mode
  • Slide mode
Edit mode View mode Book mode Slide mode
Customize slides
Note Permission
Read
Owners
  • Owners
  • Signed-in users
  • Everyone
Owners Signed-in users Everyone
Write
Owners
  • Owners
  • Signed-in users
  • Everyone
Owners Signed-in users Everyone
Comment & Invitee
Publishing
Please check the box to agree to the Community Guidelines.
Everyone on the web can find and read all notes of this public team.
After the note is published, everyone on the web can find and read this note.
See all published notes on profile page.
More (Comment, Invitee)
Commenting Enable
Disabled Forbidden Owners Signed-in users Everyone
Permission
Owners
  • Forbidden
  • Owners
  • Signed-in users
  • Everyone
Invitee
No invitee
   owned this note    owned this note      
Published Linked with GitHub
Like1 BookmarkBookmarked
Subscribed
  • Any changes
    Be notified of any changes
  • Mention me
    Be notified of mention me
  • Unsubscribe
Subscribe
--- tags: zarr, ZEPs, Meeting --- # ZEPs Bi-weekly Meetings ### **Check out the website: https://zarr.dev/zeps/meetings** Joining instructions: [https://openmicroscopy-org.zoom.us/j/82447735305?pwd=U3VXTnZBSk84T1BRNjZxaXFnZVQvZz09](https://openmicroscopy-org.zoom.us/j/82447735305?pwd=U3VXTnZBSk84T1BRNjZxaXFnZVQvZz09) Meeting ID: 82447735305 Password: 016623 GitHub repo: https://github.com/zarr-developers/zeps ## 2023-11-30 **Attending:** Sanket Verma (SV), Ward Fisher (WF), Jeremy Maitin-Shepard (JMS) **TL;DR:** **Updates:** - No updates **Open agenda (add here 👇🏻):** - WF: Recording a virtual talk for AGU next week - https://discourse.pangeo.io/t/pangeo-showcase-how-to-transform-thousands-of-cmip6-datasets-to-zarr-with-pangeo-forge-and-why-we-should-never-do-this-again/3856 - CMIP6 trying to get rid of NetCDF? - WF: Zarr as archival? - Trying to be there - much younger format - but NetCDF is a robust archival format - SV: The interoperablility b/w Zarr and NetCDF is a good thing in here - WF: - WF: Zarr implementation in FORTRAN? - JMS: Are people writing FORTRAN actively? - WF: Yes - The latest book on FORTRAN came last year - NCZarr is supported via FORTRAN - JMS: Zarr V3 FORTRAN would be good - WF: NetCDF FORTRAN is used by supercomputers across US - SV: Why end NetCDF FORTRAN? - WF: Selfish reasons - takes a lot of time fixing the modules - SV: Why supercomputers still use FORTRAN? - WF: Supercomputers unparalleled performance using FORTRAN is just great! - WF: E.g. Mathworks upgrading to newer NetCDF version is a long process - WF: - Good to go with? - [zarr-developers/zeps/#48](https://github.com/zarr-developers/zeps/pull/48) - JMS: Still need to work on this - also work on the implementation of the ZEP - TABLED - Let's go ahead with [zarr-developers/zarr-specs/#276](https://github.com/zarr-developers/zarr-specs/pull/276)? - Thoughts on [zarr-developers/zarr-specs/#205](https://github.com/zarr-developers/zarr-specs/pull/205) - Good to go with? - [zarr-developers/zarr-specs/#271](https://github.com/zarr-developers/zarr-specs/pull/271) ## 2023-11-16 **Attending:** Sanket Verma (SV), Josh Moore (JM), Ward Fisher (WF), Norman Rzepka (NR), Jeremy Maitin-Shephard (JMS) **TL;DR:** **Updates:** - Sanket recently added [PR#281](https://github.com/zarr-developers/zarr-specs/pull/281) to update ZEP2 status - Discussions around ZEP8: https://github.com/zarr-developers/zeps/pull/48 - Jeremy to update the PR - Davis update ZEP6 - Question to Davis: Status of completion? **Open agenda (add here 👇🏻):** - Cleaning of zarr-specs.readthedocs.io - Remove 'Under construction' - Remove 'Array storage transformers' - Maybe rename data types to extension data types? - JMS: [bfloat16 dtype](https://github.com/zarr-developers/zarr-specs/pull/257) should be in core spec under data type table - required and optional table separately - NR: May need some explanation on the data type - NR: Remove version number from the codecs and stores - JMS: Could track down the implementations adopting the different versions of codecs/stores - NR: No version number in metadata, so not useful - JMS: But you'd not be allowed to change the metadata - JM: Helps to write down this; for example a ZEP - JMS: Implementations might not implement all the versions - JMS: How about STAC? - SV: STAC uses incremental versions to evolve the specification - NR: Zarr V2 & V3 compatbility issues may arise in the future - JM: Flip side if we're dealing with a long list of codecs - versions may help here - JM: For example: Bumping the Blosc2 codec - JMS: Pretty rare to change the versions - JMS: Having a shorter voting period may help - SV: - JM: Having a step voting period may be troublesome - experience from NGFF and OME-Zarr - strong word for the first phase - NR: Silence in the second phase? - JM: It'll be good! - NR: How do we make the vote earlier? - JM: Once month for roll call - done reading, start implementing. finish implementing, also no vetos - JM: Graph for the progress - JMS: Make revisions after the voting - working for now but not great - JMS: A word for grace period: `Final revision deadline` - NR: Finding 1-2 champion for starting a new ZEP - JM: Having a mailing list to ask for champion - JM: Close the ZEP proposal if it's not active for sometime - JMS: C++ standard is much complicated than Zarr - only some people capable of changing the wording - SV: If someone is not able to find a champion should they not proceed with the ZEP? - Not in favour of the champion process to be the only condition to move forward - JM: List of endorses and endorsement for the ZEP process - FYI: John Bogovic may join future ZEP meeting for ZEP8 discussions - TABLED - Thoughts on [PR#276](https://github.com/zarr-developers/zarr-specs/pull/276)? ## 2023-11-02 **Attending:** Josh Moore (JM), Sanket Verma (SV), Jeremy Maitin-Shepard (JMS), Jonathan Striebel (JS) **TL;DR:** **Updates:** - [ZEP0002](https://zarr.dev/zeps/accepted/ZEP0002.html) finally accepted! 🎉 **Open agenda (add here 👇🏻):** - Changes to ZEP0 - https://github.com/zarr-developers/zeps/issues/52 - JMS: Don't think there's gonna be any changes to ZEP1 now - JM: Could be 2 voting - but how do you get everyone to implement ZEP without voting? - SV: Yeah! Good point. - JMS: Don't need to go overhead for small changes - JM: `index_location` is backwards compatible and maybe the best case scenario - JMS: But endian codec is not backward compatible - SV: Very natural to see this scenario coming up - JM: Architect building a building - they go through submittals - 20% increment - show the adoption percentage - SV: How would you define percentage? - JM: Voting could be in phases - reading phase - implementation change - grace period - *Jonathan joins* - JS: Voting encourages people to read the spec - JM: Setting a common calendar for the ZIC and ZSC - can help the author - JS: It was good to the response when we set the deadline - also the examples of implementation currently is good - JMS: Would be great to have a table of what part of spec they're current implementing would be great - JMS: Having a compatibility table would be great - e.g. Neuroglancer doesn't support boolean type - JMS: Once a ZEP is accepted the spec matters - JM: Looking at RFC for NGFF these days - JMS: Random reviews vs the expert group reviews - SV: If you're going to implement the spec and how you're going to implement the spec - Form a voting procedure around that - JS: Having a process definitely helped me to finsh the sharding - find it good as contributor! - JM: Having rebuttal would change the tone a lil' bit - JS: Telling to vote before implementation is fine - as you can find things spec - JM: Having it defined in ZEP0 would be great! - JS: Doesn't like the grace period - leave the door open a little - JS: Having an implementation phase would be good - SV: Reading notification during the voting phase - could be helpful - JM: Keeping a table would be helpful - 5 states - JMS: Three phases - - reading phase and express opinions - implementation phase and raise issues - solve issues raised in the last phase - JM: It's clear we're in agreement of phases/periods - intent - JM: People want general confidence from the audience that they're moving forward - JMS: https://chromestatus.com/feature/6213121689518080 - JMS: Getting formal feedback from the ZIC before the vote - SV: Let ZIC know when you're going to put up the ZEP for voting - JM: Worst case scenarios - no-one read the spec and someone vetos the ZEP in the initial stage - JM: Roll call helps - JS: May not need a second vote - those are implementation details - JM: Add a different state - a pre-implementation checkpoint - Endian codecs to bytes - https://github.com/zarr-developers/zarr-specs/pull/263 - JMS: Need to resolve git conflicts - Changes to ZEP2: - https://github.com/zarr-developers/zarr-specs/pull/280 (add index_location) - https://github.com/zarr-developers/zarr-specs/issues/277 (versioning) - https://github.com/zarr-developers/zarr-specs/issues/278 (tracking intermediate buffer sizes) - Adding v1 and v2 spec to zarr-specs - https://github.com/zarr-developers/zarr-specs/pull/276 ## 2023-10-19 **Attending:** Sanket Verma (SV), Mark Kittisopikul (MK), Ward Fisher (WF), Davis Bennett (DB), Norman Rzepka (NR), Isaac Virshup (IV) **TL;DR:** **Updates:** - ZEP0002 voting closes on 31st October - https://github.com/zarr-developers/zarr-specs/issues/254 **Open agenda (add here 👇🏻):** - MK: Changes to ZEP0001 are still coming in - how do we handle them? - SV: ZEP1 was provisionally accepted but not at the final stage - NR: These changes are minor and would need voting from ZIC - MK: Mention potential grace period in ZEP0000 would be helpful - NR: Needs to be written out! - MK: Zarr shards as HDF5 file - ZEP0002 should proceed as it is atm - Having a null codec to ignore the checksum would be helpful - can work on this - Recent numcodecs release helps a lot - getting Jenkins lookup checksum - Relation with ZEP0002 and Kerchunk? - NR: Don't know if there is - IV: Zarrita can read the HDF5 file using Kerchunk - IV: Multiple arrays in a single Zarr shard - NR: I don't think it'll be possible - DB: Why the chunks in the directory called C? - NR: Helps when when scanning down the groups and arrays - DB: Any questions for ZEP0006? - NR: Getting a JSON schema out of the ZEP0006 would be helpful - DB: This would also help us to get a container level validation - IV: How differs from consolidated metadata? - DB: Consolidated metadata - NR: Serializing children metadata would be helpful - could be a different ZEP - DB: Flattening array - DB: Pydantic-Zarr would have the reference implementation for ZOM - ZEP0002 discussions at Unidata - Mostly going for 'Yes' - but looking for resources who can handle and complete it - MK: NetCDF has adopted HDF filters? Making a part of Zarr filters? - WF: Would like to see spec support - supports interoperability - but hasn't considered it - MK: Between N5 and Zarr we encounter difficulties for LZ4 codec - DB: Sounds like N5 problem to me! ;) - IV: List of HDF5 filters? MK: Yes, there's a list - MK: But I think NCZarr support a select few only - WF: Yes! - MK: List of HDF5 Registered Filters: https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins - GitHub library for plugins: https://github.com/hdfGroup/hdf5_plugins - DB: Getting away from storing F9 transformations in metadata - IV: Thoughts on namespacing codecs? - NR: Would be helpful - WF: The overhead of maintenance and administration is daunting - experience from Unidata - How would I guarantee that information would be there in 10 years? - IV: Pointing at URI where the codec is defined - Having this in Zenodo would be helpful - IV: ZEP0003 progress - SV: Waiting for technical review - would help if Martin/IV could make it the ZEP meetings to raise the discussion - IV: Sure ## 2023-10-05 **Attending:** Sanket Verma (SV), Josh Moore (JM), Thomas Nicholas (TN), Jonathan Striebel (JS), Jeremy Maitin-Shepard (JMS) **TL;DR:** **Updates:** **Open agenda (add here 👇🏻):** - Introductions with favourite book - SV: ASOF by GRR Martin - TN: Dispossessed - JS: Hitchhicker's Guide to Galaxy - JM: Kingkiller Triology (Rothfuss) — can HIGHLY recommend. Beware: only 2 of the 3 books is written. - Issues and PRs to look at: - https://github.com/zarr-developers/zarr-specs/issues/266 - JS and JMS: Mostly an implementation detail - JMS: Make store less explicit - JS: Should not enforce the parameter; will send a PR after 2 weeks - https://github.com/zarr-developers/zarr-specs/issues/241 - JMS: Will try to make it more clear - https://github.com/zarr-developers/zarr-specs/pull/263 - JMS: Chris Barnes' implementation hasn't made the change yet - SV: Send an email for this to ZIC - JMS: https://github.com/zarr-developers/zarr-specs/pull/264 - JS: Fortran array needs to be inverted - JMS: C and Fortran array are contiguous in different ways - JMS: Neuroglancer and Tensorstore has 100% V3 implementation 🎉 - Working on some CI issues and will merge it - JM: https://github.com/ome/ngff/pull/206 - TN: Cubed discussions - Anything which increases the performance would be useful - interested in Jack's work - How can we get variable chunking into Zarr-Python? - SV: Needs to finalise the ZEP0003 - will go into voting soon! ## 2023-09-21 **Attending:** Sanket Verma (SV), Josh Moore (JM), Ward Fisher (WF), Davis Bennett (DB), Jeremy Maitin-Shepard (JMS) **TL;DR:** **Updates:** - ZEP0006 by Davis Bennett: https://github.com/zarr-developers/zeps/pull/46 (Zarr Object Model) - Implementation: https://github.com/zarr-developers/zarr-python/pull/1526 - ZEP0007 by Isaac Virshup: https://github.com/zarr-developers/zeps/pull/47 (String) - ZEP0008 by Jeremy Maitin-Shepard: https://github.com/zarr-developers/zeps/pull/48 (URL Syntax) **Open agenda (add here 👇🏻):** - Overview of the newly submitted ZEPs - DB: - JMS: ZEP8 Could be generalised apart from the Zarr ecosystem - provides parameters at each specific level - DB: Considering query strings? - JMS: Clearly diverging from convential syntax - - JMS: Issue with `#` syntax - interpretation will be different - a few downsides of using it - argument for using fragment identifier - JM: Descending down the attributes in N5 land? - https://github.com/ome/ngff/issues/144#issuecomment-1314299599 - DB: Query syntax like JSON - the idea was shared among and used across in N5 land - JMS: - DB: ZEP0006 (ZOM) discussions - [Tally Lambert](https://github.com/tlambert03) from Napari was looking for JSON schema for Zarr - JMS: JSON Schema for tensorstore - V2: https://github.com/google/tensorstore/blob/master/tensorstore/driver/zarr/schema.yml - V3: https://github.com/google/tensorstore/blob/master/tensorstore/driver/zarr3/schema.yml - DB: Wise thing to move towards the standardisation of the schema - DB: - JMS: Consolidated metadata - JM: Engage positively with consolidated metadata and not break it - JMS: V3 consolidated metadata not in-line with ZOM would be a bad thing :) - DB: Could be use with HDF5 as well and can be future proof for the upcoming Zarr specifications - SV: Zarr shards as a valid HDF5 - DB: Need to have a mechanism where both the formats can talk to each other - otherwise may lead to brittleness - JMS: The approach is hacky atm - Martin wants to kick-off [ZEP0003](https://zarr.dev/zeps/draft/ZEP0003.html) voting ASAP - Prototype implementation: https://github.com/zarr-developers/zarr-python/pull/1483 - Techincal review needed - DB will have at it soon! - JMS: Useful for Tensorstore ## 2023-09-07 **Attending:** Sanket Verma (SV), Norman Rzepka (NR), Ward Fisher (WF), Jeremy Maitin-Shepard (JMS) **TL;DR:** **Updates:** - ZEP0006 by Davis Bennett: https://github.com/zarr-developers/zeps/pull/46 - ZEP0007 by Isaac Virshup: https://github.com/zarr-developers/zeps/pull/47 **Open agenda (add here 👇🏻):** - WF: NASA F.15 grant could help hosting Zarr-Con over at Unidata - SV: Update on new ZEPs by Davis and Isaac - NR: Overview of the ZEP0007 - WF: Character encoding addressed? - Not implemented robustly across NetCDF - SV: Norman as co-author? - NR: No, just left some comments - JMS: Define a name for the codec - array to bytes - can be applied to raw data buffer - NR: Could model it as a data type - not clear how the translation from bytes would work in a codec - JMS: Encourage a spec PR first - make things straightforward - SV: ZEP document and spec PR - anyone can come first - depends which is the clear and straightforward way to introduce the changes - SV: ZEP0002 - JMS: Extremely close on tensorstore - NR: Zarrita.js can be added to the sharding implementation in the issue review - NR: Adding ZipStore as a ZEP - JMS: Added read-only support for ZipStore to tenstore - NR: Certain features that can be included in the ZEP - like allow different types of hierarchy - JMS: Various ways to use ZipStore in Zarr-Python - depends on different ways how want to organise your data in the Zip - NR: Maybe more of a convention - Zip on S3: How do you access it? (URL gets funky) - JMS: `s3://bucket/path/to/zip.zip|zip:path/to/array/|zarr3` - pipe URL - convey what's happening - `:` downsides - they're valid in a URL - NR: Go down further and address things further in the Zarr like array - WF: We have Zip support in NCZarr - not the similar URL style - JMS: - NR: Microscopy folks - napari folks - Saalfeld could be potential people who could work on the Zip ZEP - NR: Protocol for Google storage? GS or GCS - JMS: `gs://bucket/path` - GS - JMS: General Zip required sequential access - JMS: Standardizing some kind for URL would be good thing - NR: Getting feedback from Ward, Stephan Saalfeld would be good - WF: HTTP post style syntax in NetCDF is supported - WF: What would happen if we try to read non-Zarr store by Zarr? - JMS: Looking at the metadata file and then figuring it out? - WF: Some part of NetCDF uses HDF5 and we try to open it with Zarr and it crashed - WF: Curious to what the failed `open()` should look like? Having a defined behaviour would be good - JMS: Launch missiles if the data is malformed! 😄🚀 - WF: NetCDF have certain error code when it can't read insted of crashing the software - should be a part of the spec - JMS: Could be a good addition - WF: Just curious about the crashing! ## 2023-08-24 **Attending:** Jeremy Maitin-Shepard (JMS), Sanket Verma (SV), Josh Moore (JM) **TL;DR:** **Updates:** - ZEP0004 preliminary work for review: https://github.com/zarr-developers/zarr-specs/pull/262 - Open PRs: - https://github.com/zarr-developers/zarr-specs/pull/256 - https://github.com/zarr-developers/zarr-specs/pull/257 - https://github.com/zarr-developers/zarr-specs/pull/261 - https://github.com/zarr-developers/zarr-specs/pull/263 **Meeting notes:** - JMS: Lightweight ZEP process for adding codecs and datatypes - JM: How do you think it should look like? - JMS: Opening a PR and get the votes from ZIC and ZSC could be it - SV: Minimilastic ZEP for adding codec and data types - JM: The voting process keep going-on for codecs and data types without hinderance from the bigger ZEPs - JMS: Can work on a lightweight ZEP template for codec and dtype - JM: Certain datatypes addition may be possible blocks for some domains - having a fast-track ZEPs would help that - JMS: Would like to work on dtypes for ML use-cases - SV: Tensorstore implementation - JMS: V3 and Sharding implementation almost complete - working on some bugs - will be finalising soon! ## 2023-08-10 **Attending:** Sanket Verma (SV), Josh Moore (JM), Jonathan Striebel (JS), Norman Rzepka (NR) **TL;DR:** **Updates:** - Zarr-Python working groups - Benchmarking and performance: https://github.com/zarr-developers/zarr-python/discussions/1479 - Refactoring: https://github.com/zarr-developers/zarr-python/discussions/1480 - POC implementation of [ZEP0003](https://zarr.dev/zeps/draft/ZEP0003.html) - https://github.com/zarr-developers/zarr-python/pull/1483 - SciPy 2023 proceedings - Talk slides: https://doi.org/10.25080/gerudo-f2bc6f59-035 - Tools update slides: https://doi.org/10.25080/gerudo-f2bc6f59-038 - ZEP0001 Contributors section: https://zarr.dev/zeps/accepted/ZEP0001.html#contributors **Meeting notes:** - JM: JS are you using Zarr at work? - JS: We're convinced that it's a good idea to use Zarr! 😄 - Growing rapidly - good thing - SV: Presentation for EuroSciPy 2023! - JS: Coming up - SV: You can also cite SciPy 2023 talks - JM: Will be using the EuroSciPy 2022 poster for an upcoming meeting - SV: Tweeting about the contributors for ZEP0001 and tagging everyone - JM: Thanks to everyone! 🙏🏻 - JS: Removes the watermark from the V3 spec_ https://github.com/zarr-developers/zarr-specs/pull/260 - Figured out the CSS selector - JM: Zeiss got back to JM - SciPy 2023 discussions - James Webb Space Telescope - how they can use Zarr - JS: Misc. link: https://github.com/braingram/asdf_zarr - JS: Met with Francesc (Blosc) ? - JM: Yeah, we spend a lot of time and it was great! - How Blosc and Sharding can co-exist together - like a package - SV: Sharding can provide cloud enability to Blosc2 - discussions with Francesc - JM: Recently filled out feedback for CZI EOSS - we can do join grants as well - NR: Writing a Blosc2 codec for Zarr could help - JM: Dask chunking comes down to `.chunk()` property for the object - how about data API chunking specification around the chunks? - chunking could work across the whole PyData stack - and we can add sharding too - could help with what's the efficient access pattern for sharding chunk!? - Unified package for Zarr (Sharding), Blosc2 and Dask and other packages - NR: Interesting! Sharding has 2 access pattern - Chunk level for read and shard level for write - For Dask purposes you probably want the shard access pattern - because you're in a HPC environment - JS: Writing it as a spec and collaborating with Dask and Blosc team - NR: Agreed! - JS: Sharding also needs a lot of explanation - lots of education needed - NR: Limbo state rn - blog posts and videos can help a lot - maybe after 6 months - JM: Unifying names - block.dev? - and same documentation as well - SV: Can include HDF5 as well - JM: HDF5 could be beneficial if you're working on cluster/HPC and Zarr can help you bring that data down from the cloud to your machine - NR: Can apply to EOSS grant with same applications - JM: Less chances of getting funded - JM: Zarr can solve world hunger! 😁 - NR: Good momentum but need to deliver as well - JM: Zarr as a sister project of Napari!? - SV: Having conversations regarding fundraising for Zarr to keep the project funded - We can work on joint grant or something similar - NR: Lightweight ZEP process for codecs? - JM: Light voting procedure? - NR: Could be! - JM: A new ZEP to update ZEP0000 and add a new type of ZEP in the types and loosen the restrictions - NR: No problem with voting but how do we setup the ZEPs up for voting - anyone can do it via creating a issue but that can lead to mayhem if everyone starts doing it! - JM: We're following a chronological order but we can have a statement which can allow lightweight ZEPs to come in while big ones are on the way - Having a separate ZEPs for codecs and extension with less voting burden - How do we schedule the ZEPs for voting? - Something for https://github.com/zarr-developers/zarr-specs/pull/256 - JM: There are improvements we can make to ZEP0000 and let's keep working on that ## 2023-07-27 **Attending:** Sanket Verma (SV), Norman Rzepka (NR), Ward Fisher (WF), Jonathan Striebel (JS), Jeremy Maitin-Shepard (JMS) **TL;DR:** **Meeting notes:** - Send [ZEP2](https://zarr.dev/zeps/draft/ZEP0002.html) to the ZIC - Merged https://github.com/zarr-developers/zarr-specs/pull/253 - Need to merge https://github.com/zarr-developers/zeps/pull/40 -> SV - SV will send out the email to the ZIC - Try to fix crosslinks - JS to close [PR #152](https://github.com/zarr-developers/zarr-specs/pull/152) - NR to create an issue to gather votes and update the ZEP [PR #40](https://github.com/zarr-developers/zeps/pull/40) - SV to send an email to the ZIC after issue creation and PR merging - Everyone looking forward to it! - Mark wanting to organise a ZEP2 review call - but didn't happen - Sharding implementation - JMS working on Tensorstore implementation - V3 implementation on Tensorstore is close to completion - Zarrita had a noticeable overhead while running the benchmarks - NR: Zarr-Python sharding implementation has deviated over the time - JS: Make sense to add sharding as a codec once V3 in Zarr-Python gets in - JMS: Is there aim for having the similar API for V2 and V3 in Zarr-Python? - NR: Zarrita doesn't have various stores - JMS: Is Zarrita compatible with FSSPEC? - NR: Yes! - ZEP1 and V3 - https://github.com/zarr-developers/zarr-specs/pull/236 - https://github.com/zarr-developers/zarr-specs/pull/249 - https://github.com/zarr-developers/zarr-specs/pull/248 - https://github.com/zarr-developers/zarr-specs/pull/225 - Once the PRs are merged, SV to send out the FYI to the ZIC - SV: Any developments on Unidata side? - WF: Not yet, but it's on our roadmap ## 2023-07-13 → SciPy 2023 ## 2023-06-29 **Attending:** Sanket Verma (SV), Josh Moore (JM), Jonathan Striebel (JS), Ryan Abernathey (RA), Ward Fisher (WF), Daniel Jahn (DJ), Jeremy Maitin-Shepard (JMS), Norman Rzepka (NR) **TL;DR:** **Updates:** **Meeting notes:** - SV: ZEPs 3 and 5 ... - RA: feedback on the ZEP - Need to be implementing as we go, otherwise leads to stalling - JS: Zarr-Python has tech debt which makes it difficult to implement new stuff - *Impromptu round of introduction* - JMS: Reference implementation in any language is helpful for any new ZEPs - RA: Having explicit tweet/statement about implementation would help - JS: benchmark repo? also sample data? - NR: Sample datasets - RA: https://github.com/zarr-developers/zeps/pull/28 - best practices going forward - but a way to support old conventions - from OGC, "conformance class" determined with "conformance test" - namespacing up to the convention - would like to get it into draft form and then we can move forward with the process - SV: few open comments? - RA: just merge it and move the process forward. - still need to open a template - NR: add existing conventions at this point (OME-NGFF) - JM: Thoughts on multiscales? - RA: It should be an extension - JMS: Multiscales doesn't lead to a lot of objects - SV: ZEP0002 review process to kick-off by next week - NR: Zarrita has sharding implementation - JM: Kicking-off review process and having neat benchmarks, any idea how we could do both? - NR: Make a new issue for ZIC feedback ## 2023-06-15 **Attending:** Sanket Verma (SV), Alan Watson (AM), Jeremy Maitin-Shepard (JMS), Josh Moore (JM), Norman Rzepka (NR) **TL;DR:** **Updates:** - SV gave a talk on Zarr @ Vrije Universiteit Amsterdam - https://vu-nl.libcal.com/event/4030713 - Slides and code: https://github.com/MSanKeys963/presentations/tree/main/vu_amsterdam_6_23 **Meeting Minutes:** - Are there any command line tools for Zarr? - Found this: https://github.com/BaroudLab/zarr-tools - JM: Was working on something on this but didn't use it much - NR: There are bunch of tools which you can use OME-Zarr, having something like H5LS would be cool - JMS: Operations for small things makes sense - rechunking, copying - JM: Nextflow - workflow engine - AM: BIL is working on games - https://www.mozak.science/landing - Pushing them to use Zarr for their image (.png) data - AM: Interest and benefit in attending brain conferences - someone from the Zarr community - https://brainmeeting.swoogo.com/2023/home - AM: Allen is using Zarr for their work - SV: Recent paper out of Allen: https://www.biorxiv.org/content/10.1101/2023.05.22.541700v2 - Extensive usage at Allen have revealed some problems and it may be worth addressing them - JMS: Writing electrophysiology data in Zarr rather than tiff is a good oway to go forward - SV: Blog post on V3 coming out soon! - JMS: Fallback data types issue needs to be addressed - JMS: Not clear how it'll be specified - JMS: How do you handle it in Zarrita? - NR: Currently, we do not! - JMS: - SV: How serious it is? Implementation or spec issue? - JMS: Kind of ignoring for it now! We're in implementation phase now! - JM: If everybody is ignoring it, then it's fine - NR: Would not be straight forward to add it later! - JMS: If implementation doesn't support, it'll fail - SV: Have you started on the implementation? - JMS: Tensorstore has V3 minus sharding; planning to work on it this week - NR: Rust implementation of V3 - https://github.com/clbarnes/zarr3-rs - NR: Benchmarking in Zarrita - https://github.com/scalableminds/zarrita/blob/async/tests/test_perf.py - SV: Benchmarks in Tensorstore? - JMS: Seen bottlenecks in IO layer than array layer - NR: What about codecs? - JMS: - NR: 10x performance improvement would be great! - - NR: ZEP0002 review timeline - SV: V3 blog post, feedback for ZEP process, and then we can add ZEP0002 in the pipeline - NR: Ok! - SV: Maybe we need to invite Chris and others from the ZIC to the ZEP meeting - JM: There used to be a Zarr-Rust meeting ## 2023-06-01 **Attending:** Ward Fisher (WF), Sanket Verma (SV), Jeremy Maitin-Shepard (JMS), Ryan Abernathey (RA), Norman Rzepka(NR), Josh Moore (JM) **TL;DR:** **Updates:** **Meeting Minutes:** - SV: ZEP 4 → https://github.com/zarr-developers/zeps/pull/28 - RA: Still working on it - SV: How can we help? - RA: Need to decide if it's going to be a general convention (for bio-imaging, geospatial, genomics etc.), or a convention of Geospatial domain all - RA: Also need to decide if the dataset are conforming to a convention or not - lot of legacy data out there which doesn't conform to it - RA: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html - WF: Convention doesn't need to be broad, cf convention are based on NetCDF model - but there's nothing in the NetCDF library or code that mentions the cf conventions! - RA: The existing conventions are broad, and it's difficult to place cf in a specific place - WF: Agree with you - RA: Define what's the process to put a new convention for the community - JMS: You have group level attribute and array level attribute? - RA: Mostly yes! - WF: There - JMS: - RA: Getting all convention on the website would be a good way for cross domain and community collaboration - conventions can be composable - conventions could not be universal - WF: Conventions move slowly - take time to adopt to new things - took a good amount of time to solve SST (Sea Surface Temperature) - RA: Will get the another draft out soon - JMS: Not feasible to namespace an attribute? - RA: It would require a deep re-factoring for the software we use! - It would break Xarray, GDAL, NetCDF - Zarr-Python doesn't care about the attributes - WF: Namespacing would definitely break the NetCDF library! - RA: JMS, how do you handle conventions? - JMS: Generally, doesn't invent new conventions, and implement existing conventions - the data formats I invented, I defined those conventions - these existing conventions doesn't lack _certain_ things - NR: In the process of adopting Zarr V3, currently using `OME-Key` - WF: Attributes are strictly defined - defined to be interpretable not changeable - JMS: Maybe the best idea is to say clearly that _X_ datasets use the conventions and multiscales - JMS: https://github.com/zarr-developers/zarr-specs/issues/242 - NR: No need to separate them - my opinion: to keep it as it is - maybe Chris's comment is coming out of the Rust world and separating the codecs will be convenient for him - JMS: https://github.com/zarr-developers/zarr-specs/pull/236#issuecomment-1539188066 - JM: Cares about the codec pipeline - adding some transformations in the middle would be tricky! - NR: Feel the same! - Sharding as a single codec makes sense but adding anything would make it complicated - - JM: Current partial codec --- define the metadata format for shard codec would be great! - NR: Adding 2 partial codecs would make it tricky to implement - JMS: Blosc is kind-of sharding codec - not clear if using sharding as a partial read codec is good idea! - JMS: How do you have partial writes for the codec? - NR: - JMS: Fallback data types - JMS: Need to have a broader discussion - NR: Also, do extensions data type need to have a fallback? - JMS: No, it's optional - NR: The definition of fallback - like a tuple - having a datatype and the fallback value - JMS: Maybe it's the way to go! - NR: ## 2023-05-18 **Attending:** Sanket Verma (SV), Josh Moore (JM), Ryan Abernathey (RA), Norman Rzepka (NR), Jeremy Maitin-Shepard(JMS) **TL;DR:** **Meeting minutes:** - Discussions about climate and weather 🌡️ - [ZEP0001](https://github.com/zarr-developers/zarr-specs/issues/227) is finally accepted! 🎉 - Implementors can start working on their implementations - Will be moving the ZEP0001 under the new `Accepted` section - Will move it under `Final` section once we have atleast one complete reference implementation - SV - updates and next steps for ZEP0001 and V3: - 1 year into the process... ([2019](https://zarr.dev/blog/v3-update/) first discussion) - [gdal](https://github.com/OSGeo/gdal/pull/7706) moving quickly - will be checking in on the various implementations - zarrita as one of the most complete (in terms of code, not docs or tests), i.e. PoC - JM: Reference implementation needs to be useable! - NR: Yeah! - RA: why it's not a complete implementation? - NR: no optimizations (sharding, etc.). meant to be easy to read code. - lacks features like buffer protocol, etc. but could be used. - don't currently plan to maintain it over a long period of time. - SV: was thinking less of end-user and more of supporting all the features so others can refer to it. - NR: that probably could be done now. - NR: could write an intro for people to read. (don't want to write end-user docs) - RA: NR's production implementation? use different file format currently. must have sharding. - considering using zarrita as the implementation (for Python stack) - also have a scala stack (baked into software) - JM: ZEP0002 voting and discussions - JM: We could open up the voting for ZEP0002 and give a month/two month/full summer for voting - any open issues? - NR: None! - JM: Shard as recrusive Zarrs? - treat internals of shards like another Zarr - NR: Sharding being a codec would work that way but it's more of a implementation detail - JM: What would it look like from a URI structure? - similar to what Saalfeld is doing in N5 ecosystem - if I access a chunk inside a shard and I could treat it as a Zarr array and not as a blob - NR: Fair enough! I could have something like this in Zarrita - NR: Not have implementation details in the spec but rather point to the reference implementation for the details - RA: ZEP process should be more of an iterative process and not an ultimatum - JMS: I feel, most of the implementors would be working on sharding and V3 together - RA: flat files to virtualization (experimentation). (Once ZEP0003 lands!) - Also want to see linked referencing in metadata (to allow browsing through HTTP). Better than consolidated. For e.g. infinite hierarchies, allows browsing like a catalog - Allow parent to list it's children in Zarr groups - Also multiscales. Relates the arrays. Within the array directory? - NR: listing parent/children - NR: paths... - RA: see "link" in https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#link-object for any node (self, root, parent, child) - NR: Discussions on umaintained Zarr implementations - SV: Dissolve them with the help of the maintainers - JM: Tricky to get a hold of them - SV: Start deprecating them and then removing them from https://zarr.dev/implementations/ - JMS: But all of that is V2 - if there's going to be something on V3 we can certainly help them

Import from clipboard

Advanced permission required

Your current role can only read. Ask the system administrator to acquire write and comment permission.

This team is disabled

Sorry, this team is disabled. You can't edit this note.

This note is locked

Sorry, only owner can edit this note.

Reach the limit

Sorry, you've reached the max length this note can be.
Please reduce the content or divide it to more notes, thank you!

Import from Gist

Import from Snippet

or

Export to Snippet

Are you sure?

Do you really want to delete this note?
All users will lost their connection.

Create a note from template

Create a note from template

Oops...
This template is not available.


Upgrade

All
  • All
  • Team
No template found.

Create custom template


Upgrade

Delete template

Do you really want to delete this template?

This page need refresh

You have an incompatible client version.
Refresh to update.
New version available!
See releases notes here
Refresh to enjoy new features.
Your user state has changed.
Refresh to load new user state.

Sign in

Forgot password

or

By clicking below, you agree to our terms of service.

Sign in via Facebook Sign in via Twitter Sign in via GitHub Sign in via Dropbox

New to HackMD? Sign up

Help

  • English
  • 中文
  • Français
  • Deutsch
  • 日本語
  • Español
  • Català
  • Ελληνικά
  • Português
  • italiano
  • Türkçe
  • Русский
  • Nederlands
  • hrvatski jezik
  • język polski
  • Українська
  • हिन्दी
  • svenska
  • Esperanto
  • dansk

Documents

Tutorials

Book Mode Tutorial

Slide Mode Tutorial

YAML Metadata

Contacts

Facebook

Twitter

Discord

Feedback

Send us email

Resources

Releases

Pricing

Blog

Policy

Terms

Privacy

Cheatsheet

Syntax Example Reference
# Header Header 基本排版
- Unordered List
  • Unordered List
1. Ordered List
  1. Ordered List
- [ ] Todo List
  • Todo List
> Blockquote
Blockquote
**Bold font** Bold font
*Italics font* Italics font
~~Strikethrough~~ Strikethrough
19^th^ 19th
H~2~O H2O
++Inserted text++ Inserted text
==Marked text== Marked text
[link text](https:// "title") Link
![image alt](https:// "title") Image
`Code` Code 在筆記中貼入程式碼
```javascript
var i = 0;
```
var i = 0;
:smile: :smile: Emoji list
{%youtube youtube_id %} Externals
$L^aT_eX$ LaTeX
:::info
This is a alert area.
:::

This is a alert area.

Versions

Versions and GitHub Sync

Sign in to link this note to GitHub Learn more
This note is not linked with GitHub Learn more
 
Add badge Pull Push GitHub Link Settings
Upgrade now

Version named by    

More Less
  • Edit
  • Delete

Note content is identical to the latest version.
Compare with
    Choose a version
    No search result
    Version not found

Feedback

Submission failed, please try again

Thanks for your support.

On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?

Please give us some advice and help us improve HackMD.

 

Thanks for your feedback

Remove version name

Do you want to remove this version name and description?

Transfer ownership

Transfer to
    Warning: is a public team. If you transfer note to this team, everyone on the web can find and read this note.

      Link with GitHub

      Please authorize HackMD on GitHub

      Please sign in to GitHub and install the HackMD app on your GitHub repo. Learn more

       Sign in to GitHub

      HackMD links with GitHub through a GitHub App. You can choose which repo to install our App.

      Push the note to GitHub Push to GitHub Pull a file from GitHub

        Authorize again
       

      Choose which file to push to

      Select repo
      Refresh Authorize more repos
      Select branch
      Select file
      Select branch
      Choose version(s) to push
      • Save a new version and push
      • Choose from existing versions
      Available push count

      Upgrade

      Pull from GitHub

       
      File from GitHub
      File from HackMD

      GitHub Link Settings

      File linked

      Linked by
      File path
      Last synced branch
      Available push count

      Upgrade

      Danger Zone

      Unlink
      You will no longer receive notification when GitHub file changes after unlink.

      Syncing

      Push failed

      Push successfully