IDR-NGFF 2020-W45

--- tags: NGFF --- # IDR-NGFF 2020-W45 ## 2020-11-06 (roundtable) - J-m: flag in the Amazon SDK needed, so minio & S3 work. - working on prototype for the Java community - jzarr still can't handle the file. see https://github.com/bcdev/jzarr/issues/7 - J-m to synchronize with David next week. can re-use chunk of code. - downloading images & labels with sliders in the notebook (python) - want to re-run the analysis on an image to generate labels and compare with the stored label - opens the question of datasets generation with labels using omero-cli-zarr - Josh: likely going back to n5-zarr-s3 support - Will: tested napari supporting wells to have both napari/vizarr - PR 43 for reading the latest spec - napari also handles sparse wells - NB: need to make some breaking changes to the previous 2 specs. - cli-zarr good to review & merge? - same for zarr-py? - No commits coming. Will start on video from them. - videos to openmicroscopy channel? - Simon - working on conda packaging/workflow - Dom - about to open PR against omero-cli-zarr for masks/labels export - Petr - I2K tweets need to go out - AOB (next week & beyond) - fsspec bump coming - dask.map_ ... - HPA conversion (Jason): need **metadata**. Target after HCS? ## 2020-11-05 (roundtable) - Will - Changes showing well content. Rebased on Trevor work with grid layer. Looks quite nice. Slow zoom-in is now fixed - only tested with one set of data. Seb: also idr0094 plate - conversion with idr0002, idr0004? other plates suggested - Comments re PR/spec - Seb: happy to have it in, tag and port to spec.md - https://github.com/ome/omero-cli-zarr/pull/43 - Josh: extra capacity from tomorrow - Seb: merging by noon if no-one has raised objections - JRS - Calls for existing EOSC project ideas incl. cross-research infrastructure - Some work with idr0094 (remdesivir vs hydroxychloroquine). - idr0083 (ML method) - mobie - Seb: actionable if we want (15-20min for conversion + transfer time) - Simon - vizarr: set up own OME netlify account as opposed to Will's - Will: could even give nice URL - Need some authorization. - Alternative: host the file somewhere. Simon: require own JS build (Similar to iviewer) - Build artifact, put it on GH and download it/use it anywhere - Prefix handling - currently set to absolute by default - Postpone till next week. Maybe call with Ilan/Trevor to enquire about deployment - Dom - Keep working on plates + labels - Simon: kicked off trigger job - Seb - see above, tagging omero-ms-zarr, omero-cli-zarr - still screen reader leak :scream_cat: - J-m - reactivated code to connect to s3/embassy using Amazon SDK. Returns nothing (compared to minio). Pushing code. - Josh: - zarr call: UTC, majore take-home. HDF5 can start looking like a Zarr file - Anyone else? ## 2020-11-04 - zarr catalog (Will) - https://snoopycrimecop.github.io/idr.openmicroscopy.org/zarr - Summary: no objections but also no rush - J-m: great to have it. Viz, analysis. Later links to ITR. - Will: to be used by other clients, could put it into json format. - would need some code to generate the JSON. - Simon: could load JSON in jekyll. - Seb: adding to the menu? That will be bring promotion. - will need to evaluate - Simon: before the page goes public? - J-m: other pages are starting to use it. - JRS: so far have avoided those issues. So far the tools aren't as easy as we'd like them to be. Communicating that the effort will be broader (n5, etc.) - Josh: blog post so that it's timestamped? - ome/blog - Will: HCS updates - https://github.com/ome/omero-cli-zarr/pull/43 - https://mystifying-lalande-e12142.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/idr0033-rohban-pathways/41744_illum_corrected/pr_43/5966.zarr - Trevor has introduced a grid layout component based on Will's code. Need to look at it. (**Next**) - Will: looking for plate with layout information. Might be pushed off. Quite a bit more work for not a huge. - Josh: layers possible? Haven't looked into it. As a short term fix, might use hoover events. - Will: also multiple **plate acquisitions**. Open them each in a new page? Needs custom elements. - Format - Josh: so currently working? Yep - Seb: hard-coded decision on the field layout of the field. Josh: based on the screen size and could flow? Will: pre-calculated at the moment. Seb: idr0001 is good for testing. - AOB - Note: netlify URLs are broken. Josh: target freezing next week. Then we can publish versioned viewer as well. Will: also need to open that discussion on the viv/vizarr side. J-m: agree! - J-m: I2K workshop. What's to be shown? - Josh: Either mobie linked data - Petr: Possibly the "beyond local" memory - Josh: assume we can do more of this coordination for the Jan/Feb time slots. - Simon: these are much more like a hackathon - Seb: ok to review the new layout tomorrow? Will: **yes** - NB: working on screenreader leak. need to restart `idr-testing` periodically. exporter is a good test. - Be aware of others using the server before restarting idr-testing - **But** perhaps restart before you restart any long running jobs ## 2020-11-03 - Will: - https://github.com/ome/omero-ms-zarr/issues/76#issuecomment-720570094 - Cleanup of metadata (keeps growing) - Seb: name versus index - Josh: shape - Sparseness: A1-Z9 or B2-X8 - Force fields to be numerical? (cf. labels) - Demo? - Defining concepts concretely - View FoW in the client - Data generation - Will: local versus few hours remotely - Seb: Create subset? - Dom: --cache-numpy - Josh: add to git? - Table: positional information of fields - Distribute video? - Dom: plates with masks - using merge-ci. Dataset to Plate? Yes - Seb: MIAS example populates example - see https://github.com/ome/bioformats/pull/2319#issuecomment-219007029 - Josh - Freezing HCS - Simon: give ourselves 2 weeks for all changes? - Label work for I2K? - https://github.com/ome/ome-zarr-py/issues/60 - New metadata for Tischi et al - J-M: danger of introducing something we want to support - Simon: understanding them as k/v pairs i.e. free-form JSON - Will: pick the best of three options above - Open question of whether we can engage them in implementing it ? - Simon: Who has permissions/access to release [bioformats2raw 0.2.6](https://github.com/ome/conda-bioformats2raw/pull/4) ? - Tabled until next week ## 2020-11-02 ### From last week - versioning concepts (Will): one specification vs multiple - Seb: issue re: mask labels, i.e. others are coming? Likely to transfer to omero-ms-zarr - Will: tabular data. Probably requires some investigation, trade-offs etc - Support exists in Zarr? Only for homoegeneous tables e.g. features tables. - J-M: requirement for string comes pretty quickly. Workaround exist. - Simon: JSON labels-like as a short term workaround? - Main caveat is scalability. - Some sign-off must happen with 0.1 - also notify the community (image.sc, Simon: also Twitter) - registry of zarr data - Josh looking for branch - i2k & idr0001 (has multiple runs) - Seb: good target (with masks) - What changes to the spec are then needed? (Also need to export them from omero-cli-zarr). In UI, would need to allow switching - Will: missing between multiple runs & fields. About to start working on that in vizarr. - Given a URL how do you know it's a well? Need a shallow spec. Seb: for the moment, could go up three levels and find the plate. Will: general issue of then needing to load more .zattrs slows things down. - Simon: similar to how do you know what chunks exist (**sparseness**) - J-m: could apply **overrides** at lower levels. i.e. default is on plate. See also clear path for finding the channel name if it doesn't exist (emission wavelength, etc.) - Josh: demos every Friday per month? - Seb: away on Fridays, video them - Simon: CSHL ### Clients ### Datasets - idr0033 - J-M: pipelines submitted with idr0033 are not great. Looking into other options - idr0001 - Dom: plate acquisitions already in the spec and is exported with omero-cli-zarr - plate masks? Seb: idr0001 masks currently stored as polygons - Will: some code at image.sc - Simon: issue with reinterpreting the data (how to deal with diagonals) - Simon to look into other candidates plates with masks ### Formats - Seb: from last week, HCS specification next todos - multiple acquisitions/runs - well vs column vs row spec - metadata distribution/redundancy - spatial metadata for multiple fields of view - [bioformats2raw 0.2.6](https://github.com/ome/conda-bioformats2raw/pull/4) - [ome/ome-zarr-py Add class annotations and/or other metadata properties to labels](https://github.com/ome/ome-zarr-py/issues/60) - Move to omero-ms-zarr? Or a new ome-zarr-spec repo? ### Infrastructure - ---- ## Template ### Clients - ### Datasets - ### Formats - ### Infrastructure -