IDR-NGFF 2020-W47

--- tags: NGFF --- # IDR-NGFF 2020-W47 ## 2020-11-20 (demo notes) - Will (data locally) - Petr: Column detection usability - Chris: Performance (esp. from home). napari fixes coming. Petr: also improvements for timelapse hopefully - Jason: data on EBI S3? No. That was local. JRS: giving NG-screening talk in about a week. Viable? Doesn't matter napari / browser - Chris: deck.gl is better than napari? Yes, but you wait until all the tiles are coming. - Nice URL for JRS (https://github.com/hms-dbmi/vizarr/pull/43 once merged) - David (data local & remote) - Have to select file in the fileset. - Multiscale etc. metadata stored as annotations. - Likely need to rework the s3 layers. - Josh: colors (labels & "omero" block) aren't used at the moment? - Seb: s3 performance are initial load or the constant calls? - David: caching would help. e.g. every time you set series it's making a new call - Seb: does it work in QuPath? "better" (faster)? - David: There's no reason it shouldn't work. (since it works in bfcovert) - Seb: it's more multiresolution aware - ETA for a Jar? Multiscale needs fixing, then a day of tidying or so. - Petr (data at EBI) - JRS: Most profound? - Would like to hear a comparison of non-zarr with zarr. Wanted to try that this morning, e.g. from OMERO - Shows remote access to an image without OMERO. Works pretty fast. Know how painful to open ilastik locally. - Local docker is 2-3x slower, but stable. - JRS: comparison to a remote TIFF stack - Will: not showing the best of zarr because you're loading everything upfront. Unless you're only accessing one channel. - Josh: another option would be to have multiple computers - Chris: could be more compelling if we were doing something on a subresolution over several images, 50k x 50k, and only subregions that are of interest - Josh: the guided annotations - Will: ticks the box of interoperability - Cloud-enabled glue done. Probably some performance issues to look into. ## 2020-11-20 - Josh: - [Call](https://hackmd.io/HYn61s2YTTyfJE_oCrqpwA) re: data with EMBL-HD - http://doi.org/10.5281/zenodo.4282107 for review - Will - some more on CSV parsing. parsing different types - ported parsing of properties from ome-zarr into napari (i.e. draga's PR on plates) - working on having the value of each label == the ROI ID, perhaps just a short-cut. Logic for updating each pixel value is complicated - Seb back to spec; Josh: maybe in new repo - Petr: good - Dom: working on refactoring omero-cli-zarr to something like omero - David: some weirdness with S3 reading - works for demo, but not happy with the implementation ## 2020-11-19 - Josh: - updated look of https://joshmoore.github.io/ngff/. Working on text. - zarr meeting: looking for feedback re: https://github.com/intake/fsspec-reference-maker (cf. Trevor) - https://github.com/zarr-developers/zarr-python/issues/556 - Will - vizarr fixes pushed. Take a look. Hopefully done soon. - https://github.com/ome/ome-zarr-py/pull/63 - along with https://github.com/ome/omero-cli-zarr/pull/50 and Batch_ROI_xport - generates properties, allows a join on any property - Petr: what do you impose on the CSV so that it works? - need one column on the CSV that has a unique identifier for a mask - doesn't try to understand string vs. number columns - Demo: run locally, would be nice to generate data on merge-ci - Petr: outreach? user-50 or something - David - Nothing needed. 6001247 - Petr: feelin' good - Dom: nada - Simon: nope - Seb: umm.... prod90 ## 2020-11-18 - Josh: - data problems - cf. HPA small number of large files - n5 to ome-zarr (at Heidelberg) hopefully in time for I2K - Heads up: may need to use multiple data sources - Will: - Status of spec? https://github.com/ome/omero-ms-zarr/issues/77 etc - What's changing when? - Josh: consider HCS done for the moment (to the extent that no changes are needed for labels) - Seb: no capacity for spec branch before prod90 - Labels on Plate https://github.com/ome/ome-zarr-py/pull/62 - Code needs work - Behavior is pretty good. - Some hard codings: assume A1 isn't sparse (i.e. cleanup needed) - Not showing colors. Need a dictionary per well. Won't scale. - Draga's work: https://github.com/ome/ome-zarr-py/pull/61 - Will: omero-cli-zarr could populate from labels in omero - e.g. text value on a shape - Simon: do we need to generate? - Seb's https://github.com/ome/omero-ms-zarr/pull/75 needs work - Josh - spec: - https://joshmoore.github.io/ngff/ ([bikeshed](https://github.com/tabatkins/bikeshed)) - https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html ([pydata-sphinx-theme](https://pydata-sphinx-theme.readthedocs.io/en/latest/)) - Something like: ngff.openmicroscopy.org (Seb: we can redirect that) - specs.openmicroscopy.org/ome-ngff - specs.openmicroscopy.org/ome-tiff - Simon: if the URLs are the same then the build system doesn't matter - Will: won't ngff become cgff? Just zarr? - Simon: - Microservices deployed on idr test90: - https://github.com/IDR/deployment/pull/297 - omero-ms-thumbnail: - ``/webgateway/render_thumbnail/*`` - ``/webclient/render_thumbnail/*`` - ``/webgateway/render_birds_eye_view/*`` - ``/webclient/render_birds_eye_view/*`` - ``/webgateway/get_thumbnails/*`` - ``/webclient/get_thumbnails/*`` - omero-ms-imageregion (broken for some images): - ``/webgateway/render_image_region/*`` - ``/webgateway/render_image/*`` - ``/webclient/render_image_region/*`` - ``/webclient/render_image/*`` - ``/webgateway/render_shape_mask/*`` - omero-ms-zarr (broken for some images): - ``/idr/zarr/*`` - Summary: - Issues: - Using upstream Bio-Formats. - Still using caching. Nginx routing is complicated. - Not using a writable Bio-Formats cache. Slow if it doesn't exist. - Josh: fall back to using non-docker if there's an error-code. - Don't need anything - What do we do next? - Performance testing? Not yet. - Josh: How do get to production? - Test thumbnails. - J-m: can do a short test. List of images? - Use https://github.com/ome/docker-example-omero-microservices for local testing - J-M: - update ITR link to point to updated zarr/ilatisk notebook: labels vs ilastik probability map - Tidying up. (Focusing on GH Actions) - Jason: What's plan for demo video? Do we want to meet to review and comment? - J-m video. Petr reviewed; he can do both notebooks. - Will: - unsure of the state of the masks for Friday (local sample data or possibly on merge-ci) via napari - can show acquisition in vizarr (not supported in napari) - David: ZarrReader hopefully possible. (needs updates to jzarr. hacked in) - J-m: would be good to update issue - Josh to send invite - AOB - label properties from non-OMERO sources - Will: take CSV file and populates properties in ome-zarr. Need shape ID. - Seb: see https://github.com/ome/omero-metadata/pull/49. Associate ROIs and columns. - Josh: omero-ms-zarr uses ROI or shape ID as the label value ## 2020-11-17 - Josh: will look at [ome-zarr-py#57](https://github.com/ome/ome-zarr-py/pull/57) in the background today. Is a commit expected on [vizarr#43](https://github.com/hms-dbmi/vizarr/pull/43)? - Simon: Running omero-ms-thumbnail on idr test90 backend omero servers, now working on nginx front-end proxy - Dom: working on CLI export of labels - Looking into having reading from omero and writing in separate. ## 2020-11-16 - Josh - Print statements in [57](https://github.com/ome/ome-zarr-py/pull/57#pullrequestreview-529892282) - Will - vizarr PR is largely done. One bug needs fixing. - was going to look at Labels on plate - multiscaled? yes actually the number of resolutions must match the associated images as per the current spec - focusing on performance of the plates first though - Dom: nothing on the NGFF front - Simon - nothing on the microservices front - target? Josh: current assumption was thumbnail microservice running in production - David - running into issues opening files directly from S3. Should have reader available for testing shortly - covers multiscales + labels - open twitter thread about bdv-h5 -> ome-zarr - J-M: library? Currently modified version - J-M - examples of extracting subsets of Pytables - Josh: also open PR by Draga. Will: best way to test it? Might start the discussion on the PR - Seb - will dive into the spec later today (once compounds are fixed) - AOB - [Draga's tabular PR](https://github.com/ome/ome-zarr-py/pull/61/files)