---
# System prepended metadata

title: IDR-NGFF 2020-W47
tags: [NGFF]

---

---
tags: NGFF
---

# IDR-NGFF 2020-W47

## 2020-11-20 (demo notes)
- Will (data locally)
  - Petr: Column detection usability
  - Chris: Performance (esp. from home). napari fixes coming. Petr: also improvements for timelapse hopefully
  - Jason: data on EBI S3? No. That was local. JRS: giving NG-screening talk in about a week. Viable? Doesn't matter napari / browser
    - Chris: deck.gl is better than napari? Yes, but you wait until all the tiles are coming.
  - Nice URL for JRS (https://github.com/hms-dbmi/vizarr/pull/43 once merged)
- David (data local & remote)
  - Have to select file in the fileset.
  - Multiscale etc. metadata stored as annotations.
  - Likely need to rework the s3 layers.
  - Josh: colors (labels & "omero" block) aren't used at the moment?
  - Seb: s3 performance are initial load or the constant calls?
    - David: caching would help. e.g. every time you set series it's making a new call
  - Seb: does it work in QuPath? "better" (faster)?
    - David: There's no reason it shouldn't work. (since it works in bfcovert)
    - Seb: it's more multiresolution aware
  - ETA for a Jar? Multiscale needs fixing, then a day of tidying or so.
- Petr (data at EBI)
  - JRS: Most profound?
    - Would like to hear a comparison of non-zarr with zarr. Wanted to try that this morning, e.g. from OMERO
    - Shows remote access to an image without OMERO. Works pretty fast. Know how painful to open ilastik locally.
    - Local docker is 2-3x slower, but stable.
    - JRS: comparison to a remote TIFF stack
      - Will: not showing the best of zarr because you're loading everything upfront. Unless you're only accessing one channel.
      - Josh: another option would be to have multiple computers
      - Chris: could be more compelling if we were doing something on a subresolution over several images, 50k x 50k, and only subregions that are of interest
      - Josh: the guided annotations
      - Will: ticks the box of interoperability
      - Cloud-enabled glue done. Probably some performance issues to look into.

## 2020-11-20
- Josh:
  - [Call](https://hackmd.io/HYn61s2YTTyfJE_oCrqpwA) re: data with EMBL-HD
  - http://doi.org/10.5281/zenodo.4282107 for review
- Will
  - some more on CSV parsing. parsing different types
  - ported parsing of properties from ome-zarr into napari (i.e. draga's PR on plates)
    - working on having the value of each label == the ROI ID, perhaps just a short-cut. Logic for updating each pixel value is complicated
- Seb back to spec; Josh: maybe in new repo
- Petr: good
- Dom: working on refactoring omero-cli-zarr to something like omero
- David: some weirdness with S3 reading
  - works for demo, but not happy with the implementation

## 2020-11-19
- Josh:
  - updated look of https://joshmoore.github.io/ngff/. Working on text.
  - zarr meeting: looking for feedback re: https://github.com/intake/fsspec-reference-maker (cf. Trevor)
    - https://github.com/zarr-developers/zarr-python/issues/556
- Will
  - vizarr fixes pushed. Take a look. Hopefully done soon.
  - https://github.com/ome/ome-zarr-py/pull/63 
    - along with https://github.com/ome/omero-cli-zarr/pull/50 and Batch_ROI_xport
    - generates properties, allows a join on any property
    - Petr: what do you impose on the CSV so that it works?
      - need one column on the CSV that has a unique identifier for a mask
      - doesn't try to understand string vs. number columns
  - Demo: run locally, would be nice to generate data on merge-ci
    - Petr: outreach? user-50 or something
- David
  - Nothing needed. 6001247
- Petr: feelin' good
- Dom: nada
- Simon: nope
- Seb: umm.... prod90

## 2020-11-18
- Josh:
  - data problems
    - cf. HPA small number of large files
    - n5 to ome-zarr (at Heidelberg) hopefully in time for I2K
    - Heads up: may need to use multiple data sources
- Will:
  - Status of spec? https://github.com/ome/omero-ms-zarr/issues/77 etc
    - What's changing when?
    - Josh: consider HCS done for the moment (to the extent that no changes are needed for labels)
    - Seb: no capacity for spec branch before prod90
  - Labels on Plate https://github.com/ome/ome-zarr-py/pull/62
    - Code needs work
    - Behavior is pretty good.
      - Some hard codings: assume A1 isn't sparse (i.e. cleanup needed)
      - Not showing colors. Need a dictionary per well. Won't scale.
  - Draga's work: https://github.com/ome/ome-zarr-py/pull/61
    - Will: omero-cli-zarr could populate from labels in omero
    - e.g. text value on a shape
    - Simon: do we need to generate?
  - Seb's https://github.com/ome/omero-ms-zarr/pull/75 needs work
- Josh
  - spec:
    - https://joshmoore.github.io/ngff/ ([bikeshed](https://github.com/tabatkins/bikeshed))
    - https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html ([pydata-sphinx-theme](https://pydata-sphinx-theme.readthedocs.io/en/latest/))
    - Something like: ngff.openmicroscopy.org (Seb: we can redirect that)
      - specs.openmicroscopy.org/ome-ngff
      - specs.openmicroscopy.org/ome-tiff
    - Simon: if the URLs are the same then the build system doesn't matter
    - Will: won't ngff become cgff? Just zarr?
- Simon:
  - Microservices deployed on idr test90:
  - https://github.com/IDR/deployment/pull/297
  - omero-ms-thumbnail:
    - ``/webgateway/render_thumbnail/*``
    - ``/webclient/render_thumbnail/*``
    - ``/webgateway/render_birds_eye_view/*``
    - ``/webclient/render_birds_eye_view/*``
    - ``/webgateway/get_thumbnails/*``
    - ``/webclient/get_thumbnails/*``
  - omero-ms-imageregion (broken for some images):
    - ``/webgateway/render_image_region/*``
    - ``/webgateway/render_image/*``
    - ``/webclient/render_image_region/*``
    - ``/webclient/render_image/*``
    - ``/webgateway/render_shape_mask/*``
  - omero-ms-zarr (broken for some images):
    - ``/idr/zarr/*``
  - Summary:
    - Issues:
      - Using upstream Bio-Formats.
      - Still using caching. Nginx routing is complicated.
      - Not using a writable Bio-Formats cache. Slow if it doesn't exist.
        - Josh: fall back to using non-docker if there's an error-code.
    - Don't need anything
    - What do we do next?
      - Performance testing? Not yet.
    - Josh: How do get to production?
      - Test thumbnails.
      - J-m: can do a short test. List of images?
  - Use https://github.com/ome/docker-example-omero-microservices for local testing
- J-M:
    - update ITR link to point to updated zarr/ilatisk notebook: labels vs ilastik probability map
    - Tidying up. (Focusing on GH Actions)
- Jason: What's plan for demo video?  Do we want to meet to review and comment?
  - J-m video. Petr reviewed; he can do both notebooks.
  - Will:
    - unsure of the state of the masks for Friday (local sample data or possibly on merge-ci) via napari
    - can show acquisition in vizarr (not supported in napari)
  - David: ZarrReader hopefully possible. (needs updates to jzarr. hacked in)
    - J-m: would be good to update issue
  - Josh to send invite
- AOB
  - label properties from non-OMERO sources
    - Will: take CSV file and populates properties in ome-zarr. Need shape ID.
    - Seb: see https://github.com/ome/omero-metadata/pull/49. Associate ROIs and columns.
    - Josh: omero-ms-zarr uses ROI or shape ID as the label value
 
## 2020-11-17
- Josh: will look at [ome-zarr-py#57](https://github.com/ome/ome-zarr-py/pull/57) in the background today. Is a commit expected on [vizarr#43](https://github.com/hms-dbmi/vizarr/pull/43)?
- Simon: Running omero-ms-thumbnail on idr test90 backend omero servers, now working on nginx front-end proxy
- Dom: working on CLI export of labels
    - Looking into having reading from omero and writing in separate.

## 2020-11-16

- Josh
  - Print statements in [57](https://github.com/ome/ome-zarr-py/pull/57#pullrequestreview-529892282)
- Will
  - vizarr PR is largely done. One bug needs fixing.
  - was going to look at Labels on plate
    - multiscaled? yes actually the number of resolutions must match the associated images as per the current spec
  - focusing on performance of the plates first though
- Dom: nothing on the NGFF front
- Simon
  - nothing on the microservices front
  - target? Josh: current assumption was thumbnail microservice running in production
- David
  - running into issues opening files directly from S3. Should have reader available for testing shortly
    - covers multiscales + labels
    - open twitter thread about bdv-h5 -> ome-zarr
  - J-M: library? Currently modified version
- J-M
  - examples of extracting subsets of Pytables
    - Josh: also open PR by Draga. Will: best way to test it? Might start the discussion on the PR
- Seb
  - will dive into the spec later today (once compounds are fixed)

- AOB
  - [Draga's tabular PR](https://github.com/ome/ome-zarr-py/pull/61/files)