---
# System prepended metadata

title: IDR-NGFF 2020-W45
tags: [NGFF]

---

---
tags: NGFF
---

# IDR-NGFF 2020-W45

## 2020-11-06 (roundtable)
- J-m: flag in the Amazon SDK needed, so minio & S3 work.
  - working on prototype for the Java community
  - jzarr still can't handle the file. see https://github.com/bcdev/jzarr/issues/7
  - J-m to synchronize with David next week. can re-use chunk of code.
  - downloading images & labels with sliders in the notebook (python)
  - want to re-run the analysis on an image to generate labels and compare with the stored label
  -  opens the question of datasets generation with labels using omero-cli-zarr
- Josh: likely going back to n5-zarr-s3 support
- Will: tested napari supporting wells to have both napari/vizarr
  - PR 43 for reading the latest spec
  - napari also handles sparse wells
  - NB: need to make some breaking changes to the previous 2 specs.
  - cli-zarr good to review & merge?
  - same for zarr-py?
  - No commits coming. Will start on video from them.
  - videos to openmicroscopy channel?
- Simon
    - working on conda packaging/workflow
- Dom
    - about to open PR against omero-cli-zarr for masks/labels export
- Petr
    - I2K tweets need to go out
- AOB (next week & beyond)
  - fsspec bump coming
  - dask.map_ ...
  - HPA conversion (Jason): need **metadata**. Target after HCS?
  
## 2020-11-05 (roundtable)

- Will
    - Changes showing well content. Rebased on Trevor work with grid layer. Looks quite nice. Slow zoom-in is now fixed
    - only tested with one set of data. Seb: also idr0094 plate
    - conversion with idr0002, idr0004? other plates suggested
    - Comments re PR/spec
        - Seb: happy to have it in, tag and port to spec.md
        - https://github.com/ome/omero-cli-zarr/pull/43
        - Josh: extra capacity from tomorrow
        - Seb: merging by noon if no-one has raised objections
- JRS
    - Calls for existing EOSC project ideas incl. cross-research infrastructure
        - Some work with idr0094 (remdesivir vs hydroxychloroquine).
        - idr0083 (ML method)
        - mobie
    - Seb: actionable if we want (15-20min for conversion + transfer time)
- Simon
    - vizarr: set up own OME netlify account as opposed to Will's
        - Will: could even give nice URL
        - Need some authorization.
        - Alternative: host the file somewhere. Simon: require own JS build (Similar to iviewer)
        - Build artifact, put it on GH and download it/use it anywhere
            - Prefix handling - currently set to absolute by default
        - Postpone till next week. Maybe call with Ilan/Trevor to enquire about deployment
- Dom
    - Keep working on plates + labels
    - Simon: kicked off trigger job
- Seb
    - see above, tagging omero-ms-zarr, omero-cli-zarr
    - still screen reader leak :scream_cat: 
- J-m
    - reactivated code to connect to s3/embassy using Amazon SDK. Returns nothing (compared to minio). Pushing code.
- Josh:
    - zarr call: UTC, majore take-home. HDF5 can start looking like a Zarr file
- Anyone else?

## 2020-11-04

- zarr catalog (Will)
  - https://snoopycrimecop.github.io/idr.openmicroscopy.org/zarr
  - Summary: no objections but also no rush
  - J-m: great to have it. Viz, analysis. Later links to ITR.
  - Will: to be used by other clients, could put it into json format.
    - would need some code to generate the JSON.
    - Simon: could load JSON in jekyll.
  - Seb: adding to the menu? That will be bring promotion.
    - will need to evaluate
    - Simon: before the page goes public?
    - J-m: other pages are starting to use it.
    - JRS: so far have avoided those issues. So far the tools aren't as easy as we'd like them to be. Communicating that the effort will be broader (n5, etc.)
    - Josh: blog post so that it's timestamped?
    - ome/blog
- Will: HCS updates
  - https://github.com/ome/omero-cli-zarr/pull/43
  - https://mystifying-lalande-e12142.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/idr0033-rohban-pathways/41744_illum_corrected/pr_43/5966.zarr
  - Trevor has introduced a grid layout component based on Will's code. Need to look at it. (**Next**)
  - Will: looking for plate with layout information. Might be pushed off. Quite a bit more work for not a huge.
  - Josh: layers possible? Haven't looked into it. As a short term fix, might use hoover events.
  - Will: also multiple **plate acquisitions**. Open them each in a new page? Needs custom elements.
- Format
  - Josh: so currently working? Yep
  - Seb: hard-coded decision on the field layout of the field. Josh: based on the screen size and could flow? Will: pre-calculated at the moment. Seb: idr0001 is good for testing.
- AOB
  - Note: netlify URLs are broken. Josh: target freezing next week. Then we can publish versioned viewer as well. Will: also need to open that discussion on the viv/vizarr side. J-m: agree!
  - J-m: I2K workshop. What's to be shown?
    - Josh: Either mobie linked data
    - Petr: Possibly the "beyond local" memory
    - Josh: assume we can do more of this coordination for the Jan/Feb time slots.
    - Simon: these are much more like a hackathon
 - Seb: ok to review the new layout tomorrow? Will: **yes**
 - NB: working on screenreader leak. need to restart `idr-testing` periodically. exporter is a good test.
   - Be aware of others using the server before restarting idr-testing
   - **But** perhaps restart before you restart any long running jobs

## 2020-11-03

- Will:
  - https://github.com/ome/omero-ms-zarr/issues/76#issuecomment-720570094
  - Cleanup of metadata (keeps growing)
  - Seb: name versus index
  - Josh: shape
  - Sparseness: A1-Z9 or B2-X8
  - Force fields to be numerical? (cf. labels)
  - Demo?
    - Defining concepts concretely
    - View FoW in the client
    - Data generation
      - Will: local versus few hours remotely
      - Seb: Create subset?
      - Dom: --cache-numpy
      - Josh: add to git?
    - Table: positional information of fields
    - Distribute  video?
- Dom: plates with masks
  - using merge-ci. Dataset to Plate? Yes
  - Seb: MIAS example populates example - see https://github.com/ome/bioformats/pull/2319#issuecomment-219007029 
- Josh
  - Freezing HCS
      - Simon: give ourselves 2 weeks for all changes? 
  - Label work for I2K?
      - https://github.com/ome/ome-zarr-py/issues/60
      - New metadata for Tischi et al
          - J-M: danger of introducing something we want to support
          - Simon: understanding them as k/v pairs i.e. free-form JSON
          - Will: pick the best of three options above
      - Open question of whether we can engage them in implementing it ?
- Simon: Who has permissions/access to release [bioformats2raw 0.2.6](https://github.com/ome/conda-bioformats2raw/pull/4) ?
  - Tabled until next week

## 2020-11-02

### From last week
- versioning concepts (Will): one specification vs multiple
    - Seb: issue re: mask labels, i.e. others are coming? Likely to transfer to omero-ms-zarr
    - Will: tabular data. Probably requires some investigation, trade-offs etc
        - Support exists in Zarr? Only for homoegeneous tables e.g. features tables. 
        - J-M: requirement for string comes pretty quickly. Workaround exist.
        - Simon: JSON labels-like as a short term workaround?
        - Main caveat is scalability.
        - Some sign-off must happen with 0.1 - also notify the community (image.sc, Simon: also Twitter)
- registry of zarr data
    - Josh looking for branch
- i2k & idr0001 (has multiple runs)
  - Seb: good target (with masks)
  - What changes to the spec are then needed? (Also need to export them from omero-cli-zarr). In UI, would need to allow switching
  - Will: missing between multiple runs & fields. About to start working on that in vizarr.
    - Given a URL how do you know it's a well? Need a shallow spec. Seb: for the moment, could go up three levels and find the plate. Will: general issue of then needing to load more .zattrs slows things down.
      - Simon: similar to how do you know what chunks exist (**sparseness**)
      - J-m: could apply **overrides** at lower levels. i.e. default is on plate. See also clear path for finding the channel name if it doesn't exist (emission wavelength, etc.)
- Josh: demos every Friday per month?
   - Seb: away on Fridays, video them
   - Simon: CSHL

### Clients

### Datasets

- idr0033
    - J-M: pipelines submitted with idr0033 are not great. Looking into other options
- idr0001
    - Dom: plate acquisitions already in the spec and is exported with omero-cli-zarr
    - plate masks? Seb: idr0001 masks currently stored as polygons
    - Will: some code at image.sc
    - Simon: issue with reinterpreting the data (how to deal with diagonals)
    - Simon to look into other candidates plates with masks

### Formats

- Seb: from last week, HCS specification next todos
    - multiple acquisitions/runs
    - well vs column vs row spec
    - metadata distribution/redundancy
    - spatial metadata for multiple fields of view
- [bioformats2raw 0.2.6](https://github.com/ome/conda-bioformats2raw/pull/4)
- [ome/ome-zarr-py Add class annotations and/or other metadata properties to labels](https://github.com/ome/ome-zarr-py/issues/60)
  - Move to omero-ms-zarr? Or a new ome-zarr-spec repo?
### Infrastructure
-

----

## Template
### Clients
-
### Datasets
-
### Formats
-
### Infrastructure
-