---
# System prepended metadata

title: IDR-NGFF 2020-W41
tags: [NGFF]

---

---
tags: NGFF
---

# IDR-NGFF 2020-W41

[![hackmd-github-sync-badge](https://hackmd.io/GwjreFrvTBSxEPd0gXVH2Q/badge)](https://hackmd.io/GwjreFrvTBSxEPd0gXVH2Q)

## 2020-10-09

Attending: J-M, Petr, David, Frances, Dom

### Clients

N/A


### Datasets

- idr0033 notebooks
    - not much success with re-adding some modules to the pipeline

### Format

David: very little progress
J-M: not much activity on jzarr in the past few months so maybe not a good candidate


### Microservices
feedback from simon from call with Chris on Monday, Dom did not attend the call.

## 2020-10-08

Attending: Seb, Frances, J-M, Dom, Petr, Simon, David

### Clients

N/A

### Datasets

- idr0033 notebooks
    - J-M: cleaned submitted pipeline (Analysis.cppipe incl. other steps like QC). Need to pass corresponding image in illumination corrected plate
        - should be able to generate CSV files (nuclei, ...)
        - Seb: something we can use e.g. for segmentations
        - Everything stored in the SQL database. Need to check what is re-usable
        - Modifications required to the pipeline
    - J-M to push in-progress work to idr0033 repo

### Format

- Seb: opened https://github.com/ome/omero-ms-zarr/issues/73 following yesterday's discussion
    - Typical number of files for large plates: 10K-100K
    - Simon: option 2 mentioned as complex but metadata would be the same?
        - Seb: assuming the HCS metadata is largely identical between both options but distributed differently
        - Also question of whether having more intermediate specs makes it harder to implement for consumers
    - Simon: Number of requests for S3 - cf scalability
        - Metadata distribution between 2 options: Well, WellSample 
    - J-M: consideration about metadata performance. Reference image
        - Potentially open a separate issue
        - Seb: generically apply to collection of images (HCS or non HCS)
        - Raises the question of metadata inheritance
        - J-M: reduces the amount of metadata to fetch (cf issues with omero-marshal and JSON)
        - Simon: mandatory for HCS spec?
        - Seb: current status quo is to duplicate rendering at each image level
        - Blocker for public announcement or first milestone of spec. J-M: not for internal milestones but for external consumption.
    - Decisions to be made
        - HCS group layout
        - HCS metadata (new keys)
        - hierarchical/distributed metadata (rendering)
        - all of the above driven by benchmarking/performance considerations
        - optional vs mandatory keys e.g. plate acquisition but also well sample/image
    - Keep the issue above as the main placeholder for all HCS discussions for now
        - Simon/J-M to add comments
- David: starting to investigate libraries for reader. Probably to be discussed on Monday

### Microservices

- omero-ms-image-region working for Dom (local), not in Docker environment (Simon)
    - upcoming call with Chris to clarify configuration

## 2020-10-07

Attending: Jason, Seb, Frances, J-M, Dom, Petr, Simon, David

### Clients

- Another use case: playing all timelapse wells in a plate, or visualising all wells in a plate, simultaneously
    - cf image.sc thread with mitocheck well
    - J-M HCS NGFF format could be applied to Mitocheck after idr0033
    - Napari, VTK possible viewers
    - J-M: consumer not blocked at the moment. 
    - J-M to get in touch
    - Jason: validity of Mitocheck biologically in question (cf cell lines, perturbations..)
        - J-M: mostly proof of concept

### Format

- flat vs nested structure
    - cf https://github.com/ome/omero-guide-cellprofiler/blob/master/notebooks/idr0002_zarr.ipynb
    - Seb: to create issue on omero-ms-zarr discussing various layouts
        - Down to number of groups
    - Simon: number of images? Seb: 1k-10K typical
        - caveat of how many files/directories under a single parent when not stored on S3

### Datasets

- idr0033 original masks
    - https://github.com/IDR/idr0033-rohban-pathways/pull/1 ready for review
    - Petr: shoudl outlines be converted as masks
        - Seb: only outlines. Converting back to masks is a computational work
    - Masks vs NGFFs
        - strategy: moving away from using the DB to store masks (and ROIs)
        - viewer? to be investigated. vizarr is one option
        - micro-service retrieving ROI from S3/Zarr and translating it into the current expectation
- idr0033 pipelines
    - J-M: looked at pipelines. Version 3, human-readable format
        - Will look at setting up the infrastructure
        - Retrieve the binary data from OMERO first.
        - Consume data from S3/Zarr in a second step
    - channel name critical for analysis
    - using study repository also advantageous for ITR. Simon: can also pin its own dependencies

### Microservices

- Dom got image-region micro-service working locally
    - Simon to consume Dom's config changes in example Docker stack
    - J-M: try it out in daily CI server next
    - Java versions?
    - Feedback for GS. Which stable branches? roadmap?

## 2020-10-06

Attending: Mark, Seb, Petr, Frances, Simon, Dom, David, J-M

### Clients

### Datasets

- idr0033 original masks
    - Seb: should have something up for review tomorrow
- notebooks
    - Updating the process to simplify things. Connection from idr-py using WS not working. Originally tested for mybinder
    - idr-notebooks (idr-analysis) working fine. 
    - Simon: target infrastructure PRs for prod90? J-M: no objection
    - Goal is to get connection logic out of idr-py, maybe in omero-py directly
      - https://github.com/ome/design/issues/106
    - Training will be consuming this work as well 
    - Useful for institution with firewalls. Websockets became the de-facto default
- idr0033 notebooks
    - J-M: could not find pipelines on NFS
    - Look into email, Trello archives
    - notebook might not live within idr-notebooks
        - Seb: could even live in idr0033-rohban-pathways repository
        - Simon: be mindful of repo size for MyBinder.
        - Should be fine in that case (34M) but to be reviewed for other repos 
        - idr-notebooks == demo of using the API
    - One of the original Trello cards (pre-dates Redmine) https://trello.com/c/7JNEWNcW/577-idr0033-rohban-pathways-target-accelerator-orf-data-set-carpenter-lab
    - CellProfiler pipelines `/uod/idr/filesets/idr0033-rohban-pathways/20170214-original/pipelines/`

### Format

- N/A

### Microservices
- omero-ms-image-region:
  - https://github.com/manics/docker-example-omero-microservices updated again, ms-image-region still broken (Vertx or Java issue? Anyone know?)
  - fixed bug on Docker compose
  - paste error in Slack/GitHub for GS?
  - Dom still trying locally 

https://github.com/ome/design/issues/106
## 2020-10-05
Attending: Petr, Seb, Dom, J-M, David, Simon, Frances

### Clients

### Datasets

- idr0033 original masks (Seb)
    -  working on script to in-place attach to the relevant illumination corrected images
    - targetting next IDR release (prod90) unless stated otherwise
- notebooks (J-M)
    - Working on making notebooks binderhub compatible - to be discussed with Simon
    - Might require looking into CI (Travis-ci.org vs travis-ci.com vs GitHub actions)
    - idr-py does not work with websockets. Simon: should fix it
    - to be continued during IDR call
    - Goal is to run CellProfile pipeline. Versions issues. Pipeline might have to be modified
    - First identify whether existing the pipeline works with CP 3.3
    - Upcoming work on guides

### Format

- Josh setting up infrastructure for testing pipelines (conversion/import/export)
    - Simon asking CB about similar examples

### Microservices

- Simon:
    - thumbnail micro-service working
    - image-ms not working (0.5.2 artifact) running, returns 403
    - Dom: tested building locally but failed. Simon: need to update config accordingly
    - Still a bit unclear about relationship between image-ms-region and thumbnails 
    - Simon: use as building blocks

### AOB

- Simon: have weekly notes rather than daily?
    - Seb: will test this week
