---
tags: NGFF
---

# IDR-NGFF 2020-W41

[](https://hackmd.io/GwjreFrvTBSxEPd0gXVH2Q)

## 2020-10-09

Attending: J-M, Petr, David, Frances, Dom

### Clients

N/A

### Datasets

- idr0033 notebooks
- not much success with re-adding some modules to the pipeline

### Format

David: very little progress
J-M: not much activity on jzarr in the past few months so maybe not a good candidate

### Microservices
feedback from simon from call with Chris on Monday, Dom did not attend the call.

## 2020-10-08

Attending: Seb, Frances, J-M, Dom, Petr, Simon, David

### Clients

N/A

### Datasets

- idr0033 notebooks
- J-M: cleaned submitted pipeline (Analysis.cppipe incl. other steps like QC). Need to pass corresponding image in illumination corrected plate
- should be able to generate CSV files (nuclei, ...)
- Seb: something we can use e.g. for segmentations
- Everything stored in the SQL database. Need to check what is re-usable
- Modifications required to the pipeline
- J-M to push in-progress work to idr0033 repo

### Format

- Seb: opened https://github.com/ome/omero-ms-zarr/issues/73 following yesterday's discussion
- Typical number of files for large plates: 10K-100K
- Simon: option 2 mentioned as complex but metadata would be the same?
- Seb: assuming the HCS metadata is largely identical between both options but distributed differently
- Also question of whether having more intermediate specs makes it harder to implement for consumers
- Simon: Number of requests for S3 - cf scalability
- Metadata distribution between 2 options: Well, WellSample
- J-M: consideration about metadata performance. Reference image
- Potentially open a separate issue
- Seb: generically apply to collection of images (HCS or non HCS)
- Raises the question of metadata inheritance
- J-M: reduces the amount of metadata to fetch (cf issues with omero-marshal and JSON)
- Simon: mandatory for HCS spec?
- Seb: current status quo is to duplicate rendering at each image level
- Blocker for public announcement or first milestone of spec. J-M: not for internal milestones but for external consumption.
- Decisions to be made
- HCS group layout
- HCS metadata (new keys)
- hierarchical/distributed metadata (rendering)
- all of the above driven by benchmarking/performance considerations
- optional vs mandatory keys e.g. plate acquisition but also well sample/image
- Keep the issue above as the main placeholder for all HCS discussions for now
- Simon/J-M to add comments
- David: starting to investigate libraries for reader. Probably to be discussed on Monday

### Microservices

- omero-ms-image-region working for Dom (local), not in Docker environment (Simon)
- upcoming call with Chris to clarify configuration

## 2020-10-07

Attending: Jason, Seb, Frances, J-M, Dom, Petr, Simon, David

### Clients

- Another use case: playing all timelapse wells in a plate, or visualising all wells in a plate, simultaneously
- cf image.sc thread with mitocheck well
- J-M HCS NGFF format could be applied to Mitocheck after idr0033
- Napari, VTK possible viewers
- J-M: consumer not blocked at the moment.
- J-M to get in touch
- Jason: validity of Mitocheck biologically in question (cf cell lines, perturbations..)
- J-M: mostly proof of concept

### Format

- flat vs nested structure
- cf https://github.com/ome/omero-guide-cellprofiler/blob/master/notebooks/idr0002_zarr.ipynb
- Seb: to create issue on omero-ms-zarr discussing various layouts
- Down to number of groups
- Simon: number of images? Seb: 1k-10K typical
- caveat of how many files/directories under a single parent when not stored on S3

### Datasets

- idr0033 original masks
- https://github.com/IDR/idr0033-rohban-pathways/pull/1 ready for review
- Petr: shoudl outlines be converted as masks
- Seb: only outlines. Converting back to masks is a computational work
- Masks vs NGFFs
- strategy: moving away from using the DB to store masks (and ROIs)
- viewer? to be investigated. vizarr is one option
- micro-service retrieving ROI from S3/Zarr and translating it into the current expectation
- idr0033 pipelines
- J-M: looked at pipelines. Version 3, human-readable format
- Will look at setting up the infrastructure
- Retrieve the binary data from OMERO first.
- Consume data from S3/Zarr in a second step
- channel name critical for analysis
- using study repository also advantageous for ITR. Simon: can also pin its own dependencies

### Microservices

- Dom got image-region micro-service working locally
- Simon to consume Dom's config changes in example Docker stack
- J-M: try it out in daily CI server next
- Java versions?
- Feedback for GS. Which stable branches? roadmap?

## 2020-10-06

Attending: Mark, Seb, Petr, Frances, Simon, Dom, David, J-M

### Clients

### Datasets

- idr0033 original masks
- Seb: should have something up for review tomorrow
- notebooks
- Updating the process to simplify things. Connection from idr-py using WS not working. Originally tested for mybinder
- idr-notebooks (idr-analysis) working fine.
- Simon: target infrastructure PRs for prod90? J-M: no objection
- Goal is to get connection logic out of idr-py, maybe in omero-py directly
- https://github.com/ome/design/issues/106
- Training will be consuming this work as well
- Useful for institution with firewalls. Websockets became the de-facto default
- idr0033 notebooks
- J-M: could not find pipelines on NFS
- Look into email, Trello archives
- notebook might not live within idr-notebooks
- Seb: could even live in idr0033-rohban-pathways repository
- Simon: be mindful of repo size for MyBinder.
- Should be fine in that case (34M) but to be reviewed for other repos
- idr-notebooks == demo of using the API
- One of the original Trello cards (pre-dates Redmine) https://trello.com/c/7JNEWNcW/577-idr0033-rohban-pathways-target-accelerator-orf-data-set-carpenter-lab
- CellProfiler pipelines `/uod/idr/filesets/idr0033-rohban-pathways/20170214-original/pipelines/`

### Format

- N/A

### Microservices
- omero-ms-image-region:
- https://github.com/manics/docker-example-omero-microservices updated again, ms-image-region still broken (Vertx or Java issue? Anyone know?)
- fixed bug on Docker compose
- paste error in Slack/GitHub for GS?
- Dom still trying locally

https://github.com/ome/design/issues/106
## 2020-10-05
Attending: Petr, Seb, Dom, J-M, David, Simon, Frances

### Clients

### Datasets

- idr0033 original masks (Seb)
- working on script to in-place attach to the relevant illumination corrected images
- targetting next IDR release (prod90) unless stated otherwise
- notebooks (J-M)
- Working on making notebooks binderhub compatible - to be discussed with Simon
- Might require looking into CI (Travis-ci.org vs travis-ci.com vs GitHub actions)
- idr-py does not work with websockets. Simon: should fix it
- to be continued during IDR call
- Goal is to run CellProfile pipeline. Versions issues. Pipeline might have to be modified
- First identify whether existing the pipeline works with CP 3.3
- Upcoming work on guides

### Format

- Josh setting up infrastructure for testing pipelines (conversion/import/export)
- Simon asking CB about similar examples

### Microservices

- Simon:
- thumbnail micro-service working
- image-ms not working (0.5.2 artifact) running, returns 403
- Dom: tested building locally but failed. Simon: need to update config accordingly
- Still a bit unclear about relationship between image-ms-region and thumbnails
- Simon: use as building blocks

### AOB

- Simon: have weekly notes rather than daily?
- Seb: will test this week