---
# System prepended metadata

title: IDR-NGFF 2020-W46
tags: [NGFF]

---

---
tags: NGFF
---

# IDR-NGFF 2020-W46

## 2020-11-13
- Josh
  - n5 conversion fun incl. fixing a zarr bug. Transfer speeds from EMBL aren't great.
  - If we're done with PRs for the moment, then can reconvert data and move it to S3.
- Will
  - napari fixed to handle missing plate acquisitions (as vizarr). It can handle the existing acquisitions (to support recent exports)
  - Getting napari working with 5D pyramid through dask. Previously was getting segfaults. Working on local example that doesn't need a done of data to work. Goal is to test the performance of all the datasetes.
  - Josh: https://github.com/napari/napari/issues/1349 (or similar) for the segfault issues. Raise a flag if it's still happening.
- J-m:
  - merging PRs for analysis.
  - focusing on GH actions on the various repositories.
  - David, need anything on the Java front?
- David:
  - snippet was useful. Testing download and looks like it's working fine (on multiscale)
  - Moving modified version of jzarr (header problem)
  - Getting that cleaned up today for a first version.
  - Josh: export with something other than blosc?
- Dom
  - https://github.com/ome/omero-cli-zarr/pull/44 going in
  - Josh: `bin/omero segment` perhaps? :smile:
  - Will: added a loop around all the images in a plate (after Dataset_to_Plate)
  - Will: also have code from creating a demo for omero-parade
  - J-m: could go over Allen Cell Segmenter
  - **omero-segmenter**!
- AOB
  - Microservices? (Josh) --> Monday.
  - Draga & tabular data --> what example data do we want?
    - Josh: think he's teaching ome-zarr-py to show tabular in napari
    - Will: Have omero-cli-zarr export it?
  - Blog: publish as is or include plates? (Josh)
    - J-m: ok to get it out.
  - Jason (in absentia): do we want to schedule internal demo for next Friday, just to review where we are?  Defer to you all when would be good time.
    - Will: Plate acquisitions & tabular data
    - J-m: perhaps also existing versus new labels (Josh: can we make the scripts more usable?)

## 2020-11-12
- Will
  - Done https://github.com/hms-dbmi/vizarr/pull/43#issuecomment-725504246 "Test data"
    - Seb: assume we will regenerate the old data with the new layout before copying to S3 
  - Doing: Look at Dom's masks PR, back to napari to support new acquisitions spec
- Josh (in absentia)
  - See [thread](https://openmicroscopy.slack.com/archives/C0K5WAD8A/p1605102077229600) about json-ld styling. Happy to discuss, otherwise will look towards this as a proposal.
  - Likely going back to converting n5 to zarr today and mobie. Tischi is uploading TBs of data to idr-upload.
  - Can chat on specifics when I get back if needed.
- Seb:
  - Working on next spec version later today.
  - No real examples of multi-acquisitions with multipled FOVs. One sample OME-XML with 3 acquisitions, with 1 well and 3 FOVs if useful
- J-M
  - Issue with truncated Zarr? - https://github.com/ome/omero-analysis-desktop/pull/3
      - False alarm, truncated in IDR!
- Dom
  - https://github.com/ome/omero-cli-zarr/pull/44
  - First handle Will's acquisitions PR then adjust
- Petr
  - J-M: review Zarr-based notebooks for next week?
  - ITK workshops. possible upcoming discussions
- David
  - testing. Sample data? J-M to show code to download from S3 Embassy
  - https://github.com/jburel/minimal-omero-client/tree/zarr_download


## 2020-11-11 (roundtable)
- Josh:
  - Focusing on:
    - Writing down metadata (bioschemas but also zarr)
  - Could use help on if anyone is interested:
    - Script for restructuring N5 data
    - Actually loading data in Mobie GUI
- Simon:
  - Configurable vizarr build prefix https://github.com/hms-dbmi/vizarr/pull/49
    - E.g. ``VIZARR_PREFIX=/idr/vizarr/test-prefix npm run export``
    - https://minio-dev.openmicroscopy.org/idr/vizarr/test-prefix/index.html?source=https://minio-dev.openmicroscopy.org/idr/zarr/v0.1/9822151.zarr
  - Relative prefix seems to work for now but in general this not guaranteed to work https://github.com/vercel/next.js/issues/2581
    - ``VIZARR_PREFIX=./ npm run export``
    - https://minio-dev.openmicroscopy.org/idr/vizarr/test-dot-slash-prefix/index.html?source=https://minio-dev.openmicroscopy.org/idr/zarr/v0.1/9822151.zarr
  - Feedback: :tada: :heart:
  - Question: could the source be relative as well?
    - minio says ``"<Message>Resource name contains bad components such as ".." or ".".</Message>"`` but vizarr could resolve something like https://minio-dev.openmicroscopy.org/idr/vizarr/test-dot-slash-prefix/index.html?source=../../zarr/v0.1/9822151.zarr
- Seb
  - working on spec group rewriting
  - rephrasing to be clearer. starting at images and working up.
- J-M
  - looking into writing the analysis output as derived labels
  - will write code on the side and then can look at refaatoring
- Dom
  - Will- PR looking good. Little more testing.
- Will
  - https://github.com/ome/omero-ms-zarr/issues/78 -- **comments welcome ASAP**
  - Spent yesterday testing. Model decision on acquisitions.
  - Also discussion about how to show n the clients. May not be able to do anything too elaborate in vizarr. Had prototyped a dropdown for choosing between them. Pretty easy to add. Layout is currently limited to a grid. Next level would be to look at properly positiioning (which we'll want to do in the future anyway)
  - Also looking at napari
    - labels on a plate
    - performance of stitching
- David
  - Missed much? HCS still in flux. Melissa is implementing so could re-use that Java code.
  - J-m: have some code for reading zarr from s3
    - jzarr has a problem with blosc

## 2020-11-10
### Clients  
- notebooks with labels and analysis available, ready for review:
    - [Petr/Seb] [labels vs stardist prediction model(default)](https://github.com/ome/omero-guide-python/pull/26)   P:Done
      - Model could be improved
    - [Petr/Seb] [labels vs ilastik probability map](https://github.com/ome/omero-guide-ilastik/pull/22)
    - [Petr] [napari/zarr labels in binder](https://github.com/ome/omero-analysis-desktop/pull/3)
      - Napari version bumped.
    - Video attached for review (as zip)
    - No more warning in binderhub at EBI. (Good for performance comparison)
    - Seb: when do we want labels to show up in S3 for derived labels?
      - J-m: agreed. Can push in that direction.
      - See http://45.86.170.228/ (EBI binder)
- Will: Changes in vizarr to switch between acquisitions
  - Walk up the path if there are no zattrs
  - In the future could also add metadata to the acquisitions. Seb: most likely would be timestamp. Josh: and name.
    - Will: what about duplication? Seb: cf. what we currently done for Wells. Start with none, then we will have to decide.
    - Will: does the path value need to match the acquisition name? **Undecided**
    - Simon: do we want metadata at every level? **Undecided**
    - Josh: potentially defer these decisions to the next round of changes? Any of them blocking the current work?
- Will: https://github.com/ome/omero-cli-zarr/pull/43
  - Bug for multiple acquisitions where later acquisitions start at higher fields
### Datasets
- Seb: searching for multiple fields per acquisition
- Dom: getting his PR before the changes come (Then need re-generation)
### Formats
- jzarr: working on ``blosc`` support in November [#8](https://github.com/bcdev/jzarr/issues/8) (blocker for us).
  - J-m: keeping an eye on it. They are using an older version of the library for S3 access.
### Infrastructure
- Simon: FYI-- EBI `pilot-zarr1-dev`, 100 GB root partition, 1500GB `/data`:
  ```
  $ ssh -A idr-pilot.openmicroscopy.org
  $ ssh zarr1-dev
  ```
  - https://github.com/openmicroscopy/management_tools/pull/1308
  - https://github.com/IDR/deployment/pull/293
    - Todo: ``idr-ngff-1`` ssh alias
  - Note: ``/uod/idr/filesets`` don't exist yet. (``/nfs/bioimage`` does)
- J-m: good place for making ilastik labels
    - i.e. need to think about where those data sits (tabled)
    - raises the multiple s3 issue

### Misc
- Josh
   - netzdf/Java: hopefully in progress (wrapping C library in Java)
   - BioHackathon: bioschemas & S3
     - sitemap & content-type on S3 all work.
   - Cloud Optimised GeoTiff slack: https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999
     - https://github.com/pangeo-data/cog-best-practices
     - https://twitter.com/_VincentS_/status/1325823512393691136 etc. etc.
     - NB: J-m Ilaria had some geo-tiff that spun forever. Have them locally. (Rest are on nightshade)


## 2020-11-09
### Clients
- Will: not much since the video. One thing to look re: click-handling on large plates. (Number of layers exceeds a limit. Trevor has posted a workaround). **Next:** multiple acquisitions for plates.
- Feedback:
  - Amazing. Impressive. Yadda yadda. Nice to have it for various BioImaging discussions.
  - Decorative: all running locally. How much of it is home network? ("It even works in Fife!")
    - When all the data is in the browser it seems quick (even if could be improved)
    - There's still unexplained differences.
    - Josh: try on virtual desktop?
    - JRS: important demos that others will be interested in. But some will not understand why it's "slow"
    - Josh: Juan's experience running remotely from Australia
    - Simon: running web server locally?
    - JRS: good for diagnosing but the data needs to be remote
    - Will: others to try the demo?
  - Otherwise, acquisitions this week and then get it to the community?
  - Will: looking at overlay grids in vizarr like in napari, but not yet working.
### Datasets
- Dom: opened PR
- Will: listed all datasets
  - idr0033
  - idr0002
  - idr0004
  - Might need manual fix to rename plateAcquisitions as acquisitions. Also int -> string for columns and rows
- Blog post
  - Some ideas of the per-plate conversion time would be useful for the blog post
  - Will: when is the blog post published? v2?
  - Josh: something as soon as we have the plates ready. Also some more systematic registry.
  - Blog post vs image.sc post
  - All images should work with existing vizarr viewer
    - Simon: Might bring us back to upgrading under known location with given prefix
    - Josh: embed into the blog?
    - Having something in our control would be good in all cases
### Formats
- Seb: updated PR for the spec to capture the current PRs. haven't done the links to the datasets etc. per the above conversation. Need versioned/timestamped folders and various links to go around.
### Infrastructure
- Simon: Planning to bring up a zarr dev machine in EBI, probably behind pilot-proxy
  - roughly, "ebi-zarr-1" to do conversions on Embassy.

----

## Template
### Clients
-
### Datasets
-
### Formats
-
### Infrastructure
-