IDR-NGFF 2020-W46

--- tags: NGFF --- # IDR-NGFF 2020-W46 ## 2020-11-13 - Josh - n5 conversion fun incl. fixing a zarr bug. Transfer speeds from EMBL aren't great. - If we're done with PRs for the moment, then can reconvert data and move it to S3. - Will - napari fixed to handle missing plate acquisitions (as vizarr). It can handle the existing acquisitions (to support recent exports) - Getting napari working with 5D pyramid through dask. Previously was getting segfaults. Working on local example that doesn't need a done of data to work. Goal is to test the performance of all the datasetes. - Josh: https://github.com/napari/napari/issues/1349 (or similar) for the segfault issues. Raise a flag if it's still happening. - J-m: - merging PRs for analysis. - focusing on GH actions on the various repositories. - David, need anything on the Java front? - David: - snippet was useful. Testing download and looks like it's working fine (on multiscale) - Moving modified version of jzarr (header problem) - Getting that cleaned up today for a first version. - Josh: export with something other than blosc? - Dom - https://github.com/ome/omero-cli-zarr/pull/44 going in - Josh: `bin/omero segment` perhaps? :smile: - Will: added a loop around all the images in a plate (after Dataset_to_Plate) - Will: also have code from creating a demo for omero-parade - J-m: could go over Allen Cell Segmenter - **omero-segmenter**! - AOB - Microservices? (Josh) --> Monday. - Draga & tabular data --> what example data do we want? - Josh: think he's teaching ome-zarr-py to show tabular in napari - Will: Have omero-cli-zarr export it? - Blog: publish as is or include plates? (Josh) - J-m: ok to get it out. - Jason (in absentia): do we want to schedule internal demo for next Friday, just to review where we are? Defer to you all when would be good time. - Will: Plate acquisitions & tabular data - J-m: perhaps also existing versus new labels (Josh: can we make the scripts more usable?) ## 2020-11-12 - Will - Done https://github.com/hms-dbmi/vizarr/pull/43#issuecomment-725504246 "Test data" - Seb: assume we will regenerate the old data with the new layout before copying to S3 - Doing: Look at Dom's masks PR, back to napari to support new acquisitions spec - Josh (in absentia) - See [thread](https://openmicroscopy.slack.com/archives/C0K5WAD8A/p1605102077229600) about json-ld styling. Happy to discuss, otherwise will look towards this as a proposal. - Likely going back to converting n5 to zarr today and mobie. Tischi is uploading TBs of data to idr-upload. - Can chat on specifics when I get back if needed. - Seb: - Working on next spec version later today. - No real examples of multi-acquisitions with multipled FOVs. One sample OME-XML with 3 acquisitions, with 1 well and 3 FOVs if useful - J-M - Issue with truncated Zarr? - https://github.com/ome/omero-analysis-desktop/pull/3 - False alarm, truncated in IDR! - Dom - https://github.com/ome/omero-cli-zarr/pull/44 - First handle Will's acquisitions PR then adjust - Petr - J-M: review Zarr-based notebooks for next week? - ITK workshops. possible upcoming discussions - David - testing. Sample data? J-M to show code to download from S3 Embassy - https://github.com/jburel/minimal-omero-client/tree/zarr_download ## 2020-11-11 (roundtable) - Josh: - Focusing on: - Writing down metadata (bioschemas but also zarr) - Could use help on if anyone is interested: - Script for restructuring N5 data - Actually loading data in Mobie GUI - Simon: - Configurable vizarr build prefix https://github.com/hms-dbmi/vizarr/pull/49 - E.g. ``VIZARR_PREFIX=/idr/vizarr/test-prefix npm run export`` - https://minio-dev.openmicroscopy.org/idr/vizarr/test-prefix/index.html?source=https://minio-dev.openmicroscopy.org/idr/zarr/v0.1/9822151.zarr - Relative prefix seems to work for now but in general this not guaranteed to work https://github.com/vercel/next.js/issues/2581 - ``VIZARR_PREFIX=./ npm run export`` - https://minio-dev.openmicroscopy.org/idr/vizarr/test-dot-slash-prefix/index.html?source=https://minio-dev.openmicroscopy.org/idr/zarr/v0.1/9822151.zarr - Feedback: :tada: :heart: - Question: could the source be relative as well? - minio says ``"<Message>Resource name contains bad components such as ".." or ".".</Message>"`` but vizarr could resolve something like https://minio-dev.openmicroscopy.org/idr/vizarr/test-dot-slash-prefix/index.html?source=../../zarr/v0.1/9822151.zarr - Seb - working on spec group rewriting - rephrasing to be clearer. starting at images and working up. - J-M - looking into writing the analysis output as derived labels - will write code on the side and then can look at refaatoring - Dom - Will- PR looking good. Little more testing. - Will - https://github.com/ome/omero-ms-zarr/issues/78 -- **comments welcome ASAP** - Spent yesterday testing. Model decision on acquisitions. - Also discussion about how to show n the clients. May not be able to do anything too elaborate in vizarr. Had prototyped a dropdown for choosing between them. Pretty easy to add. Layout is currently limited to a grid. Next level would be to look at properly positiioning (which we'll want to do in the future anyway) - Also looking at napari - labels on a plate - performance of stitching - David - Missed much? HCS still in flux. Melissa is implementing so could re-use that Java code. - J-m: have some code for reading zarr from s3 - jzarr has a problem with blosc ## 2020-11-10 ### Clients - notebooks with labels and analysis available, ready for review: - [Petr/Seb] [labels vs stardist prediction model(default)](https://github.com/ome/omero-guide-python/pull/26) P:Done - Model could be improved - [Petr/Seb] [labels vs ilastik probability map](https://github.com/ome/omero-guide-ilastik/pull/22) - [Petr] [napari/zarr labels in binder](https://github.com/ome/omero-analysis-desktop/pull/3) - Napari version bumped. - Video attached for review (as zip) - No more warning in binderhub at EBI. (Good for performance comparison) - Seb: when do we want labels to show up in S3 for derived labels? - J-m: agreed. Can push in that direction. - See http://45.86.170.228/ (EBI binder) - Will: Changes in vizarr to switch between acquisitions - Walk up the path if there are no zattrs - In the future could also add metadata to the acquisitions. Seb: most likely would be timestamp. Josh: and name. - Will: what about duplication? Seb: cf. what we currently done for Wells. Start with none, then we will have to decide. - Will: does the path value need to match the acquisition name? **Undecided** - Simon: do we want metadata at every level? **Undecided** - Josh: potentially defer these decisions to the next round of changes? Any of them blocking the current work? - Will: https://github.com/ome/omero-cli-zarr/pull/43 - Bug for multiple acquisitions where later acquisitions start at higher fields ### Datasets - Seb: searching for multiple fields per acquisition - Dom: getting his PR before the changes come (Then need re-generation) ### Formats - jzarr: working on ``blosc`` support in November [#8](https://github.com/bcdev/jzarr/issues/8) (blocker for us). - J-m: keeping an eye on it. They are using an older version of the library for S3 access. ### Infrastructure - Simon: FYI-- EBI `pilot-zarr1-dev`, 100 GB root partition, 1500GB `/data`: ``` $ ssh -A idr-pilot.openmicroscopy.org $ ssh zarr1-dev ``` - https://github.com/openmicroscopy/management_tools/pull/1308 - https://github.com/IDR/deployment/pull/293 - Todo: ``idr-ngff-1`` ssh alias - Note: ``/uod/idr/filesets`` don't exist yet. (``/nfs/bioimage`` does) - J-m: good place for making ilastik labels - i.e. need to think about where those data sits (tabled) - raises the multiple s3 issue ### Misc - Josh - netzdf/Java: hopefully in progress (wrapping C library in Java) - BioHackathon: bioschemas & S3 - sitemap & content-type on S3 all work. - Cloud Optimised GeoTiff slack: https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999 - https://github.com/pangeo-data/cog-best-practices - https://twitter.com/_VincentS_/status/1325823512393691136 etc. etc. - NB: J-m Ilaria had some geo-tiff that spun forever. Have them locally. (Rest are on nightshade) ## 2020-11-09 ### Clients - Will: not much since the video. One thing to look re: click-handling on large plates. (Number of layers exceeds a limit. Trevor has posted a workaround). **Next:** multiple acquisitions for plates. - Feedback: - Amazing. Impressive. Yadda yadda. Nice to have it for various BioImaging discussions. - Decorative: all running locally. How much of it is home network? ("It even works in Fife!") - When all the data is in the browser it seems quick (even if could be improved) - There's still unexplained differences. - Josh: try on virtual desktop? - JRS: important demos that others will be interested in. But some will not understand why it's "slow" - Josh: Juan's experience running remotely from Australia - Simon: running web server locally? - JRS: good for diagnosing but the data needs to be remote - Will: others to try the demo? - Otherwise, acquisitions this week and then get it to the community? - Will: looking at overlay grids in vizarr like in napari, but not yet working. ### Datasets - Dom: opened PR - Will: listed all datasets - idr0033 - idr0002 - idr0004 - Might need manual fix to rename plateAcquisitions as acquisitions. Also int -> string for columns and rows - Blog post - Some ideas of the per-plate conversion time would be useful for the blog post - Will: when is the blog post published? v2? - Josh: something as soon as we have the plates ready. Also some more systematic registry. - Blog post vs image.sc post - All images should work with existing vizarr viewer - Simon: Might bring us back to upgrading under known location with given prefix - Josh: embed into the blog? - Having something in our control would be good in all cases ### Formats - Seb: updated PR for the spec to capture the current PRs. haven't done the links to the datasets etc. per the above conversation. Need versioned/timestamped folders and various links to go around. ### Infrastructure - Simon: Planning to bring up a zarr dev machine in EBI, probably behind pilot-proxy - roughly, "ebi-zarr-1" to do conversions on Embassy. ---- ## Template ### Clients - ### Datasets - ### Formats - ### Infrastructure -