I2K NGFF Script

--- tags: I2K, NGFF --- # I2K NGFF Script - See also: - [file format descriptions](https://docs.google.com/document/d/1pgvLaJAUpC3fIH_hs3K78aq8RmsVpM_T7aPbzduBBto/edit) ## Introduction (30m) * Introduction * What is this workshop about * What is the structure * length, breaks, .. * Who are the presenters * Name * Current job * Role in and motivation for this project * Role in this workshop * Living document () * Need to use it (try it out; critical need) * Agreement to migrate * Acknowledge: (slide?) * https://docs.google.com/presentation/d/1jMXbl1McM3DeLWfVSy-Ky51VseXGkoIWUHw6wa0Y3p4/edit?usp=sharing * Constantin Pape (EMBL) * David Gault * Nicolas Chiaruttini (EPFL) * Tobias Pietzsch * Stephan Saalfeld * Juan Napari * Round table? Everyone 2 sentences? (10m) * Name * Current job? * Why did I join this workshop? * Josh: this may no longer be possible if we hit 35. * Tischi: Some ideas * Make breakout rooms with pairs of 2 introduction themselves to each other for 2 + 2 minutes * Does not help us but creates some social contact for the participants, which is important * Let them introduce themselves in a hackMD * Carpentries in breakout * 3 rooms of 13? * Challenges of browsing and sharing large 3D image data set (Talk) (Tischi) (10m) * https://docs.google.com/presentation/d/1L0Pxaou7ggsV8gpQAPga_JxvMvSN9I3-L1MPG18H6iU/edit?usp=sharing - Biological motivation - Platynereis atlas dataset - https://www.embl.org/news/science/finding-your-way-around-platynereis-dumerilii/ - https://www.biorxiv.org/content/10.1101/2020.02.26.961037v1 - TB sized - multi-scale - 200+ images - Limitations of current standards (ome.tif and ImageJ viewer) - one cannot freely browse in 3D - ImageJ Viewer does not support it - Tiff does not support it either - ImageJ viewer cannot overlay data of different resolutions - one would need to resave the fluorescence data (500 nm)^3 in the EM resolution (10 nm)^3, which is not good if the EM data is several TB in size and you have 200+ fluorescence data sets - Requirements for a new standard (TODO: Tischi: I may not talk about it?!) - Chunking (factilitates lazy loading) - Multi-resolution (facilitates fast visualisation) - Multi-scale (saving on storage space and data transfer bandwidth) - more a viewer than a file format issue - Cloud compatible (for data sharing) - IT departments generally don't allow opening up a file-server to the public - Q&A * Sharing large images (Talk) (Josh) (10m) - "My data" versus others data (other people look) - can't access file formats - opening file server to public may be a security issue - you don't really want people downloading - OMERO but must import - ... cloud (explain HDF5) - N5, zarr (NGFF for the cloud) - ngff.openmicroscopy.org/0.1 - Current efforts and the path towards a community standard - Metadata? (so others can use it) - Discussions on forum.image.sc - Scope: raw images, segmentation images, segmentation features - Current status - Q&A :::info _First break and installation?_ (5 min) ::: ## Demonstrations (1h) * Accessing cloud hosted image data * Python notebooks on data * explain zarr, s3, etc. * Will to put together on ome-zarr * https://github.com/ome/omero-guide-python/blob/master/notebooks/zarr-public-s3-multiscale.ipynb * Q&A * Python: napari (20m) * using remote desktop for napari? * Petr: could see about what is napari, explain how to open a new image * Python blocks of parade-style queries * List of data? * Q&A * Other (Javascript, CLI) (20m) * Something simple there, but then type in a new `z` or whatever - https://jsfiddle.net/will_j_moore/tvjsxgw4/6/ - No labels (yet) * List of data? * Q&A * Java: BigDataViewer/MoBIE (Tischi) (20m) * install update site (at most) * TODO: Link to installation instructions. * show the minimal ome.zarr example * binary gene expression map * open in BDV, using a Groovy script * show the chunked loading with setLogLevel( LogLevel.DEBUG ) option TODO * platynereis em raw with labels * (show how the data set looks internally, including the .zattr files) * point out that there is an internal label image (segmented cells) * open on top of the binary image in bdv, using a Groovy script * the resolutions (em and binary) are different, but shown properly in the same coordinate system => multi-scale * first show the grayscale label mask * gray scale is not a good coloring * there is an issue with interpolation for label masks * neighboring colors can clash * show the improved label mask * glasbey * no interpolation * interpolation only changes for intensity data * color shuffling * show that there is caching * when going back to the same slice the high-res images appears fast * what's missing still: loading of annotations, but * possible in MoBIE (not yet ome.zarr based, but n5 and github) * show the annotation browsing movie from the twitter * TODO: make another movie for a bookmark: Fig 2B: Epithelial... * mention: MoBIE framework used for several other datasets * https://github.com/mobie * Q&A ## Break (15m) :::info _last break_ ::: ## Code practicals * Make your own data cloud accessible * using bfconvert to create two ome.zarr data sets * using mc to copy them into the cloud * using mc to inspect the cloud store (ls) * using mc to download from the cloud * lazy-loading the ome.zarr.s3 into different viewers * python/napari * javascript/? * Groovy ImageJ/BDV * https://github.com/joshmoore/ome-guide-zarr/tree/master/notebooks * Josh: spec here or in the pptx? * Will: mention omero-cli-zarr (Simon??) * Josh: be ready for csv, etc. * Questions - use 2 extra hours? - break outs? - languages? - etc. ### Session 1 :::info break ::: ### Session 2 ## Instructions (for new doc) - Send upfront (Tweet them) - Warning about Big Sur - notebooks - conda installation * If we want users to install - needs to be easy: * e.g. conda create -n ome-zarr -c conda-forge -c ome ome-zarr-py * https://docs.conda.io/en/latest/miniconda.html - conda environment file - Fiji installation - Fiji update site