---
# System prepended metadata

title: I2K NGFF Script
tags: [' NGFF', I2K]

---

---
tags: I2K, NGFF
---

# I2K NGFF Script

- See also:
  - [file format descriptions](https://docs.google.com/document/d/1pgvLaJAUpC3fIH_hs3K78aq8RmsVpM_T7aPbzduBBto/edit)

## Introduction (30m)

* Introduction
    * What is this workshop about
    * What is the structure
        * length, breaks, ..
    * Who are the presenters
        * Name
        * Current job
        * Role in and motivation for this project
        * Role in this workshop
    * Living document ()
        * Need to use it (try it out; critical need)
        * Agreement to migrate
    * Acknowledge: (slide?)
        * https://docs.google.com/presentation/d/1jMXbl1McM3DeLWfVSy-Ky51VseXGkoIWUHw6wa0Y3p4/edit?usp=sharing
        * Constantin Pape (EMBL)
        * David Gault
        * Nicolas Chiaruttini (EPFL)
        * Tobias Pietzsch
        * Stephan Saalfeld
        * Juan Napari

* Round table? Everyone 2 sentences? (10m)
    * Name
    * Current job?
    * Why did I join this workshop?
    * Josh: this may no longer be possible if we hit 35.
        * Tischi: Some ideas
            * Make breakout rooms with pairs of 2 introduction themselves to each other for 2 + 2 minutes
                * Does not help us but creates some social contact for the participants, which is important
            * Let them introduce themselves in a hackMD
        * Carpentries in breakout
        * 3 rooms of 13?

* Challenges of browsing and sharing large 3D image data set (Talk) (Tischi) (10m)
    * https://docs.google.com/presentation/d/1L0Pxaou7ggsV8gpQAPga_JxvMvSN9I3-L1MPG18H6iU/edit?usp=sharing
    - Biological motivation
        - Platynereis atlas dataset
            - https://www.embl.org/news/science/finding-your-way-around-platynereis-dumerilii/
            - https://www.biorxiv.org/content/10.1101/2020.02.26.961037v1
            - TB sized
            - multi-scale
            - 200+ images 
    - Limitations of current standards (ome.tif and ImageJ viewer)
        - one cannot freely browse in 3D
            - ImageJ Viewer does not support it
            - Tiff does not support it either
        - ImageJ viewer cannot overlay data of different resolutions
            - one would need to resave the fluorescence data (500 nm)^3 in the EM resolution (10 nm)^3, which is not good if the EM data is several TB in size and you have 200+ fluorescence data sets
    - Requirements for a new standard (TODO: Tischi: I may not talk about it?!)
        - Chunking (factilitates lazy loading)
        - Multi-resolution (facilitates fast visualisation)
        - Multi-scale (saving on storage space and data transfer bandwidth)
            - more a viewer than a file format issue
        - Cloud compatible (for data sharing)
            - IT departments generally don't allow opening up a file-server to the public
    - Q&A
     
* Sharing large images (Talk) (Josh) (10m)
  - "My data" versus others data (other people look)
    - can't access file formats
        - opening file server to public may be a security issue
        - you don't really want people downloading
    - OMERO but must import
    - ... cloud (explain HDF5)
  - N5, zarr (NGFF for the cloud)
    - ngff.openmicroscopy.org/0.1
  - Current efforts and the path towards a community standard
    - Metadata? (so others can use it)
    - Discussions on forum.image.sc
    - Scope: raw images, segmentation images, segmentation features
    - Current status 
  - Q&A
  
:::info
_First break and installation?_ (5 min)
:::

## Demonstrations (1h)

* Accessing cloud hosted image data

  * Python notebooks on data
      * explain zarr, s3, etc.
      * Will to put together on ome-zarr
      * https://github.com/ome/omero-guide-python/blob/master/notebooks/zarr-public-s3-multiscale.ipynb
      * Q&A
      
  * Python: napari (20m)
      * using remote desktop for napari?
      * Petr: could see about what is napari, explain how to open a new image
      * Python blocks of parade-style queries
      * List of data?
      * Q&A

  * Other (Javascript, CLI)  (20m)
      * Something simple there, but then type in a new `z` or whatever 
        - https://jsfiddle.net/will_j_moore/tvjsxgw4/6/ 
        - No labels (yet)
      * List of data?
      * Q&A
     
  * Java: BigDataViewer/MoBIE (Tischi) (20m)
      * install update site (at most)
      * TODO: Link to installation instructions.
      * show the minimal ome.zarr example
          * binary gene expression map
              * open in BDV, using a Groovy script
              * show the chunked loading with setLogLevel( LogLevel.DEBUG ) option   TODO
          * platynereis em raw with labels
              * (show how the data set looks internally, including the .zattr files)
              * point out that there is an internal label image (segmented cells)
              * open on top of the binary image in bdv, using a Groovy script
              * the resolutions (em and binary) are different, but shown properly in the same coordinate system => multi-scale
              * first show the grayscale label mask
                  * gray scale is not a good coloring
                  * there is an issue with interpolation for label masks
                  * neighboring colors can clash
              * show the improved label mask
                  * glasbey
                  * no interpolation
                      * interpolation only changes for intensity data
                  * color shuffling
              * show that there is caching
                  * when going back to the same slice the high-res images appears fast
      * what's missing still: loading of annotations, but
          * possible in MoBIE (not yet ome.zarr based, but n5 and github)
          * show the annotation browsing movie from the twitter
          * TODO: make another movie for a bookmark: Fig 2B: Epithelial...
          * mention: MoBIE framework used for several other datasets
              * https://github.com/mobie
      * Q&A
      

## Break (15m)

:::info
_last break_
:::

## Code practicals

* Make your own data cloud accessible
    * using bfconvert to create two ome.zarr data sets
    * using mc to copy them into the cloud
    * using mc to inspect the cloud store (ls)
    * using mc to download from the cloud
    * lazy-loading the ome.zarr.s3 into different viewers
        * python/napari
        * javascript/?
        * Groovy ImageJ/BDV
    * https://github.com/joshmoore/ome-guide-zarr/tree/master/notebooks
    * Josh: spec here or in the pptx?
    * Will: mention omero-cli-zarr (Simon??)
    * Josh: be ready for csv, etc.

* Questions
  - use 2 extra hours?
  - break outs?
  - languages?
  - etc.
  
### Session 1

:::info
break
:::

### Session 2


## Instructions (for new doc)

- Send upfront (Tweet them)
- Warning about Big Sur
- notebooks
- conda installation
  * If we want users to install - needs to be easy:
    * e.g. conda create -n ome-zarr -c conda-forge -c ome ome-zarr-py
    * https://docs.conda.io/en/latest/miniconda.html
- conda environment file
- Fiji installation
- Fiji update site