---
tags: WIP, NGFF
---

Josh: would be good to know how much of this could be moved to either
- https://github.com/ome/omero-cli-zarr
- https://github.com/IDR/SubmissionWorkflow

Comments from Petr for integrating:
- 1. say whether or not the IMAGE_PATH can point to any old image or does it have to be an image inside the ManagedRepo of OMERO or otherwise “special” image
- 2. say what is the output of the workflow and give a hint about how to follow-up (…and now open locally in napari) as applicable
- (basically the need to explain the bigger framework, why are the omero steps there, and why and when can I omit/ignore them)

# NGFF workflow, summarised

A summarised version of
https://github.com/IDR/SubmissionWorkflow/blob/master/zarr.md (private)
that will be used to construct a workflow.

Input parameters:
- `IMAGE_ID`: OMERO Image ID
- `IMAGE_PATH`: Path to raw data on NFS
- `OUTPUT_DIR`: Output parent directory for Zarrs

Temporary files passed between steps:
- `SERIES_CSV`: CSV including image IDs
- `ROIS_CSV`: CSV including ROI IDs

Conda Environment:
- bioformats2raw
- omero-cli-zarr
- omero-py
- zarr

Current workflow:

## 1. Obtain OMERO all image IDs associated with `${IMAGE_ID}`, save to ${SERIES_CSV}:

omero hql --style=plain -q "select i.id, i.series from Image i where i.fileset.id = (select i.fileset.id from Image i where i.id = ${IMAGE_ID}) order by i.series asc" | cut -f2,3 -d, | tee ${SERIES_CSV}

## 2. Convert raw image data to Zarr

bioformats2raw --file_type=zarr ${SERIES_CSV} ${IMAGE_PATH} ${OUTPUT_DIR}

## 3. Add OMERO metadata

curl -o- http://idr.openmicroscopy.org/webclient/imgData/${IMAGE_ID}/ > ${OUTPUT_DIR}/${IMAGE_ID}.zarr/omero.json

(Presumably need to loop through all image IDs in `${SERIES_CSV}`)

## 4. Merge external OMERO metadata into the Zarr

`merge.py`: https://github.com/IDR/idr-zarr-tools/blob/master/merge.py

./merge.py {OUTPUT_DIR}/${IMAGE_ID}.zarr

(Should we delete `${OUTPUT_DIR}/${IMAGE_ID}.zarr/omero.json`?)

## 5. Get mask IDs

omero hql --style=plain "select distinct s.textValue, s.roi.id from Shape s where s.roi.image.id = ${IMAGE_ID}" --limit=-1 | tee ${ROIS_CSV}

(Presumably need to loop through all image IDs in `${SERIES_CSV}`)

## 6. Add masks

omero zarr masks Image:${IMAGE_ID} --mask-map=${ROIS_CSV}

----
Looks like steps 3-6 should be in a loop over the output of step 2. One question is whether to keep the loop in the same workflow, or to branch into a different workflow.

# omero-cli-zarr export using Docker

Based on https://github.com/ome/omero-cli-zarr/pull/29#issuecomment-713516616 and https://github.com/ome/omero-cli-zarr/pull/38#pullrequestreview-518083982

$ ssh -A ome-zarr-dev1.openmicroscopy.org

Or use

ssh idr-ngff-1

if you have:

Host idr-ngff-1
    ProxyCommand ssh idr-pilot.openmicroscopy.org -W zarr1-dev:%p

Can run from your home directory...

$ mkdir idr0033_zarr_export_docker
$ cd idr0033_zarr_export_docker
$ vi Dockerfile

NB: Paste in this docker file, editing the omero-cli-zarr branch you want to use:

FROM centos:7

RUN yum install -y git python3 fontconfig

RUN useradd -ms /bin/bash converter
USER converter

RUN python3 -m venv /tmp/venv && /tmp/venv/bin/pip install -U pip wheel && /tmp/venv/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.2.1/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl
RUN /tmp/venv/bin/pip install git+https://github.com/ome/omero-cli-zarr.git@refs/pull/38/merge
# or RUN /tmp/venv/bin/pip install git+git://github.com/joshmoore/omero-cli-zarr.git@nested#egg=omero-cli-zarr

COPY export.omero .
ENV OMERO_USERDIR /tmp/omero
ENTRYPOINT ["/tmp/venv/bin/omero", "load", "export.omero"]

Create a companion export.omero file

vi export.omero

And add export commands to connect to idr-testing...

login -C [e-mail] -w public
zarr --output /data export Plate:5966

Then build with

$ sudo docker build --tag idr0033_plate5966 .

We want files to end up at e.g. (named after current or last-merged PR)
https://minio-dev.openmicroscopy.org/idr/idr0033-rohban-pathways/41744_illum_corrected/pr35/5966.zarr
Location is on /uod/idr/objectstore/minio/idr/ but this is now
accessible from ome-zarr-dev1.openmicroscopy.org.

Make sure target location exists...

$ cd /uod/idr/objectstore/minio/idr/idr0033-rohban-pathways/41744_illum_corrected
$ mkdir pr_35

Want to run with your own user ID, so you can modify output files

$ id wmoore  # shows user ID - e.g. 5098

$ screen -xRR

NB: had some permissions problems (fixed) but the session created when the Dockerfile was built above
then expired, so needed to run the container interactively and re-login to idr:

[wmoore@idr1-slot2 idr0002]$ sudo docker run -it --rm -u 5098 -e OMERO_USERDIR=/tmp/omero -v /uod/idr/objectstore/minio/idr/idr0033-rohban-pathways/41744_illum_corrected/pr_35:/data idr0033_plate5966
WARNING: Could not load OpenGL library.
WARNING:vispy:Could not load OpenGL library.
Server: [localhost:4064]idr.openmicroscopy.org
Username: [root]public
Password:
Created session for [e-mail]:4064. Idle timeout: 10 min. Current group: Public
Exporting to /data/5966.zarr
0.06% done, ETA: 05:20:46

Should now be able to see files appearing on idr0-slot3
e.g. to see how many Wells have been exported:

$ cd /uod/idr/objectstore/minio/idr/idr0033-rohban-pathways/41744_illum_corrected/pr_35/5966.zarr/0
$ find . -maxdepth 2 -type d | grep [0-9] | wc
     64      64     430

# export with Z-downsampling

Export and then downsample, working with Temp state of
PR: https://github.com/ome/ome-zarr-py/pull/71#issuecomment-759404371

ssh -A ome-zarr-dev1.openmicroscopy.org
mkdir idr0077_zarr_z-downsample
cd idr0077_zarr_z-downsample

Dockerfile:

FROM centos:7

RUN yum install -y git python3 fontconfig

RUN useradd -ms /bin/bash converter
USER converter

RUN python3 -m venv /tmp/venv && /tmp/venv/bin/pip install -U pip wheel && /tmp/venv/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.2.1/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl
RUN /tmp/venv/bin/pip install git+https://github.com/will-moore/omero-cli-zarr.git@use_ome_zarr_to_build_pyramid
RUN /tmp/venv/bin/pip install git+https://github.com/will-moore/ome-zarr-py.git@downsample_Z

COPY export_zarr .
ENV OMERO_USERDIR /tmp/omero
ENTRYPOINT ["bash", "export_zarr"]

Create an `export_zarr` bash script:

/tmp/venv/bin/omero login -C [e-mail] -w public
/tmp/venv/bin/omero zarr --output /data export Image:9836831
/tmp/venv/bin/ome_zarr scale /data/9836831.zarr /data/9836831_zscale.zarr --downsample_z

Build and run

$ sudo docker build --tag idr0077_9836831_zscale_02 .

$ sudo mkdir -p /uod/idr/objectstore/minio/idr/idr0077-valuchova-flowerlightsheet/zscale_02
$ sudo chown wmoore /uod/idr/objectstore/minio/idr/idr0077-valuchova-flowerlightsheet/zscale_02

$ id wmoore     # 5098

$ screen -xRR

$ sudo docker run -it --rm -u 5098 -e OMERO_USERDIR=/tmp/omero -v /uod/idr/objectstore/minio/idr/idr0077-valuchova-flowerlightsheet/zscale_02:/data idr0077_9836831_zscale_02

# export via conda environment

Use:

ssh -A ome-zarr-dev1.openmicroscopy.org

Or with this config:

# /.ssh/config

Host pilot-zarr*
    ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h.openmicroscopy.org:%p   ForwardAgent yes

Use:

ssh pilot-zarr1-dev

First install Conda. Using first link in list at https://docs.conda.io/en/latest/miniconda.html#linux-installers

$ wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.10.3-Linux-x86_64.sh

$ bash Miniconda3-py39_4.10.3-Linux-x86_64.sh
...
Miniconda3 will now be installed into this location:
/lifesci/groups/jrs/wmoore/miniconda3
...

After re-opening a Terminal:

$ which conda
~/miniconda3/bin/conda

Using `environment.yml` at https://github.com/ome/NGFF-ELMI-2021-Workshop/blob/main/binder/environment.yml ...

wget https://raw.githubusercontent.com/ome/NGFF-ELMI-2021-Workshop/main/binder/environment.yml

$ conda create -f environment.yml

CondaValueError: The target prefix is the base prefix. Aborting.

$ conda env create -f environment.yml

CondaValueError: prefix already exists: /lifesci/groups/jrs/wmoore/miniconda3

$ conda env create --prefix ./envs -f environment.yml

NB: could use -n myenv instead of --prefix?

...
ERROR conda.core.link:_execute(699): An error occurred while installing package 'conda-forge::pyopengl-3.1.5-py_0'.
Rolling back transaction: done

LinkError: post-link script failed for package conda-forge::pyopengl-3.1.5-py_0
location of failed script: /lifesci/groups/jrs/wmoore/envs/bin/.pyopengl-post-link.sh
==> script messages <==
<None>
==> script output <==
stdout: Warning: Missing OpenGL driver, install with yum install mesa-libGL-devel or equivalent
...

$ sudo yum install mesa-libGL-devel

# try again...

$ conda env create -f environment.yml -n omero_zarr_export

Seemed to work:

$ conda env list
# conda environments:
#
                         /lifesci/groups/jrs/wmoore/envs
base                  *  /lifesci/groups/jrs/wmoore/miniconda3
omero_zarr_export        /lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export

Activate and install version of omero-cli-zarr we want to use

$ conda activate omero_zarr_export
$ pip uninstall omero-cli-zarr
$ pip install git+https://github.com/will-moore/omero-cli-zarr.git@export_v0.3_ome_ngff

$ cd /uod/idr/objectstore/minio/idr/v0.3/idr0062-blin-nuclearsegmentation
$ omero zarr export Image:6001240

Available at:
- 2D (YX): https://minio-dev.openmicroscopy.org/idr/v0.3/idr0094-ellinger-sarscov2/10503791.zarr
- 3D (CXY): https://minio-dev.openmicroscopy.org/idr/v0.3/idr0077-valuchova-flowerlightsheet/9836842.zarr
- 4D (TCYX) https://minio-dev.openmicroscopy.org/idr/v0.3/idr0002-heriche-condensation/179758.zarr
- 4D (CZYX): https://minio-dev.openmicroscopy.org/idr/v0.3/idr0062-blin-nuclearsegmentation/6001240.zarr
- 4D (TZYX): idr0099-jain-beetlelightsheet/12557113.zarr (not done)
- 4D (TCYX): https://minio-dev.openmicroscopy.org/idr/v0.3/idr0077-valuchova-flowerlightsheet/9836849.zarr
- 5D (TCZYX): https://minio-dev.openmicroscopy.org/idr/v0.3/idr0077-valuchova-flowerlightsheet/9836847.zarr (not done)

Ended with:

OSError: [Errno 28] No space left on device: '/uod/idr/objectstore/minio/idr/v0.3/idr0099-jain-beetlelightsheet/12557113.zarr/0/0/54/0/0'

Seb: looks like we have used all the Inodes on the objectstore GPFS fileset at UoD
so options are 1- clean up data on GPFS, 2- increase the number of inodes, 3- use EBI S3 rather than our minio

Others:
- https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-24/plates/idr0004
- https://minio-dev.openmicroscopy.org/idr/v0.4/2022-02-03/idr0062/6001240.zarr
- https://minio-dev.openmicroscopy.org/idr/v0.4/2022-02-03/idr0101/13457227.zarr (13457537.zarr 13457539.zarr)
- From bioformats2raw, includes OME/OME.xml for MIF https://minio-dev.openmicroscopy.org/idr/bf2raw/mdb/martin/sample_files.zarr/.zattrs is `{"bioformats2raw.layout" : 3}`

# export vizarr & deploy on s3
Checkout https://github.com/hms-dbmi/vizarr/pull/43

$ cd vizarr
$ npm run export # to /out directory

# export via bioformats2raw

Using pilot-zarr1-dev (ngff data directory: `/data`) or pilot-zarr2-dev (`/data/ngff`).

Install bioformats2raw via conda:

conda install -c ome bioformats2raw

This is actually just for getting the dependencies installed.
Get the actual bioformats2raw from this PR and just unzip it into your home directory:
https://github.com/IDR/bioformats2raw/pull/1

Create a directory for the idr project and memo files (if it's not already there),
and change into the idr directory.
For example for idr0051:

mkdir idr0051 
mkdir memo
cd idr0051

Find out where the pattern, screen or companion files are. For example:
`/nfs/bioimage/drop/idr0051-fulton-tailbudlightsheet/patterns/`

Then run the conversion (using the bioformat2raw from the PR!):

for i in `ls /nfs/bioimage/drop/idr0051-fulton-tailbudlightsheet/patterns/`; do echo $i; /home/<USER>/bioformats2raw-0.7.0-SNAPSHOT/bin/bioformats2raw --memo-directory ../memo /nfs/bioimage/drop/idr0051-fulton-tailbudlightsheet/patterns/$i ${i%.*}.ome.zarr; done

(`$i` is the pattern file, `${i%.*}.ome.zarr` strips the .pattern file extension and adds .ome.zarr; this should work for pattern, screen and also companion file extensions)

# upload to uk1 s3

Install minio and configure...

ssh -A ome-zarr-dev1.openmicroscopy.org

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc

(base) [wmoore@ome-zarr-dev1 ~]$ ./mc config host add uk1s3 https://uk1s3.embassy.ebi.ac.uk
Enter Access Key: X8GE11ZKP71A8529XFAE
Enter Secret Key: 
Added `uk1s3` successfully.

$ ./mc ls uk1s3/idr/zarr
[2022-10-13 11:07:56 BST]     0B test-data/
[2022-10-13 11:07:56 BST]     0B v0.1/
[2022-10-13 11:07:56 BST]     0B v0.2/
[2022-10-13 11:07:56 BST]     0B v0.3/
[2022-10-13 11:07:56 BST]     0B v0.4/


$ cd /uod/idr/objectstore/minio/idr/v0.4/2022-02-03/
$ /lifesci/groups/jrs/wmoore/mc cp -r idr0101/ uk1s3/idr/zarr/v0.4/idr0101A/
...idr0101/13457539.zarr/labels/0/2/0/0/0/0/0: 13.21 GiB / 13.21 GiB ━━━━━━━

Also installed at on idrtesting-pilot to allow direct upload without conversion...

ssh -A idr-pilot.openmicroscopy.org
ssh -A idrtesting-omeroreadwrite

# install and config as above...

$ cd /uod/idr/filesets/idr0138-lohoff-seqfish/20210917-Globus/ngff/TimEmbryos-102219/HybCycle_0/
$ /home/wmoore/mc cp -r MMStack_Pos0.ome.zarr/ uk1s3/idr/zarr/v0.4/idr0138A/TimEmbryos-102219/HybCycle_0/MMStack_Pos0.ome.zarr/
.../MMStack_Pos0.ome.zarr/labels/nuclei/4/0/0/0: 186.41 MiB / 186.41 MiB ━━━━━━━

# Copy files from IDR

rsync -av  -e "ssh -A idr-pilot.openmicroscopy.org ssh" "idrtesting-omeroreadwrite:/uod/idr/filesets/idr0070-kerwin-hdbr/20200414-Batch3-ftp/HDBR_BMP4_ISH/" . -P

# Test import to OMERO

$ ssh idr2-slot3.openmicroscopy.org

$ cd /uod/idr-scratch/will-test/

$ aws s3 sync --no-sign-request --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr/zarr/v0.4/idr0050A/4995115.zarr 4995115.zarr

$ conda activate omeropy

$ echo $OMERODIR
/lifesci/groups/jrs/wmoore/OMERO.server-5.6.5-ice36-b233

$ omero import --depth=100 4995115.zarr

# Update Jan 2023

See https://forum.image.sc/t/converting-other-idr-images-into-public-zarr/44025/12
Want to convert /uod/idr/filesets/idr0048-abdeladim-chroms/20181217-ftp/Astrop65_BDV/ (astroP65.h5 (73GB) and astroP65.xml)

Seb: "`pilot-zarr1-dev` or `pilot-zarr2-dev`
that has the advantage of being on EBI Embassy (so easier to upload to S3)"

ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' pilot-zarr1-dev

Installed conda and bioformats2raw as above in `/home/wmoore/miniconda`, and installed & configured `mc` client as above.

rsync -av  -e "ssh -A idr-pilot.openmicroscopy.org ssh" "idrtesting-omeroreadwrite:/uod/idr/filesets/idr0048-abdeladim-chroms/20181217-ftp/Astrop65_BDV/" -P .

$ bioformats2raw --version
Version = 0.5.0
Bio-Formats version = 6.10.1
NGFF specification version = 0.4

$ ls -lh
total 73G
-rw-rw----. 1 wmoore idrnfs  73G Dec 17  2018 astroP65.h5
-rw-rw----. 1 wmoore idrnfs 2.3K Dec 17  2018 astroP65.xml


$ bioformats2raw astroP65.xml 9846151.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp3473542386768814439/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.


$ /home/wmoore/mc cp -r 9846151.zarr/ uk1s3/idr/zarr/v0.4/idr0048A/9846151.zarr/

**conda install**

conda create -n omero_zarr_export -c ome python=3.9 zeroc-ice36-python omero-py

conda activate omero_zarr_export

# custom versions of ome-zarr-py and omero-cli-zarr
pip install git+https://github.com/ome/ome-zarr-py.git@refs/pull/244/head

pip install git+https://github.com/ome/omero-cli-zarr.git@refs/pull/134/head

# March 2023

Rsync local data to minio:

rsync -rvP --progress 13425213.zarr ome-zarr-dev1.openmicroscopy.org:/uod/idr/objectstore/minio/idr/idr0113-bottes-opcclones/

# IDR NGFF conversion - make buckets - April 2023

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0010

Seb: set the policy on the bucket as described in https://github.com/IDR/deployment/blob/master/docs/object-store.md#policy ? this will be mandatory for accessing it from the pilots (without keys)

Will: also need to configure CORS as described there too.

Update - April 2025

Minio hosted at `ome-dckr-ap1` which is at...

ssh 134.36.7.234
ls -alh /uod/idr/objectstore/minio/idr/Testing/sample_files.zarr/

NB: Dom had to move the data there as when I put it at
`/uod/idr/objectstore/minio/idr/ome-ngff-tools/sample_files.zarr/`
it wasn't accessible via minio!

NB: tried to create a new top-level bucket (sibling to idr) but couldn't fix CORS...

aws s3api put-bucket-cors --endpoint-url https://minio-dev.openmicroscopy.org --bucket ome-ngff-tools --cors-configuration file://cors.json

... as I don't know aws credentials!

Update: May 2025:
`pilot-zarr1`, `pilot-zarr2` & `pilot-zarr3` will be replaced by `pilot-zarr`: see https://github.com/openmicroscopy/management_tools/pull/1798/files

# August 2025

Want to install `mc` locally to upload to https://uk1s3.embassy.ebi.ac.uk...

Install mc with:

$ conda create -n mc python=3.12
$ conda activate mc
$ conda install "go-mc==2020.11.17" -c joshmoore

But then...

$ mc config host add uk1s3 https://uk1s3.embassy.ebi.ac.uk
mc: Configuration written to `/Users/wmoore/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/Users/wmoore/.mc/share`.
mc: Initialized share uploads `/Users/wmoore/.mc/share/uploads.json` file.
mc: Initialized share downloads `/Users/wmoore/.mc/share/downloads.json` file.
Enter Access Key: X8GE11ZKP71A8529XFAE
Enter Secret Key: 
mc: <ERROR> Unable to initialize new alias from the provided credentials. Get https://uk1s3.embassy.ebi.ac.uk/probe-bucket-sign-deollovggy2v/?location=: x509: certificate signed by unknown authority.