# OME-NGFF samples

## Current status

The table at the bottom captures the current location of the known OME-NGFF samples (v0.1 -> v0.4) across different locations: UoD GPFS (data_repo), EMBL-EBI idr S3 bucket, EMBL i2k-2020 S3 bucket.

## Objectives

The goal is to design a unified layout to consolidate the OME-NGFF samples and meet the following requirements:

- samples should come from various sources and span multiple versions of the specification
- their organisation should be logical and easily extendable
- the naming itself does not need to be self-explanatory: this should be the responsibility of either metadata and/or an external registry
- the structure should be compatible both with object storage and a standard file system
- being able to archive a dataset/a group of datasets e.g. as a Zenodo record would be a plus

A lot of these principles where applied for the OME-TIFF representative samples (https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/) although the naming structure of these samples was a bit ad-hoc and primarily controlled by the Dundee team.

## Proposal

The simplest layout for mapping all OME-NGFF samples would be `<version>/<folder>/<name>.zarr`:

- `version` is the version of the OME-NGFF specification i.e. 0.1, 0.2, 0.3
- `folder` (for the lack of the better term) is a container of one-to-many OME-NGFF datasets grouped by provenance
- `name` is the name of individual OME-NGFF datasets e.g. identifiers for IDR or more explicit names

With such layout, all samples could be stored:

- under a `curated/ome-ngff` folder of the OME data repository at UoD (for integration in the CI testing)
- under a new `ome-ngff` bucket on the [EMBL-EBI Embassy object store](https://uk1s3.embassy.ebi.ac.uk)
- optionally replicated on third-party object storage/file systems

Depending on the decision on the `<folder>` nomenclature, the datasets under each `<folder>` could be zipped as `<name>.zip` and archived as a Zenodo record.

## Questions

The primary question with the layout above is the name of the `<folder>`. Discussing the possible options and their pros/cons

* `idr/embl` (see `data_repo` in the table)
* Pros:
* simplest way to map the existing set of samples
* Cons:
* IDR samples already introduced another layer of hierarchy to differentiate images/plates
* Feels like binding samples to an entity which might not be representative e.g. how would converted OME-TIFF samples be stored?
* accession e.g. `idr0016`
* Pros:
* create some form of provenance
* allow a finer grouping for multiple datasets from the same source
* Cons:
* can create confusion as it is a subset of a study
* does not easily apply to datasets without accession e.g. ome-ngff-prototypes, OME-TIFF
* DOI e.g. generated by Zenodo
* Pros: would force an archival workflow & create DOIs
* Cons:
* DOIs create another artificial layer of hierarchy e.g. 10.5281/zenodo*/<name>.zarr
* how to deal with >50G datasets?
* Some more ad-hoc form of folder e.g. `Hackathon_20210222`
* Pros:
* probably the easiest way to append new samples to the set. Only constraint is the unicity of the
* Cons:
* defining some rules would be advantageous to allow others to contribute
* is this allowing any form of nested layout?

| Version | Source | data_repo (current) | EMBL/EBI bucket (current) |
|---------|---------------------|-----------------------------------------------------|----------------------------------------------------------------------|
| 0.1 | idr0106 | unsupported/ome-ngff/0.1/idr/12689244.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/12689244.zarr |
| 0.1 | idr0002 | unsupported/ome-ngff/0.1/idr/179706.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr |
| 0.1 | idr0021 | unsupported/ome-ngff/0.1/idr/1884807.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/1884807.zarr |
| 0.1 | idr0023 | unsupported/ome-ngff/0.1/idr/1885619.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/1885619.zarr |
| 0.1 | idr0044 | | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/4007801.zarr |
| 0.1 | idr0053 | unsupported/ome-ngff/0.1/idr/4495402.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/4495402.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001237.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001237.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001238.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001238.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001239.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001239.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001240.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001241.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001241.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001242.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001242.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001243.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001243.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001244.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001244.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001245.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001245.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001246.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001246.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001247.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001247.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001248.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001248.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001249.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001249.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001250.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001250.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001251.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001251.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001252.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001252.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001253.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001253.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001254.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001254.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001255.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001255.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001256.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001256.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001257.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001257.zarr |
| 0.1 | idr0062 | unsupported/ome-ngff/0.1/idr/6001258.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001258.zarr |
| 0.1 | idr0073 | unsupported/ome-ngff/0.1/idr/9798462.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9798462.zarr |
| 0.1 | idr0083 | unsupported/ome-ngff/0.1/idr/9822151.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9822151.zarr |
| 0.1 | idr0083 | unsupported/ome-ngff/0.1/idr/9822152.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9822152.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836831.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836831.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836832.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836832.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836833.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836833.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836834.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836834.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836835.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836835.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836836.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836836.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836837.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836837.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836838.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836838.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836839.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836839.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836840.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836840.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836841.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836841.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836842.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836842.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836843.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836843.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836844.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836844.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836845.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836845.zarr |
| 0.1 | idr0077 | unsupported/ome-ngff/0.1/idr/9836846.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836846.zarr |
| 0.1 | idr0079 | unsupported/ome-ngff/0.1/idr/9836950.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836950.zarr |
| 0.1 | idr0004 | unsupported/ome-ngff/0.1/idr/plates/1751.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/1751.zarr |
| 0.1 | idr0001 | unsupported/ome-ngff/0.1/idr/plates/2551.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/2551.zarr |
| 0.1 | idr0002 | unsupported/ome-ngff/0.1/idr/plates/422.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/422.zarr |
| 0.1 | idr0033 | unsupported/ome-ngff/0.1/idr/plates/5966.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/5966.zarr |
| 0.1 | idr0094 | unsupported/ome-ngff/0.1/idr/plates/7825.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr |
| 0.2 | idr0062 | unsupported/ome-ngff/0.2/idr/6001240.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.2/6001240.zarr |
| 0.2 | idr0062 | unsupported/ome-ngff/0.2/idr/6001247.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.2/6001247.zarr |
| 0.3 | idr0077 | unsupported/ome-ngff/0.3/idr/9836842.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/9836842.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0040A/3491626.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0040A/3491626.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0051A/4007817.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0051A/4007817.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0052A/5514375.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0052A/5514375.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0075A/9528933.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0075A/9528933.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0079A/9836998.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0079A/9836998.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0094A/7751.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0095B/11511419.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511419.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0095B/11511420.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511420.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0095B/11511421.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511421.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0095B/11511422.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511422.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0095B/11511423.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511423.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0095B/11511424.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511424.zarr |
| 0.3 | ome-ngff-prototypes | unsupported/ome-ngff/0.3/idr/idr0109A/12922361.zarr | https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0109A/12922361.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/cyx.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/cyx.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/multi-image.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/embl/multi-image.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/tcyx.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/embl/tcyx.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/tczyx.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/embl/tczyx.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/tyx.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/embl/tyx.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/yx.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/embl/yx.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/embl/zyx.ome.zarr | https://s3.embl.de/ngff-example-data/v0.4/embl/zyx.ome.zarr |
| 0.4 | ome-ngff-prototypes | unsupported/ome-ngff/0.4/idr/9512.ome.zarr | |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/cyx.ome.zarr |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/embl/multi-image.ome.zarr |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/embl/tcyx.ome.zarr |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/embl/tczyx.ome.zarr |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/embl/tyx.ome.zarr |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/embl/yx.ome.zarr |
| 0.3 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.3/embl/zyx.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/cyx.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/embl/multi-image.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/embl/tcyx.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/embl/tczyx.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/embl/tyx.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/embl/yx.ome.zarr |
| 0.2 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.2/embl/zyx.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/cyx.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/embl/multi-image.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/embl/tcyx.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/embl/tczyx.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/embl/tyx.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/embl/yx.ome.zarr |
| 0.1 | ome-ngff-prototypes | | https://s3.embl.de/ngff-example-data/v0.1/embl/zyx.ome.zarr |