Try   HackMD

OME-NGFF samples

Current status

The table at the bottom captures the current location of the known OME-NGFF samples (v0.1 -> v0.4) across different locations: UoD GPFS (data_repo), EMBL-EBI idr S3 bucket, EMBL i2k-2020 S3 bucket.

Objectives

The goal is to design a unified layout to consolidate the OME-NGFF samples and meet the following requirements:

  • samples should come from various sources and span multiple versions of the specification
  • their organisation should be logical and easily extendable
  • the naming itself does not need to be self-explanatory: this should be the responsibility of either metadata and/or an external registry
  • the structure should be compatible both with object storage and a standard file system
  • being able to archive a dataset/a group of datasets e.g. as a Zenodo record would be a plus

A lot of these principles where applied for the OME-TIFF representative samples (https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/) although the naming structure of these samples was a bit ad-hoc and primarily controlled by the Dundee team.

Proposal

The simplest layout for mapping all OME-NGFF samples would be <version>/<folder>/<name>.zarr:

  • version is the version of the OME-NGFF specification i.e. 0.1, 0.2, 0.3
  • folder (for the lack of the better term) is a container of one-to-many OME-NGFF datasets grouped by provenance
  • name is the name of individual OME-NGFF datasets e.g. identifiers for IDR or more explicit names

With such layout, all samples could be stored:

  • under a curated/ome-ngff folder of the OME data repository at UoD (for integration in the CI testing)
  • under a new ome-ngff bucket on the EMBL-EBI Embassy object store
  • optionally replicated on third-party object storage/file systems

Depending on the decision on the <folder> nomenclature, the datasets under each <folder> could be zipped as <name>.zip and archived as a Zenodo record.

Questions

The primary question with the layout above is the name of the <folder>. Discussing the possible options and their pros/cons

  • idr/embl (see data_repo in the table)
    • Pros:
      • simplest way to map the existing set of samples
    • Cons:
      • IDR samples already introduced another layer of hierarchy to differentiate images/plates
      • Feels like binding samples to an entity which might not be representative e.g. how would converted OME-TIFF samples be stored?
  • accession e.g. idr0016
    • Pros:
      • create some form of provenance
      • allow a finer grouping for multiple datasets from the same source
    • Cons:
      • can create confusion as it is a subset of a study
      • does not easily apply to datasets without accession e.g. ome-ngff-prototypes, OME-TIFF
  • DOI e.g. generated by Zenodo
    • Pros: would force an archival workflow & create DOIs
    • Cons:
      • DOIs create another artificial layer of hierarchy e.g. 10.5281/zenodo*/<name>.zarr
      • how to deal with >50G datasets?
  • Some more ad-hoc form of folder e.g. Hackathon_20210222
    • Pros:
      • probably the easiest way to append new samples to the set. Only constraint is the unicity of the
    • Cons:
      • defining some rules would be advantageous to allow others to contribute
      • is this allowing any form of nested layout?
Version Source data_repo (current) EMBL/EBI bucket (current)
0.1 idr0106 unsupported/ome-ngff/0.1/idr/12689244.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/12689244.zarr
0.1 idr0002 unsupported/ome-ngff/0.1/idr/179706.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/179706.zarr
0.1 idr0021 unsupported/ome-ngff/0.1/idr/1884807.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/1884807.zarr
0.1 idr0023 unsupported/ome-ngff/0.1/idr/1885619.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/1885619.zarr
0.1 idr0044 https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/4007801.zarr
0.1 idr0053 unsupported/ome-ngff/0.1/idr/4495402.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/4495402.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001237.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001237.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001238.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001238.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001239.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001239.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001240.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001241.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001241.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001242.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001242.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001243.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001243.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001244.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001244.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001245.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001245.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001246.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001246.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001247.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001247.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001248.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001248.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001249.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001249.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001250.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001250.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001251.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001251.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001252.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001252.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001253.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001253.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001254.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001254.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001255.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001255.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001256.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001256.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001257.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001257.zarr
0.1 idr0062 unsupported/ome-ngff/0.1/idr/6001258.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001258.zarr
0.1 idr0073 unsupported/ome-ngff/0.1/idr/9798462.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9798462.zarr
0.1 idr0083 unsupported/ome-ngff/0.1/idr/9822151.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9822151.zarr
0.1 idr0083 unsupported/ome-ngff/0.1/idr/9822152.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9822152.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836831.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836831.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836832.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836832.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836833.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836833.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836834.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836834.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836835.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836835.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836836.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836836.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836837.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836837.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836838.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836838.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836839.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836839.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836840.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836840.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836841.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836841.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836842.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836842.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836843.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836843.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836844.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836844.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836845.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836845.zarr
0.1 idr0077 unsupported/ome-ngff/0.1/idr/9836846.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836846.zarr
0.1 idr0079 unsupported/ome-ngff/0.1/idr/9836950.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/9836950.zarr
0.1 idr0004 unsupported/ome-ngff/0.1/idr/plates/1751.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/1751.zarr
0.1 idr0001 unsupported/ome-ngff/0.1/idr/plates/2551.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/2551.zarr
0.1 idr0002 unsupported/ome-ngff/0.1/idr/plates/422.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/422.zarr
0.1 idr0033 unsupported/ome-ngff/0.1/idr/plates/5966.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/5966.zarr
0.1 idr0094 unsupported/ome-ngff/0.1/idr/plates/7825.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/7825.zarr
0.2 idr0062 unsupported/ome-ngff/0.2/idr/6001240.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.2/6001240.zarr
0.2 idr0062 unsupported/ome-ngff/0.2/idr/6001247.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.2/6001247.zarr
0.3 idr0077 unsupported/ome-ngff/0.3/idr/9836842.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/9836842.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0040A/3491626.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0040A/3491626.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0051A/4007817.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0051A/4007817.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0052A/5514375.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0052A/5514375.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0075A/9528933.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0075A/9528933.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0079A/9836998.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0079A/9836998.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0094A/7751.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0094A/7751.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0095B/11511419.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511419.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0095B/11511420.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511420.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0095B/11511421.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511421.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0095B/11511422.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511422.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0095B/11511423.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511423.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0095B/11511424.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0095B/11511424.zarr
0.3 ome-ngff-prototypes unsupported/ome-ngff/0.3/idr/idr0109A/12922361.zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0109A/12922361.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/cyx.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/cyx.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/multi-image.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/embl/multi-image.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/tcyx.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/embl/tcyx.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/tczyx.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/embl/tczyx.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/tyx.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/embl/tyx.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/yx.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/embl/yx.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/embl/zyx.ome.zarr https://s3.embl.de/ngff-example-data/v0.4/embl/zyx.ome.zarr
0.4 ome-ngff-prototypes unsupported/ome-ngff/0.4/idr/9512.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/cyx.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/embl/multi-image.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/embl/tcyx.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/embl/tczyx.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/embl/tyx.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/embl/yx.ome.zarr
0.3 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.3/embl/zyx.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/cyx.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/embl/multi-image.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/embl/tcyx.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/embl/tczyx.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/embl/tyx.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/embl/yx.ome.zarr
0.2 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.2/embl/zyx.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/cyx.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/embl/multi-image.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/embl/tcyx.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/embl/tczyx.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/embl/tyx.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/embl/yx.ome.zarr
0.1 ome-ngff-prototypes https://s3.embl.de/ngff-example-data/v0.1/embl/zyx.ome.zarr