---
# System prepended metadata

title: '2022-02-01'
tags: [meeting, biohub, storage, data]

---

2022-02-01
===

###### tags: `meeting` `storage` `biohub` `data`

:::info
- **Date:** 2022-02-01
- **Agenda**
    - data workflow, requirements
- **Attendees**
    - Nathan Clack
    - Cameron Foltz
    - Ivan Ivanov
- **Follow up**
    - [ ] keep in touch!
    - [ ] Nathan should visit
:::

# Data requirements with Ivan and Cameron

## Bandwidth and latency

- 2x10Gbps from cameras
- to PC 8xraid nvme, acquisition pc
- to on prem (40 Gbps)
    - conversion to chunked (zarr)

- during setup of microscope
    - recon every shot
        - single image plane (4ch) 
        - full volume: 30 to 60s.
            - 350 images, 5 Mpx at 16 bit, 4-5 polarizations
- during production
    - recon every 10 minutes
    - 24 hr development every 10 minute snapshots
    - 16 positions w fluorscence

## Data lifecycle

- micromanager pycromanager
    - determines Tiff based workflow
- convert to zarr, w compression ideally
- delete original tiffs
- multiple rounds of analysis
    - artifacts live as zarr float32
- metadata
    - important to have parameters from original acquisition
        - umanager metadata is in tiff
        - wavelengths etc is in a separate json
            - computer's not in the loop for the filter placement
            - ends up being input as a parameter into the 
              reconstruction snapshot tool...so that goes into the 
              json.
        - try to use OME-TIFF where possible
            - other techniques have different metadata standards
            - there's a bigger **data hub project** that's forming 
              at the biohub around unifying across domains and 
              sharing.
    - reconstruction parameters
        - custom yaml
            - plugin related
        - zarr attributes

## Choices around formats

Why zarr
    - python
    - chunked
    - OME standard
    - lazy loading/distributed compute support

Chunking 
    - (sharding here, not even concerned with partitions)
    - every image, or 8 z slices
    - determined by unit of compute
    - balance between file size/read speed
        - large files easy to transfer
        - but hard to view an individual thing

Compression
    - reduces storage cost, esp important 
    - ratios: 1.7X 
    - numcodex, zstd
    - lossy no good
    
Standards
    - accessibility
        - open source
    - standards
        - standard dimension order (tzcyx)
    - compatible tooling
        - nice to have a variety, reduce dependencies, better community
        