---
# System prepended metadata

title: zipping dicoms

---

# zipping dicoms

## links
- jira story [FLYW-16602][0]
- [`xfer.imports.models.Import`][1]
- [`fw_meta.imports.ImportRule`][2]
- [`connector.imports.scan_storage.run`][3]

[0]: https://flywheelio.atlassian.net/browse/FLYW-16602
[1]: https://gitlab.com/flywheel-io/tools/app/xfer/-/blob/main/xfer/imports/models.py#L189
[2]: https://gitlab.com/flywheel-io/tools/lib/fw-meta/-/blob/main/fw_meta/imports.py#L429
[3]: https://gitlab.com/flywheel-io/tools/app/connector/-/blob/main/connector/imports/scan_storage.py#L22

## (1d) extend fw-meta ImportRule
```python
ImportRule:
    ...
    # TODO figure out default extension table code loc
    # NOTE also auto-populates file.type (priority?)
    type: opt[str] = null  # enum(?), auto=from-extension
    # NOTE validator to set group_by:null -> dir IF type=dicom
    group_by: opt[str] = null
    # NOTE validator to set zip:null -> true IF type=dicom
    zip: opt[bool] = null  # pro: generic, symmetric w/ export
```

## (1d) bump cli, expose args, update docs
```bash
fw-beta import run -t/--type dicom ...
fw-beta import run --group-by dir ...
fw-beta import run --zip ...
```

## (1d) bump xfer, update docs
- add test for exercizing new import attributes
- ensure / adjust models to display in redoc nicely

## (3d) bump connector deps, implement scan/group_by
- scan to respect the matching rule's group_by if set
- eval the group_by template
- kick off process_item based on the group value
- store last group value
- skip subsequent items until the group value is the same
- (low effort mock test to be dumped via next task...)

## (3d) in connector, implement proc/zip
- process_item to support payloads where the path is a dir
  - (coming from a scan/group_by)
- process_item should list the storage and
- get all files under dir that match the rule
- if rule.zip is true
  - zip all files retrieved (TODO finalize top dir)
  - get correct upload metadata (name, size)
  - upload as a single file
- otherwise upload files one-by-one
  - get correct upload metadata (name, size) for each
  - (future consider slash handling...)

## status quo

current connector import scan/proc assumes:
- include/exclude can be fully evaluated @scan time
- 1 source file == 1 dest file == 1 process item

### approach 1: distinguish eager/lazy filters
- scan/filter - implement lazy/eager field distinction
  - fw-storage fields are always available (path, name, dir)
  - anything else should be considered "lazy"
  - lazy fields not yet available @scan time, only @proc
  - scan can match() if eager (ie. none of the fields are "lazy")
  - otherwise must use exclude_only (for future data fields)
  - ^ this already feels **crazy-complex**
- scan: if zip=true, put_item on dir changes (and at the end!)

### approach 2: grouping via config
- import **must** eval rules @scan time
- import **should** simply eval rules as usual **IF**
  - **all filters/group_by use base fields ONLY: path/name/dir**
  - could be a simple property on the filter
- import **should** get and parse the file header **IF**
  - **any filters/group_by use non-base (header) fields**
- import scan to allow grouping
  - mapped from base fields, eg. w/ a template:
    - `rule.group[_by]: "{dir}"`
    - `x/y/z/acq.dcm` => `x/y/z`
  - allows for starting proc on first match
    - passing item=group=dir
  - can skip 2nd+ match if grouping and group=prev-group
  - **where to specify grouping? as an import property?**
  - **what's the impact on `process_item()`?**
- if multi &&  rule.zip: zip, upload as one packed file
- if multi && !rule.zip: upload separately (consider slashes)

## misc TODOs to piggyback
- fw-storage future refact cont. (s3)
- wait for snapshot in xfer scheduler
- export progress / websocket rewrite
