---
# System prepended metadata

title: Pipeline Automation Planning

---

# Pipeline Automation Planning

## Goals

1. Define Automation Use Cases
2. Outline scenarios for trial cases
3. Define point-of-contact for automation integrations
4. Action plan for streaming vs batch processing needs

## Orchestration Use Cases
 * File Ingestion
     (contact: Karunakara Seelam, Bing She)
    1. web-upload (Deposit Mangaer)
    2. web-upload (low-latency sychronous) — summer program, resume
    3. large file batch upload — MAPS, flint registry — long-lived upload server for SFTP? (per Matt Nizol, consider https://aws.amazon.com/aws-transfer-family/)
    4. pulled data via external API (scheduled or streaming) – ingests ongoing streams of data from 3rd party services — *future use case*

 * Curation
    
    1. context-specific manual curation — SASS / SPSF scripts — create a shared library of common tasks
    2. data handoff — Hermes, Turnover — hand-coded bash pipelines

 * Metrics Aggregation / BI Piping
    
    1. internal utilization metrics cleanup into simple data warehouse
    2. BI data, possibly store in data lake, requires more complex prep

 * VDE Provisioning
 * DB Parity
 * Data Exports

## Streaming vs Batch

 * Upload for smaller files must be low-latency (< 1s overhead)
 * Box latency expectations by file size:
     * &lt; 10MB
     * 10-100MB
     * &gt; 100MB
 * Data Integrity / Verification toolset (no more lost files)
 * Very large files are causing failures due to component bottle-necks

The majority of use-cases allow for an asynchronous upload of files, while a smaller subset might expect a fast upload for specific scenarios (ReDMAS, resume upload — small files)

### Streaming Strageties
 * Parallel pipeline for streaming
 * Tool that is capable of streaming and batch
 * Event-triggered low-latency batch pipeline ("fake" streaming)
     * potentially could use orchestration tool to spin up long-lived containers based on demand, and process jobs via agent on the long-lived container
 * Traditional upload pipeline with additional user-facing reporting on ingestion progress


### Reporting Mechanisms

For standard file ingestion, front-end should asynchronously poll backend to update user-visible progress status