IIBMP2022 Sapporo Demo

Link

• Slide: https://docs.google.com/presentation/d/1M7EtFuSwC0p31pVHOxRzb2j854KZB0eCg1wMDCqDayI/edit?usp=sharing
• Repositories:
  • sapporo-wes/sapporo-service
  • sapporo-wes/sapporo-web
• Getting Started: https://github.com/sapporo-wes/sapporo/blob/main/docs/GettingStarted.md

Demo

Sapporo-service

WES API クライアントである Sapporo-service の起動

$ curl -O https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/docker-compose.yml
$ docker compose up -d
[+] Running 1/1
 ⠿ app Pulled                                                                                        0.9s
[+] Running 1/1
 ⠿ Container sapporo-service  Started                                                                1.0s
$ curl -s localhost:1122/service-info | jq .
{
  "auth_instructions_url": "https://github.com/sapporo-wes/sapporo-service",
  "contact_info_url": "https://github.com/sapporo-wes/sapporo-service",
  "default_workflow_engine_parameters": {
    "nextflow": [
      {
        "default_value": "",
        "name": "-dsl1",
        "type": "str"
      }
    ],
    "snakemake": [
      {
        "default_value": 1,
        "name": "--cores",
        "type": "int"
      },
      {
        "default_value": "",
        "name": "--use-conda",
        "type": "str"
      }
    ]
  },
  "supported_filesystem_protocols": [
    "http",
    "https",
    "file",
    "s3"
  ],
  "supported_wes_versions": [
    "sapporo-wes-1.0.1"
  ],
  "system_state_counts": {},
  "tags": {
    "get_runs": true,
    "news_content": "",
    "registered_only_mode": false,
    "sapporo-version": "1.4.4",
    "wes-name": "sapporo",
    "workflow_attachment": true
  },
  "workflow_engine_versions": {
    "cromwell": "80",
    "cwltool": "3.1.20220628170238",
    "ep3 (experimental)": "v1.7.0",
    "nextflow": "22.04.4",
    "snakemake": "v7.8.3",
    "streamflow": "0.1.3",
    "toil (experimental)": "4.1.0"
  },
  "workflow_type_versions": {
    "CWL": {
      "workflow_type_version": [
        "v1.0",
        "v1.1",
        "v1.2"
      ]
    },
    "NFL": {
      "workflow_type_version": [
        "1.0",
        "DSL2"
      ]
    },
    "SMK": {
      "workflow_type_version": [
        "1.0"
      ]
    },
    "StreamFlow": {
      "workflow_type_version": [
        "v1.0"
      ]
    },
    "WDL": {
      "workflow_type_version": [
        "1.0"
      ]
    }
  }
}

WES API エンドポイントの定義は https://app.swaggerhub.com/apis/suecharo/sapporo-wes/sapporo-wes-1.0.1-oas3 を参照

現状の状態で、API リクエストにより、workflow のジョブの実行が可能である

• Example of API request:
  • https://github.com/sapporo-wes/sapporo-service/tree/main/tests/curl_example
  • https://github.com/sapporo-wes/sapporo-service/blob/main/tests/curl_example/post_runs/cwltool/attach_all_files/post_runs.sh

$ curl -s localhost:1122/runs
{"next_page_token":"","runs":[]}

現状対応している Workflow Engine は:

• cwltool (CWL)
• cromwell (CWL, WDL)
• Nextflow
• Snakemake

実際に workflow 実行の処理の手順としては:

• 1. WES server (Sapporo-service) に Run Request が POST される
• 2. Request の validation や、添付されたファイルのダウンロードなどを行う (saved to Run_Dir)
• 3. run.sh が fork される
• 4. run.sh 内で、workflow engine の command が実行される

全ての component は Docker コンテナとして起動される:

• Sapporo-service
• Workflow Engine
• Bioinfomatics Tools

Sapporo-web

Sapporo-web は、既に SPA として deploy されているため、起動の必要がない

https://sapporo-wes.github.io/sapporo-web/

WES の登録

• 一つの Sapporo-web にいくつもの Sapporo-service (WES) を登録できる
• 情報は全てブラウザの localStorage に保存される

実行したい Workflow の登録

• Example Workflow: https://github.com/sapporo-wes/sapporo-service/blob/main/tests/resources/cwltool/trimming_and_qc.cwl

Run Request の記述

• Worklfow Attachment: Workflow Engine の実行 directory に配置されるファイル群
  • Remote URL と Local File Upload の両方に対応している
    • fastq1: https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/ERR034597_1.small.fq.gz
    • fastq2: https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/ERR034597_2.small.fq.gz

{
  "fastq_1": {
    "class": "File",
    "location": "ERR034597_1.small.fq.gz"
  },
  "fastq_2": {
    "class": "File",
    "location": "ERR034597_2.small.fq.gz"
  }
}

cwltool は、ジョブ定義ファイルに remote url を記述できるため、workflow attachment を使わなくても良い

{
  "fastq_1": {
    "class": "File",
    "location": "https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/ERR034597_1.small.fq.gz"
  },
  "fastq_2": {
    "class": "File",
    "location": "https://raw.githubusercontent.com/sapporo-wes/sapporo-service/main/tests/resources/cwltool/ERR034597_2.small.fq.gz"
  }
}

Public Instance

Deployed at DDBJ super-computer system

• sapporo-service: https://ddbj.nig.ac.jp/ga4gh/wes/v1/service-info
• sapporo-web: https://ddbj.nig.ac.jp/wes/