---
# System prepended metadata

title: NVIDIA / Parabricks / docker inspect ...
tags: [Clara, 基因體, 二級分析, NVIDIA, Parabricks, Parabricks-v3.1]

---

NVIDIA / Parabricks / docker inspect ...
===
###### tags: `Parabricks-v3.1`
###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析`

<br>

[TOC]

<br>

## v3: pbrun fq2bam

### [v3] 直接執行 pbrun fq2bam
> 執行位置：/home/ubuntu/tj_tsai/parabricks/dataset
```bash
$ pbrun fq2bam \
    --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
    --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \
    --out-bam output.bam
```
- 實際上，是透過底下 docker 指令去執行

<br>

### [v3] 透過 docker 去執行 fq2bam
> 執行位置：/home/ubuntu/tj_tsai/parabricks/dataset
```bash
# create a working folder with a random name
mkdir 8XEILIEA

# run the docker engine
nvidia-docker run \
  -u=1000:1000 \
  --rm \
  -w=/home/ubuntu/tj_tsai/parabricks/dataset \
  --net=host \
  -v /opt/parabricks:/INSTALL/ \
  -v /home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA:/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA \
  -v /home/ubuntu/tj_tsai/parabricks/dataset:/home/ubuntu/tj_tsai/parabricks/dataset \
  -v /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref \
  -v /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data \
  parabricks/release:v3.1.1 \
  fq2bam \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \
  --out-bam output.bam \
  --tmp-dir /home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA
```
- 執行前，會先建立暫時的工作目錄：8XEILIEA，並掛載到容器


<br>

### [v3] fq2bam 執行結果
```bash=
ubuntu@alprworker-1203417-iaas:~/tj_tsai/parabricks/dataset$ pbrun fq2bam \
>     --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
>     --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \
>     --out-bam output.bam
Please visit https://www.nvidia.com/en-us/docs/parabricks/ for detailed documentation


Please visit https://www.nvidia.com/en-us/docs/parabricks/ for detailed documentation


[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for
/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz and
/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
------------------------------------------------------------------------------
||                 Parabricks accelerated Genomics Pipeline                 ||
||                              Version v3.1.1                              ||
||                       GPU-BWA mem, Sorting Phase-I                       ||
||                  Contact: Parabricks-Support@nvidia.com                  ||
------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs

GPU-BWA mem
ProgressMeter	Reads		Base Pairs Aligned


WARNING
The system has 8 threads, however recommended number of threads with 2 GPU is 24.
The run might not finish or might have less than expected performance.
[04:11:32]	5043564		580000000
[04:12:00]	10087128	1160000000
[04:12:27]	15130692	1750000000
[04:12:57]	20174256	2320000000
[04:13:25]	25217820	2900000000
[04:13:54]	30261384	3480000000
[04:14:21]	35304948	4060000000
[04:14:51]	40348512	4660000000
[04:15:19]	45392076	5210000000
[04:15:49]	50435640	5790000000

GPU-BWA Mem time: 322.795364 seconds
GPU-BWA Mem is finished.

GPU Sorting, Marking Dups, BQSR
ProgressMeter	SAM Entries Completed

Total GPU-BWA Mem + Sorting + MarkingDups + BQSR Generation + BAM writing
Processing time: 322.797858 seconds

[main] CMD: PARABRICKS mem -Z ./pbOpts.txt /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 329.689 sec; CPU: 2388.747 sec
------------------------------------------------------------------------------
||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
||        Version:                                            v3.1.1        ||
||        Start Time:                       Mon Nov  2 04:10:45 2020        ||
||        End Time:                         Mon Nov  2 04:16:15 2020        ||
||        Total Time:                           5 minutes 30 seconds        ||
------------------------------------------------------------------------------
------------------------------------------------------------------------------
||                 Parabricks accelerated Genomics Pipeline                 ||
||                              Version v3.1.1                              ||
||                             Sorting Phase-II                             ||
||                  Contact: Parabricks-Support@nvidia.com                  ||
------------------------------------------------------------------------------
progressMeter - Percentage
[04:16:17]	0.0	 0.00 GB
[04:16:27]	75.4	 0.38 GB
Sorting and Marking: 20.001 seconds
------------------------------------------------------------------------------
||        Program:                                  Sorting Phase-II        ||
||        Version:                                            v3.1.1        ||
||        Start Time:                       Mon Nov  2 04:16:17 2020        ||
||        End Time:                         Mon Nov  2 04:16:37 2020        ||
||        Total Time:                                     20 seconds        ||
------------------------------------------------------------------------------
------------------------------------------------------------------------------
||                 Parabricks accelerated Genomics Pipeline                 ||
||                              Version v3.1.1                              ||
||                         Marking Duplicates, BQSR                         ||
||                  Contact: Parabricks-Support@nvidia.com                  ||
------------------------------------------------------------------------------
Error happened when parsing args :-1 Exiting...
Please contact Parabricks-Support@nvidia.com for any questions. Exiting...

Could not run fq2bam
Exiting pbrun ...
```
- Marking Duplicates 執行失敗，向 Nvidia 詢問問題解法
    - [[Nvidia/Parabricks] got an error on running Marking Duplicates (with the official Parabricks samples)](https://forums.developer.nvidia.com/t/nvidia-parabricks-got-an-error-on-running-marking-duplicates-with-the-official-parabricks-samples/158340)

<br>

### 查看 image 組態
```
$ docker inspect --type=image parabricks/release:v3.1.1
```
- :::spoiler **展開結果**
    ```bash=
    [
        {
            "Id": "sha256:71c75a606a991e8c4354978da9b0f288284a71a37141cbf3f4f0eec60d891936",
            "RepoTags": [
                "parabricks/release:v3.1.1",
                "registry.gitlab.com/pbuser/release/x86_64:v3.1.1"
            ],
            "RepoDigests": [
                "registry.gitlab.com/pbuser/release/x86_64@sha256:aeed0fc58ce8ff7c8dc03c8905752f794b0d33faa1e851d1bf785bf853de0e61"
            ],
            "Parent": "",
            "Comment": "",
            "Created": "2020-09-25T17:53:50.534038874Z",
            "Container": "9a84b8eb22eae54c3ea1c75b75c8350431b092ca11ca7c98863fb07fc0c123f3",
            "ContainerConfig": {
                "Hostname": "057799ec4d2e",
                "Domainname": "",
                "User": "",
                "AttachStdin": true,
                "AttachStdout": true,
                "AttachStderr": true,
                "Tty": true,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "CUDA_VERSION=9.0.176",
                    "CUDA_PKG_VERSION=9-0=9.0.176-1",
                    "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                    "NVIDIA_VISIBLE_DEVICES=all",
                    "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                    "NVIDIA_REQUIRE_CUDA=cuda>=9.0",
                    "ENV=~/.profile"
                ],
                "Cmd": [
                    "/bin/sh",
                    "-c",
                    "#(nop) ",
                    "ENTRYPOINT [\"/parabricks/run_pipeline.py\"]"
                ],
                "Image": "sha256:dbf06e2a2aa24903edcf69605b284124792784d71a997de4e6f5cb8c05b90f37",
                "Volumes": null,
                "WorkingDir": "",
                "Entrypoint": [
                    "/parabricks/run_pipeline.py"
                ],
                "OnBuild": null,
                "Labels": {
                    "com.nvidia.cuda.version": "9.0.176",
                    "com.nvidia.volumes.needed": "nvidia_driver",
                    "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>"
                }
            },
            "DockerVersion": "18.09.7",
            "Author": "",
            "Config": {
                "Hostname": "057799ec4d2e",
                "Domainname": "",
                "User": "",
                "AttachStdin": true,
                "AttachStdout": true,
                "AttachStderr": true,
                "Tty": true,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "CUDA_VERSION=9.0.176",
                    "CUDA_PKG_VERSION=9-0=9.0.176-1",
                    "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                    "NVIDIA_VISIBLE_DEVICES=all",
                    "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                    "NVIDIA_REQUIRE_CUDA=cuda>=9.0",
                    "ENV=~/.profile"
                ],
                "Cmd": null,
                "Image": "sha256:dbf06e2a2aa24903edcf69605b284124792784d71a997de4e6f5cb8c05b90f37",
                "Volumes": null,
                "WorkingDir": "",
                "Entrypoint": [
                    "/parabricks/run_pipeline.py"
                ],
                "OnBuild": null,
                "Labels": {
                    "com.nvidia.cuda.version": "9.0.176",
                    "com.nvidia.volumes.needed": "nvidia_driver",
                    "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>"
                }
            },
            "Architecture": "amd64",
            "Os": "linux",
            "Size": 2726159518,
            "VirtualSize": 2726159518,
            "GraphDriver": {
                "Data": {
                    "LowerDir": "/var/lib/docker/overlay2/a67c874b385813ce8bcce45cbee03e341493e8db87253dbd7a7678e79ee4b76a/diff:/var/lib/docker/overlay2/cd6e506fa6bb6c6a33a199e2dd8fd19b4f2b87244de84fbe4f19b95ccc4ecc00/diff:/var/lib/docker/overlay2/21dfc1e60980855a1f92981cc5f5f934763d5b61fd1e037aacaecfcb197075ae/diff:/var/lib/docker/overlay2/fbf931220479988a22fc51c4d01de72b8ec720235aaa3cdc09a4d3d9977d6caf/diff:/var/lib/docker/overlay2/a79838bb12f18fa5fc3a829b86e4929fa13b99d991462410831b6d4743097c12/diff:/var/lib/docker/overlay2/db9b267b863aa0c3f51f9ffce850ceac01931a43b480564bba096e2ae6ea8572/diff:/var/lib/docker/overlay2/dc50241604382bdf0989bdb63e31879b73922a529ca8c810e7e2d6e44cc54ce8/diff:/var/lib/docker/overlay2/5da32dda6d04595672fee51ebd66ca62dddaee03e4362b82bc94feb524dbcd63/diff:/var/lib/docker/overlay2/5474e8fbfbfc5abb3c45e3f15ba838421e2481e7ca7e9e589e3f7b6926e9f943/diff:/var/lib/docker/overlay2/d9a3b3de0631dbe681e2a642159f4cd7c35b6fd92ac1c356985c428cf1faab09/diff:/var/lib/docker/overlay2/22000ac02963206210fc177c9f1771fae2dc9dc0147ebd5fd94629083bee18a4/diff:/var/lib/docker/overlay2/d17a61a008f265623833b83bbc9a9b7150c06199d1a0525b2eb54af6fa208507/diff:/var/lib/docker/overlay2/e72312d3c4b8d4e7601020d22fec47fe9ae2725e8b00df789b031931a629a43b/diff:/var/lib/docker/overlay2/54a1da7311ab99d3b5a525ef3147280cbd5edf43a0e47bee9554596db8f3ffb4/diff:/var/lib/docker/overlay2/52b6a0fe293bf8d4e2099de9a831491f91b144612d607c7fc0297c6f28e66994/diff:/var/lib/docker/overlay2/f6d404cb304a01c60152f70ae2b3ab00b1147c591d740de7ab4c964e8800f8eb/diff:/var/lib/docker/overlay2/3f8e7054f4ddc5afb7c4a5670c8b6179e1737b35b12556cf2d9ed921ffdef2a6/diff:/var/lib/docker/overlay2/9354fc5d589e0a4e0e6b63673e8e733c427054ef0e3f720e02db19d379e6e37f/diff:/var/lib/docker/overlay2/4de98792cac072adf7d506aa0014ef877a0915c7075d578495438cee918f1a9e/diff:/var/lib/docker/overlay2/d1fc59ed462ae931fd3a57a3e58f0326ec930a5bec36568f17be7027ae1a475e/diff:/var/lib/docker/overlay2/b7b6073ca3b5229028dcf508b7e5dec9ec36504e3e87b50cf14ad783a41cbef7/diff:/var/lib/docker/overlay2/e18dea8264373c4ba1e9b727a9a88c0a7601309609f00cd190c953a341f8e154/diff:/var/lib/docker/overlay2/2b36d9478b22bdf61de8e9aa184e3148a5c1bb122e5de7c9c2bbc285f96b52e5/diff:/var/lib/docker/overlay2/b701e5b243257bd3e0fa061f1437b00ba8b146ca28a9a99be82011a037bf09fa/diff:/var/lib/docker/overlay2/dac2ed621131d1f935c339ebbbfd9fde48e704fcc18e0ee6804d11c7bd21da6b/diff:/var/lib/docker/overlay2/c23853e258029f75245d825de9703d768b1da043a8b29cf1fee10b50923635a8/diff:/var/lib/docker/overlay2/91a7dec18eb2ee6c714e8e64ec0e3b62e64f3d942776cd5031698eb32ea51ce1/diff:/var/lib/docker/overlay2/5fa7853acfdf3c16b488f9ee00dd15beab910e8728bfd2099b8937fd33cc9ff9/diff:/var/lib/docker/overlay2/f41ad68bac6c742537f6bd2b8df0bc4900a0aecedfb2f88f5300d2c40812299f/diff:/var/lib/docker/overlay2/b8500e13ead3d7212e79eb8db9812a2a7c41108263ecca78dbe46e4a22cef11a/diff",
                    "MergedDir": "/var/lib/docker/overlay2/651288797f5d2fd5a7a89e0fac1468cbfc27293736d79f593949937f17b6aae3/merged",
                    "UpperDir": "/var/lib/docker/overlay2/651288797f5d2fd5a7a89e0fac1468cbfc27293736d79f593949937f17b6aae3/diff",
                    "WorkDir": "/var/lib/docker/overlay2/651288797f5d2fd5a7a89e0fac1468cbfc27293736d79f593949937f17b6aae3/work"
                },
                "Name": "overlay2"
            },
            "RootFS": {
                "Type": "layers",
                "Layers": [
                    "sha256:0de2edf7bff41238438e25b6f2de055b97c7fb6fe095cd560c2095b8dd70fc99",
                    ...
                    "sha256:2088719d10fd7d07b9ba644a981860976ab7144e282b2af4a145337a563319d8"
                ]
            },
            "Metadata": {
                "LastTagTime": "2020-11-02T17:18:22.384742702+08:00"
            }
        }
    ]

    ```
    :::

<br>

### 查看當前容器組態
```bash
$ docker inspect a61ef4fb8f9ea61ef4fb8f9e
```
- :::spoiler **展開結果**
    ```json=
    [
        {
            "Id": "a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f",
            "Created": "2020-10-30T09:54:43.05952969Z",
            "Path": "/parabricks/run_pipeline.py",
            "Args": [
                "fq2bam",
                "--ref",
                "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta",
                "--in-fq",
                "parabricks_sample/Data/sample_1.fq.gz",
                "parabricks_sample/Data/sample_2.fq.gz",
                "--out-bam",
                "output.bam",
                "--tmp-dir",
                "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA"
            ],
            "State": {
                "Status": "running",
                "Running": true,
                "Paused": false,
                "Restarting": false,
                "OOMKilled": false,
                "Dead": false,
                "Pid": 31707,
                "ExitCode": 0,
                "Error": "",
                "StartedAt": "2020-10-30T09:54:43.74889488Z",
                "FinishedAt": "0001-01-01T00:00:00Z"
            },
            "Image": "sha256:71c75a606a991e8c4354978da9b0f288284a71a37141cbf3f4f0eec60d891936",
            "ResolvConfPath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/resolv.conf",
            "HostnamePath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/hostname",
            "HostsPath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/hosts",
            "LogPath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f-json.log",
            "Name": "/thirsty_pascal",
            "RestartCount": 0,
            "Driver": "overlay2",
            "Platform": "linux",
            "MountLabel": "",
            "ProcessLabel": "",
            "AppArmorProfile": "docker-default",
            "ExecIDs": null,
            "HostConfig": {
                "Binds": [
                    "/opt/parabricks:/INSTALL/",
                    "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA:/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA",
                    "/home/ubuntu/tj_tsai/parabricks/dataset:/home/ubuntu/tj_tsai/parabricks/dataset",
                    "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref",
                    "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data"
                ],
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "json-file",
                    "Config": {
                        "max-size": "100m"
                    }
                },
                "NetworkMode": "host",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": true,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "CapAdd": null,
                "CapDrop": null,
                "Capabilities": null,
                "Dns": [],
                "DnsOptions": [],
                "DnsSearch": [],
                "ExtraHosts": null,
                "GroupAdd": null,
                "IpcMode": "private",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 67108864,
                "Runtime": "nvidia",
                "ConsoleSize": [
                    0,
                    0
                ],
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": [],
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [],
                "DeviceCgroupRules": null,
                "DeviceRequests": null,
                "KernelMemory": 0,
                "KernelMemoryTCP": 0,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": [
                    "/proc/asound",
                    "/proc/acpi",
                    "/proc/kcore",
                    "/proc/keys",
                    "/proc/latency_stats",
                    "/proc/timer_list",
                    "/proc/timer_stats",
                    "/proc/sched_debug",
                    "/proc/scsi",
                    "/sys/firmware"
                ],
                "ReadonlyPaths": [
                    "/proc/bus",
                    "/proc/fs",
                    "/proc/irq",
                    "/proc/sys",
                    "/proc/sysrq-trigger"
                ]
            },
            "GraphDriver": {
                "Data": {
                    "LowerDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db-init/diff:/var/lib/docker/overlay2/77e60ade2461ba9bc003f19a01144271fd616fb8b044d931bf4a70e303e57452/diff:/var/lib/docker/overlay2/6ac7e8d56a09a50c36ef26bbedffc6fa4ac5e70673e27fd3a956ec249e653d97/diff:/var/lib/docker/overlay2/02a4360a34cd1f33241c6de230b26287247b7d5510d0cab46a2e8130cd60fc48/diff:/var/lib/docker/overlay2/6c0095b52542058a8a3e139a20de360a910ca55dfbe607711b616fc5606a9c65/diff:/var/lib/docker/overlay2/781837541e7a7491be9341538ade1a6029bb147762fc70751f514670c7868088/diff:/var/lib/docker/overlay2/cd9dbea8a69bde0fe6ca69761013c002a137466abdeaca0e252b403f7723813d/diff:/var/lib/docker/overlay2/b69ba23128b9d2ad304217ec9dbaf9cf9faf318a6c8f8c516d2938969ca8981a/diff:/var/lib/docker/overlay2/d993e8f92faf4675a68a296a21182e6428624b8619e483f902e7e702568dae6b/diff:/var/lib/docker/overlay2/141acd74c35cf4f8ce83ac2f2d888ff410a34700eb89e2eed7e930e82a147d9f/diff:/var/lib/docker/overlay2/5e99943781094e09f156431adcab4216218b5372a9e1dc94c7b6862b307a259a/diff:/var/lib/docker/overlay2/a5f006bd1cf1d1f7c2911bb2a93b27987f4e236e100867fb9fb92d47ad125a95/diff:/var/lib/docker/overlay2/28dce71f3f876a9ebd3a80d9f0c01d473b9abe674bc1d5cb967e5be1974297d2/diff:/var/lib/docker/overlay2/d7f3c38cbe40950f67e2c6ca2b0a3ffcdc59a8cf978b259476ecfe1261897b32/diff:/var/lib/docker/overlay2/bd0d56dfca54e5b0ee38c82fb5d0eaad5d33960b241d1c086ea4f4a918a70b06/diff:/var/lib/docker/overlay2/5f50bb6cf3f2c5e7d021139a9277199657f80bfae990294598a116e26eaa0442/diff:/var/lib/docker/overlay2/28f1deac386a7b16e74780d0728934b12e794c1d0b1bdab6730cbc72f78072cf/diff:/var/lib/docker/overlay2/319bc7b9cf27b96e3d2483ffecd27f08d1d17aa98a9fb78890e1a278ddb0ffda/diff:/var/lib/docker/overlay2/a5d408fb3c7185cbcca95cfee677c6d144f7e320d0be1501cb3cb42f8d7eb88e/diff:/var/lib/docker/overlay2/e37a4d905be6f4ff70a89dfad4d6f17137ab884920968cb288bfcdffad7f89bb/diff:/var/lib/docker/overlay2/b951fdc815034bb00988c705c562740a5e2ca5589f3d2473e6d31650b60f258c/diff:/var/lib/docker/overlay2/cb952b9691ab20f97485d27c6615dd583062ebac14defc5cdd6d41762584b9d2/diff:/var/lib/docker/overlay2/5077f6758b1dba84a809623aedd3a8ce8c2dc54a4192a41df28949b9c6e1e34d/diff:/var/lib/docker/overlay2/08a387e8750376d9ba680bad39f8af5db4c6dbf8fb16f01823c1a3c2c147c2dc/diff:/var/lib/docker/overlay2/d608f451dfc0b0852714100bb64e7241eb7d84a2e892262831ad37a0d5466f7e/diff:/var/lib/docker/overlay2/cdff16dc620d533857971d6ed67b3c605b799f85f7630d2033219f34656254b5/diff:/var/lib/docker/overlay2/26f606671ac040ac0d6c9614d990f6a0499c3a12325d10ef332bb7aa7adb020d/diff:/var/lib/docker/overlay2/a1cfe223bfba6da71b50348edc8c52912d33ebac9d1760c3c1ecc0bdeada5ffc/diff:/var/lib/docker/overlay2/6d5d03992e6404f975023f7c0fa010b996a4fca3b582c234025ec4524cf0ddd5/diff:/var/lib/docker/overlay2/a8feeca3a1b79c89aa6cd0048a3ee5596165f69be8da26382f14744802046868/diff:/var/lib/docker/overlay2/bc3968c03bd10c6c01d043fb55c43da001e59c57d0c655c1125f8e89ce57b6da/diff:/var/lib/docker/overlay2/9c40fce9c5ea2cafddac3497d70687ac91cbc67e910efa174ecb4b53239ae21f/diff",
                    "MergedDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db/merged",
                    "UpperDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db/diff",
                    "WorkDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db/work"
                },
                "Name": "overlay2"
            },
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/opt/parabricks",
                    "Destination": "/INSTALL",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA",
                    "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/home/ubuntu/tj_tsai/parabricks/dataset",
                    "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref",
                    "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data",
                    "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                }
            ],
            "Config": {
                "Hostname": "alprworker-1203417-iaas",
                "Domainname": "",
                "User": "1000:1000",
                "AttachStdin": false,
                "AttachStdout": true,
                "AttachStderr": true,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "CUDA_VERSION=9.0.176",
                    "CUDA_PKG_VERSION=9-0=9.0.176-1",
                    "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                    "NVIDIA_VISIBLE_DEVICES=all",
                    "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                    "NVIDIA_REQUIRE_CUDA=cuda>=9.0",
                    "ENV=~/.profile"
                ],
                "Cmd": [
                    "fq2bam",
                    "--ref",
                    "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta",
                    "--in-fq",
                    "parabricks_sample/Data/sample_1.fq.gz",
                    "parabricks_sample/Data/sample_2.fq.gz",
                    "--out-bam",
                    "output.bam",
                    "--tmp-dir",
                    "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA"
                ],
                "Image": "parabricks/release:v3.1.1",
                "Volumes": null,
                "WorkingDir": "/home/ubuntu/tj_tsai/parabricks/dataset",
                "Entrypoint": [
                    "/parabricks/run_pipeline.py"
                ],
                "OnBuild": null,
                "Labels": {
                    "com.nvidia.cuda.version": "9.0.176",
                    "com.nvidia.volumes.needed": "nvidia_driver",
                    "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>"
                }
            },
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "bfbf77ef4bd8acaa4a13f71c2b3b37c06305adc059ca4f2ee4880e8496b090bf",
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "Ports": {},
                "SandboxKey": "/var/run/docker/netns/default",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "",
                "Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "MacAddress": "",
                "Networks": {
                    "host": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": null,
                        "NetworkID": "62c0aed62f8a64fc4b33a9951869575a548cd4a7a4f10a6662c39b8628f1baf7",
                        "EndpointID": "7f5d546487145186fd4711d9bc482c5f0f42deddcf1f1bf0b9d367bf6ecff333",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "MacAddress": "",
                        "DriverOpts": null
                    }
                }
            }
        }
    ]
    ```
    :::

<br>

<hr>
<hr>

<br>


## v2: pbrun germline

### 首先，要將 v2 image 偽裝成 v3 image
```
# 將 parabricks/release:v3.1.1 備份起來
docker tag parabricks/release:v3.1.1 parabricks/release:v3.1.1.bak

# 將 nvcr.io/hpc/parabricks:v2.5.0 命名成 parabricks/release:v3.1.1
docker tag nvcr.io/hpc/parabricks:v2.5.0 parabricks/release:v3.1.1
```
- 偽裝結果
    ![](https://i.imgur.com/N67QqR6.png)

<br>

### 接著，修改 pb_compose.py，移除在 v2 無法支援的參數 ```--memory-limit```
```
class germline(object):
    ...
    #alignmentCmd.extend(["--memory-limit", str(pipelineArgs.memory_limit)])
```

<br>

### 然後，開始執行 pbrun germline
> 執行位置：mnt/parabricks/dataset/parabricks_sample  (2020/11/11)
```bash
$ pbrun germline \
    --ref Ref/Homo_sapiens_assembly38.fasta \
    --in-fq Data/sample_1.fq.gz Data/sample_2.fq.gz \
    --knownSites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
    --out-bam output.bam \
    --out-variants output.vcf \
    --out-recal-file report.txt \
    --x3
```
- 實際上，是透過底下「多次」的 docker run 指令，
    去串接多項 single standalone 工具，
    來完成 germline pipeline

<br>

### [Step1] 透過 docker 去執行 fq2bam
```bash
# create a working folder with a random name
$ mkdir 4TLSTRB4

# run the docker engine
$ nvidia-docker run \
    -u=1000:1000 \
    --rm \
    -w=/mnt/parabricks/dataset/parabricks_sample \
    --net=host \
    -v /opt/parabricks:/INSTALL/ \
    -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \
    -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample \
    -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref \
    -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data \
    parabricks/release:v3.1.1 \
    fq2bam \
    --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
    --in-fq /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz \
            /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz \
            "@RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1" \
    --knownSites /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
    --out-bam /mnt/parabricks/dataset/parabricks_sample/output.bam \
    --out-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt \
    --num-cpu-threads 0 \
    --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \
    --num-gpus 2 \
    --x3 \
```
- 執行前，會先建立暫時的工作目錄：4TLSTRB4，並掛載到容器
- 與單獨執行 fq2bam 的差別，多了：
    - ```--knownSites Homo_sapiens_assembly38.known_indels.vcf.gz```
    - ```--out-recal-file output.bam```
    - ```--num-cpu-threads 0```
    - ```--num-gpus 2```
- [v2] docker 內部執行的程式
    ```
    /parabricks/bwa mem \
        /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
        /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz \
        /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz \
        "@RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1" \
        -Z ./pbOpts.txt
    ```
    - 此程式會分別列出底下執行時間
        - GPU-BWA Mem
        - GPU Sorting, Marking Dups, BQSR

<br>

### [Step2] 透過 docker 去執行 applybqsr ==(germline pipeline 並不會執行)==
```bash
nvidia-docker run \
    -u=1000:1000 \
    --rm \
    --net=host \
    -v /opt/parabricks:/INSTALL/ \
    -v /mnt/parabricks/dataset/parabricks_sample:/dataset \
    -w=/dataset \
    parabricks/release:v3.1.1 \
    applybqsr \
    --ref Ref/Homo_sapiens_assembly38.fasta \
    --in-bam output.bam \
    --in-recal-file report.txt \
    --out-bam output_new.bam \
    --tmp-dir tmp-dir \
    --num-gpus 2 \
    --x3
```

- docker 內部執行的程式
    ```
    /parabricks/applyBQSR \
        /dataset/output.bam \
        /dataset/Ref/Homo_sapiens_assembly38.fasta \
        /dataset/report.txt \
        /dataset/output_new.bam 2 \
        -n 8
    ```

<br>

### [Step3] 透過 docker 去執行 haplotypecaller
```bash=
nvidia-docker run \
    -u=1000:1000 \
    --rm \
    -w=/mnt/parabricks/dataset/parabricks_sample \
    --net=host \
    -v /opt/parabricks:/INSTALL/ \
    -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \
    -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample \
    -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref \
    -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data \
    parabricks/release:v3.1.1 \
    haplotypecaller \
    --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
    --in-bam /mnt/parabricks/dataset/parabricks_sample/output.bam \
    --out-variants /mnt/parabricks/dataset/parabricks_sample/output.vcf \
    --ploidy 2 \
    --num-htvc-threads 5 \
    --in-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt \
    --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \
    --num-gpus 2 \
    --x3 \
```

- docker 內部執行的程式
    ```
    /parabricks/htvc \
        /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
        /mnt/parabricks/dataset/parabricks_sample/output.bam 2 \
        -o /mnt/parabricks/dataset/parabricks_sample/output.vcf
        -nt 5
        -a /mnt/parabricks/dataset/parabricks_sample/report.txt
    ```

<br>

### [v2] germline 執行結果
```=
/mnt/parabricks/dataset/parabricks_sample$ pbrun germline \
>     --ref Ref/Homo_sapiens_assembly38.fasta \
>     --in-fq Data/sample_1.fq.gz Data/sample_2.fq.gz \
>     --knownSites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
>     --out-bam output.bam \
>     --out-variants output.vcf \
>     --out-recal-file report.txt \
>     --x3
Please visit https://www.nvidia.com/en-us/docs/parabricks/ for detailed documentation


[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz and
/mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1

nvidia-docker run -u=1000:1000 --rm -w=/mnt/parabricks/dataset/parabricks_sample --net=host -v /opt/parabricks:/INSTALL/ -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data parabricks/release:v3.1.1 fq2bam --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-fq /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 --knownSites /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz --out-bam /mnt/parabricks/dataset/parabricks_sample/output.bam --out-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt --num-cpu-threads 0 --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 --num-gpus 2 --x3
[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Read group created for /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz and
/mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
g 2 b 0 B 2 P 4 s 1 r 0 o 2 m 1 z 4 f 2 v 0 M 2 name /mnt/parabricks/dataset/parabricks_sample/output.bam report /mnt/parabricks/dataset/parabricks_sample/report.txt K /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz
/parabricks/bwa mem /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 -Z ./pbOpts.txt
------------------------------------------------------------------------------
||                 Parabricks accelerated Genomics Pipeline                 ||
||                              Version v2.5.0                              ||
||              GPU-BWA mem, Sorting, Marking Duplicates, BQSR              ||
||                  Contact: Parabricks-Support@nvidia.com                  ||
------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs

GPU-BWA mem
ProgressMeter	Reads		Base Pairs Aligned
[06:04:19]	5043564		570000000
[06:04:41]	10087128		1170000000
[06:05:03]	15130692		1750000000
[06:05:25]	20174256		2320000000
[06:05:47]	25217820		2910000000
[06:06:12]	30261384		3480000000
[06:06:35]	35304948		4060000000
[06:07:05]	40348512		4630000000
[06:07:39]	45392076		5220000000
[06:08:23]	50435640		5810000000

GPU-BWA Mem time: 320.785074 seconds
GPU-BWA Mem is finished.

GPU Sorting, Marking Dups, BQSR
ProgressMeter	SAM Entries Completed
[06:10:43]	5000000
[06:10:55]	10000000
[06:11:10]	15000000
[06:11:22]	20000000
[06:11:33]	25000000
[06:11:46]	30000000
[06:11:58]	35000000
[06:12:09]	40000000
[06:12:20]	45000000
[06:12:30]	50000000

Total GPU-BWA Mem + Sorting + MarkingDups + BQSR Generation + BAM writing
Processing time: 554.307597 seconds

[main] CMD: PARABRICKS mem -Z ./pbOpts.txt /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 579.484 sec; CPU: 3512.903 sec
------------------------------------------------------------------------------
||        Program:    GPU-BWA mem, Sorting, Marking Duplicates, BQSR        ||
||        Version:                                            v2.5.0        ||
||        Start Time:                       Wed Nov 11 06:03:16 2020        ||
||        End Time:                         Wed Nov 11 06:12:56 2020        ||
||        Total Time:                           9 minutes 40 seconds        ||
------------------------------------------------------------------------------
nvidia-docker run -u=1000:1000 --rm -w=/mnt/parabricks/dataset/parabricks_sample --net=host -v /opt/parabricks:/INSTALL/ -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data parabricks/release:v3.1.1 haplotypecaller --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-bam /mnt/parabricks/dataset/parabricks_sample/output.bam --out-variants /mnt/parabricks/dataset/parabricks_sample/output.vcf --ploidy 2 --num-htvc-threads 5 --in-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 --num-gpus 2 --x3
/parabricks/htvc /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /mnt/parabricks/dataset/parabricks_sample/output.bam 2 -o /mnt/parabricks/dataset/parabricks_sample/output.vcf -nt 5 -a /mnt/parabricks/dataset/parabricks_sample/report.txt
------------------------------------------------------------------------------
||                 Parabricks accelerated Genomics Pipeline                 ||
||                              Version v2.5.0                              ||
||                         GPU-GATK4 HaplotypeCaller                        ||
||                  Contact: Parabricks-Support@nvidia.com                  ||
------------------------------------------------------------------------------
ProgressMeter -	Current-Locus	Elapsed-Minutes	Regions-Processed	Regions/Minute
0 /mnt/parabricks/dataset/parabricks_sample/output.bam /mnt/parabricks/dataset/parabricks_sample/output.vcf
[06:13:41]	chr1:44025192	0.2	186462	1118772
[06:13:51]	chr1:120047401	0.3	506651	1519953
[06:14:01]	chr1:212664256	0.5	807809	1615618
[06:14:11]	chr2:30225516	0.7	1083834	1625751
[06:14:21]	chr2:113423954	0.8	1417596	1701115
[06:14:31]	chr2:202497574	1.0	1789922	1789922
[06:14:41]	chr3:23853493	1.2	2059339	1765147
[06:14:51]	chr3:88430228	1.3	2332229	1749171
[06:15:01]	chr3:164063784	1.5	2638414	1758942
[06:15:11]	chr4:40181156	1.7	2947562	1768537
[06:15:21]	chr4:135335070	1.8	3334613	1818879
[06:15:31]	chr5:20620641	2.0	3647921	1823960
[06:15:41]	chr5:109422287	2.2	3995759	1844196
[06:15:51]	chr5:169770920	2.3	4249075	1821032
[06:16:01]	chr6:68818865	2.5	4562383	1824953
[06:16:11]	chr6:153781792	2.7	4914247	1842842
[06:16:21]	chr7:53001504	2.8	5209805	1838754
[06:16:31]	chr7:144706059	3.0	5576395	1858798
[06:16:41]	chr8:25370543	3.2	5737251	1811763
[06:16:51]	chr8:96134342	3.3	6024200	1807260
[06:17:01]	chr9:18340715	3.5	6305279	1801508
[06:17:11]	chr9:107663978	3.7	6587580	1796612
[06:17:21]	chr10:29345105	3.8	6845977	1785907
[06:17:31]	chr10:112539685	4.0	7188383	1797095
[06:17:41]	chr11:34828594	4.2	7424252	1781820
[06:17:51]	chr11:96441495	4.3	7669672	1769924
[06:18:01]	chr12:30686173	4.5	7961091	1769131
[06:18:11]	chr12:118239773	4.7	8318854	1782611
[06:18:21]	chr13:78705460	4.8	8643172	1788242
[06:18:31]	chr14:68303992	5.0	9001898	1800379
[06:18:41]	chr15:74308571	5.2	9365561	1812689
[06:18:51]	chr16:60485626	5.3	9683210	1815601
[06:19:01]	chr17:51988581	5.5	9999505	1818091
[06:19:11]	chr18:39220687	5.7	10277526	1813681
[06:19:21]	chr19:46865573	5.8	10636583	1823414
[06:19:31]	chr20:63575952	6.0	10945566	1824261
[06:19:41]	chr22:30416502	6.2	11165927	1810690
[06:19:51]	chrX:82593563	6.3	11551238	1823879
[06:20:01]	HLA-DRB1*10:01:01:8401	6.5	12001422	1846372
Total time taken: 404.827
------------------------------------------------------------------------------
||        Program:                         GPU-GATK4 HaplotypeCaller        ||
||        Version:                                            v2.5.0        ||
||        Start Time:                       Wed Nov 11 06:13:01 2020        ||
||        End Time:                         Wed Nov 11 06:20:12 2020        ||
||        Total Time:                           7 minutes 11 seconds        ||
------------------------------------------------------------------------------
```


| 回合   | fq2bam | HaplotypeCaller | 總計 |
| ----- | ------ | -------- | ---------- |
| 1st   | 579.5s | 431s | 1010.5s (16m 50.5s) |
| 2nd   | 489.8s | 411s | 900.8s (15m 0.8s) |
| 3rd   | 497.3s | 423s | 920.3s (15m 20.3s) |
| 4th   | 522.3s | 432s | 954.3s (15m 54.3s) |
| avg   |        |      | 3785.9s (15m 46.5s)|



