NVIDIA / Parabricks / docker inspect ... === ###### tags: `Parabricks-v3.1` ###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析` <br> [TOC] <br> ## v3: pbrun fq2bam ### [v3] 直接執行 pbrun fq2bam > 執行位置：/home/ubuntu/tj_tsai/parabricks/dataset ```bash $ pbrun fq2bam \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \ --out-bam output.bam ``` - 實際上，是透過底下 docker 指令去執行 <br> ### [v3] 透過 docker 去執行 fq2bam > 執行位置：/home/ubuntu/tj_tsai/parabricks/dataset ```bash # create a working folder with a random name mkdir 8XEILIEA # run the docker engine nvidia-docker run \ -u=1000:1000 \ --rm \ -w=/home/ubuntu/tj_tsai/parabricks/dataset \ --net=host \ -v /opt/parabricks:/INSTALL/ \ -v /home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA:/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA \ -v /home/ubuntu/tj_tsai/parabricks/dataset:/home/ubuntu/tj_tsai/parabricks/dataset \ -v /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref \ -v /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data \ parabricks/release:v3.1.1 \ fq2bam \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \ --out-bam output.bam \ --tmp-dir /home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA ``` - 執行前，會先建立暫時的工作目錄：8XEILIEA，並掛載到容器 <br> ### [v3] fq2bam 執行結果 ```bash= ubuntu@alprworker-1203417-iaas:~/tj_tsai/parabricks/dataset$ pbrun fq2bam \ > --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ > --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \ > --out-bam output.bam Please visit https://www.nvidia.com/en-us/docs/parabricks/ for detailed documentation Please visit https://www.nvidia.com/en-us/docs/parabricks/ for detailed documentation [Parabricks Options Mesg]: Checking argument compatibility [Parabricks Options Mesg]: Automatically generating ID prefix [Parabricks Options Mesg]: Read group created for /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz and /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz [Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.1.1 || || GPU-BWA mem, Sorting Phase-I || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ [M::bwa_idx_load_from_disk] read 0 ALT contigs GPU-BWA mem ProgressMeter Reads Base Pairs Aligned WARNING The system has 8 threads, however recommended number of threads with 2 GPU is 24. The run might not finish or might have less than expected performance. [04:11:32] 5043564 580000000 [04:12:00] 10087128 1160000000 [04:12:27] 15130692 1750000000 [04:12:57] 20174256 2320000000 [04:13:25] 25217820 2900000000 [04:13:54] 30261384 3480000000 [04:14:21] 35304948 4060000000 [04:14:51] 40348512 4660000000 [04:15:19] 45392076 5210000000 [04:15:49] 50435640 5790000000 GPU-BWA Mem time: 322.795364 seconds GPU-BWA Mem is finished. GPU Sorting, Marking Dups, BQSR ProgressMeter SAM Entries Completed Total GPU-BWA Mem + Sorting + MarkingDups + BQSR Generation + BAM writing Processing time: 322.797858 seconds [main] CMD: PARABRICKS mem -Z ./pbOpts.txt /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 [main] Real time: 329.689 sec; CPU: 2388.747 sec ------------------------------------------------------------------------------ || Program: GPU-BWA mem, Sorting Phase-I || || Version: v3.1.1 || || Start Time: Mon Nov 2 04:10:45 2020 || || End Time: Mon Nov 2 04:16:15 2020 || || Total Time: 5 minutes 30 seconds || ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.1.1 || || Sorting Phase-II || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ progressMeter - Percentage [04:16:17] 0.0 0.00 GB [04:16:27] 75.4 0.38 GB Sorting and Marking: 20.001 seconds ------------------------------------------------------------------------------ || Program: Sorting Phase-II || || Version: v3.1.1 || || Start Time: Mon Nov 2 04:16:17 2020 || || End Time: Mon Nov 2 04:16:37 2020 || || Total Time: 20 seconds || ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.1.1 || || Marking Duplicates, BQSR || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ Error happened when parsing args :-1 Exiting... Please contact Parabricks-Support@nvidia.com for any questions. Exiting... Could not run fq2bam Exiting pbrun ... ``` - Marking Duplicates 執行失敗，向 Nvidia 詢問問題解法 - [[Nvidia/Parabricks] got an error on running Marking Duplicates (with the official Parabricks samples)](https://forums.developer.nvidia.com/t/nvidia-parabricks-got-an-error-on-running-marking-duplicates-with-the-official-parabricks-samples/158340) <br> ### 查看 image 組態 ``` $ docker inspect --type=image parabricks/release:v3.1.1 ``` - :::spoiler **展開結果** ```bash= [ { "Id": "sha256:71c75a606a991e8c4354978da9b0f288284a71a37141cbf3f4f0eec60d891936", "RepoTags": [ "parabricks/release:v3.1.1", "registry.gitlab.com/pbuser/release/x86_64:v3.1.1" ], "RepoDigests": [ "registry.gitlab.com/pbuser/release/x86_64@sha256:aeed0fc58ce8ff7c8dc03c8905752f794b0d33faa1e851d1bf785bf853de0e61" ], "Parent": "", "Comment": "", "Created": "2020-09-25T17:53:50.534038874Z", "Container": "9a84b8eb22eae54c3ea1c75b75c8350431b092ca11ca7c98863fb07fc0c123f3", "ContainerConfig": { "Hostname": "057799ec4d2e", "Domainname": "", "User": "", "AttachStdin": true, "AttachStdout": true, "AttachStderr": true, "Tty": true, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "CUDA_VERSION=9.0.176", "CUDA_PKG_VERSION=9-0=9.0.176-1", "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64", "NVIDIA_VISIBLE_DEVICES=all", "NVIDIA_DRIVER_CAPABILITIES=compute,utility", "NVIDIA_REQUIRE_CUDA=cuda>=9.0", "ENV=~/.profile" ], "Cmd": [ "/bin/sh", "-c", "#(nop) ", "ENTRYPOINT [\"/parabricks/run_pipeline.py\"]" ], "Image": "sha256:dbf06e2a2aa24903edcf69605b284124792784d71a997de4e6f5cb8c05b90f37", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/parabricks/run_pipeline.py" ], "OnBuild": null, "Labels": { "com.nvidia.cuda.version": "9.0.176", "com.nvidia.volumes.needed": "nvidia_driver", "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>" } }, "DockerVersion": "18.09.7", "Author": "", "Config": { "Hostname": "057799ec4d2e", "Domainname": "", "User": "", "AttachStdin": true, "AttachStdout": true, "AttachStderr": true, "Tty": true, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "CUDA_VERSION=9.0.176", "CUDA_PKG_VERSION=9-0=9.0.176-1", "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64", "NVIDIA_VISIBLE_DEVICES=all", "NVIDIA_DRIVER_CAPABILITIES=compute,utility", "NVIDIA_REQUIRE_CUDA=cuda>=9.0", "ENV=~/.profile" ], "Cmd": null, "Image": "sha256:dbf06e2a2aa24903edcf69605b284124792784d71a997de4e6f5cb8c05b90f37", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/parabricks/run_pipeline.py" ], "OnBuild": null, "Labels": { "com.nvidia.cuda.version": "9.0.176", "com.nvidia.volumes.needed": "nvidia_driver", "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>" } }, "Architecture": "amd64", "Os": "linux", "Size": 2726159518, "VirtualSize": 2726159518, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/a67c874b385813ce8bcce45cbee03e341493e8db87253dbd7a7678e79ee4b76a/diff:/var/lib/docker/overlay2/cd6e506fa6bb6c6a33a199e2dd8fd19b4f2b87244de84fbe4f19b95ccc4ecc00/diff:/var/lib/docker/overlay2/21dfc1e60980855a1f92981cc5f5f934763d5b61fd1e037aacaecfcb197075ae/diff:/var/lib/docker/overlay2/fbf931220479988a22fc51c4d01de72b8ec720235aaa3cdc09a4d3d9977d6caf/diff:/var/lib/docker/overlay2/a79838bb12f18fa5fc3a829b86e4929fa13b99d991462410831b6d4743097c12/diff:/var/lib/docker/overlay2/db9b267b863aa0c3f51f9ffce850ceac01931a43b480564bba096e2ae6ea8572/diff:/var/lib/docker/overlay2/dc50241604382bdf0989bdb63e31879b73922a529ca8c810e7e2d6e44cc54ce8/diff:/var/lib/docker/overlay2/5da32dda6d04595672fee51ebd66ca62dddaee03e4362b82bc94feb524dbcd63/diff:/var/lib/docker/overlay2/5474e8fbfbfc5abb3c45e3f15ba838421e2481e7ca7e9e589e3f7b6926e9f943/diff:/var/lib/docker/overlay2/d9a3b3de0631dbe681e2a642159f4cd7c35b6fd92ac1c356985c428cf1faab09/diff:/var/lib/docker/overlay2/22000ac02963206210fc177c9f1771fae2dc9dc0147ebd5fd94629083bee18a4/diff:/var/lib/docker/overlay2/d17a61a008f265623833b83bbc9a9b7150c06199d1a0525b2eb54af6fa208507/diff:/var/lib/docker/overlay2/e72312d3c4b8d4e7601020d22fec47fe9ae2725e8b00df789b031931a629a43b/diff:/var/lib/docker/overlay2/54a1da7311ab99d3b5a525ef3147280cbd5edf43a0e47bee9554596db8f3ffb4/diff:/var/lib/docker/overlay2/52b6a0fe293bf8d4e2099de9a831491f91b144612d607c7fc0297c6f28e66994/diff:/var/lib/docker/overlay2/f6d404cb304a01c60152f70ae2b3ab00b1147c591d740de7ab4c964e8800f8eb/diff:/var/lib/docker/overlay2/3f8e7054f4ddc5afb7c4a5670c8b6179e1737b35b12556cf2d9ed921ffdef2a6/diff:/var/lib/docker/overlay2/9354fc5d589e0a4e0e6b63673e8e733c427054ef0e3f720e02db19d379e6e37f/diff:/var/lib/docker/overlay2/4de98792cac072adf7d506aa0014ef877a0915c7075d578495438cee918f1a9e/diff:/var/lib/docker/overlay2/d1fc59ed462ae931fd3a57a3e58f0326ec930a5bec36568f17be7027ae1a475e/diff:/var/lib/docker/overlay2/b7b6073ca3b5229028dcf508b7e5dec9ec36504e3e87b50cf14ad783a41cbef7/diff:/var/lib/docker/overlay2/e18dea8264373c4ba1e9b727a9a88c0a7601309609f00cd190c953a341f8e154/diff:/var/lib/docker/overlay2/2b36d9478b22bdf61de8e9aa184e3148a5c1bb122e5de7c9c2bbc285f96b52e5/diff:/var/lib/docker/overlay2/b701e5b243257bd3e0fa061f1437b00ba8b146ca28a9a99be82011a037bf09fa/diff:/var/lib/docker/overlay2/dac2ed621131d1f935c339ebbbfd9fde48e704fcc18e0ee6804d11c7bd21da6b/diff:/var/lib/docker/overlay2/c23853e258029f75245d825de9703d768b1da043a8b29cf1fee10b50923635a8/diff:/var/lib/docker/overlay2/91a7dec18eb2ee6c714e8e64ec0e3b62e64f3d942776cd5031698eb32ea51ce1/diff:/var/lib/docker/overlay2/5fa7853acfdf3c16b488f9ee00dd15beab910e8728bfd2099b8937fd33cc9ff9/diff:/var/lib/docker/overlay2/f41ad68bac6c742537f6bd2b8df0bc4900a0aecedfb2f88f5300d2c40812299f/diff:/var/lib/docker/overlay2/b8500e13ead3d7212e79eb8db9812a2a7c41108263ecca78dbe46e4a22cef11a/diff", "MergedDir": "/var/lib/docker/overlay2/651288797f5d2fd5a7a89e0fac1468cbfc27293736d79f593949937f17b6aae3/merged", "UpperDir": "/var/lib/docker/overlay2/651288797f5d2fd5a7a89e0fac1468cbfc27293736d79f593949937f17b6aae3/diff", "WorkDir": "/var/lib/docker/overlay2/651288797f5d2fd5a7a89e0fac1468cbfc27293736d79f593949937f17b6aae3/work" }, "Name": "overlay2" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:0de2edf7bff41238438e25b6f2de055b97c7fb6fe095cd560c2095b8dd70fc99", ... "sha256:2088719d10fd7d07b9ba644a981860976ab7144e282b2af4a145337a563319d8" ] }, "Metadata": { "LastTagTime": "2020-11-02T17:18:22.384742702+08:00" } } ] ``` ::: <br> ### 查看當前容器組態 ```bash $ docker inspect a61ef4fb8f9ea61ef4fb8f9e ``` - :::spoiler **展開結果** ```json= [ { "Id": "a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f", "Created": "2020-10-30T09:54:43.05952969Z", "Path": "/parabricks/run_pipeline.py", "Args": [ "fq2bam", "--ref", "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta", "--in-fq", "parabricks_sample/Data/sample_1.fq.gz", "parabricks_sample/Data/sample_2.fq.gz", "--out-bam", "output.bam", "--tmp-dir", "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 31707, "ExitCode": 0, "Error": "", "StartedAt": "2020-10-30T09:54:43.74889488Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:71c75a606a991e8c4354978da9b0f288284a71a37141cbf3f4f0eec60d891936", "ResolvConfPath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/resolv.conf", "HostnamePath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/hostname", "HostsPath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/hosts", "LogPath": "/var/lib/docker/containers/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f/a61ef4fb8f9e4d26501ed4b1651ac2302502abe469a1ad8b689d52bc5dc10f1f-json.log", "Name": "/thirsty_pascal", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "docker-default", "ExecIDs": null, "HostConfig": { "Binds": [ "/opt/parabricks:/INSTALL/", "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA:/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA", "/home/ubuntu/tj_tsai/parabricks/dataset:/home/ubuntu/tj_tsai/parabricks/dataset", "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref", "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data:/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": { "max-size": "100m" } }, "NetworkMode": "host", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": true, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "nvidia", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db-init/diff:/var/lib/docker/overlay2/77e60ade2461ba9bc003f19a01144271fd616fb8b044d931bf4a70e303e57452/diff:/var/lib/docker/overlay2/6ac7e8d56a09a50c36ef26bbedffc6fa4ac5e70673e27fd3a956ec249e653d97/diff:/var/lib/docker/overlay2/02a4360a34cd1f33241c6de230b26287247b7d5510d0cab46a2e8130cd60fc48/diff:/var/lib/docker/overlay2/6c0095b52542058a8a3e139a20de360a910ca55dfbe607711b616fc5606a9c65/diff:/var/lib/docker/overlay2/781837541e7a7491be9341538ade1a6029bb147762fc70751f514670c7868088/diff:/var/lib/docker/overlay2/cd9dbea8a69bde0fe6ca69761013c002a137466abdeaca0e252b403f7723813d/diff:/var/lib/docker/overlay2/b69ba23128b9d2ad304217ec9dbaf9cf9faf318a6c8f8c516d2938969ca8981a/diff:/var/lib/docker/overlay2/d993e8f92faf4675a68a296a21182e6428624b8619e483f902e7e702568dae6b/diff:/var/lib/docker/overlay2/141acd74c35cf4f8ce83ac2f2d888ff410a34700eb89e2eed7e930e82a147d9f/diff:/var/lib/docker/overlay2/5e99943781094e09f156431adcab4216218b5372a9e1dc94c7b6862b307a259a/diff:/var/lib/docker/overlay2/a5f006bd1cf1d1f7c2911bb2a93b27987f4e236e100867fb9fb92d47ad125a95/diff:/var/lib/docker/overlay2/28dce71f3f876a9ebd3a80d9f0c01d473b9abe674bc1d5cb967e5be1974297d2/diff:/var/lib/docker/overlay2/d7f3c38cbe40950f67e2c6ca2b0a3ffcdc59a8cf978b259476ecfe1261897b32/diff:/var/lib/docker/overlay2/bd0d56dfca54e5b0ee38c82fb5d0eaad5d33960b241d1c086ea4f4a918a70b06/diff:/var/lib/docker/overlay2/5f50bb6cf3f2c5e7d021139a9277199657f80bfae990294598a116e26eaa0442/diff:/var/lib/docker/overlay2/28f1deac386a7b16e74780d0728934b12e794c1d0b1bdab6730cbc72f78072cf/diff:/var/lib/docker/overlay2/319bc7b9cf27b96e3d2483ffecd27f08d1d17aa98a9fb78890e1a278ddb0ffda/diff:/var/lib/docker/overlay2/a5d408fb3c7185cbcca95cfee677c6d144f7e320d0be1501cb3cb42f8d7eb88e/diff:/var/lib/docker/overlay2/e37a4d905be6f4ff70a89dfad4d6f17137ab884920968cb288bfcdffad7f89bb/diff:/var/lib/docker/overlay2/b951fdc815034bb00988c705c562740a5e2ca5589f3d2473e6d31650b60f258c/diff:/var/lib/docker/overlay2/cb952b9691ab20f97485d27c6615dd583062ebac14defc5cdd6d41762584b9d2/diff:/var/lib/docker/overlay2/5077f6758b1dba84a809623aedd3a8ce8c2dc54a4192a41df28949b9c6e1e34d/diff:/var/lib/docker/overlay2/08a387e8750376d9ba680bad39f8af5db4c6dbf8fb16f01823c1a3c2c147c2dc/diff:/var/lib/docker/overlay2/d608f451dfc0b0852714100bb64e7241eb7d84a2e892262831ad37a0d5466f7e/diff:/var/lib/docker/overlay2/cdff16dc620d533857971d6ed67b3c605b799f85f7630d2033219f34656254b5/diff:/var/lib/docker/overlay2/26f606671ac040ac0d6c9614d990f6a0499c3a12325d10ef332bb7aa7adb020d/diff:/var/lib/docker/overlay2/a1cfe223bfba6da71b50348edc8c52912d33ebac9d1760c3c1ecc0bdeada5ffc/diff:/var/lib/docker/overlay2/6d5d03992e6404f975023f7c0fa010b996a4fca3b582c234025ec4524cf0ddd5/diff:/var/lib/docker/overlay2/a8feeca3a1b79c89aa6cd0048a3ee5596165f69be8da26382f14744802046868/diff:/var/lib/docker/overlay2/bc3968c03bd10c6c01d043fb55c43da001e59c57d0c655c1125f8e89ce57b6da/diff:/var/lib/docker/overlay2/9c40fce9c5ea2cafddac3497d70687ac91cbc67e910efa174ecb4b53239ae21f/diff", "MergedDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db/merged", "UpperDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db/diff", "WorkDir": "/var/lib/docker/overlay2/279afe5e684d5dd7550aa7a0e29a5d28b43eeea0f95c51c56d67777be98fd7db/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/opt/parabricks", "Destination": "/INSTALL", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA", "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/home/ubuntu/tj_tsai/parabricks/dataset", "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref", "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Ref", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data", "Destination": "/home/ubuntu/tj_tsai/parabricks/dataset/parabricks_sample/Data", "Mode": "", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "alprworker-1203417-iaas", "Domainname": "", "User": "1000:1000", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "CUDA_VERSION=9.0.176", "CUDA_PKG_VERSION=9-0=9.0.176-1", "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64", "NVIDIA_VISIBLE_DEVICES=all", "NVIDIA_DRIVER_CAPABILITIES=compute,utility", "NVIDIA_REQUIRE_CUDA=cuda>=9.0", "ENV=~/.profile" ], "Cmd": [ "fq2bam", "--ref", "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta", "--in-fq", "parabricks_sample/Data/sample_1.fq.gz", "parabricks_sample/Data/sample_2.fq.gz", "--out-bam", "output.bam", "--tmp-dir", "/home/ubuntu/tj_tsai/parabricks/dataset/8XEILIEA" ], "Image": "parabricks/release:v3.1.1", "Volumes": null, "WorkingDir": "/home/ubuntu/tj_tsai/parabricks/dataset", "Entrypoint": [ "/parabricks/run_pipeline.py" ], "OnBuild": null, "Labels": { "com.nvidia.cuda.version": "9.0.176", "com.nvidia.volumes.needed": "nvidia_driver", "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "bfbf77ef4bd8acaa4a13f71c2b3b37c06305adc059ca4f2ee4880e8496b090bf", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/default", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "host": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "62c0aed62f8a64fc4b33a9951869575a548cd4a7a4f10a6662c39b8628f1baf7", "EndpointID": "7f5d546487145186fd4711d9bc482c5f0f42deddcf1f1bf0b9d367bf6ecff333", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null } } } } ] ``` ::: <br> <hr> <hr> <br> ## v2: pbrun germline ### 首先，要將 v2 image 偽裝成 v3 image ``` # 將 parabricks/release:v3.1.1 備份起來 docker tag parabricks/release:v3.1.1 parabricks/release:v3.1.1.bak # 將 nvcr.io/hpc/parabricks:v2.5.0 命名成 parabricks/release:v3.1.1 docker tag nvcr.io/hpc/parabricks:v2.5.0 parabricks/release:v3.1.1 ``` - 偽裝結果 ![](https://i.imgur.com/N67QqR6.png) <br> ### 接著，修改 pb_compose.py，移除在 v2 無法支援的參數 ```--memory-limit``` ``` class germline(object): ... #alignmentCmd.extend(["--memory-limit", str(pipelineArgs.memory_limit)]) ``` <br> ### 然後，開始執行 pbrun germline > 執行位置：mnt/parabricks/dataset/parabricks_sample (2020/11/11) ```bash $ pbrun germline \ --ref Ref/Homo_sapiens_assembly38.fasta \ --in-fq Data/sample_1.fq.gz Data/sample_2.fq.gz \ --knownSites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ --out-bam output.bam \ --out-variants output.vcf \ --out-recal-file report.txt \ --x3 ``` - 實際上，是透過底下「多次」的 docker run 指令， 去串接多項 single standalone 工具， 來完成 germline pipeline <br> ### [Step1] 透過 docker 去執行 fq2bam ```bash # create a working folder with a random name $ mkdir 4TLSTRB4 # run the docker engine $ nvidia-docker run \ -u=1000:1000 \ --rm \ -w=/mnt/parabricks/dataset/parabricks_sample \ --net=host \ -v /opt/parabricks:/INSTALL/ \ -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \ -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample \ -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref \ -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data \ parabricks/release:v3.1.1 \ fq2bam \ --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz \ /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz \ "@RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1" \ --knownSites /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ --out-bam /mnt/parabricks/dataset/parabricks_sample/output.bam \ --out-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt \ --num-cpu-threads 0 \ --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \ --num-gpus 2 \ --x3 \ ``` - 執行前，會先建立暫時的工作目錄：4TLSTRB4，並掛載到容器 - 與單獨執行 fq2bam 的差別，多了： - ```--knownSites Homo_sapiens_assembly38.known_indels.vcf.gz``` - ```--out-recal-file output.bam``` - ```--num-cpu-threads 0``` - ```--num-gpus 2``` - [v2] docker 內部執行的程式 ``` /parabricks/bwa mem \ /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz \ /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz \ "@RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1" \ -Z ./pbOpts.txt ``` - 此程式會分別列出底下執行時間 - GPU-BWA Mem - GPU Sorting, Marking Dups, BQSR <br> ### [Step2] 透過 docker 去執行 applybqsr ==(germline pipeline 並不會執行)== ```bash nvidia-docker run \ -u=1000:1000 \ --rm \ --net=host \ -v /opt/parabricks:/INSTALL/ \ -v /mnt/parabricks/dataset/parabricks_sample:/dataset \ -w=/dataset \ parabricks/release:v3.1.1 \ applybqsr \ --ref Ref/Homo_sapiens_assembly38.fasta \ --in-bam output.bam \ --in-recal-file report.txt \ --out-bam output_new.bam \ --tmp-dir tmp-dir \ --num-gpus 2 \ --x3 ``` - docker 內部執行的程式 ``` /parabricks/applyBQSR \ /dataset/output.bam \ /dataset/Ref/Homo_sapiens_assembly38.fasta \ /dataset/report.txt \ /dataset/output_new.bam 2 \ -n 8 ``` <br> ### [Step3] 透過 docker 去執行 haplotypecaller ```bash= nvidia-docker run \ -u=1000:1000 \ --rm \ -w=/mnt/parabricks/dataset/parabricks_sample \ --net=host \ -v /opt/parabricks:/INSTALL/ \ -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \ -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample \ -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref \ -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data \ parabricks/release:v3.1.1 \ haplotypecaller \ --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-bam /mnt/parabricks/dataset/parabricks_sample/output.bam \ --out-variants /mnt/parabricks/dataset/parabricks_sample/output.vcf \ --ploidy 2 \ --num-htvc-threads 5 \ --in-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt \ --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 \ --num-gpus 2 \ --x3 \ ``` - docker 內部執行的程式 ``` /parabricks/htvc \ /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ /mnt/parabricks/dataset/parabricks_sample/output.bam 2 \ -o /mnt/parabricks/dataset/parabricks_sample/output.vcf -nt 5 -a /mnt/parabricks/dataset/parabricks_sample/report.txt ``` <br> ### [v2] germline 執行結果 ```= /mnt/parabricks/dataset/parabricks_sample$ pbrun germline \ > --ref Ref/Homo_sapiens_assembly38.fasta \ > --in-fq Data/sample_1.fq.gz Data/sample_2.fq.gz \ > --knownSites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ > --out-bam output.bam \ > --out-variants output.vcf \ > --out-recal-file report.txt \ > --x3 Please visit https://www.nvidia.com/en-us/docs/parabricks/ for detailed documentation [Parabricks Options Mesg]: Automatically generating ID prefix [Parabricks Options Mesg]: Read group created for /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz and /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz [Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 nvidia-docker run -u=1000:1000 --rm -w=/mnt/parabricks/dataset/parabricks_sample --net=host -v /opt/parabricks:/INSTALL/ -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data parabricks/release:v3.1.1 fq2bam --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-fq /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 --knownSites /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz --out-bam /mnt/parabricks/dataset/parabricks_sample/output.bam --out-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt --num-cpu-threads 0 --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 --num-gpus 2 --x3 [Parabricks Options Mesg]: Checking argument compatibility [Parabricks Options Mesg]: Read group created for /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz and /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz [Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 g 2 b 0 B 2 P 4 s 1 r 0 o 2 m 1 z 4 f 2 v 0 M 2 name /mnt/parabricks/dataset/parabricks_sample/output.bam report /mnt/parabricks/dataset/parabricks_sample/report.txt K /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz /parabricks/bwa mem /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 -Z ./pbOpts.txt ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v2.5.0 || || GPU-BWA mem, Sorting, Marking Duplicates, BQSR || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ [M::bwa_idx_load_from_disk] read 0 ALT contigs GPU-BWA mem ProgressMeter Reads Base Pairs Aligned [06:04:19] 5043564 570000000 [06:04:41] 10087128 1170000000 [06:05:03] 15130692 1750000000 [06:05:25] 20174256 2320000000 [06:05:47] 25217820 2910000000 [06:06:12] 30261384 3480000000 [06:06:35] 35304948 4060000000 [06:07:05] 40348512 4630000000 [06:07:39] 45392076 5220000000 [06:08:23] 50435640 5810000000 GPU-BWA Mem time: 320.785074 seconds GPU-BWA Mem is finished. GPU Sorting, Marking Dups, BQSR ProgressMeter SAM Entries Completed [06:10:43] 5000000 [06:10:55] 10000000 [06:11:10] 15000000 [06:11:22] 20000000 [06:11:33] 25000000 [06:11:46] 30000000 [06:11:58] 35000000 [06:12:09] 40000000 [06:12:20] 45000000 [06:12:30] 50000000 Total GPU-BWA Mem + Sorting + MarkingDups + BQSR Generation + BAM writing Processing time: 554.307597 seconds [main] CMD: PARABRICKS mem -Z ./pbOpts.txt /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /mnt/parabricks/dataset/parabricks_sample/Data/sample_1.fq.gz /mnt/parabricks/dataset/parabricks_sample/Data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1 [main] Real time: 579.484 sec; CPU: 3512.903 sec ------------------------------------------------------------------------------ || Program: GPU-BWA mem, Sorting, Marking Duplicates, BQSR || || Version: v2.5.0 || || Start Time: Wed Nov 11 06:03:16 2020 || || End Time: Wed Nov 11 06:12:56 2020 || || Total Time: 9 minutes 40 seconds || ------------------------------------------------------------------------------ nvidia-docker run -u=1000:1000 --rm -w=/mnt/parabricks/dataset/parabricks_sample --net=host -v /opt/parabricks:/INSTALL/ -v /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4:/mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 -v /mnt/parabricks/dataset/parabricks_sample:/mnt/parabricks/dataset/parabricks_sample -v /mnt/parabricks/dataset/parabricks_sample/Ref:/mnt/parabricks/dataset/parabricks_sample/Ref -v /mnt/parabricks/dataset/parabricks_sample/Data:/mnt/parabricks/dataset/parabricks_sample/Data parabricks/release:v3.1.1 haplotypecaller --ref /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-bam /mnt/parabricks/dataset/parabricks_sample/output.bam --out-variants /mnt/parabricks/dataset/parabricks_sample/output.vcf --ploidy 2 --num-htvc-threads 5 --in-recal-file /mnt/parabricks/dataset/parabricks_sample/report.txt --tmp-dir /mnt/parabricks/dataset/parabricks_sample/4TLSTRB4 --num-gpus 2 --x3 /parabricks/htvc /mnt/parabricks/dataset/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /mnt/parabricks/dataset/parabricks_sample/output.bam 2 -o /mnt/parabricks/dataset/parabricks_sample/output.vcf -nt 5 -a /mnt/parabricks/dataset/parabricks_sample/report.txt ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v2.5.0 || || GPU-GATK4 HaplotypeCaller || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ ProgressMeter - Current-Locus Elapsed-Minutes Regions-Processed Regions/Minute 0 /mnt/parabricks/dataset/parabricks_sample/output.bam /mnt/parabricks/dataset/parabricks_sample/output.vcf [06:13:41] chr1:44025192 0.2 186462 1118772 [06:13:51] chr1:120047401 0.3 506651 1519953 [06:14:01] chr1:212664256 0.5 807809 1615618 [06:14:11] chr2:30225516 0.7 1083834 1625751 [06:14:21] chr2:113423954 0.8 1417596 1701115 [06:14:31] chr2:202497574 1.0 1789922 1789922 [06:14:41] chr3:23853493 1.2 2059339 1765147 [06:14:51] chr3:88430228 1.3 2332229 1749171 [06:15:01] chr3:164063784 1.5 2638414 1758942 [06:15:11] chr4:40181156 1.7 2947562 1768537 [06:15:21] chr4:135335070 1.8 3334613 1818879 [06:15:31] chr5:20620641 2.0 3647921 1823960 [06:15:41] chr5:109422287 2.2 3995759 1844196 [06:15:51] chr5:169770920 2.3 4249075 1821032 [06:16:01] chr6:68818865 2.5 4562383 1824953 [06:16:11] chr6:153781792 2.7 4914247 1842842 [06:16:21] chr7:53001504 2.8 5209805 1838754 [06:16:31] chr7:144706059 3.0 5576395 1858798 [06:16:41] chr8:25370543 3.2 5737251 1811763 [06:16:51] chr8:96134342 3.3 6024200 1807260 [06:17:01] chr9:18340715 3.5 6305279 1801508 [06:17:11] chr9:107663978 3.7 6587580 1796612 [06:17:21] chr10:29345105 3.8 6845977 1785907 [06:17:31] chr10:112539685 4.0 7188383 1797095 [06:17:41] chr11:34828594 4.2 7424252 1781820 [06:17:51] chr11:96441495 4.3 7669672 1769924 [06:18:01] chr12:30686173 4.5 7961091 1769131 [06:18:11] chr12:118239773 4.7 8318854 1782611 [06:18:21] chr13:78705460 4.8 8643172 1788242 [06:18:31] chr14:68303992 5.0 9001898 1800379 [06:18:41] chr15:74308571 5.2 9365561 1812689 [06:18:51] chr16:60485626 5.3 9683210 1815601 [06:19:01] chr17:51988581 5.5 9999505 1818091 [06:19:11] chr18:39220687 5.7 10277526 1813681 [06:19:21] chr19:46865573 5.8 10636583 1823414 [06:19:31] chr20:63575952 6.0 10945566 1824261 [06:19:41] chr22:30416502 6.2 11165927 1810690 [06:19:51] chrX:82593563 6.3 11551238 1823879 [06:20:01] HLA-DRB1*10:01:01:8401 6.5 12001422 1846372 Total time taken: 404.827 ------------------------------------------------------------------------------ || Program: GPU-GATK4 HaplotypeCaller || || Version: v2.5.0 || || Start Time: Wed Nov 11 06:13:01 2020 || || End Time: Wed Nov 11 06:20:12 2020 || || Total Time: 7 minutes 11 seconds || ------------------------------------------------------------------------------ ``` | 回合 | fq2bam | HaplotypeCaller | 總計 | | ----- | ------ | -------- | ---------- | | 1st | 579.5s | 431s | 1010.5s (16m 50.5s) | | 2nd | 489.8s | 411s | 900.8s (15m 0.8s) | | 3rd | 497.3s | 423s | 920.3s (15m 20.3s) | | 4th | 522.3s | 432s | 954.3s (15m 54.3s) | | avg | | | 3785.9s (15m 46.5s)|