[log] container 內部的細節 === ###### tags: `Parabricks-v3.5` ###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析` <br> [TOC] <br> :::warning :bulb: **啟發** - 原本是由 pbrun 啟動容器 - 拿這個容器的 image 當作 base - 再把 pbrun 複製到 image - 原本 pbrun 是執行 docker run, 改成 python3 /parabricks/run_pipeline.py ::: ## 建置 image :::warning :bulb: 需先前查看 [container 的組態](#inspect) ::: ### 準備 Dockerfile ```dockerfile= FROM parabricks/release:v3.5.0 ENTRYPOINT ["/bin/bash"] ``` <br> ### 執行 docker build ```bash= docker build -t parabricks/release:v3.5.0-by-tj . ``` - `.` 表示當前目錄，從當前目錄尋找 `Dockerfile` <br> ### 執行 docker run ```bash= docker run --rm -it parabricks/release:v3.5.0-by-tj ``` - v3.5 版本 :+1: - 沒有拔掉 pwd, ls, find, grep, ... [![](https://i.imgur.com/VOeIOxj.png)](https://i.imgur.com/VOeIOxj.png) - user 是 root ``` # id uid=0(root) gid=0(root) groups=0(root) ``` - 可以自行安裝套件 ```bash= apt install nano ``` - 舊版 - pwd, ls, find, grep, ... 等所有的指令，都是空殼 - 要執行這些指令，必須從外部掛進去 <br> ## 掛載資料測試 ### 進入到容器 > 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 18:50) ```bash= # 切到含有 parabricks_sample 子目錄的目錄 rm -rf tmp-dir mkdir tmp-dir WORKSPACE=`pwd` docker run \ --gpus all \ --rm \ -it \ --net=host \ -w=$WORKSPACE \ -v $WORKSPACE:$WORKSPACE \ -v /opt/parabricks:/INSTALL/ \ parabricks/release:v3.5.0-by-tj ``` - 這邊要移除掉 `-u=1000:1000`參數 因為 user 是 `uid=0(root) gid=0(root) groups=0(root) ` 否則會有底下的 error： `groups: cannot find name for group ID 1000` - `--gpus all` - 主機若沒有 CPU，會遇到下列錯誤： > docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown. - 即使有安裝 Nvidia runtime (CUDA Toolkit)、nvidia-container-toolkit 仍然一樣 - `nvidia-smi` ``` $ nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. ``` - 解決方式 - 暫時不要使用 `--gpus all` 參數 <br> ### [在容器中] 執行 fq2bam (簡單版) > 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 19:15) ```bash= python3 /parabricks/run_pipeline.py fq2bam \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq parabricks_sample/Data/sample_1.fq.gz \ parabricks_sample/Data/sample_2.fq.gz \ --out-bam output.bam \ --tmp-dir tmp-dir \ --x3 ``` - 是 `python3` 程式，不是 `python` 否則會執行到 Python 2.7.17 - 指令同 [`pbrun fq2bam`](/on6QkW93SKyxvV2codV7GA##執行-pbrun-fq2bam)，但要加上 `--tmp-dir` 參數 否則會有底下的 error： ![](https://i.imgur.com/a8c4NF7.png) `ValueError: '--tmp-dir' is not in list` - 曾經遇到： - 硬碟空間不足，但實際上空間很大(1TB)，目前無法重製 ``` ... progressMeter - Percentage [08:10:31] 0.0 0.00 GB Received signal: 11 Please make sure you have enough disk space in the output and temp directory Contact Parabricks-Support@nvidia.com for support Please contact Parabricks-Support@nvidia.com for any questions There is a forum for Q&A as well at https://forums.developer.nvidia.com/c/healthcare/Parabricks/290 Exiting... ``` - Please make sure you have enough disk space. <br> ### [在容器中] 執行 fq2bam (複雜版) > 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 19:30) ```bash= python3 /parabricks/run_pipeline.py fq2bam \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq parabricks_sample/Data/sample_1.fq.gz \ parabricks_sample/Data/sample_2.fq.gz \ "@RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1" \ --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ --out-bam output.bam \ --out-recal-file report.txt \ --tmp-dir tmp-dir \ --x3 ``` <br> ### [在容器中] 執行 germline :::warning :warning: **底下指令不存在** :no_entry_sign: ```bash= python3 /parabricks/run_pipeline.py germline \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \ --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ --out-bam output.bam \ --out-variants output.vcf \ --out-recal-file report.txt \ --x3 ``` ![](https://i.imgur.com/HSnvoCl.png) - germline 指令是由 pbrun 控制 - 容器內部並無 germline 指令 ::: > 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 19:43) ``` python3 /parabricks/run_pipeline.py haplotypecaller \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-bam output.bam \ --in-recal-file report.txt \ --out-variants output.vcf \ --ploidy 2 \ --num-htvc-threads 5 \ --num-gpus 2 \ --x3 \ --tmp-dir tmp-dir ``` <br> <hr> <br> ## inspect ### [`pbrun fq2bam`](/on6QkW93SKyxvV2codV7GA#pbrun-fq2bam) ```bash= pbrun fq2bam \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq parabricks_sample/Data/sample_1.fq.gz \ parabricks_sample/Data/sample_2.fq.gz \ --out-bam output.bam \ --x3 ``` ```bash= # 透過 pbrun 正常啟動的容器 $ docker inspect db3d02a38ec0 ``` - Path `"/parabricks/run_pipeline.py"` - Args ```json= [ "fq2bam", "--ref", "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta", "--in-fq", "parabricks_sample/Data/sample_1.fq.gz", "parabricks_sample/Data/sample_2.fq.gz", "--out-bam", "output.bam", "--x3", "--tmp-dir", "/uploads/workspace/9DBIL9ME" ] ``` - Cmd ```bash= [ "fq2bam", "--ref", "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta", "--in-fq", "parabricks_sample/Data/sample_1.fq.gz", "parabricks_sample/Data/sample_2.fq.gz", "--out-bam", "output.bam", "--x3", "--tmp-dir", "/uploads/workspace/9DBIL9ME" ] ``` - Entrypoint ```json= "/parabricks/run_pipeline.py" ``` - 執行結果 :::spoiler ```json [ { "Id": "db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf", "Created": "2021-06-10T03:14:57.398476975Z", "Path": "/parabricks/run_pipeline.py", "Args": [ "fq2bam", "--ref", "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta", "--in-fq", "parabricks_sample/Data/sample_1.fq.gz", "parabricks_sample/Data/sample_2.fq.gz", "--out-bam", "output.bam", "--x3", "--tmp-dir", "/uploads/workspace/9DBIL9ME" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 24839, "ExitCode": 0, "Error": "", "StartedAt": "2021-06-10T03:14:59.631886014Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:da882a2f26c9e3858c92e2dc8ac4c05d3dfbab69702dc75e4d218369e6c3cfe6", "ResolvConfPath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/resolv.conf", "HostnamePath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/hostname", "HostsPath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/hosts", "LogPath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf-json.log", "Name": "/distracted_mendeleev", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "docker-default", "ExecIDs": null, "HostConfig": { "Binds": [ "/uploads/workspace/parabricks_sample/Data:/uploads/workspace/parabricks_sample/Data", "/opt/parabricks:/INSTALL/", "/uploads/workspace/9DBIL9ME:/uploads/workspace/9DBIL9ME", "/uploads/workspace:/uploads/workspace", "/uploads/workspace/parabricks_sample/Ref:/uploads/workspace/parabricks_sample/Ref" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "host", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": true, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "CgroupnsMode": "host", "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "private", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": [ { "Driver": "", "Count": -1, "DeviceIDs": null, "Capabilities": [ [ "gpu" ] ], "Options": {} } ], "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab-init/diff:/var/lib/docker/overlay2/95100646f1ff8316d1c06a7b0b3451439dca5bdf4e33bde6f584065c96866c09/diff:/var/lib/docker/overlay2/c9158a87787badbb60960de32a39cfa18bfc56f2b22e502cbda712f5cb99be42/diff:/var/lib/docker/overlay2/b6f618b9e13e49ab439c08f179dc7ddf2e756ac1cd53bbdc31b3872f01f4d3d4/diff:/var/lib/docker/overlay2/8ce465351f799fd7e0c9284cbeb1619e594b963713b79516fd5f861da11f8c7c/diff:/var/lib/docker/overlay2/5f51dbee605647a571a609ff9f33d411c5d944503356ac3f20ce44a8e6421a65/diff:/var/lib/docker/overlay2/e3a6db43fff265a671c939b6a28574bb0599e96603ef72aa77bdb325e29468da/diff:/var/lib/docker/overlay2/dbf16f87259a59e72ac5edef6594586296916c0c969c928648441d64b4b03b9e/diff:/var/lib/docker/overlay2/93478e0a56e909c6026d2cb020fa33c26633b9661f0be122be6b9b528f3437ee/diff:/var/lib/docker/overlay2/e3209e1ad5328df416136513d36e9bfafd1f28cde75ac54be142d25090185177/diff:/var/lib/docker/overlay2/417b5ab64d2485fddc0a6369434e9acdde7d40a360aab00b8a9087ed7803ce00/diff:/var/lib/docker/overlay2/aff613731f0d6d16380efe3573d38ae44985c4adcf303de9af822ace571ab5e3/diff:/var/lib/docker/overlay2/204381c326d056c06c381c9e3f80d6ff753999aeecd90fbea0c3e32f2aa3018d/diff:/var/lib/docker/overlay2/1f53eb8731ff13cf6389da0f0b91cb7ae75dfae974d727baf718167c685ec90c/diff:/var/lib/docker/overlay2/2cbce77b8816ba28ca718757c351060a5fc31d3e4e01f90fe685a4745fbface8/diff:/var/lib/docker/overlay2/adb4dc92c8c2a038a2eb3aefdb59eaae2ca6a9f13dbe46efad3b1c8e79c35456/diff:/var/lib/docker/overlay2/98513893aa6968b97d1b0df0937407e76e41b92accad4b6f8a87093eaaaa125d/diff:/var/lib/docker/overlay2/b82cc3de5486d714f0babf1ba64caba690dede00c10d60adcb736635edc3eaf3/diff:/var/lib/docker/overlay2/187bea67669cd8d2de88f6ba252751249243b6a3a3db21a1f78e12e7182b8558/diff:/var/lib/docker/overlay2/44efbdf16447b790efaf613ced36a7801203006beba5d53f7728d76bf9a1a723/diff", "MergedDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab/merged", "UpperDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab/diff", "WorkDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/uploads/workspace/parabricks_sample/Data", "Destination": "/uploads/workspace/parabricks_sample/Data", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/opt/parabricks", "Destination": "/INSTALL", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/uploads/workspace/9DBIL9ME", "Destination": "/uploads/workspace/9DBIL9ME", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/uploads/workspace", "Destination": "/uploads/workspace", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/uploads/workspace/parabricks_sample/Ref", "Destination": "/uploads/workspace/parabricks_sample/Ref", "Mode": "", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "GPU-VM-NC12s-v2", "Domainname": "", "User": "1000:1000", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/miniconda/bin", "CUDA_VERSION=10.1.243", "CUDA_PKG_VERSION=10-1=10.1.243-1", "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64", "NVIDIA_VISIBLE_DEVICES=all", "NVIDIA_DRIVER_CAPABILITIES=compute,utility", "NVIDIA_REQUIRE_CUDA=cuda>=10.1 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419" ], "Cmd": [ "fq2bam", "--ref", "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta", "--in-fq", "parabricks_sample/Data/sample_1.fq.gz", "parabricks_sample/Data/sample_2.fq.gz", "--out-bam", "output.bam", "--x3", "--tmp-dir", "/uploads/workspace/9DBIL9ME" ], "Image": "parabricks/release:v3.5.0", "Volumes": null, "WorkingDir": "/uploads/workspace", "Entrypoint": [ "/parabricks/run_pipeline.py" ], "OnBuild": null, "Labels": { "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "3ccfc9126c93fb12933d89f28314c3719180f8c6169cf4adecd8b59da4a6a549", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/default", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "host": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "e93977ab9b9b8ab0ba8b127243f8b6d66bca2596428a25857a131f68b5ee2b41", "EndpointID": "0de75fab853cb4be062e1eae031598814599f0030e050c389c0c964644c9e139", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null } } } } ] ``` ::: <br> <hr> <br> ## binary 檔的真正位置在 :::warning **資料來源：[[log] WGS + Azure:GPU-NC12s-v2(P100-16GB x2)](/fS60joh8TNqroKm2L_0WYg)** - log 中出現的 binary 檔位置，並非是真正位置 ``` /usr/local/cuda/.pb/binaries//bin/bwa ``` ::: ``` /usr/local/cuda-10.1/.pb/binaries/bin ``` ```bash= usr/local/cuda-10.1/.pb/binaries/bin$ ls -ls total 467644 360 -rwxrwxrwx 367256 Feb 23 applyBQSR 1192 -rwxrwxrwx 1218248 Feb 23 bamreadcount 296 -rwxrwxrwx 301024 Feb 23 bcftoolscall 432 -rwxrwxrwx 440368 Feb 23 bcftoolsmpileup 96 -rwxrwxrwx 96584 Feb 23 bedcov 500 -rwxrwxrwx 510696 Feb 23 bqsr 2424 -rwxrwxrwx 2481896 Feb 23 bwa 708 -rwxrwxrwx 723584 Feb 23 cnnscorevariants 220 -rwxrwxrwx 223192 Feb 23 cnvkit 844 -rwxrwxrwx 863712 Feb 23 collectmultiplemetrics 336 -rwxrwxrwx 342408 Feb 23 coverage 272 -rwxrwxrwx 276456 Feb 23 dbsnp 2252 -rwxrwxrwx 2305216 Feb 23 deepvariant 652 -rwxrwxrwx 664832 Feb 23 deviceQuery 296520 -rwxrwxrwx 303633183 Feb 23 gatk-package-4.1.0.0-local.jar 20 -rwxrwxrwx 19197 Feb 23 gatk_cpu 572 -rwxrwxrwx 584152 Feb 23 genotypegvcf 146552 -rwxrwxrwx 150068464 Feb 23 glnexus 2132 -rwxrwxrwx 2182792 Feb 23 htvc 252 -rwxrwxrwx 255960 Feb 23 indexgvcf 220 -rwxrwxrwx 223168 Feb 23 licenseManagerTool 324 -rwxrwxrwx 329688 Feb 23 licenseinfo 220 -rwxrwxrwx 223104 Feb 23 licensereturn 44 -rwxrwxrwx 42968 Feb 23 markQueryName 32 -rwxrwxrwx 31848 Feb 23 mergegvcf_humanpar 1824 -rwxrwxrwx 1867488 Feb 23 mutect 12 -rwxrwxrwx 10120 Feb 23 pb_driver 1000 -rwxrwxrwx 1022960 Feb 23 postsort 316 -rwxrwxrwx 321512 Feb 23 samtoolsmpileup 340 -rwxrwxrwx 346616 Feb 23 somaticsniper 432 -rwxrwxrwx 440296 Feb 23 sort 364 -rwxrwxrwx 370976 Feb 23 splitncigar 2920 -rwxrwxrwx 2989784 Feb 23 star 624 -rwxrwxrwx 636896 Feb 23 starfusion 468 -rwxrwxrwx 477168 Feb 23 trioCombineGVCF 768 -rwxrwxrwx 784640 Feb 23 variantfiltration 312 -rwxrwxrwx 317400 Feb 23 varscan 792 -rwxrwxrwx 809120 Feb 23 vqsr ```