[log] WGS + Azure:GPU-NC12s-v2(P100-16GB x2) === ###### tags: `Parabricks-v3.5` ###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析` <br> [TOC] <br> ## 資料集來源 [WGS-LIS-AI018A](/Bx3R_i2wSfmDl1cQE0w3OA#WGS) <br> ## Azure VM 硬體 ### GPU 位於「**N 系列**」底下 [![](https://i.imgur.com/y0h3agW.png)](https://i.imgur.com/y0h3agW.png) - 標準 NC12s_v2 ![](https://i.imgur.com/XrQn3Kg.png) ### GPU 資訊 - GP100GL [Tesla P100 PCIe 16GB] x 2張 ``` description: 3D controller product: GP100GL [Tesla P100 PCIe 16GB] vendor: NVIDIA Corporation physical id: 2 bus info: pci@0001:00:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: bus_master cap_list configuration: driver=nvidia latency=0 ``` - 單張價格 https://www.ruten.com.tw/item/show?22112358039141 ![](https://i.imgur.com/k34Cuix.png) - [官方介紹(NVIDIA® TESLA® P100 GPU 加速器)](https://www.nvidia.com/content/dam/en-zz/zh_tw/Solutions/design-visualization/grid-vpc-vapps/235448_Tesla_P100_PCle_DS_NV_R10_A4_PressReady_TW.PDF) ![](https://i.imgur.com/gPujh0B.png) <br> ### CPU 資訊 - model name: `Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz` - 1(插槽) x 12(核/插槽) x 1(超執行緒/核) = 12條超執行緒 - physical id: 0 - cpu cores: 12 - processor: 0~11 - 超執行緒/核: 1 (每核是單執行緒) <br> <hr> <br> ## VM 之間的效能比較 | Program | NC6s-v2 | ==NC12s-v2== | NC24s-v2 | NC24s-v3 | NC24rs-v3 | AWS(T4x8) | | -------- | -------- | -------- | -------- | -------- | -------- | -------- | | GPU-BWA mem,<br>Sorting Phase-I | 5h 27m 04s | 2h 23m 23s | 0h 68m 16s | 0h 44m 39s | 0h 45m 34s | 0h 34m 32s | | Sorting Phase-II | 0h 07m 00s | 0h 04m 00s | 0h 02m20 s | 0h 02m 20s | 0h 02m 30s | 0h 01m 50s | | Marking Duplicates,<br>BQSR | 0h 44m 51s | 0h 16m 31s | 0h 07m 18s | 0h 05m 58s | 0h 08m 52s | 0h 03m 21s | | GPU-GATK4 HaplotypeCaller | 1h 48m 47s | 0h 47m 49s | 0h 23m 17s | 0h 22m 48s | 0h 20m 17s | 0h 14m 3s | | **Total** | **8h 07m 42s** | **3h 31m 43s** | **1h 41m 11s** | **1h 15m 45s** | **1h 17m 12s** | **0h 53m 46s** | - ### AWS(T4x8) (2021/07/14) - **GPU** Tesla T4 (15109MiB) x 8張 - **CPU** 1(插槽) x 77(核/插槽) x 1(超執行緒/核) = 77條超執行緒 (硬體執行緒) ( Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz ) - **RAM** 320GB <br> <hr> <br> ## Log (VM: NC12s-v2 + germline) ```bash= pbrun germline \ --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ --in-fq dataset/WGS-LIS-AI018A_R1.fastq.gz \ dataset/WGS-LIS-AI018A_R2.fastq.gz \ --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ --out-bam output.bam \ --out-variants output.vcf \ --out-recal-file report.txt \ --x3 ``` ```= $ pbrun germline \ > --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \ > --in-fq dataset/WGS-LIS-AI018A_R1.fastq.gz \ > dataset/WGS-LIS-AI018A_R2.fastq.gz \ > --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \ > --out-bam output.bam \ > --out-variants output.vcf \ > --out-recal-file report.txt \ > --x3 Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation [Parabricks Options Mesg]: Automatically generating ID prefix [Parabricks Options Mesg]: Read group created for /uploads/workspace/dataset/WGS-LIS-AI018A_R1.fastq.gz and /uploads/workspace/dataset/WGS-LIS-AI018A_R2.fastq.gz [Parabricks Options Mesg]: @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1 docker run --gpus all -u=1000:1000 --rm -w=/uploads/workspace --net=host -v /opt/parabricks:/INSTALL/ -v /uploads/workspace/HKQ9D0AZ:/uploads/workspace/HKQ9D0AZ -v /uploads/workspace:/uploads/workspace -v /uploads/workspace/parabricks_sample/Ref:/uploads/workspace/parabricks_sample/Ref -v /uploads/workspace/dataset:/uploads/workspace/dataset parabricks/release:v3.5.0 fq2bam --ref /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-fq /uploads/workspace/dataset/WGS-LIS-AI018A_R1.fastq.gz /uploads/workspace/dataset/WGS-LIS-AI018A_R2.fastq.gz @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1 --knownSites /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz --out-bam /uploads/workspace/output.bam --out-recal-file /uploads/workspace/report.txt --memory-limit 110 --num-cpu-threads 0 --tmp-dir /uploads/workspace/HKQ9D0AZ --num-gpus 2 --x3 Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation [Parabricks Options Mesg]: Checking argument compatibility [Parabricks Options Mesg]: Read group created for /uploads/workspace/dataset/WGS-LIS-AI018A_R1.fastq.gz and /uploads/workspace/dataset/WGS-LIS-AI018A_R2.fastq.gz [Parabricks Options Mesg]: @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1 g 2 b 0 B 2 P 4 s 1 r 0 o 2 m 1 z 4 f 2 v 0 M 2 name /uploads/workspace/output.bam report /uploads/workspace/report.txt K /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz /usr/local/cuda/.pb/binaries//bin/bwa mem /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /uploads/workspace/dataset/WGS-LIS-AI018A_R1.fastq.gz /uploads/workspace/dataset/WGS-LIS-AI018A_R2.fastq.gz @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1 -Z ./pbOpts.txt ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.5.0 || || GPU-BWA mem, Sorting Phase-I || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ [M::bwa_idx_load_from_disk] read 0 ALT contigs GPU-BWA mem ProgressMeter Reads Base Pairs Aligned WARNING The system has 12 threads, however recommended number of threads with 2 GPU is 24. The run might not finish or might have less than expected performance. [04:58:31] 5033176 760000000 [04:59:17] 10066352 1510000000 [05:00:04] 15099528 2270000000 [05:00:50] 20132704 3030000000 [05:01:36] 25165880 3800000000 [05:02:21] 30199056 4570000000 [05:03:06] 35232232 5320000000 [05:03:54] 40265408 6080000000 [05:04:40] 45298584 6830000000 [05:05:27] 50331760 7590000000 [05:06:13] 55364936 8350000000 [05:07:01] 60398112 9110000000 [05:07:48] 65431288 9890000000 [05:08:34] 70464464 10630000000 [05:09:21] 75497640 11400000000 [05:10:07] 80530816 12160000000 [05:10:55] 85563992 12920000000 [05:11:42] 90597168 13670000000 [05:12:28] 95630344 14430000000 [05:13:15] 100663520 15200000000 [05:14:01] 105696696 15960000000 [05:14:47] 110729872 16720000000 [05:15:34] 115763048 17480000000 [05:16:20] 120796224 18240000000 [05:17:04] 125829400 19000000000 [05:17:50] 130862576 19760000000 [05:18:36] 135895752 20520000000 [05:19:23] 140928928 21280000000 [05:20:11] 145962104 22040000000 [05:20:56] 150995280 22810000000 [05:21:41] 156028456 23550000000 [05:22:28] 161061632 24320000000 [05:23:16] 166094808 25070000000 [05:24:01] 171127984 25840000000 [05:24:47] 176161160 26620000000 [05:25:34] 181194336 27350000000 [05:26:22] 186227512 28120000000 [05:27:08] 191260688 28880000000 [05:27:54] 196293864 29640000000 [05:28:41] 201327040 30400000000 [05:29:27] 206360216 31150000000 [05:30:13] 211393392 31920000000 [05:30:59] 216426568 32680000000 [05:31:45] 221459744 33440000000 [05:32:31] 226492920 34210000000 [05:33:16] 231526096 34960000000 [05:34:02] 236559272 35730000000 [05:34:48] 241592448 36480000000 [05:35:33] 246625624 37240000000 [05:36:19] 251658800 37990000000 [05:37:05] 256691976 38760000000 [05:37:50] 261725152 39520000000 [05:38:36] 266758328 40280000000 [05:39:21] 271791504 41040000000 [05:40:08] 276824680 41790000000 [05:40:54] 281857856 42560000000 [05:41:40] 286891032 43310000000 [05:42:27] 291924208 44080000000 [05:43:12] 296957384 44830000000 [05:43:57] 301990560 45600000000 [05:44:44] 307023736 46360000000 [05:45:30] 312056912 47120000000 [05:46:16] 317090088 47880000000 [05:47:01] 322123264 48640000000 [05:47:46] 327156440 49400000000 [05:48:32] 332189616 50170000000 [05:49:18] 337222792 50920000000 [05:50:03] 342255968 51680000000 [05:50:49] 347289144 52440000000 [05:51:36] 352322320 53210000000 [05:52:22] 357355496 53960000000 [05:53:09] 362388672 54730000000 [05:53:55] 367421848 55480000000 [05:54:43] 372455024 56240000000 [05:55:29] 377488200 57000000000 [05:56:15] 382521376 57770000000 [05:57:02] 387554552 58520000000 [05:57:49] 392587728 59280000000 [05:58:35] 397620904 60030000000 [05:59:20] 402654080 60800000000 [06:00:06] 407687256 61550000000 [06:00:52] 412720432 62320000000 [06:01:39] 417753608 63080000000 [06:02:25] 422786784 63830000000 [06:03:10] 427819960 64590000000 [06:03:58] 432853136 65360000000 [06:04:43] 437886312 66120000000 [06:05:30] 442919488 66880000000 [06:06:16] 447952664 67650000000 [06:07:02] 452985840 68390000000 [06:07:49] 458019016 69160000000 [06:08:34] 463052192 69910000000 [06:09:20] 468085368 70680000000 [06:10:07] 473118544 71450000000 [06:10:55] 478151720 72200000000 [06:11:40] 483184896 72960000000 [06:12:28] 488218072 73720000000 [06:13:15] 493251248 74480000000 [06:14:01] 498284424 75240000000 [06:14:46] 503317600 76000000000 [06:15:34] 508350776 76770000000 [06:16:20] 513383952 77510000000 [06:17:06] 518417128 78270000000 [06:17:53] 523450304 79050000000 [06:18:40] 528483480 79810000000 [06:19:25] 533516656 80560000000 [06:20:11] 538549832 81330000000 [06:20:57] 543583008 82090000000 [06:21:43] 548616184 82840000000 [06:22:29] 553649360 83590000000 [06:23:14] 558682536 84360000000 [06:24:01] 563715712 85120000000 [06:24:47] 568748888 85880000000 [06:25:32] 573782064 86630000000 [06:26:18] 578815240 87390000000 [06:27:05] 583848416 88160000000 [06:27:51] 588881592 88920000000 [06:28:37] 593914768 89680000000 [06:29:23] 598947944 90450000000 [06:30:09] 603981120 91190000000 [06:30:55] 609014296 91960000000 [06:31:43] 614047472 92730000000 [06:32:28] 619080648 93480000000 [06:33:14] 624113824 94240000000 [06:34:00] 629147000 95000000000 [06:34:45] 634180176 95760000000 [06:35:31] 639213352 96510000000 [06:36:17] 644246528 97260000000 [06:37:03] 649279704 98040000000 [06:37:49] 654312880 98810000000 [06:38:34] 659346056 99550000000 [06:39:20] 664379232 100310000000 [06:40:05] 669412408 101080000000 [06:40:51] 674445584 101840000000 [06:41:36] 679478760 102610000000 [06:42:23] 684511936 103350000000 [06:43:09] 689545112 104120000000 [06:43:56] 694578288 104870000000 [06:44:42] 699611464 105650000000 [06:45:29] 704644640 106390000000 [06:46:14] 709677816 107160000000 [06:47:01] 714710992 107910000000 [06:47:47] 719744168 108680000000 [06:48:34] 724777344 109450000000 [06:49:21] 729810520 110200000000 [06:50:06] 734843696 110960000000 [06:50:53] 739876872 111710000000 [06:51:39] 744910048 112480000000 [06:52:25] 749943224 113230000000 [06:53:11] 754976400 114010000000 [06:53:59] 760009576 114770000000 [06:54:45] 765042752 115520000000 [06:55:30] 770075928 116280000000 [06:56:16] 775109104 117050000000 [06:57:03] 780142280 117800000000 [06:57:48] 785175456 118560000000 [06:58:35] 790208632 119320000000 [06:59:21] 795241808 120090000000 [07:00:08] 800274984 120850000000 [07:00:54] 805308160 121600000000 [07:01:41] 810341336 122360000000 [07:02:28] 815374512 123120000000 [07:03:14] 820407688 123890000000 [07:03:59] 825440864 124640000000 [07:04:45] 830474040 125400000000 [07:05:30] 835507216 126160000000 [07:06:16] 840540392 126910000000 [07:07:01] 845573568 127680000000 [07:07:47] 850606744 128450000000 [07:08:32] 855639920 129190000000 [07:09:18] 860673096 129960000000 [07:10:04] 865706272 130720000000 [07:10:49] 870739448 131470000000 [07:11:35] 875772624 132230000000 [07:12:21] 880805800 133000000000 [07:13:08] 885838976 133770000000 [07:13:54] 890872152 134530000000 [07:14:40] 895905328 135280000000 [07:15:27] 900938504 136050000000 [07:16:12] 905971680 136790000000 [07:16:58] 911004856 137560000000 [07:17:43] 916038032 138320000000 [07:18:31] 921071208 139070000000 [07:19:19] 926104384 139840000000 [07:20:04] 931137560 140600000000 GPU-BWA Mem time: 8595.545476 seconds GPU-BWA Mem is finished. GPU Sorting, Marking Dups, BQSR ProgressMeter SAM Entries Completed Total GPU-BWA Mem + Sorting + MarkingDups + BQSR Generation + BAM writing Processing time: 8595.546564 seconds [main] CMD: PARABRICKS mem -Z ./pbOpts.txt /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /uploads/workspace/dataset/WGS-LIS-AI018A_R1.fastq.gz /uploads/workspace/dataset/WGS-LIS-AI018A_R2.fastq.gz @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1 [main] Real time: 8598.889 sec; CPU: 102580.657 sec ------------------------------------------------------------------------------ || Program: GPU-BWA mem, Sorting Phase-I || || Version: v3.5.0 || || Start Time: Fri Jun 11 04:57:27 2021 || || End Time: Fri Jun 11 07:20:50 2021 || || Total Time: 143 minutes 23 seconds || ------------------------------------------------------------------------------ /usr/local/cuda/.pb/binaries//bin/sort -sort_unmapped -ft 10 -gb 110 ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.5.0 || || Sorting Phase-II || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ progressMeter - Percentage [07:20:51] 0.0 0.00 GB [07:21:01] 3.0 3.25 GB [07:21:11] 6.8 3.87 GB [07:21:21] 11.2 3.52 GB [07:21:31] 14.8 3.44 GB [07:21:41] 19.4 3.61 GB [07:21:51] 23.0 3.35 GB [07:22:01] 26.6 3.21 GB [07:22:11] 30.7 3.22 GB [07:22:21] 35.3 3.28 GB [07:22:31] 39.4 3.25 GB [07:22:41] 43.8 3.33 GB [07:22:51] 48.7 3.37 GB [07:23:01] 52.1 3.43 GB [07:23:11] 56.2 3.34 GB [07:23:21] 60.7 2.98 GB [07:23:31] 63.9 3.26 GB [07:23:41] 70.2 3.13 GB [07:23:51] 73.9 3.29 GB [07:24:01] 78.0 3.31 GB [07:24:11] 82.8 3.45 GB [07:24:21] 86.7 3.29 GB [07:24:31] 92.1 2.20 GB [07:24:41] 99.9 0.60 GB Sorting and Marking: 240.015 seconds ------------------------------------------------------------------------------ || Program: Sorting Phase-II || || Version: v3.5.0 || || Start Time: Fri Jun 11 07:20:51 2021 || || End Time: Fri Jun 11 07:24:51 2021 || || Total Time: 4 minutes 0 seconds || ------------------------------------------------------------------------------ /usr/local/cuda/.pb/binaries//bin/postsort /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta -o /uploads/workspace/output.bam -sort_unmapped -ft 4 -wt 2 -zt 3 -bq 2 -gb 110 -a /uploads/workspace/report.txt /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.5.0 || || Marking Duplicates, BQSR || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ progressMeter - Percentage [07:25:02] 0.0 9.71 GB [07:25:12] 0.0 15.22 GB [07:25:22] 0.0 19.86 GB [07:25:32] 0.0 26.28 GB [07:25:42] 0.0 33.82 GB [07:25:52] 0.0 45.06 GB [07:26:02] 0.0 54.58 GB [07:26:12] 0.0 60.92 GB [07:26:22] 0.0 64.16 GB [07:26:32] 0.0 66.54 GB [07:26:42] 0.0 68.99 GB [07:26:52] 0.0 71.76 GB [07:27:02] 0.0 74.56 GB [07:27:12] 0.0 80.13 GB [07:27:22] 0.0 84.38 GB [07:27:32] 0.0 89.65 GB [07:27:42] 0.0 94.51 GB [07:27:52] 0.0 96.05 GB [07:28:02] 0.0 99.95 GB [07:28:12] 0.0 102.75 GB [07:28:22] 0.0 102.75 GB [07:28:32] 2.5 95.61 GB [07:28:42] 5.5 88.09 GB [07:28:52] 8.1 81.00 GB [07:29:02] 9.8 79.29 GB [07:29:12] 12.6 74.03 GB [07:29:22] 15.0 68.13 GB [07:29:32] 17.5 60.15 GB [07:29:42] 20.1 51.89 GB [07:29:52] 22.5 45.13 GB [07:30:02] 25.4 37.34 GB [07:30:12] 28.0 32.15 GB [07:30:22] 30.7 27.70 GB [07:30:32] 33.0 24.10 GB [07:30:42] 35.9 14.58 GB [07:30:52] 38.5 5.88 GB [07:31:02] 40.4 1.55 GB [07:31:12] 41.5 1.91 GB [07:31:22] 43.0 1.91 GB [07:31:32] 44.4 1.87 GB [07:31:42] 45.1 1.99 GB [07:31:52] 45.6 1.47 GB [07:32:02] 46.5 2.36 GB [07:32:12] 47.2 2.25 GB [07:32:22] 49.0 2.66 GB [07:32:32] 50.3 1.94 GB [07:32:42] 51.6 1.48 GB [07:32:52] 52.0 1.48 GB [07:33:02] 52.4 1.86 GB [07:33:12] 53.1 3.51 GB [07:33:22] 54.6 2.37 GB [07:33:32] 55.9 1.87 GB [07:33:42] 57.1 1.60 GB [07:33:52] 57.5 1.51 GB [07:34:02] 57.9 1.49 GB [07:34:12] 58.7 1.54 GB [07:34:22] 59.5 2.64 GB [07:34:32] 60.9 2.40 GB [07:34:42] 62.1 1.87 GB [07:34:52] 62.9 1.52 GB [07:35:02] 63.3 1.72 GB [07:35:12] 64.4 1.49 GB [07:35:22] 65.3 1.83 GB [07:35:32] 66.1 1.85 GB [07:35:42] 67.8 1.86 GB [07:35:52] 69.0 1.93 GB [07:36:02] 70.8 1.56 GB [07:36:12] 71.3 1.50 GB [07:36:22] 71.6 1.51 GB [07:36:32] 72.4 1.56 GB [07:36:42] 73.4 1.93 GB [07:36:52] 74.3 2.56 GB [07:37:02] 75.6 2.03 GB [07:37:12] 76.3 1.56 GB [07:37:22] 76.6 2.03 GB [07:37:32] 77.4 1.83 GB [07:37:42] 78.5 2.06 GB [07:37:52] 79.7 1.50 GB [07:38:02] 80.9 1.56 GB [07:38:12] 81.3 1.56 GB [07:38:22] 81.8 2.05 GB [07:38:32] 82.6 2.26 GB [07:38:42] 83.7 2.74 GB [07:38:52] 84.9 2.47 GB [07:39:02] 85.3 4.05 GB [07:39:12] 86.6 1.75 GB [07:39:22] 87.4 1.59 GB [07:39:32] 87.9 1.87 GB [07:39:42] 88.8 1.84 GB [07:39:52] 90.1 1.84 GB [07:40:02] 91.2 1.85 GB [07:40:12] 92.1 1.92 GB [07:40:22] 92.5 1.89 GB [07:40:32] 94.5 3.47 GB [07:40:42] 94.6 5.37 GB [07:40:52] 95.7 6.45 GB [07:41:02] 98.0 3.46 GB [07:41:12] 98.2 0.53 GB [07:41:22] 100.0 0.00 GB BQSR and writing final BAM: 990.037 seconds ------------------------------------------------------------------------------ || Program: Marking Duplicates, BQSR || || Version: v3.5.0 || || Start Time: Fri Jun 11 07:24:51 2021 || || End Time: Fri Jun 11 07:41:22 2021 || || Total Time: 16 minutes 31 seconds || ------------------------------------------------------------------------------ docker run --gpus all -u=1000:1000 --rm -w=/uploads/workspace --net=host -v /opt/parabricks:/INSTALL/ -v /uploads/workspace/HKQ9D0AZ:/uploads/workspace/HKQ9D0AZ -v /uploads/workspace:/uploads/workspace -v /uploads/workspace/parabricks_sample/Ref:/uploads/workspace/parabricks_sample/Ref -v /uploads/workspace/dataset:/uploads/workspace/dataset parabricks/release:v3.5.0 haplotypecaller --ref /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-bam /uploads/workspace/output.bam --out-variants /uploads/workspace/output.vcf --ploidy 2 --num-htvc-threads 5 --in-recal-file /uploads/workspace/report.txt --tmp-dir /uploads/workspace/HKQ9D0AZ --num-gpus 2 --x3 Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation /usr/local/cuda/.pb/binaries//bin/htvc /uploads/workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /uploads/workspace/output.bam 2 -o /uploads/workspace/output.vcf -nt 5 -a /uploads/workspace/report.txt ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.5.0 || || GPU-GATK4 HaplotypeCaller || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ ProgressMeter - Current-Locus Elapsed-Minutes Regions-Processed Regions/Minute 0 /uploads/workspace/output.bam /uploads/workspace/output.vcf [07:42:01] chr1:14500688 0.2 80087 480522 [07:42:11] chr1:24657456 0.3 140124 420372 [07:42:21] chr1:38174393 0.5 214146 428292 [07:42:31] chr1:52449540 0.7 291040 436560 [07:42:41] chr1:66383965 0.8 371021 445225 [07:42:51] chr1:81038143 1.0 453722 453722 [07:43:01] chr1:93038224 1.2 522109 447522 [07:43:11] chr1:105311910 1.3 592586 444439 [07:43:21] chr1:119860505 1.5 674281 449520 [07:43:31] chr1:124535925 1.7 704559 422735 [07:43:41] chr1:143615837 1.8 711889 388303 [07:43:51] chr1:148233544 2.0 733308 366654 [07:44:01] chr1:149899153 2.2 741226 342104 [07:44:11] chr1:156753506 2.3 778438 333616 [07:44:21] chr1:169814297 2.5 854861 341944 [07:44:31] chr1:182025357 2.7 923245 346216 [07:44:41] chr1:194663999 2.8 994698 351069 [07:44:51] chr1:209356556 3.0 1075639 358546 [07:45:01] chr1:221390230 3.2 1144635 361463 [07:45:11] chr1:231988742 3.3 1206298 361889 [07:45:21] chr1:242025418 3.5 1266321 361806 [07:45:31] chr2:5385472 3.7 1330806 362947 [07:45:41] chr2:16411137 3.8 1394659 363824 [07:45:51] chr2:27878392 4.0 1457695 364423 [07:46:01] chr2:39979188 4.2 1528556 366853 [07:46:11] chr2:52497568 4.3 1602701 369854 [07:46:21] chr2:64756699 4.5 1671458 371435 [07:46:31] chr2:76060601 4.7 1734365 371649 [07:46:41] chr2:87993559 4.8 1800890 372597 [07:46:51] chr2:101059165 5.0 1863482 372696 [07:47:01] chr2:114407935 5.2 1936724 374849 [07:47:11] chr2:126393520 5.3 2004906 375919 [07:47:21] chr2:139915067 5.5 2081657 378483 [07:47:31] chr2:151127977 5.7 2145613 378637 [07:47:41] chr2:162441526 5.8 2209048 378693 [07:47:51] chr2:174916778 6.0 2280205 380034 [07:48:01] chr2:187339159 6.2 2351043 381250 [07:48:11] chr2:199295946 6.3 2418041 381795 [07:48:21] chr2:211732625 6.5 2485899 382446 [07:48:31] chr2:223151979 6.7 2551423 382713 [07:48:41] chr2:234633532 6.8 2617648 383070 [07:48:51] chr3:2659014 7.0 2677052 382436 [07:49:01] chr3:14260579 7.2 2746016 383165 [07:49:11] chr3:27225552 7.3 2818476 384337 [07:49:21] chr3:40300778 7.5 2891801 385573 [07:49:31] chr3:54302246 7.7 2966285 386906 [07:49:41] chr3:67603192 7.8 3045721 388815 [07:49:51] chr3:79094316 8.0 3112022 389002 [07:50:01] chr3:91151967 8.2 3182367 389677 [07:50:11] chr3:104030140 8.3 3250400 390048 [07:50:21] chr3:115588770 8.5 3317864 390336 [07:50:31] chr3:128884645 8.7 3391866 391369 [07:50:41] chr3:140020578 8.8 3454414 391065 [07:50:51] chr3:152351866 9.0 3521982 391331 [07:51:01] chr3:165671877 9.2 3595149 392198 [07:51:11] chr3:180532719 9.3 3674589 393705 [07:51:21] chr3:194054298 9.5 3750995 394841 [07:51:31] chr4:6547156 9.7 3811477 394290 [07:51:41] chr4:15700590 9.8 3867032 393257 [07:51:51] chr4:28593328 10.0 3939166 393916 [07:52:01] chr4:39595098 10.2 4003230 393760 [07:52:11] chr4:49943718 10.3 4061990 393095 [07:52:21] chr4:57124608 10.5 4098301 390314 [07:52:31] chr4:67315100 10.7 4158756 389883 [07:52:41] chr4:79468630 10.8 4227538 390234 [07:52:51] chr4:91646329 11.0 4296215 390565 [07:53:01] chr4:106079802 11.2 4374414 391738 [07:53:11] chr4:118415939 11.3 4445002 392206 [07:53:21] chr4:131630185 11.5 4521339 393159 [07:53:31] chr4:143222388 11.7 4590107 393437 [07:53:41] chr4:154602902 11.8 4654058 393300 [07:53:51] chr4:165652533 12.0 4718720 393226 [07:54:01] chr4:177273313 12.2 4787003 393452 [07:54:11] chr4:187401484 12.3 4849125 393172 [07:54:21] chr5:6835160 12.5 4902425 392194 [07:54:31] chr5:18359857 12.7 4969584 392335 [07:54:41] chr5:30215804 12.8 5037279 392515 [07:54:51] chr5:41457583 13.0 5101484 392421 [07:55:01] chr5:51134205 13.2 5144004 390683 [07:55:11] chr5:60234910 13.3 5196181 389713 [07:55:21] chr5:76271979 13.5 5275340 390765 [07:55:31] chr5:89639738 13.7 5349242 391407 [07:55:41] chr5:103161462 13.8 5421517 391916 [07:55:51] chr5:114345391 14.0 5484227 391730 [07:56:01] chr5:126839911 14.2 5553554 392015 [07:56:11] chr5:138950150 14.3 5618782 392008 [07:56:21] chr5:151113533 14.5 5684492 392033 [07:56:31] chr5:162820578 14.7 5748580 391948 [07:56:41] chr5:173841572 14.8 5812421 391848 [07:56:51] chr6:2174357 15.0 5869930 391328 [07:57:01] chr6:13099201 15.2 5934357 391276 [07:57:11] chr6:24095879 15.3 5999868 391295 [07:57:21] chr6:39239708 15.5 6065193 391302 [07:57:31] chr6:51763120 15.7 6136623 391699 [07:57:41] chr6:66043158 15.8 6212565 392372 [07:57:51] chr6:76804587 16.0 6273352 392084 [07:58:01] chr6:88463928 16.2 6341271 392243 [07:58:11] chr6:101111852 16.3 6414254 392709 [07:58:21] chr6:112592122 16.5 6479721 392710 [07:58:31] chr6:123057448 16.7 6540159 392409 [07:58:41] chr6:138523068 16.8 6620869 393318 [07:58:51] chr6:150815990 17.0 6688369 393433 [07:59:01] chr6:162417538 17.2 6756212 393565 [07:59:11] chr7:1123051 17.3 6809359 392847 [07:59:21] chr7:10987151 17.5 6870476 392598 [07:59:31] chr7:20207963 17.7 6926752 392080 [07:59:41] chr7:30129338 17.8 6984407 391648 [07:59:51] chr7:41639883 18.0 7049658 391647 [08:00:01] chr7:52742366 18.2 7114215 391608 [08:00:11] chr7:64492688 18.3 7175762 391405 [08:00:21] chr7:74404712 18.5 7234246 391040 [08:00:31] chr7:86289350 18.7 7303051 391234 [08:00:41] chr7:99105435 18.8 7374483 391565 [08:00:51] chr7:111052550 19.0 7441449 391655 [08:01:01] chr7:124526344 19.2 7512482 391955 [08:01:11] chr7:137635062 19.3 7585855 392371 [08:01:21] chr7:151089312 19.5 7656670 392649 [08:01:31] chr8:287874 19.7 7707403 391901 [08:01:41] chr8:9662285 19.8 7759820 391251 [08:01:51] chr8:17279769 20.0 7809806 390490 [08:02:01] chr8:28478363 20.2 7877124 390601 [08:02:11] chr8:41342239 20.3 7947893 390879 [08:02:21] chr8:54892798 20.5 8018077 391125 [08:02:31] chr8:67396740 20.7 8086401 391277 [08:02:41] chr8:79891195 20.8 8154631 391422 [08:02:51] chr8:91257348 21.0 8216994 391285 [08:03:01] chr8:105451035 21.2 8293247 391806 [08:03:11] chr8:119231857 21.3 8366882 392197 [08:03:21] chr8:131015989 21.5 8431585 392166 [08:03:31] chr8:141268771 21.7 8493136 391990 [08:03:41] chr9:7171130 21.8 8556734 391911 [08:03:51] chr9:17121561 22.0 8617745 391715 [08:04:01] chr9:29534331 22.2 8689822 392022 [08:04:11] chr9:41145384 22.3 8757621 392132 [08:04:21] chr9:64636773 22.5 8801009 391155 [08:04:31] chr9:75278174 22.7 8856207 390715 [08:04:41] chr9:88809569 22.8 8932383 391199 [08:04:51] chr9:99297494 23.0 8992000 390956 [08:05:01] chr9:112094310 23.2 9062329 391179 [08:05:11] chr9:123849335 23.3 9128069 391202 [08:05:21] chr9:134347118 23.5 9189444 391040 [08:05:31] chr10:5716800 23.7 9248261 390771 [08:05:41] chr10:15129445 23.8 9306456 390480 [08:05:51] chr10:25689406 24.0 9365305 390221 [08:06:01] chr10:37204618 24.2 9432999 390330 [08:06:11] chr10:43641597 24.3 9467344 389068 [08:06:21] chr10:49655960 24.5 9499560 387737 [08:06:31] chr10:58411180 24.7 9551834 387236 [08:06:41] chr10:69004578 24.8 9613293 387112 [08:06:51] chr10:81196725 25.0 9680930 387237 [08:07:01] chr10:94171005 25.2 9754419 387592 [08:07:11] chr10:105527970 25.3 9816688 387500 [08:07:21] chr10:117499160 25.5 9885942 387684 [08:07:31] chr10:127487955 25.7 9945376 387482 [08:07:41] chr11:4074910 25.8 10004600 387274 [08:07:51] chr11:13670231 26.0 10062124 387004 [08:08:01] chr11:23404791 26.2 10120031 386752 [08:08:11] chr11:35039882 26.3 10187890 386881 [08:08:21] chr11:45465454 26.5 10248907 386751 [08:08:31] chr11:58655873 26.7 10322730 387102 [08:08:41] chr11:70550335 26.8 10389725 387194 [08:08:51] chr11:82699128 27.0 10458028 387334 [08:09:01] chr11:95183950 27.2 10529903 387603 [08:09:11] chr11:105404230 27.3 10591935 387509 [08:09:21] chr11:118113579 27.5 10663472 387762 [08:09:31] chr11:127828778 27.7 10720235 387478 [08:09:41] chr12:3772600 27.8 10782360 387390 [08:09:51] chr12:13617594 28.0 10838371 387084 [08:10:01] chr12:25204741 28.2 10906134 387200 [08:10:11] chr12:35140535 28.3 10964515 386982 [08:10:21] chr12:48115201 28.5 11037386 387276 [08:10:31] chr12:59572653 28.7 11100294 387219 [08:10:41] chr12:70362957 28.8 11162926 387153 [08:10:51] chr12:82252700 29.0 11229541 387225 [08:11:01] chr12:94555142 29.2 11296964 387324 [08:11:11] chr12:106257326 29.3 11364015 387409 [08:11:21] chr12:118113357 29.5 11429804 387450 [08:11:31] chr12:128428591 29.7 11490245 387311 [08:11:41] chr13:17433335 29.8 11533593 386600 [08:11:51] chr13:23620616 30.0 11571234 385707 [08:12:01] chr13:32678185 30.2 11625059 385361 [08:12:11] chr13:42877058 30.3 11685127 385223 [08:12:21] chr13:54364675 30.5 11750218 385253 [08:12:31] chr13:66182336 30.7 11817325 385347 [08:12:41] chr13:76636796 30.8 11878766 385257 [08:12:51] chr13:88338973 31.0 11946401 385367 [08:13:01] chr13:100002100 31.2 12014000 385475 [08:13:11] chr13:109991927 31.3 12074522 385357 [08:13:21] chr14:23102151 31.5 12128035 385016 [08:13:31] chr14:34675139 31.7 12195247 385113 [08:13:41] chr14:46651191 31.8 12265052 385289 [08:13:51] chr14:56860729 32.0 12325565 385173 [08:14:01] chr14:69162929 32.2 12394196 385311 [08:14:11] chr14:80155066 32.3 12456642 385256 [08:14:21] chr14:95054293 32.5 12532511 385615 [08:14:31] chr15:19084501 32.7 12599256 385691 [08:14:41] chr15:26303958 32.8 12638761 384936 [08:14:51] chr15:41654385 33.0 12716069 385335 [08:15:01] chr15:53956745 33.2 12782704 385408 [08:15:11] chr15:64262294 33.3 12843781 385313 [08:15:21] chr15:76622127 33.5 12910621 385391 [08:15:31] chr15:89183873 33.7 12980934 385572 [08:15:41] chr15:99119935 33.8 13040292 385427 [08:15:51] chr16:6239850 34.0 13094052 385119 [08:16:01] chr16:17903960 34.2 13159860 385166 [08:16:11] chr16:30575895 34.3 13230399 385351 [08:16:21] chr16:46843169 34.5 13264044 384465 [08:16:31] chr16:50356552 34.7 13281858 383130 [08:16:41] chr16:50534401 34.8 13282816 381324 [08:16:51] chr16:51033382 35.0 13285332 379580 [08:17:01] chr16:56668620 35.2 13317997 378710 [08:17:11] chr16:66340739 35.3 13372556 378468 [08:17:21] chr16:78494349 35.5 13441307 378628 [08:17:31] chr16:86764727 35.7 13495850 378388 [08:17:41] chr17:6004797 35.8 13550193 378144 [08:17:51] chr17:16103817 36.0 13611035 378084 [08:18:01] chr17:27335879 36.2 13665109 377837 [08:18:11] chr17:32567997 36.3 13691820 376839 [08:18:21] chr17:48100505 36.5 13762051 377042 [08:18:31] chr17:60715188 36.7 13833119 377266 [08:18:41] chr17:72743813 36.8 13899135 377352 [08:18:51] chr17:82857459 37.0 13959148 377274 [08:19:01] chr18:5035095 37.2 13992067 376468 [08:19:11] chr18:15772749 37.3 14055776 376494 [08:19:21] chr18:31862299 37.5 14137727 377006 [08:19:31] chr18:44404582 37.7 14206800 377171 [08:19:41] chr18:56145515 37.8 14272027 377234 [08:19:51] chr18:66806343 38.0 14334896 377234 [08:20:01] chr18:77450328 38.2 14400238 377298 [08:20:11] chr19:7204757 38.3 14458848 377187 [08:20:21] chr19:17092774 38.5 14515977 377038 [08:20:31] chr19:29687961 38.7 14571827 376857 [08:20:41] chr19:40055768 38.8 14629502 376725 [08:20:51] chr19:51993534 39.0 14696335 376829 [08:21:01] chr20:3734129 39.2 14753431 376683 [08:21:11] chr20:15580673 39.3 14822370 376839 [08:21:21] chr20:29164737 39.5 14893885 377060 [08:21:31] chr20:30969577 39.7 14904999 375756 [08:21:41] chr20:35390372 39.8 14927354 374745 [08:21:51] chr20:47510383 40.0 14993875 374846 [08:22:01] chr20:57993306 40.2 15055192 374818 [08:22:11] chr21:9887876 40.3 15104460 374490 [08:22:21] chr21:13003128 40.5 15111400 373120 [08:22:31] chr21:14630337 40.7 15121399 371837 [08:22:41] chr21:23375851 40.8 15171597 371549 [08:22:51] chr21:34766365 41.0 15236716 371627 [08:23:01] chr21:44735926 41.2 15294144 371517 [08:23:11] chr22:15566304 41.3 15320039 370646 [08:23:21] chr22:17812772 41.5 15334045 369495 [08:23:31] chr22:28266998 41.7 15391065 369385 [08:23:41] chr22:40363170 41.8 15457621 369504 [08:23:51] chr22:50231903 42.0 15515605 369419 [08:24:01] chrX:13080926 42.2 15588723 369693 [08:24:11] chrX:27940608 42.3 15663186 369996 [08:24:21] chrX:43070178 42.5 15739141 370332 [08:24:31] chrX:57364732 42.7 15809464 370534 [08:24:41] chrX:78412566 42.8 15904009 371299 [08:24:51] chrX:93091067 43.0 15982345 371682 [08:25:01] chrX:111542201 43.2 16072805 372342 [08:25:11] chrX:125447990 43.3 16142724 372524 [08:25:21] chrX:143433569 43.5 16233034 373173 [08:25:31] chrY:10934194 43.7 16323623 373823 [08:25:41] chr4_GL000008v2_random:139014 43.8 16365892 373366 [08:25:51] chr14_GL000225v1_random:129581 44.0 16372043 372091 [08:26:01] chr16_KI270728v1_random:926386 44.2 16378486 370833 [08:26:11] chr22_KI270738v1_random:47797 44.3 16391650 369736 [08:26:21] chrUn_KI270438v1:9583 44.5 16396350 368457 [08:26:31] chrUn_KI270538v1:57337 44.7 16398692 367134 [08:26:41] chrUn_KI270741v1:76717 44.8 16405209 365915 [08:26:51] chrUn_KI270747v1:134350 45.0 16410003 364666 [08:27:01] chrUn_KI270756v1:23713 45.2 16414833 363428 [08:27:11] chr1_KI270766v1_alt:19167 45.3 16420437 362215 [08:27:21] chr4_KI270788v1_alt:62388 45.5 16429483 361087 [08:27:31] chr15_KI270905v1_alt:4891153 45.7 16525403 361870 [08:27:41] chrUn_JTFH01000004v1_decoy:13531 45.8 16567966 361482 [08:27:51] chrUn_JTFH01000361v1_decoy:1740 46.0 16572964 360281 [08:28:01] chrUn_JTFH01000982v1_decoy:3011 46.2 16577135 359071 [08:28:11] chrUn_JTFH01001743v1_decoy:4708 46.3 16583142 357909 [08:28:21] HLA-DRB1*15:02:01:10276 46.5 16588330 356738 [08:28:31] HLA-DRB1*15:02:01:10276 46.7 16588330 355464 [08:28:41] HLA-DRB1*15:02:01:10276 46.8 16588330 354199 [08:28:51] HLA-DRB1*15:02:01:10276 47.0 16588330 352943 [08:29:01] HLA-DRB1*15:02:01:10276 47.2 16588330 351696 [08:29:11] HLA-DRB1*15:02:01:10276 47.3 16588330 350457 Total time taken: 2855.28 ------------------------------------------------------------------------------ || Program: GPU-GATK4 HaplotypeCaller || || Version: v3.5.0 || || Start Time: Fri Jun 11 07:41:35 2021 || || End Time: Fri Jun 11 08:29:24 2021 || || Total Time: 47 minutes 49 seconds || ------------------------------------------------------------------------------ ``` - log 中出現的 binary 檔位置，並非是真正位置 ``` /usr/local/cuda/.pb/binaries//bin/bwa ``` - binary 檔的真正位置在 ``` /usr/local/cuda-10.1/.pb/binaries/bin ``` ```bash= usr/local/cuda-10.1/.pb/binaries/bin$ ls -ls total 467644 360 -rwxrwxrwx 367256 Feb 23 applyBQSR 1192 -rwxrwxrwx 1218248 Feb 23 bamreadcount 296 -rwxrwxrwx 301024 Feb 23 bcftoolscall 432 -rwxrwxrwx 440368 Feb 23 bcftoolsmpileup 96 -rwxrwxrwx 96584 Feb 23 bedcov 500 -rwxrwxrwx 510696 Feb 23 bqsr 2424 -rwxrwxrwx 2481896 Feb 23 bwa 708 -rwxrwxrwx 723584 Feb 23 cnnscorevariants 220 -rwxrwxrwx 223192 Feb 23 cnvkit 844 -rwxrwxrwx 863712 Feb 23 collectmultiplemetrics 336 -rwxrwxrwx 342408 Feb 23 coverage 272 -rwxrwxrwx 276456 Feb 23 dbsnp 2252 -rwxrwxrwx 2305216 Feb 23 deepvariant 652 -rwxrwxrwx 664832 Feb 23 deviceQuery 296520 -rwxrwxrwx 303633183 Feb 23 gatk-package-4.1.0.0-local.jar 20 -rwxrwxrwx 19197 Feb 23 gatk_cpu 572 -rwxrwxrwx 584152 Feb 23 genotypegvcf 146552 -rwxrwxrwx 150068464 Feb 23 glnexus 2132 -rwxrwxrwx 2182792 Feb 23 htvc 252 -rwxrwxrwx 255960 Feb 23 indexgvcf 220 -rwxrwxrwx 223168 Feb 23 licenseManagerTool 324 -rwxrwxrwx 329688 Feb 23 licenseinfo 220 -rwxrwxrwx 223104 Feb 23 licensereturn 44 -rwxrwxrwx 42968 Feb 23 markQueryName 32 -rwxrwxrwx 31848 Feb 23 mergegvcf_humanpar 1824 -rwxrwxrwx 1867488 Feb 23 mutect 12 -rwxrwxrwx 10120 Feb 23 pb_driver 1000 -rwxrwxrwx 1022960 Feb 23 postsort 316 -rwxrwxrwx 321512 Feb 23 samtoolsmpileup 340 -rwxrwxrwx 346616 Feb 23 somaticsniper 432 -rwxrwxrwx 440296 Feb 23 sort 364 -rwxrwxrwx 370976 Feb 23 splitncigar 2920 -rwxrwxrwx 2989784 Feb 23 star 624 -rwxrwxrwx 636896 Feb 23 starfusion 468 -rwxrwxrwx 477168 Feb 23 trioCombineGVCF 768 -rwxrwxrwx 784640 Feb 23 variantfiltration 312 -rwxrwxrwx 317400 Feb 23 varscan 792 -rwxrwxrwx 809120 Feb 23 vqsr ``` <br> <hr> <br> ## Log (VM: NC12s-v2) (pipeline: deepvariant_germline) ```= ... BQSR and writing final BAM: 190.039 seconds ------------------------------------------------------------------------------ || Program: Marking Duplicates, BQSR || || Version: v3.5.0 || || Start Time: Tue Jun 22 09:00:34 2021 || || End Time: Tue Jun 22 09:03:45 2021 || || Total Time: 3 minutes 11 seconds || ------------------------------------------------------------------------------ Detected 2 CUDA Capable device(s), considering 2 device(s) CUDA Driver Version / Runtime Version 11.3 / 10.1 Using model for CUDA Capability Major/Minor version number: 60 ------------------------------------------------------------------------------ || Parabricks accelerated Genomics Pipeline || || Version v3.5.0 || || deepvariant || || Contact: Parabricks-Support@nvidia.com || ------------------------------------------------------------------------------ Invalid modelFileName /usr/local/cuda/.pb/binaries//model/60/shortread/deepvariant.eng. Exiting... Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation Please contact Parabricks-Support@nvidia.com for any questions There is a forum for Q&A as well at https://forums.developer.nvidia.com/c/healthcare/Parabricks/290 Exiting... Could not run deepvariant as part of deepvariant_germline pipeline Exiting pbrun ... ``` - Invalid modelFileName /usr/local/cuda/.pb/binaries//model/60/shortread/deepvariant.eng - 官方文件尚未公布 [`pbrun deepvariant_germline`](https://docs.nvidia.com/clara/parabricks/v3.5/text/deep_pipeline.html) 指令，有可能表示還在維護中