---
# System prepended metadata

title: '[log] germline: WGS + NHRI: A100x4'
tags: [Clara, Parabricks-v3.8, 基因體, 二級分析, NVIDIA, Parabricks]

---

[log] germline: WGS + NHRI: A100x4
===
###### tags: `Parabricks-v3.8`
###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析`

<br>

[TOC]

<br>

## 硬體資源
### Flavor
- ### [246900] a100-parabricks-4v
    - **GPU:** a100-40gb x 4張
        :::spoiler `$ nvidia-smi -L`
        ```= 
        GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-835407b2-058f-0ff9-938e-fca462f5beb7)
        GPU 1: NVIDIA A100-SXM4-40GB (UUID: GPU-fd747277-363e-0e90-1866-9512d2cc265e)
        GPU 2: NVIDIA A100-SXM4-40GB (UUID: GPU-48e6767e-5165-8dea-010c-56c016a9d6ad)
        GPU 3: NVIDIA A100-SXM4-40GB (UUID: GPU-c4595343-9ada-e53e-0032-ac1a3ec4ad4c)
        ```
        :::
        :::spoiler `$ lspci | grep -i nvida`
        ```= 
        07:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        0f:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        47:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        4e:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        87:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        90:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        b7:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        bd:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
        c4:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
        c5:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
        c6:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
        c7:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
        c8:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
        c9:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
        ```
        :::
    - **CPU:** 126 cores
        [全：256 = 2插槽 x 64(核/插槽) x 2(超執行緒/核)]
        :::spoiler `$ lscpu`
        ```= 
        Architecture:        x86_64
        CPU op-mode(s):      32-bit, 64-bit
        Byte Order:          Little Endian
        CPU(s):              256
        On-line CPU(s) list: 0-255
        Thread(s) per core:  2
        Core(s) per socket:  64
        Socket(s):           2
        NUMA node(s):        8
        Vendor ID:           AuthenticAMD
        CPU family:          23
        Model:               49
        Model name:          AMD EPYC 7742 64-Core Processor
        Stepping:            0
        CPU MHz:             3392.706
        CPU max MHz:         2250.0000
        CPU min MHz:         1500.0000
        BogoMIPS:            4491.41
        Virtualization:      AMD-V
        L1d cache:           32K
        L1i cache:           32K
        L2 cache:            512K
        L3 cache:            16384K
        NUMA node0 CPU(s):   0-15,128-143
        NUMA node1 CPU(s):   16-31,144-159
        NUMA node2 CPU(s):   32-47,160-175
        NUMA node3 CPU(s):   48-63,176-191
        NUMA node4 CPU(s):   64-79,192-207
        NUMA node5 CPU(s):   80-95,208-223
        NUMA node6 CPU(s):   96-111,224-239
        NUMA node7 CPU(s):   112-127,240-255
        Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
        ```
        :::
    - **RAM:** 480 GiB
        [全：1 TB]
        :::spoiler `$ lsmem`
        ```= 
        RANGE                                   SIZE  STATE REMOVABLE     BLOCK
        0x0000000000000000-0x00000000afffffff   2.8G online       yes      0-21
        0x0000000100000000-0x000000e04fffffff 893.3G online       yes   32-7177
        0x0000010000000000-0x0000011fffffffff   128G online       yes 8192-9215

        Memory block size:       128M
        Total online memory:       1T
        Total offline memory:      0B
        ```
        :::
        :::spoiler `$ cat /proc/meminfo`
        ```= 
        MemTotal:       1056634792 kB
        MemFree:        370156980 kB
        MemAvailable:   1021699836 kB
        Buffers:         6302856 kB
        Cached:         631340356 kB
        SwapCached:            0 kB
        Active:         124370696 kB
        Inactive:       525632312 kB
        Active(anon):   12042852 kB
        Inactive(anon):    70352 kB
        Active(file):   112327844 kB
        Inactive(file): 525561960 kB
        Unevictable:       19228 kB
        Mlocked:           19228 kB
        SwapTotal:             0 kB
        SwapFree:              0 kB
        Dirty:            519576 kB
        Writeback:        121432 kB
        AnonPages:      12347456 kB
        Mapped:          4525364 kB
        Shmem:            134200 kB
        KReclaimable:   19075040 kB
        Slab:           25545236 kB
        SReclaimable:   19075040 kB
        SUnreclaim:      6470196 kB
        KernelStack:       92496 kB
        PageTables:        68424 kB
        NFS_Unstable:          0 kB
        Bounce:                0 kB
        WritebackTmp:          0 kB
        CommitLimit:    528317396 kB
        Committed_AS:   49159716 kB
        VmallocTotal:   34359738367 kB
        VmallocUsed:     1672904 kB
        VmallocChunk:          0 kB
        Percpu:          1764352 kB
        HardwareCorrupted:     0 kB
        AnonHugePages:     47104 kB
        ShmemHugePages:        0 kB
        ShmemPmdMapped:        0 kB
        FileHugePages:         0 kB
        FilePmdMapped:         0 kB
        CmaTotal:              0 kB
        CmaFree:               0 kB
        HugePages_Total:       0
        HugePages_Free:        0
        HugePages_Rsvd:        0
        HugePages_Surp:        0
        Hugepagesize:       2048 kB
        Hugetlb:               0 kB
        DirectMap4k:    12491992 kB
        DirectMap2M:    232714240 kB
        DirectMap1G:    828375040 kB
        ```
        :::
    - **Network:** 100Gb

### Storage
:::spoiler `$ fio`
```
$ fio \
  --randrepeat=1 \
  --ioengine=libaio \
  --direct=1 \
  --gtod_reduce=1 \
  --name=fiotest \
  --filename=testfio \
  --bs=4k \
  --iodepth=64 \
  --size=8G \
  --readwrite=randrw \
  --rwmixread=75
```
:::
- ### 實際效能測試1
    |   | **速度** | **IOPS** |
    |---|---------|----------|
    | **Read** | 144 MiB/s (151MB/s) | 36.9k |
    | **Write** | 48.1MiB/s (50.4MB/s) | 12.3k |
    
- ### 實際效能測試2 (不同時間點)
    |   | **速度** | **IOPS** |
    |---|---------|----------|
    | **Read** | 142MiB/s (149MB/s) | 36.4k |
    | **Write** | 47.5MiB/s (49.8MB/s) | 12.1k |


<br>

<hr>

<br>

## 效能評估
| Programs | Round 1 | Round 2 | Round 3 | Round 4 |
|----------|--------:|--------:|--------:|--------:|
| GPU-BWA mem, Sorting Phase-I | 23m 07s | 23m 23s | 23m 19s | m s |
| Sorting Phase-II             |  1m 40s |  8m 10s | 9m 40s | m s |
| Marking Dmuplicates, BQSR    |  4m 51s | 17m 12s | 16m 51s | m s |
| GPU-GATK4 HaplotypeCaller    | 14m 03s | 15m 54s | 126m 54s |
| | | |
| **Total**                    | 43m 41s | 64m 39s | 2h 56m 44s |
| **Actual**                    | 44m 20s | 65m 12s | 2h 57m 23s |

- [[繁中] 官方效能](https://www.nvidia.com/zh-tw/clara/genomics/)
    [![](https://i.imgur.com/h5qNcsf.png)](https://i.imgur.com/h5qNcsf.png)
- [Accelerated Whole Genome Sequencing Analysis](https://www.robusthpc.com/nvidia-clara-parabricks/)
    ![](https://i.imgur.com/0oplNc1.png)


<br>

<hr>

<br>

## log
:::spoiler 點此查看完整 log

```=
# sh datasets/germline/WGS-LIS-AI018A_germline_pipeline.sh 
Tue May 24 18:02:55 CST 2022
start_time: 1653386575
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation


[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for /workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz and
/workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz
[Parabricks Options Mesg]: @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1


[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Read group created for /workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz and
/workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz
[Parabricks Options Mesg]: @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1
Namespace(append=None, bwa_options=None, fix_mate=None, fq_list=None, gpu_devices='all', in_fq=[['/workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz', '/workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz', '@RG\\tID:H3WFJDSXX.1\\tLB:lib1\\tPL:bar\\tSM:sample\\tPU:H3WFJDSXX.1']], in_fq_list=None, in_se_fq=None, interval=None, interval_file=None, interval_padding=None, keep_tmp=None, knownSites=['/workspace/datasets/ref/Homo_sapiens_assembly38.known_indels.vcf.gz'], license_file=None, logfile=None, markdups_assume_sortorder_queryname=None, markdups_picard_version_2182=None, memory_limit=503, no_markdups=None, no_seccomp_override=None, no_warnings=None, num_cpu_threads=0, num_gpus=4, optical_duplicate_pixel_distance=None, out_bam='/workspace/output/output.bam', out_duplicate_metrics=None, out_qc_metrics_dir=None, out_recal_file='/workspace/output/report.txt', read_group_id_prefix=None, read_group_lb=None, read_group_pl=None, read_group_sm=None, ref='/workspace/datasets/ref/Homo_sapiens_assembly38.fasta', standalone_bqsr=None, tmp_dir='/workspace/ZU287CHD', verbose=None, version=None, with_petagene_dir=None, x1=None, x2=None, x3=None, x4=None, x5=None)
[PB Info 2022-May-24 18:02:56] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:02:56] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-May-24 18:02:56] ||                          Version 3.8.0-1.ampere                          ||
[PB Info 2022-May-24 18:02:56] ||                       GPU-BWA mem, Sorting Phase-I                       ||
[PB Info 2022-May-24 18:02:56] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-May-24 18:02:56] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Info 2022-May-24 18:03:02] GPU-BWA mem
[PB Info 2022-May-24 18:03:02] ProgressMeter    Reads           Base Pairs Aligned
[PB Info 2022-May-24 18:03:17] 5033176          760000000
[PB Info 2022-May-24 18:03:24] 10066352 1510000000
[PB Info 2022-May-24 18:03:32] 15099528 2310000000
[PB Info 2022-May-24 18:03:40] 20132704 3040000000
[PB Info 2022-May-24 18:03:47] 25165880 3800000000
[PB Info 2022-May-24 18:03:54] 30199056 4580000000
[PB Info 2022-May-24 18:04:02] 35232232 5330000000
[PB Info 2022-May-24 18:04:09] 40265408 6070000000
[PB Info 2022-May-24 18:04:17] 45298584 6840000000
[PB Info 2022-May-24 18:04:24] 50331760 7610000000
[PB Info 2022-May-24 18:04:31] 55364936 8360000000
[PB Info 2022-May-24 18:04:38] 60398112 9110000000
[PB Info 2022-May-24 18:04:46] 65431288 9870000000
[PB Info 2022-May-24 18:04:53] 70464464 10670000000
[PB Info 2022-May-24 18:05:00] 75497640 11410000000
[PB Info 2022-May-24 18:05:08] 80530816 12150000000
[PB Info 2022-May-24 18:05:15] 85563992 12920000000
[PB Info 2022-May-24 18:05:22] 90597168 13700000000
[PB Info 2022-May-24 18:05:30] 95630344 14440000000
[PB Info 2022-May-24 18:05:37] 100663520        15210000000
[PB Info 2022-May-24 18:05:44] 105696696        15960000000
[PB Info 2022-May-24 18:05:51] 110729872        16710000000
[PB Info 2022-May-24 18:05:59] 115763048        17470000000
[PB Info 2022-May-24 18:06:06] 120796224        18230000000
[PB Info 2022-May-24 18:06:13] 125829400        19010000000
[PB Info 2022-May-24 18:06:21] 130862576        19750000000
[PB Info 2022-May-24 18:06:28] 135895752        20520000000
[PB Info 2022-May-24 18:06:36] 140928928        21290000000
[PB Info 2022-May-24 18:06:43] 145962104        22040000000
[PB Info 2022-May-24 18:06:50] 150995280        22810000000
[PB Info 2022-May-24 18:06:57] 156028456        23560000000
[PB Info 2022-May-24 18:07:05] 161061632        24320000000
[PB Info 2022-May-24 18:07:12] 166094808        25080000000
[PB Info 2022-May-24 18:07:19] 171127984        25840000000
[PB Info 2022-May-24 18:07:27] 176161160        26630000000
[PB Info 2022-May-24 18:07:35] 181194336        27370000000
[PB Info 2022-May-24 18:07:42] 186227512        28110000000
[PB Info 2022-May-24 18:07:49] 191260688        28900000000
[PB Info 2022-May-24 18:07:57] 196293864        29660000000
[PB Info 2022-May-24 18:08:04] 201327040        30400000000
[PB Info 2022-May-24 18:08:12] 206360216        31160000000
[PB Info 2022-May-24 18:08:19] 211393392        31930000000
[PB Info 2022-May-24 18:08:26] 216426568        32680000000
[PB Info 2022-May-24 18:08:33] 221459744        33440000000
[PB Info 2022-May-24 18:08:41] 226492920        34200000000
[PB Info 2022-May-24 18:08:48] 231526096        34960000000
[PB Info 2022-May-24 18:08:55] 236559272        35720000000
[PB Info 2022-May-24 18:09:02] 241592448        36470000000
[PB Info 2022-May-24 18:09:10] 246625624        37240000000
[PB Info 2022-May-24 18:09:17] 251658800        37980000000
[PB Info 2022-May-24 18:09:24] 256691976        38750000000
[PB Info 2022-May-24 18:09:31] 261725152        39510000000
[PB Info 2022-May-24 18:09:39] 266758328        40260000000
[PB Info 2022-May-24 18:09:46] 271791504        41040000000
[PB Info 2022-May-24 18:09:53] 276824680        41800000000
[PB Info 2022-May-24 18:10:01] 281857856        42560000000
[PB Info 2022-May-24 18:10:08] 286891032        43320000000
[PB Info 2022-May-24 18:10:15] 291924208        44100000000
[PB Info 2022-May-24 18:10:22] 296957384        44840000000
[PB Info 2022-May-24 18:10:29] 301990560        45620000000
[PB Info 2022-May-24 18:10:37] 307023736        46360000000
[PB Info 2022-May-24 18:10:44] 312056912        47130000000
[PB Info 2022-May-24 18:10:52] 317090088        47880000000
[PB Info 2022-May-24 18:10:59] 322123264        48640000000
[PB Info 2022-May-24 18:11:06] 327156440        49390000000
[PB Info 2022-May-24 18:11:13] 332189616        50170000000
[PB Info 2022-May-24 18:11:20] 337222792        50910000000
[PB Info 2022-May-24 18:11:28] 342255968        51660000000
[PB Info 2022-May-24 18:11:35] 347289144        52440000000
[PB Info 2022-May-24 18:11:43] 352322320        53170000000
[PB Info 2022-May-24 18:11:50] 357355496        53950000000
[PB Info 2022-May-24 18:11:57] 362388672        54710000000
[PB Info 2022-May-24 18:12:05] 367421848        55470000000
[PB Info 2022-May-24 18:12:12] 372455024        56230000000
[PB Info 2022-May-24 18:12:19] 377488200        57000000000
[PB Info 2022-May-24 18:12:27] 382521376        57780000000
[PB Info 2022-May-24 18:12:34] 387554552        58520000000
[PB Info 2022-May-24 18:12:41] 392587728        59280000000
[PB Info 2022-May-24 18:12:49] 397620904        60030000000
[PB Info 2022-May-24 18:12:56] 402654080        60820000000
[PB Info 2022-May-24 18:13:03] 407687256        61560000000
[PB Info 2022-May-24 18:13:10] 412720432        62290000000
[PB Info 2022-May-24 18:13:18] 417753608        63070000000
[PB Info 2022-May-24 18:13:25] 422786784        63840000000
[PB Info 2022-May-24 18:13:32] 427819960        64600000000
[PB Info 2022-May-24 18:13:40] 432853136        65360000000
[PB Info 2022-May-24 18:13:47] 437886312        66130000000
[PB Info 2022-May-24 18:13:54] 442919488        66890000000
[PB Info 2022-May-24 18:14:01] 447952664        67660000000
[PB Info 2022-May-24 18:14:08] 452985840        68400000000
[PB Info 2022-May-24 18:14:16] 458019016        69180000000
[PB Info 2022-May-24 18:14:23] 463052192        69920000000
[PB Info 2022-May-24 18:14:31] 468085368        70680000000
[PB Info 2022-May-24 18:14:38] 473118544        71440000000
[PB Info 2022-May-24 18:14:46] 478151720        72210000000
[PB Info 2022-May-24 18:14:53] 483184896        72960000000
[PB Info 2022-May-24 18:15:00] 488218072        73720000000
[PB Info 2022-May-24 18:15:08] 493251248        74500000000
[PB Info 2022-May-24 18:15:15] 498284424        75250000000
[PB Info 2022-May-24 18:15:22] 503317600        76010000000
[PB Info 2022-May-24 18:15:29] 508350776        76770000000
[PB Info 2022-May-24 18:15:36] 513383952        77530000000
[PB Info 2022-May-24 18:15:44] 518417128        78300000000
[PB Info 2022-May-24 18:15:51] 523450304        79040000000
[PB Info 2022-May-24 18:15:59] 528483480        79800000000
[PB Info 2022-May-24 18:16:06] 533516656        80570000000
[PB Info 2022-May-24 18:16:13] 538549832        81310000000
[PB Info 2022-May-24 18:16:21] 543583008        82070000000
[PB Info 2022-May-24 18:16:28] 548616184        82820000000
[PB Info 2022-May-24 18:16:35] 553649360        83600000000
[PB Info 2022-May-24 18:16:43] 558682536        84350000000
[PB Info 2022-May-24 18:16:50] 563715712        85120000000
[PB Info 2022-May-24 18:16:57] 568748888        85850000000
[PB Info 2022-May-24 18:17:05] 573782064        86640000000
[PB Info 2022-May-24 18:17:12] 578815240        87400000000
[PB Info 2022-May-24 18:17:20] 583848416        88170000000
[PB Info 2022-May-24 18:17:27] 588881592        88920000000
[PB Info 2022-May-24 18:17:34] 593914768        89700000000
[PB Info 2022-May-24 18:17:42] 598947944        90440000000
[PB Info 2022-May-24 18:17:49] 603981120        91200000000
[PB Info 2022-May-24 18:17:57] 609014296        91960000000
[PB Info 2022-May-24 18:18:04] 614047472        92730000000
[PB Info 2022-May-24 18:18:11] 619080648        93470000000
[PB Info 2022-May-24 18:18:19] 624113824        94250000000
[PB Info 2022-May-24 18:18:26] 629147000        95000000000
[PB Info 2022-May-24 18:18:33] 634180176        95750000000
[PB Info 2022-May-24 18:18:40] 639213352        96520000000
[PB Info 2022-May-24 18:18:47] 644246528        97280000000
[PB Info 2022-May-24 18:18:55] 649279704        98050000000
[PB Info 2022-May-24 18:19:02] 654312880        98800000000
[PB Info 2022-May-24 18:19:09] 659346056        99560000000
[PB Info 2022-May-24 18:19:16] 664379232        100300000000
[PB Info 2022-May-24 18:19:23] 669412408        101070000000
[PB Info 2022-May-24 18:19:31] 674445584        101840000000
[PB Info 2022-May-24 18:19:38] 679478760        102600000000
[PB Info 2022-May-24 18:19:45] 684511936        103350000000
[PB Info 2022-May-24 18:19:52] 689545112        104130000000
[PB Info 2022-May-24 18:20:00] 694578288        104880000000
[PB Info 2022-May-24 18:20:07] 699611464        105640000000
[PB Info 2022-May-24 18:20:15] 704644640        106400000000
[PB Info 2022-May-24 18:20:22] 709677816        107170000000
[PB Info 2022-May-24 18:20:29] 714710992        107920000000
[PB Info 2022-May-24 18:20:37] 719744168        108680000000
[PB Info 2022-May-24 18:20:44] 724777344        109440000000
[PB Info 2022-May-24 18:20:51] 729810520        110200000000
[PB Info 2022-May-24 18:20:58] 734843696        110970000000
[PB Info 2022-May-24 18:21:06] 739876872        111730000000
[PB Info 2022-May-24 18:21:13] 744910048        112460000000
[PB Info 2022-May-24 18:21:21] 749943224        113280000000
[PB Info 2022-May-24 18:21:28] 754976400        114000000000
[PB Info 2022-May-24 18:21:35] 760009576        114780000000
[PB Info 2022-May-24 18:21:42] 765042752        115530000000
[PB Info 2022-May-24 18:21:50] 770075928        116280000000
[PB Info 2022-May-24 18:21:57] 775109104        117040000000
[PB Info 2022-May-24 18:22:04] 780142280        117820000000
[PB Info 2022-May-24 18:22:12] 785175456        118560000000
[PB Info 2022-May-24 18:22:20] 790208632        119330000000
[PB Info 2022-May-24 18:22:27] 795241808        120110000000
[PB Info 2022-May-24 18:22:34] 800274984        120840000000
[PB Info 2022-May-24 18:22:42] 805308160        121600000000
[PB Info 2022-May-24 18:22:50] 810341336        122360000000
[PB Info 2022-May-24 18:22:57] 815374512        123120000000
[PB Info 2022-May-24 18:23:04] 820407688        123890000000
[PB Info 2022-May-24 18:23:12] 825440864        124640000000
[PB Info 2022-May-24 18:23:19] 830474040        125430000000
[PB Info 2022-May-24 18:23:26] 835507216        126160000000
[PB Info 2022-May-24 18:23:33] 840540392        126930000000
[PB Info 2022-May-24 18:23:41] 845573568        127670000000
[PB Info 2022-May-24 18:23:48] 850606744        128420000000
[PB Info 2022-May-24 18:23:55] 855639920        129200000000
[PB Info 2022-May-24 18:24:03] 860673096        129930000000
[PB Info 2022-May-24 18:24:10] 865706272        130730000000
[PB Info 2022-May-24 18:24:17] 870739448        131500000000
[PB Info 2022-May-24 18:24:24] 875772624        132230000000
[PB Info 2022-May-24 18:24:32] 880805800        133000000000
[PB Info 2022-May-24 18:24:39] 885838976        133740000000
[PB Info 2022-May-24 18:24:46] 890872152        134520000000
[PB Info 2022-May-24 18:24:54] 895905328        135270000000
[PB Info 2022-May-24 18:25:01] 900938504        136050000000
[PB Info 2022-May-24 18:25:08] 905971680        136800000000
[PB Info 2022-May-24 18:25:16] 911004856        137560000000
[PB Info 2022-May-24 18:25:23] 916038032        138330000000
[PB Info 2022-May-24 18:25:31] 921071208        139080000000
[PB Info 2022-May-24 18:25:38] 926104384        139850000000
[PB Info 2022-May-24 18:25:46] 931137560        140600000000
[PB Info 2022-May-24 18:26:03] 
GPU-BWA Mem time: 1380.876101 seconds
[PB Info 2022-May-24 18:26:03] GPU-BWA Mem is finished.


[main] CMD: PARABRICKS mem -Z ./pbOpts.txt /workspace/datasets/ref/Homo_sapiens_assembly38.fasta /workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz /workspace/datasets/germline/WGS-LIS-AI018A_R1.fastq.gz @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1
[main] Real time: 1386.678 sec; CPU: 39483.336 sec
[PB Info 2022-May-24 18:26:03] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:26:03] ||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
[PB Info 2022-May-24 18:26:03] ||        Version:                                    3.8.0-1.ampere        ||
[PB Info 2022-May-24 18:26:03] ||        Start Time:                       Tue May 24 18:02:56 2022        ||
[PB Info 2022-May-24 18:26:03] ||        End Time:                         Tue May 24 18:26:03 2022        ||
[PB Info 2022-May-24 18:26:03] ||        Total Time:                           23 minutes 7 seconds        ||
[PB Info 2022-May-24 18:26:03] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:26:06] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:26:06] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-May-24 18:26:06] ||                          Version 3.8.0-1.ampere                          ||
[PB Info 2022-May-24 18:26:06] ||                             Sorting Phase-II                             ||
[PB Info 2022-May-24 18:26:06] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-May-24 18:26:06] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:26:06] progressMeter - Percentage
[PB Info 2022-May-24 18:26:06] 0.0       0.00 GB
[PB Info 2022-May-24 18:26:16] 6.0       3.00 GB
[PB Info 2022-May-24 18:26:26] 17.2      3.00 GB
[PB Info 2022-May-24 18:26:36] 27.0      3.00 GB
[PB Info 2022-May-24 18:26:46] 38.6      3.00 GB
[PB Info 2022-May-24 18:26:56] 49.4      3.00 GB
[PB Info 2022-May-24 18:27:06] 60.0      3.00 GB
[PB Info 2022-May-24 18:27:16] 72.7      3.00 GB
[PB Info 2022-May-24 18:27:26] 83.0      3.00 GB
[PB Info 2022-May-24 18:27:36] 99.6      1.00 GB
[PB Info 2022-May-24 18:27:46] Sorting and Marking: 100.004 seconds
[PB Info 2022-May-24 18:27:46] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:27:46] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2022-May-24 18:27:46] ||        Version:                                    3.8.0-1.ampere        ||
[PB Info 2022-May-24 18:27:46] ||        Start Time:                       Tue May 24 18:26:06 2022        ||
[PB Info 2022-May-24 18:27:46] ||        End Time:                         Tue May 24 18:27:46 2022        ||
[PB Info 2022-May-24 18:27:46] ||        Total Time:                            1 minute 40 seconds        ||
[PB Info 2022-May-24 18:27:46] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:27:46] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:27:46] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-May-24 18:27:46] ||                          Version 3.8.0-1.ampere                          ||
[PB Info 2022-May-24 18:27:46] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2022-May-24 18:27:46] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-May-24 18:27:46] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:27:47] progressMeter -  Percentage
[PB Info 2022-May-24 18:27:57] 0.0       48.03 GB
[PB Info 2022-May-24 18:28:07] 0.0       93.73 GB
[PB Info 2022-May-24 18:28:17] 0.4       135.85 GB
[PB Info 2022-May-24 18:28:27] 2.5       169.24 GB
[PB Info 2022-May-24 18:28:37] 5.9       195.30 GB
[PB Info 2022-May-24 18:28:47] 8.6       226.34 GB
[PB Info 2022-May-24 18:28:57] 11.3      258.30 GB
[PB Info 2022-May-24 18:29:07] 15.0      289.23 GB
[PB Info 2022-May-24 18:29:17] 19.4      300.92 GB
[PB Info 2022-May-24 18:29:27] 23.7      283.09 GB
[PB Info 2022-May-24 18:29:37] 28.1      265.47 GB
[PB Info 2022-May-24 18:29:47] 32.4      248.70 GB
[PB Info 2022-May-24 18:29:57] 36.3      233.23 GB
[PB Info 2022-May-24 18:30:07] 40.7      215.42 GB
[PB Info 2022-May-24 18:30:17] 44.9      198.51 GB
[PB Info 2022-May-24 18:30:27] 49.4      182.24 GB
[PB Info 2022-May-24 18:30:37] 52.2      170.51 GB
[PB Info 2022-May-24 18:30:47] 55.8      155.64 GB
[PB Info 2022-May-24 18:30:57] 58.4      144.86 GB
[PB Info 2022-May-24 18:31:07] 60.9      134.92 GB
[PB Info 2022-May-24 18:31:17] 64.0      123.84 GB
[PB Info 2022-May-24 18:31:27] 68.0      109.63 GB
[PB Info 2022-May-24 18:31:37] 72.2      95.28 GB
[PB Info 2022-May-24 18:31:47] 76.3      78.44 GB
[PB Info 2022-May-24 18:31:57] 81.1      59.02 GB
[PB Info 2022-May-24 18:32:07] 85.1      43.30 GB
[PB Info 2022-May-24 18:32:17] 89.8      24.87 GB
[PB Info 2022-May-24 18:32:27] 94.6      11.05 GB
[PB Info 2022-May-24 18:32:37] 100.0     0.00 GB
[PB Info 2022-May-24 18:32:37] BQSR and writing final BAM:  290.082 seconds
[PB Info 2022-May-24 18:32:37] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:32:37] ||        Program:                          Marking Duplicates, BQSR        ||
[PB Info 2022-May-24 18:32:37] ||        Version:                                    3.8.0-1.ampere        ||
[PB Info 2022-May-24 18:32:37] ||        Start Time:                       Tue May 24 18:27:46 2022        ||
[PB Info 2022-May-24 18:32:37] ||        End Time:                         Tue May 24 18:32:37 2022        ||
[PB Info 2022-May-24 18:32:37] ||        Total Time:                           4 minutes 51 seconds        ||
[PB Info 2022-May-24 18:32:37] ------------------------------------------------------------------------------
/usr/local/parabricks/.pb/binaries//bin/htvc /workspace/datasets/ref/Homo_sapiens_assembly38.fasta /workspace/output/output.bam 4 -o /workspace/output/output.vcf -nt 5 -a /workspace/output/report.txt
[PB Info 2022-May-24 18:33:11] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:33:11] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-May-24 18:33:11] ||                          Version 3.8.0-1.ampere                          ||
[PB Info 2022-May-24 18:33:11] ||                         GPU-GATK4 HaplotypeCaller                        ||
[PB Info 2022-May-24 18:33:11] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-May-24 18:33:11] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:33:24] 0 /workspace/output/output.bam/workspace/output/output.vcf
[PB Info 2022-May-24 18:33:24] ProgressMeter -  Current-Locus   Elapsed-MinutesRegions-Processed        Regions/Minute
[PB Info 2022-May-24 18:33:34] chr1:42652801    0.2     255182  1531074
[PB Info 2022-May-24 18:33:44] chr1:61718401    0.3     346556  1039668
[PB Info 2022-May-24 18:33:54] chr1:73795201    0.5     415832  831660
[PB Info 2022-May-24 18:34:04] chr1:88224001    0.7     497220  745830
[PB Info 2022-May-24 18:34:14] chr1:109180801   0.8     617064  740476
[PB Info 2022-May-24 18:34:24] chr1:167539201   1.0     826379  826379
[PB Info 2022-May-24 18:34:34] chr1:201307201   1.2     1013619 868816
[PB Info 2022-May-24 18:34:44] chr1:227640001   1.3     1165645 874233
[PB Info 2022-May-24 18:34:54] chr2:11337601    1.5     1347088 898058
[PB Info 2022-May-24 18:35:04] chr2:42672001    1.7     1526643 915985
[PB Info 2022-May-24 18:35:14] chr2:79939201    1.8     1739431 948780
[PB Info 2022-May-24 18:35:24] chr2:119448001   2.0     1935026 967513
[PB Info 2022-May-24 18:35:34] chr2:152414401   2.2     2121671 979232
[PB Info 2022-May-24 18:35:44] chr2:187540801   2.3     2319932 994256
[PB Info 2022-May-24 18:35:54] chr2:221404801   2.5     2509807 1003922
[PB Info 2022-May-24 18:36:04] chr3:12302401    2.7     2702866 1013574
[PB Info 2022-May-24 18:36:14] chr3:45072001    2.8     2884748 1018143
[PB Info 2022-May-24 18:36:24] chr3:79828801    3.0     3082745 1027581
[PB Info 2022-May-24 18:36:34] chr3:116976001   3.2     3287114 1038036
[PB Info 2022-May-24 18:36:44] chr3:150720001   3.3     3473801 1042140
[PB Info 2022-May-24 18:36:54] chr3:183139201   3.5     3648364 1042389
[PB Info 2022-May-24 18:37:04] chr4:18537601    3.7     3843360 1048189
[PB Info 2022-May-24 18:37:14] chr4:56472001    3.8     4048657 1056171
[PB Info 2022-May-24 18:37:24] chr4:90643201    4.0     4243640 1060910
[PB Info 2022-May-24 18:37:34] chr4:124478401   4.2     4433236 1063976
[PB Info 2022-May-24 18:37:44] chr4:156547201   4.3     4617614 1065603
[PB Info 2022-May-24 18:37:54] chr5:1272001     4.5     4818605 1070798
[PB Info 2022-May-24 18:38:04] chr5:36561601    4.7     5025918 1076982
[PB Info 2022-May-24 18:38:14] chr5:77596801    4.8     5226415 1081327
[PB Info 2022-May-24 18:38:24] chr5:113712001   5.0     5422373 1084474
[PB Info 2022-May-24 18:38:34] chr5:146683201   5.2     5603598 1084567
[PB Info 2022-May-24 18:38:44] chr6:796801      5.3     5802018 1087877
[PB Info 2022-May-24 18:38:54] chr6:40588801    5.5     6009406 1092619
[PB Info 2022-May-24 18:39:04] chr6:77836801    5.7     6213718 1096538
[PB Info 2022-May-24 18:39:14] chr6:112728001   5.8     6412760 1099330
[PB Info 2022-May-24 18:39:24] chr6:149001601   6.0     6609195 1101532
[PB Info 2022-May-24 18:39:34] chr7:13305601    6.2     6816555 1105387
[PB Info 2022-May-24 18:39:44] chr7:45940801    6.3     7004427 1105962
[PB Info 2022-May-24 18:39:54] chr7:87523201    6.5     7229211 1112186
[PB Info 2022-May-24 18:40:04] chr7:123369601   6.7     7424689 1113703
[PB Info 2022-May-24 18:40:14] chr8:3115201     6.8     7630785 1116700
[PB Info 2022-May-24 18:40:24] chr8:41212801    7.0     7859542 1122791
[PB Info 2022-May-24 18:40:34] chr8:80030401    7.2     8061798 1124902
[PB Info 2022-May-24 18:40:44] chr8:115958401   7.3     8255292 1125721
[PB Info 2022-May-24 18:40:54] chr9:4622401     7.5     8445700 1126092
[PB Info 2022-May-24 18:41:04] chr9:39806401    7.7     8655294 1128950
[PB Info 2022-May-24 18:41:14] chr9:100881601   7.8     8877581 1133308
[PB Info 2022-May-24 18:41:24] chr9:135667201   8.0     9073718 1134214
[PB Info 2022-May-24 18:41:34] chr10:31233601   8.2     9277419 1136010
[PB Info 2022-May-24 18:41:44] chr10:70396801   8.3     9492442 1139093
[PB Info 2022-May-24 18:41:54] chr10:104500801  8.5     9681305 1138977
[PB Info 2022-May-24 18:42:04] chr11:5937601    8.7     9884872 1140562
[PB Info 2022-May-24 18:42:14] chr11:41078401   8.8     10091265        1142407
[PB Info 2022-May-24 18:42:24] chr11:77760001   9.0     10290075        1143341
[PB Info 2022-May-24 18:42:34] chr11:114220801  9.2     10500771        1145538
[PB Info 2022-May-24 18:42:44] chr12:14136001   9.3     10699289        1146352
[PB Info 2022-May-24 18:42:54] chr12:51057601   9.5     10902904        1147674
[PB Info 2022-May-24 18:43:04] chr12:86851201   9.7     11106169        1148914
[PB Info 2022-May-24 18:43:14] chr12:122356801  9.8     11306209        1149783
[PB Info 2022-May-24 18:43:24] chr13:42590401   10.0    11525302        1152530
[PB Info 2022-May-24 18:43:34] chr13:79036801   10.2    11733750        1154139
[PB Info 2022-May-24 18:43:44] chr14:20491201   10.3    11949307        1156384
[PB Info 2022-May-24 18:43:54] chr14:55656001   10.5    12155722        1157687
[PB Info 2022-May-24 18:44:04] chr14:89059201   10.7    12344248        1157273
[PB Info 2022-May-24 18:44:14] chr15:45921601   10.8    12557598        1159162
[PB Info 2022-May-24 18:44:24] chr15:82084801   11.0    12763598        1160327
[PB Info 2022-May-24 18:44:34] chr16:20553601   11.2    12986350        1162956
[PB Info 2022-May-24 18:44:44] chr16:68524801   11.3    13187013        1163559
[PB Info 2022-May-24 18:44:54] chr17:13195201   11.5    13396048        1164873
[PB Info 2022-May-24 18:45:04] chr17:55915201   11.7    13595451        1165324
[PB Info 2022-May-24 18:45:14] chr18:5217601    11.8    13782533        1164721
[PB Info 2022-May-24 18:45:24] chr18:45648001   12.0    13990766        1165897
[PB Info 2022-May-24 18:45:34] chr19:2971201    12.2    14207126        1167708
[PB Info 2022-May-24 18:45:44] chr19:41169601   12.3    14409672        1168351
[PB Info 2022-May-24 18:45:54] chr20:19152001   12.5    14617645        1169410
[PB Info 2022-May-24 18:46:04] chr20:57388801   12.7    14820050        1170003
[PB Info 2022-May-24 18:46:14] chr21:41539201   12.8    15042581        1172149
[PB Info 2022-May-24 18:46:24] chr22:48916801   13.0    15260697        1173899
[PB Info 2022-May-24 18:46:34] chrX:35419201    13.2    15454760        1173778
[PB Info 2022-May-24 18:46:44] chrX:78806401    13.3    15650993        1173824
[PB Info 2022-May-24 18:46:54] chrX:118982401   13.5    15852480        1174257
[PB Info 2022-May-24 18:47:04] chrY:11289601    13.7    16088000        1177170
[PB Info 2022-May-24 18:47:14] Total time taken: 830.321212
[PB Info 2022-May-24 18:47:14] ------------------------------------------------------------------------------
[PB Info 2022-May-24 18:47:14] ||        Program:                         GPU-GATK4 HaplotypeCaller        ||
[PB Info 2022-May-24 18:47:14] ||        Version:                                    3.8.0-1.ampere        ||
[PB Info 2022-May-24 18:47:14] ||        Start Time:                       Tue May 24 18:33:11 2022        ||
[PB Info 2022-May-24 18:47:14] ||        End Time:                         Tue May 24 18:47:14 2022        ||
[PB Info 2022-May-24 18:47:14] ||        Total Time:                           14 minutes 3 seconds        ||
[PB Info 2022-May-24 18:47:14] ------------------------------------------------------------------------------
Tue May 24 18:47:15 CST 2022
end_time: 1653389235
output/
├── [ 37G]  output.bam
├── [9.0M]  output.bam.bai
├── [885M]  output.vcf
├── [ 90K]  output_chrs.txt
└── [112K]  report.txt

0 directories, 5 files
Total time: 0h 44m 20s
```
:::
- program1:
    - GPU:
        ![](https://i.imgur.com/2Ejul6r.png)

- program4: GPU-GATK4 HaplotypeCaller
    - #### CPU: 利用率低
        ![](https://i.imgur.com/02X8iY4.png)
    - #### GPU: 利用率非常低
        ![](https://i.imgur.com/KunfInx.png)
        ![](https://i.imgur.com/UxgcdYt.png)
        ![](https://i.imgur.com/u3BXBtE.png)
        ![](https://i.imgur.com/ccKSqyd.png)

