---
# System prepended metadata

title: '[log] germline: WGS + AWS: g4dn.4xlarge (T4 + 16vCPU + 64GB)'
tags: [Clara, Parabricks-v3.8, 基因體, 二級分析, NVIDIA, Parabricks]

---

[log] germline: WGS + AWS: g4dn.4xlarge (T4 + 16vCPU + 64GB)
===
###### tags: `Parabricks-v3.8`
###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析`

<br>

[TOC]

<br>

## 硬體資源：EC2 / g4dn.4xlarge
> 查閱方式：[[HackMD] AWS / 容器執行個體(EC2) / EC2 / g4dn.4xlarge](https://hackmd.io/QavypnMBRoW8P9aH4gw8FA#EC2--g4dnxlarge)

[![](https://i.imgur.com/7mWwSMW.png)](https://i.imgur.com/7mWwSMW.png)

### GPU
- GPU 0: Tesla T4 (15360MiB)

### CPU
- Intel® Xeon® Platinum 8259CL CPU @ 2.50GHz
- 1(插槽) x 8(核/插槽) x 2(超執行緒/核) = 16條執行緒 (16vCPU)
- CPU MHz: 3114
- BogoMIPS: 5000
- :::spoiler `$ lscpu`
    ```
    $ lscpu
    Architecture:        x86_64
    CPU op-mode(s):      32-bit, 64-bit
    Byte Order:          Little Endian
    CPU(s):              16
    On-line CPU(s) list: 0-15
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    NUMA node(s):        1
    Vendor ID:           GenuineIntel
    CPU family:          6
    Model:               85
    Model name:          Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
    Stepping:            7
    CPU MHz:             3113.942 <--- 會一直變動
    BogoMIPS:            4999.99
    Hypervisor vendor:   KVM
    Virtualization type: full
    L1d cache:           32K
    L1i cache:           32K
    L2 cache:            1024K
    L3 cache:            36608K
    NUMA node0 CPU(s):   0-15
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni
    ```
    :::

### RAM
- 64GB

### Storage


- **規格**：`General Purpose SSD (gp3)`, `3000IOPS`
- **儲存容量**
    - `/`: 50GB
    - `/workspace`: 500GB
- **實際效能測試**
    |   | 速度 | IOPS |
    |---|-----|------|
    | Read | 8.8 MiB/s | 2252 |
    | Write | 2.94 MiB/s | 752 |
    ```bash=
    $ apt-get install -y fio

    $ fio \
      --randrepeat=1 \
      --ioengine=libaio \
      --direct=1 \
      --gtod_reduce=1 \
      --name=fiotest \
      --filename=testfio \
      --bs=4k \
      --iodepth=64 \
      --size=8G \
      --readwrite=randrw \
      --rwmixread=75
    ```
    :::spoiler 點此查看 **執行結果**
    ```
    fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
    fio-3.1
    Starting 1 process
    fiotest: Laying out IO file (1 file / 8192MiB)
    Jobs: 1 (f=1): [m(1)][100.0%][r=9021KiB/s,w=2982KiB/s][r=2255,w=745 IOPS][eta 00m:00s]
    fiotest: (groupid=0, jobs=1): err= 0: pid=14500: Fri Mar 18 01:33:50 2022
       read: IOPS=2252, BW=9009KiB/s (9225kB/s)(6141MiB/698043msec)
       bw (  KiB/s): min= 8600, max=26552, per=100.00%, avg=9008.19, stdev=489.47, samples=1396
       iops        : min= 2150, max= 6638, avg=2252.04, stdev=122.37, samples=1396
      write: IOPS=752, BW=3008KiB/s (3081kB/s)(2051MiB/698043msec)
       bw (  KiB/s): min= 2520, max= 9200, per=100.00%, avg=3008.15, stdev=215.20, samples=1396
       iops        : min=  630, max= 2300, avg=752.02, stdev=53.80, samples=1396
      cpu          : usr=0.68%, sys=2.03%, ctx=691431, majf=0, minf=8
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued rwt: total=1572145,525007,0, short=0,0,0, dropped=0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
       READ: bw=9009KiB/s (9225kB/s), 9009KiB/s-9009KiB/s (9225kB/s-9225kB/s), io=6141MiB (6440MB), run=698043-698043msec
      WRITE: bw=3008KiB/s (3081kB/s), 3008KiB/s-3008KiB/s (3081kB/s-3081kB/s), io=2051MiB (2150MB), run=698043-698043msec

    Disk stats (read/write):
      nvme2n1: ios=1571732/524908, merge=0/23, ticks=15631663/5277839, in_queue=16745612, util=100.00%
    ```
    :::
<br>

<hr>

<br>

## script 
```bash=
# cmd: sh germline.sh 2>&1 | tee germline.sh.log &

date && start_time=`date +%s` && echo "start_time:" $start_time

pbrun germline \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-fq \
    datasets/WGS-LIS-AI018A_R1.fastq.gz \
    datasets/WGS-LIS-AI018A_R2.fastq.gz \
  --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
  --out-bam        output/output.bam \
  --out-variants   output/output.vcf \
  --out-recal-file output/report.txt; \
  echo "exit code:" $!; \
  echo "Done!"

date && end_time=`date +%s` && echo "end_time:" $end_time

total=$((end_time - start_time))
time_h=$((total/3600))
time_m=$(( (total - time_h*3600) / 60 ))
time_s=$(( total - time_h*3600 - time_m*60 ))
echo "Total time: ${time_h}h ${time_m}m ${time_s}s"
```

<br>

<hr>

<br>

## Log
> **g4dn.4xlarge (T4 + 16vCPU + 64GB)**

| Programs | Round-1 | Round-2 | Average |
|----------|--------:|--------:|--------:|
| GPU-BWA mem, Sorting Phase-I | 4h 55m 30s | 4h 55m 45s | 4h 55m 38s |
| Sorting Phase-II             |     5m 40s |     5m 41s |     5m 40s |
| Marking Dmuplicates, BQSR    |    29m 30s |    29m 46s |    29m 38s |
| GPU-GATK4 HaplotypeCaller    |    62m 29s |    62m 48s |    62m 38s |
| | | |
| **Total**                    | 6h 33m 31s | 6h 34m 08s | 6h 33m 50s |

```=
$ sh germline.sh 2>&1 | tee germline.sh.log &
[1] 5789
Thu Mar 17 06:49:22 UTC 2022
start_time: 1647499762
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation


[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for /workspace/datasets/WGS-LIS-AI018A_R1.fastq.gz and
/workspace/datasets/WGS-LIS-AI018A_R2.fastq.gz
[Parabricks Options Mesg]: @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1

[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Read group created for /workspace/datasets/WGS-LIS-AI018A_R1.fastq.gz and
/workspace/datasets/WGS-LIS-AI018A_R2.fastq.gz
[Parabricks Options Mesg]: @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1
[PB Info 2022-Mar-17 06:49:23] Logger not initialized! 
[PB Info 2022-Mar-17 06:49:23] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 06:49:23] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-Mar-17 06:49:23] ||                              Version 3.7.0-1                             ||
[PB Info 2022-Mar-17 06:49:23] ||                       GPU-BWA mem, Sorting Phase-I                       ||
[PB Info 2022-Mar-17 06:49:23] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-Mar-17 06:49:23] ------------------------------------------------------------------------------
gpg: keyblock resource '/home/ubuntu/.gnupg/pubring.kbx': Permission denied
[PB Info 2022-Mar-17 06:49:23] Logger already initialized, continuing with current settings.
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2022-Mar-17 06:50:07][ParaBricks/src/pbOpts.cu:316] 

WARNING
The system has 62 GB, however recommended RAM with 1 GPU is 64 GB.
The run might not finish or might have less than expected performance.

[PB Info 2022-Mar-17 06:50:07] GPU-BWA mem
[PB Info 2022-Mar-17 06:50:07] ProgressMeter	Reads		Base Pairs Aligned

(base) ubuntu@ip-172-31-17-84:/workspace$ [PB Info 2022-Mar-17 06:51:42] 5033176		760000000
[PB Info 2022-Mar-17 06:53:14] 10066352	1520000000
[PB Info 2022-Mar-17 06:54:46] 15099528	2280000000
[PB Info 2022-Mar-17 06:56:21] 20132704	3040000000
[PB Info 2022-Mar-17 06:57:58] 25165880	3800000000
[PB Info 2022-Mar-17 06:59:36] 30199056	4560000000
[PB Info 2022-Mar-17 07:01:13] 35232232	5320000000
[PB Info 2022-Mar-17 07:02:51] 40265408	6080000000
[PB Info 2022-Mar-17 07:04:31] 45298584	6840000000
[PB Info 2022-Mar-17 07:06:06] 50331760	7600000000
[PB Info 2022-Mar-17 07:07:38] 55364936	8360000000
[PB Info 2022-Mar-17 07:09:13] 60398112	9120000000
[PB Info 2022-Mar-17 07:10:47] 65431288	9880000000
[PB Info 2022-Mar-17 07:12:20] 70464464	10640000000
[PB Info 2022-Mar-17 07:13:54] 75497640	11400000000
[PB Info 2022-Mar-17 07:15:29] 80530816	12160000000
[PB Info 2022-Mar-17 07:17:04] 85563992	12920000000
[PB Info 2022-Mar-17 07:18:39] 90597168	13680000000
[PB Info 2022-Mar-17 07:20:12] 95630344	14440000000
[PB Info 2022-Mar-17 07:21:47] 100663520	15200000000
[PB Info 2022-Mar-17 07:23:23] 105696696	15960000000
[PB Info 2022-Mar-17 07:25:00] 110729872	16720000000
[PB Info 2022-Mar-17 07:26:35] 115763048	17480000000
[PB Info 2022-Mar-17 07:28:08] 120796224	18240000000
[PB Info 2022-Mar-17 07:29:41] 125829400	19000000000
[PB Info 2022-Mar-17 07:31:12] 130862576	19760000000
[PB Info 2022-Mar-17 07:32:45] 135895752	20520000000
[PB Info 2022-Mar-17 07:34:22] 140928928	21280000000
[PB Info 2022-Mar-17 07:35:59] 145962104	22040000000
[PB Info 2022-Mar-17 07:37:32] 150995280	22800000000
[PB Info 2022-Mar-17 07:39:05] 156028456	23560000000
[PB Info 2022-Mar-17 07:40:44] 161061632	24320000000
[PB Info 2022-Mar-17 07:42:23] 166094808	25080000000
[PB Info 2022-Mar-17 07:44:00] 171127984	25840000000
[PB Info 2022-Mar-17 07:45:36] 176161160	26600000000
[PB Info 2022-Mar-17 07:47:13] 181194336	27360000000
[PB Info 2022-Mar-17 07:48:50] 186227512	28120000000
[PB Info 2022-Mar-17 07:50:25] 191260688	28880000000
[PB Info 2022-Mar-17 07:51:59] 196293864	29640000000
[PB Info 2022-Mar-17 07:53:36] 201327040	30400000000
[PB Info 2022-Mar-17 07:55:15] 206360216	31160000000
[PB Info 2022-Mar-17 07:56:52] 211393392	31920000000
[PB Info 2022-Mar-17 07:58:30] 216426568	32680000000
[PB Info 2022-Mar-17 08:00:09] 221459744	33440000000
[PB Info 2022-Mar-17 08:01:42] 226492920	34200000000
[PB Info 2022-Mar-17 08:03:14] 231526096	34960000000
[PB Info 2022-Mar-17 08:04:50] 236559272	35720000000
[PB Info 2022-Mar-17 08:06:24] 241592448	36480000000
[PB Info 2022-Mar-17 08:07:57] 246625624	37240000000
[PB Info 2022-Mar-17 08:09:30] 251658800	38000000000
[PB Info 2022-Mar-17 08:11:03] 256691976	38760000000
[PB Info 2022-Mar-17 08:12:36] 261725152	39520000000
[PB Info 2022-Mar-17 08:14:11] 266758328	40280000000
[PB Info 2022-Mar-17 08:15:50] 271791504	41040000000
[PB Info 2022-Mar-17 08:17:29] 276824680	41800000000
[PB Info 2022-Mar-17 08:19:07] 281857856	42560000000
[PB Info 2022-Mar-17 08:20:42] 286891032	43320000000
[PB Info 2022-Mar-17 08:22:15] 291924208	44080000000
[PB Info 2022-Mar-17 08:23:49] 296957384	44840000000
[PB Info 2022-Mar-17 08:25:25] 301990560	45600000000
[PB Info 2022-Mar-17 08:26:58] 307023736	46360000000
[PB Info 2022-Mar-17 08:28:33] 312056912	47120000000
[PB Info 2022-Mar-17 08:30:07] 317090088	47880000000
[PB Info 2022-Mar-17 08:31:39] 322123264	48640000000
[PB Info 2022-Mar-17 08:33:13] 327156440	49400000000
[PB Info 2022-Mar-17 08:34:50] 332189616	50160000000
[PB Info 2022-Mar-17 08:36:22] 337222792	50920000000
[PB Info 2022-Mar-17 08:37:54] 342255968	51680000000
[PB Info 2022-Mar-17 08:39:31] 347289144	52440000000
[PB Info 2022-Mar-17 08:41:09] 352322320	53200000000
[PB Info 2022-Mar-17 08:42:42] 357355496	53960000000
[PB Info 2022-Mar-17 08:44:14] 362388672	54720000000
[PB Info 2022-Mar-17 08:45:47] 367421848	55480000000
[PB Info 2022-Mar-17 08:47:22] 372455024	56240000000
[PB Info 2022-Mar-17 08:48:57] 377488200	57000000000
[PB Info 2022-Mar-17 08:50:29] 382521376	57760000000
[PB Info 2022-Mar-17 08:52:05] 387554552	58520000000
[PB Info 2022-Mar-17 08:53:38] 392587728	59280000000
[PB Info 2022-Mar-17 08:55:12] 397620904	60040000000
[PB Info 2022-Mar-17 08:56:44] 402654080	60800000000
[PB Info 2022-Mar-17 08:58:16] 407687256	61560000000
[PB Info 2022-Mar-17 08:59:50] 412720432	62320000000
[PB Info 2022-Mar-17 09:01:26] 417753608	63080000000
[PB Info 2022-Mar-17 09:03:02] 422786784	63840000000
[PB Info 2022-Mar-17 09:04:39] 427819960	64600000000
[PB Info 2022-Mar-17 09:06:12] 432853136	65360000000
[PB Info 2022-Mar-17 09:07:44] 437886312	66120000000
[PB Info 2022-Mar-17 09:09:15] 442919488	66880000000
[PB Info 2022-Mar-17 09:10:47] 447952664	67640000000
[PB Info 2022-Mar-17 09:12:21] 452985840	68400000000
[PB Info 2022-Mar-17 09:13:57] 458019016	69160000000
[PB Info 2022-Mar-17 09:15:31] 463052192	69920000000
[PB Info 2022-Mar-17 09:17:05] 468085368	70680000000
[PB Info 2022-Mar-17 09:18:40] 473118544	71440000000
[PB Info 2022-Mar-17 09:20:17] 478151720	72200000000
[PB Info 2022-Mar-17 09:21:50] 483184896	72960000000
[PB Info 2022-Mar-17 09:23:28] 488218072	73720000000
[PB Info 2022-Mar-17 09:25:06] 493251248	74480000000
[PB Info 2022-Mar-17 09:26:44] 498284424	75240000000
[PB Info 2022-Mar-17 09:28:18] 503317600	76000000000
[PB Info 2022-Mar-17 09:29:52] 508350776	76760000000
[PB Info 2022-Mar-17 09:31:29] 513383952	77520000000
[PB Info 2022-Mar-17 09:33:08] 518417128	78280000000
[PB Info 2022-Mar-17 09:34:45] 523450304	79040000000
[PB Info 2022-Mar-17 09:36:25] 528483480	79800000000
[PB Info 2022-Mar-17 09:38:01] 533516656	80560000000
[PB Info 2022-Mar-17 09:39:38] 538549832	81320000000
[PB Info 2022-Mar-17 09:41:15] 543583008	82080000000
[PB Info 2022-Mar-17 09:42:53] 548616184	82840000000
[PB Info 2022-Mar-17 09:44:32] 553649360	83600000000
[PB Info 2022-Mar-17 09:46:08] 558682536	84360000000
[PB Info 2022-Mar-17 09:47:48] 563715712	85120000000
[PB Info 2022-Mar-17 09:49:26] 568748888	85880000000
[PB Info 2022-Mar-17 09:51:04] 573782064	86640000000
[PB Info 2022-Mar-17 09:52:38] 578815240	87400000000
[PB Info 2022-Mar-17 09:54:13] 583848416	88160000000
[PB Info 2022-Mar-17 09:55:48] 588881592	88920000000
[PB Info 2022-Mar-17 09:57:23] 593914768	89680000000
[PB Info 2022-Mar-17 09:59:00] 598947944	90440000000
[PB Info 2022-Mar-17 10:00:39] 603981120	91200000000
[PB Info 2022-Mar-17 10:02:16] 609014296	91960000000
[PB Info 2022-Mar-17 10:03:56] 614047472	92720000000
[PB Info 2022-Mar-17 10:05:30] 619080648	93480000000
[PB Info 2022-Mar-17 10:07:03] 624113824	94240000000
[PB Info 2022-Mar-17 10:08:35] 629147000	95000000000
[PB Info 2022-Mar-17 10:10:06] 634180176	95760000000
[PB Info 2022-Mar-17 10:11:38] 639213352	96520000000
[PB Info 2022-Mar-17 10:13:11] 644246528	97280000000
[PB Info 2022-Mar-17 10:14:47] 649279704	98040000000
[PB Info 2022-Mar-17 10:16:26] 654312880	98800000000
[PB Info 2022-Mar-17 10:18:01] 659346056	99560000000
[PB Info 2022-Mar-17 10:19:37] 664379232	100320000000
[PB Info 2022-Mar-17 10:21:08] 669412408	101080000000
[PB Info 2022-Mar-17 10:22:40] 674445584	101840000000
[PB Info 2022-Mar-17 10:24:13] 679478760	102600000000
[PB Info 2022-Mar-17 10:25:48] 684511936	103360000000
[PB Info 2022-Mar-17 10:27:23] 689545112	104120000000
[PB Info 2022-Mar-17 10:28:58] 694578288	104880000000
[PB Info 2022-Mar-17 10:30:36] 699611464	105640000000
[PB Info 2022-Mar-17 10:32:14] 704644640	106400000000
[PB Info 2022-Mar-17 10:33:51] 709677816	107160000000
[PB Info 2022-Mar-17 10:35:27] 714710992	107920000000
[PB Info 2022-Mar-17 10:37:00] 719744168	108680000000
[PB Info 2022-Mar-17 10:38:33] 724777344	109440000000
[PB Info 2022-Mar-17 10:40:10] 729810520	110200000000
[PB Info 2022-Mar-17 10:41:46] 734843696	110960000000
[PB Info 2022-Mar-17 10:43:24] 739876872	111720000000
[PB Info 2022-Mar-17 10:45:01] 744910048	112480000000
[PB Info 2022-Mar-17 10:46:38] 749943224	113240000000
[PB Info 2022-Mar-17 10:48:15] 754976400	114000000000
[PB Info 2022-Mar-17 10:49:53] 760009576	114760000000
[PB Info 2022-Mar-17 10:51:30] 765042752	115520000000
[PB Info 2022-Mar-17 10:53:06] 770075928	116280000000
[PB Info 2022-Mar-17 10:54:38] 775109104	117040000000
[PB Info 2022-Mar-17 10:56:13] 780142280	117800000000
[PB Info 2022-Mar-17 10:57:49] 785175456	118560000000
[PB Info 2022-Mar-17 10:59:27] 790208632	119320000000
[PB Info 2022-Mar-17 11:01:03] 795241808	120080000000
[PB Info 2022-Mar-17 11:02:42] 800274984	120840000000
[PB Info 2022-Mar-17 11:04:21] 805308160	121600000000
[PB Info 2022-Mar-17 11:05:55] 810341336	122360000000
[PB Info 2022-Mar-17 11:07:27] 815374512	123120000000
[PB Info 2022-Mar-17 11:09:00] 820407688	123880000000
[PB Info 2022-Mar-17 11:10:31] 825440864	124640000000
[PB Info 2022-Mar-17 11:12:04] 830474040	125400000000
[PB Info 2022-Mar-17 11:13:36] 835507216	126160000000
[PB Info 2022-Mar-17 11:15:12] 840540392	126920000000
[PB Info 2022-Mar-17 11:16:43] 845573568	127680000000
[PB Info 2022-Mar-17 11:18:17] 850606744	128440000000
[PB Info 2022-Mar-17 11:19:52] 855639920	129200000000
[PB Info 2022-Mar-17 11:21:30] 860673096	129960000000
[PB Info 2022-Mar-17 11:23:07] 865706272	130720000000
[PB Info 2022-Mar-17 11:24:43] 870739448	131480000000
[PB Info 2022-Mar-17 11:26:19] 875772624	132240000000
[PB Info 2022-Mar-17 11:27:58] 880805800	133000000000
[PB Info 2022-Mar-17 11:29:30] 885838976	133760000000
[PB Info 2022-Mar-17 11:31:02] 890872152	134520000000
[PB Info 2022-Mar-17 11:32:34] 895905328	135280000000
[PB Info 2022-Mar-17 11:34:08] 900938504	136040000000
[PB Info 2022-Mar-17 11:35:41] 905971680	136800000000
[PB Info 2022-Mar-17 11:37:14] 911004856	137560000000
[PB Info 2022-Mar-17 11:38:46] 916038032	138320000000
[PB Info 2022-Mar-17 11:40:21] 921071208	139080000000
[PB Info 2022-Mar-17 11:41:55] 926104384	139840000000
[PB Info 2022-Mar-17 11:43:27] 931137560	140600000000
[PB Info 2022-Mar-17 11:44:53] 
GPU-BWA Mem time: 17686.079019 seconds
[PB Info 2022-Mar-17 11:44:53] GPU-BWA Mem is finished.


[main] CMD: PARABRICKS mem -Z ./pbOpts.txt /workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta /workspace/datasets/WGS-LIS-AI018A_R1.fastq.gz /workspace/datasets/WGS-LIS-AI018A_R2.fastq.gz @RG\tID:H3WFJDSXX.1\tLB:lib1\tPL:bar\tSM:sample\tPU:H3WFJDSXX.1
[main] Real time: 17729.759 sec; CPU: 185965.745 sec
[PB Info 2022-Mar-17 11:44:53] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 11:44:53] ||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
[PB Info 2022-Mar-17 11:44:53] ||        Version:                                           3.7.0-1        ||
[PB Info 2022-Mar-17 11:44:53] ||        Start Time:                       Thu Mar 17 06:49:23 2022        ||
[PB Info 2022-Mar-17 11:44:53] ||        End Time:                         Thu Mar 17 11:44:53 2022        ||
[PB Info 2022-Mar-17 11:44:53] ||        Total Time:                         295 minutes 30 seconds        ||
[PB Info 2022-Mar-17 11:44:53] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 11:44:55] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 11:44:55] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-Mar-17 11:44:55] ||                              Version 3.7.0-1                             ||
[PB Info 2022-Mar-17 11:44:55] ||                             Sorting Phase-II                             ||
[PB Info 2022-Mar-17 11:44:55] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-Mar-17 11:44:55] ------------------------------------------------------------------------------
gpg: keyblock resource '/home/ubuntu/.gnupg/pubring.kbx': Permission denied
[PB Info 2022-Mar-17 11:44:55] progressMeter - Percentage
[PB Info 2022-Mar-17 11:44:55] 0.0	 0.00 GB
[PB Info 2022-Mar-17 11:45:05] 0.0	 3.00 GB
[PB Info 2022-Mar-17 11:45:15] 3.7	 3.00 GB
[PB Info 2022-Mar-17 11:45:25] 8.0	 3.00 GB
[PB Info 2022-Mar-17 11:45:35] 12.0	 3.00 GB
[PB Info 2022-Mar-17 11:45:45] 15.8	 3.00 GB
[PB Info 2022-Mar-17 11:45:55] 19.8	 3.00 GB
[PB Info 2022-Mar-17 11:46:05] 23.4	 3.00 GB
[PB Info 2022-Mar-17 11:46:15] 27.5	 3.00 GB
[PB Info 2022-Mar-17 11:46:25] 31.6	 3.00 GB
[PB Info 2022-Mar-17 11:46:35] 35.6	 3.00 GB
[PB Info 2022-Mar-17 11:46:45] 39.0	 3.00 GB
[PB Info 2022-Mar-17 11:46:55] 42.9	 3.00 GB
[PB Info 2022-Mar-17 11:47:05] 45.8	 4.00 GB
[PB Info 2022-Mar-17 11:47:15] 46.1	 8.00 GB
[PB Info 2022-Mar-17 11:47:25] 46.3	 12.00 GB
[PB Info 2022-Mar-17 11:47:35] 46.5	 15.00 GB
[PB Info 2022-Mar-17 11:47:45] 46.7	 19.00 GB
[PB Info 2022-Mar-17 11:47:55] 47.0	 22.00 GB
[PB Info 2022-Mar-17 11:48:05] 48.3	 24.00 GB
[PB Info 2022-Mar-17 11:48:15] 49.6	 26.00 GB
[PB Info 2022-Mar-17 11:48:25] 50.5	 27.00 GB
[PB Info 2022-Mar-17 11:48:35] 51.6	 28.00 GB
[PB Info 2022-Mar-17 11:48:45] 53.9	 28.00 GB
[PB Info 2022-Mar-17 11:48:55] 56.1	 29.00 GB
[PB Info 2022-Mar-17 11:49:05] 58.4	 28.00 GB
[PB Info 2022-Mar-17 11:49:15] 60.8	 28.00 GB
[PB Info 2022-Mar-17 11:49:25] 63.0	 28.00 GB
[PB Info 2022-Mar-17 11:49:35] 66.7	 28.00 GB
[PB Info 2022-Mar-17 11:49:45] 70.1	 26.00 GB
[PB Info 2022-Mar-17 11:49:55] 75.8	 20.00 GB
[PB Info 2022-Mar-17 11:50:05] 81.7	 14.00 GB
[PB Info 2022-Mar-17 11:50:15] 87.9	 8.00 GB
[PB Info 2022-Mar-17 11:50:25] 95.3	 2.00 GB
[PB Info 2022-Mar-17 11:50:35] Sorting and Marking: 340.005 seconds
[PB Info 2022-Mar-17 11:50:35] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 11:50:35] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2022-Mar-17 11:50:35] ||        Version:                                           3.7.0-1        ||
[PB Info 2022-Mar-17 11:50:35] ||        Start Time:                       Thu Mar 17 11:44:55 2022        ||
[PB Info 2022-Mar-17 11:50:35] ||        End Time:                         Thu Mar 17 11:50:35 2022        ||
[PB Info 2022-Mar-17 11:50:35] ||        Total Time:                           5 minutes 40 seconds        ||
[PB Info 2022-Mar-17 11:50:35] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 11:50:36] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 11:50:36] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-Mar-17 11:50:36] ||                              Version 3.7.0-1                             ||
[PB Info 2022-Mar-17 11:50:36] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2022-Mar-17 11:50:36] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-Mar-17 11:50:36] ------------------------------------------------------------------------------
gpg: keyblock resource '/home/ubuntu/.gnupg/pubring.kbx': Permission denied
[PB Info 2022-Mar-17 11:50:36] progressMeter -	Percentage
[PB Info 2022-Mar-17 11:50:46] 0.0	 4.10 GB
[PB Info 2022-Mar-17 11:50:56] 0.0	 8.15 GB
[PB Info 2022-Mar-17 11:51:06] 0.0	 10.49 GB
[PB Info 2022-Mar-17 11:51:16] 0.0	 12.84 GB
[PB Info 2022-Mar-17 11:51:26] 0.0	 15.22 GB
[PB Info 2022-Mar-17 11:51:36] 0.0	 18.63 GB
[PB Info 2022-Mar-17 11:51:46] 0.0	 20.28 GB
[PB Info 2022-Mar-17 11:51:56] 0.0	 23.13 GB
[PB Info 2022-Mar-17 11:52:06] 0.0	 25.06 GB
[PB Info 2022-Mar-17 11:52:16] 0.0	 27.86 GB
[PB Info 2022-Mar-17 11:52:26] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:52:36] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:52:46] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:52:56] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:53:06] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:53:16] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:53:26] 0.0	 29.20 GB
[PB Info 2022-Mar-17 11:53:36] 1.2	 27.21 GB
[PB Info 2022-Mar-17 11:53:46] 2.4	 24.43 GB
[PB Info 2022-Mar-17 11:53:56] 3.4	 23.43 GB
[PB Info 2022-Mar-17 11:54:06] 3.7	 24.59 GB
[PB Info 2022-Mar-17 11:54:16] 5.8	 19.53 GB
[PB Info 2022-Mar-17 11:54:26] 6.7	 17.90 GB
[PB Info 2022-Mar-17 11:54:36] 7.7	 17.24 GB
[PB Info 2022-Mar-17 11:54:46] 8.6	 15.16 GB
[PB Info 2022-Mar-17 11:54:56] 9.1	 14.93 GB
[PB Info 2022-Mar-17 11:55:06] 9.4	 16.89 GB
[PB Info 2022-Mar-17 11:55:16] 9.7	 18.06 GB
[PB Info 2022-Mar-17 11:55:26] 10.0	 19.28 GB
[PB Info 2022-Mar-17 11:55:36] 10.2	 21.74 GB
[PB Info 2022-Mar-17 11:55:46] 10.4	 23.32 GB
[PB Info 2022-Mar-17 11:55:56] 10.7	 24.64 GB
[PB Info 2022-Mar-17 11:56:06] 11.0	 26.57 GB
[PB Info 2022-Mar-17 11:56:16] 11.3	 27.77 GB
[PB Info 2022-Mar-17 11:56:26] 11.5	 29.37 GB
[PB Info 2022-Mar-17 11:56:36] 12.1	 28.56 GB
[PB Info 2022-Mar-17 11:56:46] 12.5	 30.14 GB
[PB Info 2022-Mar-17 11:56:56] 13.0	 29.71 GB
[PB Info 2022-Mar-17 11:57:06] 13.6	 29.39 GB
[PB Info 2022-Mar-17 11:57:16] 14.2	 29.90 GB
[PB Info 2022-Mar-17 11:57:26] 14.7	 29.48 GB
[PB Info 2022-Mar-17 11:57:36] 15.1	 29.03 GB
[PB Info 2022-Mar-17 11:57:46] 15.8	 29.14 GB
[PB Info 2022-Mar-17 11:57:56] 16.3	 30.20 GB
[PB Info 2022-Mar-17 11:58:06] 16.8	 29.35 GB
[PB Info 2022-Mar-17 11:58:16] 17.4	 29.31 GB
[PB Info 2022-Mar-17 11:58:26] 17.9	 29.93 GB
[PB Info 2022-Mar-17 11:58:36] 18.4	 29.57 GB
[PB Info 2022-Mar-17 11:58:46] 19.0	 29.18 GB
[PB Info 2022-Mar-17 11:58:56] 19.5	 29.85 GB
[PB Info 2022-Mar-17 11:59:06] 20.1	 29.21 GB
[PB Info 2022-Mar-17 11:59:16] 20.7	 29.61 GB
[PB Info 2022-Mar-17 11:59:26] 21.1	 28.88 GB
[PB Info 2022-Mar-17 11:59:36] 21.8	 28.96 GB
[PB Info 2022-Mar-17 11:59:46] 22.2	 28.95 GB
[PB Info 2022-Mar-17 11:59:56] 22.7	 29.00 GB
[PB Info 2022-Mar-17 12:00:06] 23.3	 29.83 GB
[PB Info 2022-Mar-17 12:00:16] 23.8	 29.48 GB
[PB Info 2022-Mar-17 12:00:26] 24.3	 30.33 GB
[PB Info 2022-Mar-17 12:00:36] 24.9	 29.64 GB
[PB Info 2022-Mar-17 12:00:46] 25.6	 28.94 GB
[PB Info 2022-Mar-17 12:00:56] 26.1	 29.33 GB
[PB Info 2022-Mar-17 12:01:06] 26.6	 29.77 GB
[PB Info 2022-Mar-17 12:01:16] 27.1	 29.43 GB
[PB Info 2022-Mar-17 12:01:26] 27.7	 29.81 GB
[PB Info 2022-Mar-17 12:01:36] 28.3	 29.01 GB
[PB Info 2022-Mar-17 12:01:46] 28.9	 29.81 GB
[PB Info 2022-Mar-17 12:01:56] 29.5	 29.18 GB
[PB Info 2022-Mar-17 12:02:06] 30.1	 30.04 GB
[PB Info 2022-Mar-17 12:02:16] 30.7	 29.35 GB
[PB Info 2022-Mar-17 12:02:26] 31.2	 30.12 GB
[PB Info 2022-Mar-17 12:02:36] 31.8	 29.33 GB
[PB Info 2022-Mar-17 12:02:46] 32.3	 29.00 GB
[PB Info 2022-Mar-17 12:02:56] 32.8	 29.61 GB
[PB Info 2022-Mar-17 12:03:06] 33.4	 29.26 GB
[PB Info 2022-Mar-17 12:03:16] 34.0	 29.69 GB
[PB Info 2022-Mar-17 12:03:26] 34.6	 28.74 GB
[PB Info 2022-Mar-17 12:03:36] 35.1	 29.18 GB
[PB Info 2022-Mar-17 12:03:46] 35.7	 29.10 GB
[PB Info 2022-Mar-17 12:03:56] 36.2	 29.93 GB
[PB Info 2022-Mar-17 12:04:06] 36.7	 29.68 GB
[PB Info 2022-Mar-17 12:04:16] 37.6	 29.14 GB
[PB Info 2022-Mar-17 12:04:26] 37.9	 29.15 GB
[PB Info 2022-Mar-17 12:04:36] 38.3	 29.64 GB
[PB Info 2022-Mar-17 12:04:46] 39.2	 29.09 GB
[PB Info 2022-Mar-17 12:04:56] 39.5	 29.22 GB
[PB Info 2022-Mar-17 12:05:06] 40.1	 29.22 GB
[PB Info 2022-Mar-17 12:05:16] 40.7	 29.21 GB
[PB Info 2022-Mar-17 12:05:26] 41.2	 28.99 GB
[PB Info 2022-Mar-17 12:05:36] 41.8	 29.52 GB
[PB Info 2022-Mar-17 12:05:46] 42.4	 29.95 GB
[PB Info 2022-Mar-17 12:05:56] 43.0	 29.25 GB
[PB Info 2022-Mar-17 12:06:06] 43.6	 29.85 GB
[PB Info 2022-Mar-17 12:06:16] 44.0	 29.87 GB
[PB Info 2022-Mar-17 12:06:26] 44.5	 29.07 GB
[PB Info 2022-Mar-17 12:06:36] 45.1	 30.25 GB
[PB Info 2022-Mar-17 12:06:46] 45.7	 29.33 GB
[PB Info 2022-Mar-17 12:06:56] 46.2	 29.10 GB
[PB Info 2022-Mar-17 12:07:06] 46.8	 29.82 GB
[PB Info 2022-Mar-17 12:07:16] 47.3	 29.43 GB
[PB Info 2022-Mar-17 12:07:26] 47.8	 29.27 GB
[PB Info 2022-Mar-17 12:07:36] 48.9	 28.90 GB
[PB Info 2022-Mar-17 12:07:46] 49.4	 30.06 GB
[PB Info 2022-Mar-17 12:07:56] 50.0	 29.20 GB
[PB Info 2022-Mar-17 12:08:06] 50.5	 29.35 GB
[PB Info 2022-Mar-17 12:08:16] 51.0	 30.03 GB
[PB Info 2022-Mar-17 12:08:26] 51.5	 29.51 GB
[PB Info 2022-Mar-17 12:08:36] 52.0	 29.24 GB
[PB Info 2022-Mar-17 12:08:46] 52.6	 29.65 GB
[PB Info 2022-Mar-17 12:08:56] 53.1	 29.22 GB
[PB Info 2022-Mar-17 12:09:06] 53.7	 29.60 GB
[PB Info 2022-Mar-17 12:09:16] 54.3	 29.25 GB
[PB Info 2022-Mar-17 12:09:26] 54.8	 29.19 GB
[PB Info 2022-Mar-17 12:09:36] 55.3	 29.46 GB
[PB Info 2022-Mar-17 12:09:46] 55.9	 29.39 GB
[PB Info 2022-Mar-17 12:09:56] 56.4	 29.02 GB
[PB Info 2022-Mar-17 12:10:06] 57.0	 29.32 GB
[PB Info 2022-Mar-17 12:10:16] 57.5	 28.87 GB
[PB Info 2022-Mar-17 12:10:26] 58.1	 29.50 GB
[PB Info 2022-Mar-17 12:10:36] 58.6	 29.11 GB
[PB Info 2022-Mar-17 12:10:46] 59.2	 29.00 GB
[PB Info 2022-Mar-17 12:10:56] 59.7	 29.53 GB
[PB Info 2022-Mar-17 12:11:06] 60.3	 29.17 GB
[PB Info 2022-Mar-17 12:11:16] 60.9	 29.98 GB
[PB Info 2022-Mar-17 12:11:26] 61.4	 29.59 GB
[PB Info 2022-Mar-17 12:11:36] 61.9	 29.22 GB
[PB Info 2022-Mar-17 12:11:46] 62.5	 28.99 GB
[PB Info 2022-Mar-17 12:11:56] 63.0	 29.25 GB
[PB Info 2022-Mar-17 12:12:06] 64.0	 29.78 GB
[PB Info 2022-Mar-17 12:12:16] 64.4	 29.88 GB
[PB Info 2022-Mar-17 12:12:26] 65.0	 29.11 GB
[PB Info 2022-Mar-17 12:12:36] 65.6	 29.59 GB
[PB Info 2022-Mar-17 12:12:46] 66.2	 29.26 GB
[PB Info 2022-Mar-17 12:12:56] 66.7	 29.66 GB
[PB Info 2022-Mar-17 12:13:06] 67.7	 29.09 GB
[PB Info 2022-Mar-17 12:13:16] 68.2	 28.87 GB
[PB Info 2022-Mar-17 12:13:26] 68.9	 29.32 GB
[PB Info 2022-Mar-17 12:13:36] 69.3	 29.03 GB
[PB Info 2022-Mar-17 12:13:46] 69.9	 29.67 GB
[PB Info 2022-Mar-17 12:13:56] 71.0	 30.05 GB
[PB Info 2022-Mar-17 12:14:06] 71.6	 29.19 GB
[PB Info 2022-Mar-17 12:14:16] 72.2	 29.16 GB
[PB Info 2022-Mar-17 12:14:26] 72.7	 29.48 GB
[PB Info 2022-Mar-17 12:14:36] 73.3	 29.87 GB
[PB Info 2022-Mar-17 12:14:46] 73.7	 29.32 GB
[PB Info 2022-Mar-17 12:14:56] 74.2	 29.14 GB
[PB Info 2022-Mar-17 12:15:06] 74.9	 29.23 GB
[PB Info 2022-Mar-17 12:15:16] 75.5	 28.96 GB
[PB Info 2022-Mar-17 12:15:26] 75.9	 29.64 GB
[PB Info 2022-Mar-17 12:15:36] 76.5	 30.75 GB
[PB Info 2022-Mar-17 12:15:46] 77.0	 30.30 GB
[PB Info 2022-Mar-17 12:15:56] 77.7	 29.02 GB
[PB Info 2022-Mar-17 12:16:06] 78.2	 29.37 GB
[PB Info 2022-Mar-17 12:16:16] 78.7	 29.17 GB
[PB Info 2022-Mar-17 12:16:26] 79.3	 29.00 GB
[PB Info 2022-Mar-17 12:16:36] 79.8	 29.32 GB
[PB Info 2022-Mar-17 12:16:46] 80.3	 29.77 GB
[PB Info 2022-Mar-17 12:16:56] 80.9	 29.02 GB
[PB Info 2022-Mar-17 12:17:06] 81.4	 29.35 GB
[PB Info 2022-Mar-17 12:17:16] 82.1	 29.55 GB
[PB Info 2022-Mar-17 12:17:26] 82.6	 29.07 GB
[PB Info 2022-Mar-17 12:17:36] 83.2	 29.35 GB
[PB Info 2022-Mar-17 12:17:46] 83.7	 28.97 GB
[PB Info 2022-Mar-17 12:17:56] 84.3	 29.15 GB
[PB Info 2022-Mar-17 12:18:06] 84.7	 30.30 GB
[PB Info 2022-Mar-17 12:18:16] 85.3	 29.46 GB
[PB Info 2022-Mar-17 12:18:26] 85.9	 29.57 GB
[PB Info 2022-Mar-17 12:18:36] 86.6	 28.91 GB
[PB Info 2022-Mar-17 12:18:46] 87.0	 29.08 GB
[PB Info 2022-Mar-17 12:18:56] 87.7	 28.61 GB
[PB Info 2022-Mar-17 12:19:06] 88.3	 29.32 GB
[PB Info 2022-Mar-17 12:19:16] 88.9	 27.95 GB
[PB Info 2022-Mar-17 12:19:26] 89.9	 24.19 GB
[PB Info 2022-Mar-17 12:19:36] 92.3	 14.47 GB
[PB Info 2022-Mar-17 12:19:46] 95.8	 6.36 GB
[PB Info 2022-Mar-17 12:19:56] 100.0	 0.00 GB
[PB Info 2022-Mar-17 12:20:06] BQSR and writing final BAM:  1769.367 seconds
[PB Info 2022-Mar-17 12:20:06] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 12:20:06] ||        Program:                          Marking Duplicates, BQSR        ||
[PB Info 2022-Mar-17 12:20:06] ||        Version:                                           3.7.0-1        ||
[PB Info 2022-Mar-17 12:20:06] ||        Start Time:                       Thu Mar 17 11:50:36 2022        ||
[PB Info 2022-Mar-17 12:20:06] ||        End Time:                         Thu Mar 17 12:20:06 2022        ||
[PB Info 2022-Mar-17 12:20:06] ||        Total Time:                          29 minutes 30 seconds        ||
[PB Info 2022-Mar-17 12:20:06] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 12:20:23] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 12:20:23] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2022-Mar-17 12:20:23] ||                              Version 3.7.0-1                             ||
[PB Info 2022-Mar-17 12:20:23] ||                         GPU-GATK4 HaplotypeCaller                        ||
[PB Info 2022-Mar-17 12:20:23] ||                  Contact: Parabricks-Support@nvidia.com                  ||
[PB Info 2022-Mar-17 12:20:23] ------------------------------------------------------------------------------
gpg: keyblock resource '/home/ubuntu/.gnupg/pubring.kbx': Permission denied
[PB Info 2022-Mar-17 12:20:50] 0 /workspace/output/output.bam/workspace/output/output.vcf
[PB Info 2022-Mar-17 12:20:51] ProgressMeter -	Current-Locus	Elapsed-Minutes	Regions-Processed	Regions/Minute
[PB Info 2022-Mar-17 12:21:01] chr1:10382127	0.2	58049	348294
[PB Info 2022-Mar-17 12:21:11] chr1:18287934	0.3	102856	308568
[PB Info 2022-Mar-17 12:21:21] chr1:29044610	0.5	162708	325416
[PB Info 2022-Mar-17 12:21:31] chr1:40214246	0.7	225114	337671
[PB Info 2022-Mar-17 12:21:41] chr1:52358368	0.8	290739	348886
[PB Info 2022-Mar-17 12:21:51] chr1:63374284	1.0	353894	353894
[PB Info 2022-Mar-17 12:22:01] chr1:75105596	1.2	419996	359996
[PB Info 2022-Mar-17 12:22:11] chr1:86687943	1.3	486927	365195
[PB Info 2022-Mar-17 12:22:21] chr1:98433456	1.5	552665	368443
[PB Info 2022-Mar-17 12:22:31] chr1:109670257	1.7	617655	370593
[PB Info 2022-Mar-17 12:22:41] chr1:120964702	1.8	680446	371152
[PB Info 2022-Mar-17 12:22:51] chr1:123633349	2.0	697355	348677
[PB Info 2022-Mar-17 12:23:01] chr1:124252700	2.2	702766	324353
[PB Info 2022-Mar-17 12:23:11] chr1:124771159	2.3	707327	303140
[PB Info 2022-Mar-17 12:23:21] chr1:143279760	2.5	710964	284385
[PB Info 2022-Mar-17 12:23:31] chr1:143942374	2.7	714340	267877
[PB Info 2022-Mar-17 12:23:41] chr1:146102325	2.8	722773	255096
[PB Info 2022-Mar-17 12:23:51] chr1:146227003	3.0	723386	241128
[PB Info 2022-Mar-17 12:24:01] chr1:146323162	3.2	723695	228535
[PB Info 2022-Mar-17 12:24:11] chr1:146347152	3.3	723812	217143
[PB Info 2022-Mar-17 12:24:21] chr1:146577511	3.5	724885	207110
[PB Info 2022-Mar-17 12:24:31] chr1:148046062	3.7	733155	199951
[PB Info 2022-Mar-17 12:24:41] chr1:160521122	3.8	801720	209144
[PB Info 2022-Mar-17 12:24:51] chr1:172070353	4.0	868797	217199
[PB Info 2022-Mar-17 12:25:01] chr1:184435025	4.2	937599	225023
[PB Info 2022-Mar-17 12:25:11] chr1:196866914	4.3	1008354	232697
[PB Info 2022-Mar-17 12:25:21] chr1:208694302	4.5	1072807	238401
[PB Info 2022-Mar-17 12:25:31] chr1:220415955	4.7	1139947	244274
[PB Info 2022-Mar-17 12:25:41] chr1:231258993	4.8	1203290	248956
[PB Info 2022-Mar-17 12:25:51] chr1:241319806	5.0	1262786	252557
[PB Info 2022-Mar-17 12:26:01] chr2:3743840	5.2	1321541	255782
[PB Info 2022-Mar-17 12:26:11] chr2:14500563	5.3	1384803	259650
[PB Info 2022-Mar-17 12:26:21] chr2:26399731	5.5	1450728	263768
[PB Info 2022-Mar-17 12:26:31] chr2:37290940	5.7	1513855	267150
[PB Info 2022-Mar-17 12:26:41] chr2:47351849	5.8	1573442	269732
[PB Info 2022-Mar-17 12:26:51] chr2:57705521	6.0	1634908	272484
[PB Info 2022-Mar-17 12:27:01] chr2:69393556	6.2	1698437	275422
[PB Info 2022-Mar-17 12:27:11] chr2:79867200	6.3	1758277	277622
[PB Info 2022-Mar-17 12:27:21] chr2:92087742	6.5	1817422	279603
[PB Info 2022-Mar-17 12:27:31] chr2:98404723	6.7	1850662	277599
[PB Info 2022-Mar-17 12:27:41] chr2:109089583	6.8	1909970	279507
[PB Info 2022-Mar-17 12:27:51] chr2:120152493	7.0	1970491	281498
[PB Info 2022-Mar-17 12:28:01] chr2:130588605	7.2	2030378	283308
[PB Info 2022-Mar-17 12:28:11] chr2:140836742	7.3	2088873	284846
[PB Info 2022-Mar-17 12:28:21] chr2:151982185	7.5	2152089	286945
[PB Info 2022-Mar-17 12:28:31] chr2:163329425	7.7	2215602	288991
[PB Info 2022-Mar-17 12:28:41] chr2:173900897	7.8	2276123	290568
[PB Info 2022-Mar-17 12:28:51] chr2:184929527	8.0	2338998	292374
[PB Info 2022-Mar-17 12:29:01] chr2:196243121	8.2	2403492	294305
[PB Info 2022-Mar-17 12:29:11] chr2:207811125	8.3	2466079	295929
[PB Info 2022-Mar-17 12:29:21] chr2:218697397	8.5	2527637	297369
[PB Info 2022-Mar-17 12:29:31] chr2:229051191	8.7	2586888	298487
[PB Info 2022-Mar-17 12:29:41] chr2:239001510	8.8	2644299	299354
[PB Info 2022-Mar-17 12:29:51] chr3:5951970	9.0	2699802	299978
[PB Info 2022-Mar-17 12:30:01] chr3:15686354	9.2	2755883	300641
[PB Info 2022-Mar-17 12:30:11] chr3:27019049	9.3	2818762	302010
[PB Info 2022-Mar-17 12:30:21] chr3:38020687	9.5	2880413	303201
[PB Info 2022-Mar-17 12:30:31] chr3:49478382	9.7	2942406	304386
[PB Info 2022-Mar-17 12:30:41] chr3:60091103	9.8	3002050	305293
[PB Info 2022-Mar-17 12:30:51] chr3:70175776	10.0	3062323	306232
[PB Info 2022-Mar-17 12:31:01] chr3:78423007	10.2	3109717	305873
[PB Info 2022-Mar-17 12:31:11] chr3:89092609	10.3	3172789	307044
[PB Info 2022-Mar-17 12:31:21] chr3:97291094	10.5	3213935	306089
[PB Info 2022-Mar-17 12:31:31] chr3:107999830	10.7	3275771	307103
[PB Info 2022-Mar-17 12:31:41] chr3:119265534	10.8	3340129	308319
[PB Info 2022-Mar-17 12:31:51] chr3:130037001	11.0	3400230	309111
[PB Info 2022-Mar-17 12:32:01] chr3:140803052	11.2	3460454	309891
[PB Info 2022-Mar-17 12:32:11] chr3:152327037	11.3	3523748	310918
[PB Info 2022-Mar-17 12:32:21] chr3:163732535	11.5	3586445	311864
[PB Info 2022-Mar-17 12:32:31] chr3:175790249	11.7	3650666	312914
[PB Info 2022-Mar-17 12:32:41] chr3:186911743	11.8	3710885	313595
[PB Info 2022-Mar-17 12:32:51] chr3:196559703	12.0	3767592	313966
[PB Info 2022-Mar-17 12:33:01] chr4:7276631	12.2	3817865	313797
[PB Info 2022-Mar-17 12:33:11] chr4:15652536	12.3	3868787	313685
[PB Info 2022-Mar-17 12:33:21] chr4:26913360	12.5	3931101	314488
[PB Info 2022-Mar-17 12:33:31] chr4:37516524	12.7	3992161	315170
[PB Info 2022-Mar-17 12:33:41] chr4:47783937	12.8	4052680	315793
[PB Info 2022-Mar-17 12:33:51] chr4:51311859	13.0	4068366	312951
[PB Info 2022-Mar-17 12:34:01] chr4:51652721	13.2	4069662	309088
[PB Info 2022-Mar-17 12:34:11] chr4:52771181	13.3	4075292	305646
[PB Info 2022-Mar-17 12:34:21] chr4:62582172	13.5	4132358	306100
[PB Info 2022-Mar-17 12:34:31] chr4:73516749	13.7	4195337	306975
[PB Info 2022-Mar-17 12:34:41] chr4:83884797	13.8	4255042	307593
[PB Info 2022-Mar-17 12:34:51] chr4:95529587	14.0	4318191	308442
[PB Info 2022-Mar-17 12:35:01] chr4:107462399	14.2	4383818	309445
[PB Info 2022-Mar-17 12:35:11] chr4:118238215	14.3	4446170	310197
[PB Info 2022-Mar-17 12:35:21] chr4:128961570	14.5	4508577	310936
[PB Info 2022-Mar-17 12:35:31] chr4:139732623	14.7	4571926	311722
[PB Info 2022-Mar-17 12:35:41] chr4:150465462	14.8	4632728	312318
[PB Info 2022-Mar-17 12:35:51] chr4:161467040	15.0	4694876	312991
[PB Info 2022-Mar-17 12:36:01] chr4:171191914	15.2	4753538	313420
[PB Info 2022-Mar-17 12:36:11] chr4:181151746	15.3	4813642	313933
[PB Info 2022-Mar-17 12:36:21] chr5:215921	15.5	4865312	313891
[PB Info 2022-Mar-17 12:36:31] chr5:4598285	15.7	4891375	312215
[PB Info 2022-Mar-17 12:36:41] chr5:14620587	15.8	4950686	312674
[PB Info 2022-Mar-17 12:36:51] chr5:24983912	16.0	5009526	313095
[PB Info 2022-Mar-17 12:37:01] chr5:35198398	16.2	5068659	313525
[PB Info 2022-Mar-17 12:37:11] chr5:45787013	16.3	5127808	313947
[PB Info 2022-Mar-17 12:37:21] chr5:50145510	16.5	5140865	311567
[PB Info 2022-Mar-17 12:37:31] chr5:53361312	16.7	5159232	309553
[PB Info 2022-Mar-17 12:37:41] chr5:64363185	16.8	5220917	310153
[PB Info 2022-Mar-17 12:37:51] chr5:77793589	17.0	5286668	310980
[PB Info 2022-Mar-17 12:38:01] chr5:89083121	17.2	5349066	311596
[PB Info 2022-Mar-17 12:38:11] chr5:101620355	17.3	5415261	312418
[PB Info 2022-Mar-17 12:38:21] chr5:113327757	17.5	5480446	313168
[PB Info 2022-Mar-17 12:38:31] chr5:124867142	17.7	5544928	313863
[PB Info 2022-Mar-17 12:38:41] chr5:136506945	17.8	5608015	314468
[PB Info 2022-Mar-17 12:38:51] chr5:148271705	18.0	5671345	315074
[PB Info 2022-Mar-17 12:39:01] chr5:159426971	18.2	5732579	315554
[PB Info 2022-Mar-17 12:39:11] chr5:170277789	18.3	5794275	316051
[PB Info 2022-Mar-17 12:39:21] chr5:179721518	18.5	5849867	316209
[PB Info 2022-Mar-17 12:39:31] chr6:7507195	18.7	5904287	316301
[PB Info 2022-Mar-17 12:39:41] chr6:17202937	18.8	5960870	316506
[PB Info 2022-Mar-17 12:39:51] chr6:26539196	19.0	6017084	316688
[PB Info 2022-Mar-17 12:40:01] chr6:41284756	19.2	6079625	317197
[PB Info 2022-Mar-17 12:40:11] chr6:51791895	19.3	6139545	317562
[PB Info 2022-Mar-17 12:40:21] chr6:63815796	19.5	6203205	318113
[PB Info 2022-Mar-17 12:40:31] chr6:74999914	19.7	6265819	318600
[PB Info 2022-Mar-17 12:40:41] chr6:85958376	19.8	6329369	319127
[PB Info 2022-Mar-17 12:40:51] chr6:96479901	20.0	6391154	319557
[PB Info 2022-Mar-17 12:41:01] chr6:107678346	20.2	6454823	320073
[PB Info 2022-Mar-17 12:41:11] chr6:119188583	20.3	6519519	320632
[PB Info 2022-Mar-17 12:41:21] chr6:132287742	20.5	6589237	321426
[PB Info 2022-Mar-17 12:41:31] chr6:144911707	20.7	6657805	322151
[PB Info 2022-Mar-17 12:41:41] chr6:156473037	20.8	6724179	322760
[PB Info 2022-Mar-17 12:41:51] chr6:166718378	21.0	6784239	323059
[PB Info 2022-Mar-17 12:42:01] chr7:5371132	21.2	6838426	323075
[PB Info 2022-Mar-17 12:42:11] chr7:15287828	21.3	6900257	323449
[PB Info 2022-Mar-17 12:42:21] chr7:26169549	21.5	6965298	323967
[PB Info 2022-Mar-17 12:42:31] chr7:37790272	21.7	7030605	324489
[PB Info 2022-Mar-17 12:42:41] chr7:49583795	21.8	7097511	325076
[PB Info 2022-Mar-17 12:42:51] chr7:61238388	22.0	7160077	325458
[PB Info 2022-Mar-17 12:43:01] chr7:67910254	22.2	7200608	324839
[PB Info 2022-Mar-17 12:43:11] chr7:78455867	22.3	7259790	325065
[PB Info 2022-Mar-17 12:43:21] chr7:89985567	22.5	7326404	325617
[PB Info 2022-Mar-17 12:43:31] chr7:101831240	22.7	7392323	326131
[PB Info 2022-Mar-17 12:43:41] chr7:112607940	22.8	7452204	326373
[PB Info 2022-Mar-17 12:43:51] chr7:125625424	23.0	7522486	327064
[PB Info 2022-Mar-17 12:44:01] chr7:137063772	23.2	7585906	327449
[PB Info 2022-Mar-17 12:44:11] chr7:150047891	23.3	7653877	328023
[PB Info 2022-Mar-17 12:44:21] chr7:158687939	23.5	7705221	327881
[PB Info 2022-Mar-17 12:44:31] chr8:9479907	23.7	7761975	327970
[PB Info 2022-Mar-17 12:44:41] chr8:17385578	23.8	7813758	327849
[PB Info 2022-Mar-17 12:44:51] chr8:27369311	24.0	7873806	328075
[PB Info 2022-Mar-17 12:45:01] chr8:39023815	24.2	7938849	328504
[PB Info 2022-Mar-17 12:45:11] chr8:53217553	24.3	8012398	329276
[PB Info 2022-Mar-17 12:45:21] chr8:64367827	24.5	8074728	329580
[PB Info 2022-Mar-17 12:45:31] chr8:76012759	24.7	8137550	329900
[PB Info 2022-Mar-17 12:45:41] chr8:87714955	24.8	8200673	330228
[PB Info 2022-Mar-17 12:45:51] chr8:98726176	25.0	8260231	330409
[PB Info 2022-Mar-17 12:46:01] chr8:110419058	25.2	8322649	330701
[PB Info 2022-Mar-17 12:46:11] chr8:122452773	25.3	8388381	331120
[PB Info 2022-Mar-17 12:46:21] chr8:133415862	25.5	8449046	331335
[PB Info 2022-Mar-17 12:46:31] chr8:142833454	25.7	8505261	331373
[PB Info 2022-Mar-17 12:46:41] chr9:7223707	25.8	8560659	331380
[PB Info 2022-Mar-17 12:46:51] chr9:16762877	26.0	8618849	331494
[PB Info 2022-Mar-17 12:47:01] chr9:27417432	26.2	8680944	331755
[PB Info 2022-Mar-17 12:47:11] chr9:37458938	26.3	8740573	331920
[PB Info 2022-Mar-17 12:47:21] chr9:42287917	26.5	8768703	330894
[PB Info 2022-Mar-17 12:47:31] chr9:64603189	26.7	8804845	330181
[PB Info 2022-Mar-17 12:47:41] chr9:71894164	26.8	8840898	329474
[PB Info 2022-Mar-17 12:47:51] chr9:82689558	27.0	8901700	329692
[PB Info 2022-Mar-17 12:48:01] chr9:93115178	27.2	8960162	329821
[PB Info 2022-Mar-17 12:48:11] chr9:104159819	27.3	9020837	330030
[PB Info 2022-Mar-17 12:48:21] chr9:114594910	27.5	9081343	330230
[PB Info 2022-Mar-17 12:48:31] chr9:125481314	27.7	9141361	330410
[PB Info 2022-Mar-17 12:48:41] chr9:134543975	27.8	9194849	330353
[PB Info 2022-Mar-17 12:48:51] chr10:4742187	28.0	9246337	330226
[PB Info 2022-Mar-17 12:49:01] chr10:13718356	28.2	9301640	330235
[PB Info 2022-Mar-17 12:49:11] chr10:23932799	28.3	9360554	330372
[PB Info 2022-Mar-17 12:49:21] chr10:33417525	28.5	9416309	330396
[PB Info 2022-Mar-17 12:49:31] chr10:39907080	28.7	9453718	329780
[PB Info 2022-Mar-17 12:49:41] chr10:43602940	28.8	9471510	328491
[PB Info 2022-Mar-17 12:49:51] chr10:43703956	29.0	9472162	326626
[PB Info 2022-Mar-17 12:50:01] chr10:43723196	29.2	9472314	324765
[PB Info 2022-Mar-17 12:50:11] chr10:48825594	29.3	9498873	323825
[PB Info 2022-Mar-17 12:50:21] chr10:58611969	29.5	9557470	323982
[PB Info 2022-Mar-17 12:50:31] chr10:69062176	29.7	9618053	324204
[PB Info 2022-Mar-17 12:50:41] chr10:79612709	29.8	9675674	324324
[PB Info 2022-Mar-17 12:50:51] chr10:90273463	30.0	9737248	324574
[PB Info 2022-Mar-17 12:51:01] chr10:101169429	30.2	9797822	324789
[PB Info 2022-Mar-17 12:51:11] chr10:111806104	30.3	9858331	324999
[PB Info 2022-Mar-17 12:51:21] chr10:121785513	30.5	9916495	325130
[PB Info 2022-Mar-17 12:51:31] chr10:130380180	30.7	9967686	325033
[PB Info 2022-Mar-17 12:51:41] chr11:3667093	30.8	10006766	324543
[PB Info 2022-Mar-17 12:51:51] chr11:12542347	31.0	10059960	324514
[PB Info 2022-Mar-17 12:52:01] chr11:22531159	31.2	10119100	324677
[PB Info 2022-Mar-17 12:52:11] chr11:32587089	31.3	10178124	324833
[PB Info 2022-Mar-17 12:52:21] chr11:42887892	31.5	10238886	325044
[PB Info 2022-Mar-17 12:52:31] chr11:55579130	31.7	10311853	325637
[PB Info 2022-Mar-17 12:52:41] chr11:66465554	31.8	10371580	325808
[PB Info 2022-Mar-17 12:52:51] chr11:76430198	32.0	10427296	325853
[PB Info 2022-Mar-17 12:53:01] chr11:87182355	32.2	10488996	326082
[PB Info 2022-Mar-17 12:53:11] chr11:97617371	32.3	10549253	326265
[PB Info 2022-Mar-17 12:53:21] chr11:107447944	32.5	10608758	326423
[PB Info 2022-Mar-17 12:53:31] chr11:118257577	32.7	10669609	326620
[PB Info 2022-Mar-17 12:53:41] chr11:127670363	32.8	10724709	326640
[PB Info 2022-Mar-17 12:53:51] chr12:3211183	33.0	10784386	326799
[PB Info 2022-Mar-17 12:54:01] chr12:12585306	33.2	10837163	326748
[PB Info 2022-Mar-17 12:54:11] chr12:23063971	33.3	10898711	326961
[PB Info 2022-Mar-17 12:54:21] chr12:33763157	33.5	10961362	327204
[PB Info 2022-Mar-17 12:54:31] chr12:46027176	33.7	11030792	327647
[PB Info 2022-Mar-17 12:54:41] chr12:57431925	33.8	11095122	327934
[PB Info 2022-Mar-17 12:54:51] chr12:69422384	34.0	11162797	328317
[PB Info 2022-Mar-17 12:55:01] chr12:81791969	34.2	11232318	328750
[PB Info 2022-Mar-17 12:55:11] chr12:94439888	34.3	11301898	329181
[PB Info 2022-Mar-17 12:55:21] chr12:105705582	34.5	11366339	329459
[PB Info 2022-Mar-17 12:55:31] chr12:117393312	34.7	11431229	329746
[PB Info 2022-Mar-17 12:55:41] chr12:127660695	34.8	11490967	329884
[PB Info 2022-Mar-17 12:55:51] chr13:16766401	35.0	11533094	329516
[PB Info 2022-Mar-17 12:56:01] chr13:18590303	35.2	11546668	328341
[PB Info 2022-Mar-17 12:56:11] chr13:25910126	35.3	11590799	328041
[PB Info 2022-Mar-17 12:56:21] chr13:36614281	35.5	11653076	328255
[PB Info 2022-Mar-17 12:56:31] chr13:47721430	35.7	11718634	328559
[PB Info 2022-Mar-17 12:56:41] chr13:59548657	35.8	11786281	328919
[PB Info 2022-Mar-17 12:56:51] chr13:71313462	36.0	11853373	329260
[PB Info 2022-Mar-17 12:57:01] chr13:82857510	36.2	11920524	329599
[PB Info 2022-Mar-17 12:57:11] chr13:94103796	36.3	11985927	329887
[PB Info 2022-Mar-17 12:57:21] chr13:104755103	36.5	12048399	330093
[PB Info 2022-Mar-17 12:57:31] chr14:18456411	36.7	12106885	330187
[PB Info 2022-Mar-17 12:57:41] chr14:27489587	36.8	12159763	330129
[PB Info 2022-Mar-17 12:57:51] chr14:38855935	37.0	12225538	330419
[PB Info 2022-Mar-17 12:58:01] chr14:50275017	37.2	12292760	330746
[PB Info 2022-Mar-17 12:58:11] chr14:61339160	37.3	12357110	330994
[PB Info 2022-Mar-17 12:58:21] chr14:73473478	37.5	12424245	331313
[PB Info 2022-Mar-17 12:58:31] chr14:84767772	37.7	12488390	331550
[PB Info 2022-Mar-17 12:58:41] chr14:97166152	37.8	12551155	331748
[PB Info 2022-Mar-17 12:58:51] chr15:20375959	38.0	12612448	331906
[PB Info 2022-Mar-17 12:59:01] chr15:21508504	38.2	12618621	330618
[PB Info 2022-Mar-17 12:59:11] chr15:33782195	38.3	12676638	330694
[PB Info 2022-Mar-17 12:59:21] chr15:45859085	38.5	12743468	330999
[PB Info 2022-Mar-17 12:59:31] chr15:57273545	38.7	12809246	331273
[PB Info 2022-Mar-17 12:59:41] chr15:68553600	38.8	12872297	331475
[PB Info 2022-Mar-17 12:59:51] chr15:79679868	39.0	12935522	331680
[PB Info 2022-Mar-17 13:00:01] chr15:91348797	39.2	13000287	331922
[PB Info 2022-Mar-17 13:00:11] chr15:101750291	39.3	13062655	332101
[PB Info 2022-Mar-17 13:00:21] chr16:8174344	39.5	13114241	332006
[PB Info 2022-Mar-17 13:00:31] chr16:20049546	39.7	13177834	332214
[PB Info 2022-Mar-17 13:00:41] chr16:30743871	39.8	13237940	332333
[PB Info 2022-Mar-17 13:00:51] chr16:36748798	40.0	13263795	331594
[PB Info 2022-Mar-17 13:01:01] chr16:46684775	40.2	13270003	330373
[PB Info 2022-Mar-17 13:01:11] chr16:46910322	40.3	13271162	329037
[PB Info 2022-Mar-17 13:01:21] chr16:46934375	40.5	13271270	327685
[PB Info 2022-Mar-17 13:01:31] chr16:46972627	40.7	13271473	326347
[PB Info 2022-Mar-17 13:01:41] chr16:49271874	40.8	13282391	325283
[PB Info 2022-Mar-17 13:01:51] chr16:49281519	41.0	13282441	323961
[PB Info 2022-Mar-17 13:02:01] chr16:49281519	41.2	13282441	322650
[PB Info 2022-Mar-17 13:02:11] chr16:49286241	41.3	13282469	321350
[PB Info 2022-Mar-17 13:02:21] chr16:49300612	41.5	13282555	320061
[PB Info 2022-Mar-17 13:02:31] chr16:49305597	41.7	13282588	318782
[PB Info 2022-Mar-17 13:02:41] chr16:49305597	41.8	13282588	317512
[PB Info 2022-Mar-17 13:02:51] chr16:49847853	42.0	13285657	316325
[PB Info 2022-Mar-17 13:03:01] chr16:60897485	42.2	13350103	316603
[PB Info 2022-Mar-17 13:03:11] chr16:73516668	42.3	13417814	316956
[PB Info 2022-Mar-17 13:03:21] chr16:82809600	42.5	13475844	317078
[PB Info 2022-Mar-17 13:03:31] chr17:1219100	42.7	13529771	317104
[PB Info 2022-Mar-17 13:03:41] chr17:10886230	42.8	13586275	317189
[PB Info 2022-Mar-17 13:03:51] chr17:21451008	43.0	13647641	317387
[PB Info 2022-Mar-17 13:04:01] chr17:26812733	43.2	13669766	316674
[PB Info 2022-Mar-17 13:04:11] chr17:27964615	43.3	13676040	315600
[PB Info 2022-Mar-17 13:04:21] chr17:29073486	43.5	13681161	314509
[PB Info 2022-Mar-17 13:04:31] chr17:44327804	43.7	13753028	314954
[PB Info 2022-Mar-17 13:04:41] chr17:56803174	43.8	13819252	315268
[PB Info 2022-Mar-17 13:04:51] chr17:68534276	44.0	13884249	315551
[PB Info 2022-Mar-17 13:05:01] chr17:78854244	44.2	13942716	315684
[PB Info 2022-Mar-17 13:05:11] chr18:1603156	44.3	13978267	315299
[PB Info 2022-Mar-17 13:05:21] chr18:1804656	44.5	13979567	314147
[PB Info 2022-Mar-17 13:05:31] chr18:12220779	44.7	14041569	314363
[PB Info 2022-Mar-17 13:05:41] chr18:26519978	44.8	14115827	314851
[PB Info 2022-Mar-17 13:05:51] chr18:39162979	45.0	14185244	315227
[PB Info 2022-Mar-17 13:06:01] chr18:51239856	45.2	14251821	315538
[PB Info 2022-Mar-17 13:06:11] chr18:62313544	45.3	14315968	315793
[PB Info 2022-Mar-17 13:06:21] chr18:73031981	45.5	14380180	316047
[PB Info 2022-Mar-17 13:06:31] chr19:2683004	45.7	14438656	316174
[PB Info 2022-Mar-17 13:06:41] chr19:11486329	45.8	14491365	316175
[PB Info 2022-Mar-17 13:06:51] chr19:21350397	46.0	14547456	316249
[PB Info 2022-Mar-17 13:07:01] chr19:32980794	46.2	14598283	316208
[PB Info 2022-Mar-17 13:07:11] chr19:43391781	46.3	14654095	316275
[PB Info 2022-Mar-17 13:07:21] chr19:52838156	46.5	14709615	316335
[PB Info 2022-Mar-17 13:07:31] chr20:4823823	46.7	14768356	316464
[PB Info 2022-Mar-17 13:07:41] chr20:16247950	46.8	14834210	316744
[PB Info 2022-Mar-17 13:07:51] chr20:28756652	47.0	14899613	317013
[PB Info 2022-Mar-17 13:08:01] chr20:29572773	47.2	14904541	315997
[PB Info 2022-Mar-17 13:08:11] chr20:30206132	47.3	14908646	314971
[PB Info 2022-Mar-17 13:08:21] chr20:32500582	47.5	14920056	314106
[PB Info 2022-Mar-17 13:08:31] chr20:33743831	47.7	14926723	313148
[PB Info 2022-Mar-17 13:08:41] chr20:45777304	47.8	14992013	313421
[PB Info 2022-Mar-17 13:08:51] chr20:55958214	48.0	15051120	313565
[PB Info 2022-Mar-17 13:09:01] chr21:7967980	48.2	15103716	313571
[PB Info 2022-Mar-17 13:09:11] chr21:9614244	48.3	15110244	312625
[PB Info 2022-Mar-17 13:09:21] chr21:9767917	48.5	15111477	311576
[PB Info 2022-Mar-17 13:09:31] chr21:10084710	48.7	15114225	310566
[PB Info 2022-Mar-17 13:09:41] chr21:13036766	48.8	15119855	309621
[PB Info 2022-Mar-17 13:09:51] chr21:13526399	49.0	15123049	308633
[PB Info 2022-Mar-17 13:10:01] chr21:18729553	49.2	15152087	308178
[PB Info 2022-Mar-17 13:10:11] chr21:29020543	49.3	15213495	308381
[PB Info 2022-Mar-17 13:10:21] chr21:38827199	49.5	15268597	308456
[PB Info 2022-Mar-17 13:10:31] chr22:11059140	49.7	15316641	308388
[PB Info 2022-Mar-17 13:10:41] chr22:12191897	49.8	15322213	307469
[PB Info 2022-Mar-17 13:10:51] chr22:12575963	50.0	15324553	306491
[PB Info 2022-Mar-17 13:11:01] chr22:16305531	50.2	15333096	305643
[PB Info 2022-Mar-17 13:11:11] chr22:24052514	50.3	15374759	305458
[PB Info 2022-Mar-17 13:11:21] chr22:33657597	50.5	15429259	305529
[PB Info 2022-Mar-17 13:11:31] chr22:43852759	50.7	15484813	305621
[PB Info 2022-Mar-17 13:11:41] chrX:1060653	50.8	15532348	305554
[PB Info 2022-Mar-17 13:11:51] chrX:10089394	51.0	15582615	305541
[PB Info 2022-Mar-17 13:12:01] chrX:23346902	51.2	15647106	305806
[PB Info 2022-Mar-17 13:12:11] chrX:35933550	51.3	15712720	306091
[PB Info 2022-Mar-17 13:12:21] chrX:49281549	51.5	15779155	306391
[PB Info 2022-Mar-17 13:12:31] chrX:68097568	51.7	15859196	306952
[PB Info 2022-Mar-17 13:12:41] chrX:81311880	51.8	15927847	307289
[PB Info 2022-Mar-17 13:12:51] chrX:94074940	52.0	15996418	307623
[PB Info 2022-Mar-17 13:13:01] chrX:108710230	52.2	16067782	308008
[PB Info 2022-Mar-17 13:13:11] chrX:122500795	52.3	16135918	308329
[PB Info 2022-Mar-17 13:13:21] chrX:136223991	52.5	16203925	308646
[PB Info 2022-Mar-17 13:13:31] chrX:148574191	52.7	16269345	308911
[PB Info 2022-Mar-17 13:13:41] chrY:10843072	52.8	16331951	309122
[PB Info 2022-Mar-17 13:13:51] chr1_KI270706v1_random:67119	53.0	16366913	308809
[PB Info 2022-Mar-17 13:14:01] chr1_KI270713v1_random:9471	53.2	16370898	307916
[PB Info 2022-Mar-17 13:14:11] chr2_KI270715v1_random:100725	53.3	16372011	306975
[PB Info 2022-Mar-17 13:14:21] chr4_GL000008v2_random:110244	53.5	16374855	306072
[PB Info 2022-Mar-17 13:14:31] chr9_KI270719v1_random:91172	53.7	16377194	305165
[PB Info 2022-Mar-17 13:14:41] chr14_KI270722v1_random:43114	53.8	16382284	304314
[PB Info 2022-Mar-17 13:14:51] chr14_KI270724v1_random:19122	54.0	16385468	303434
[PB Info 2022-Mar-17 13:15:01] chr14_KI270725v1_random:105554	54.2	16386248	302515
[PB Info 2022-Mar-17 13:15:11] chr17_KI270729v1_random:86343	54.3	16393194	301715
[PB Info 2022-Mar-17 13:15:21] chr22_KI270734v1_random:18989	54.5	16397496	300871
[PB Info 2022-Mar-17 13:15:31] chr22_KI270734v1_random:100616	54.7	16397899	299961
[PB Info 2022-Mar-17 13:15:41] chrUn_KI270425v1:1612	54.8	16402020	299124
[PB Info 2022-Mar-17 13:15:51] chrUn_KI270442v1:14333	55.0	16402111	298220
[PB Info 2022-Mar-17 13:16:01] chrUn_KI270442v1:182332	55.2	16403373	297342
[PB Info 2022-Mar-17 13:16:11] chrUn_KI270512v1:9517	55.3	16406385	296500
[PB Info 2022-Mar-17 13:16:21] chrUn_GL000195v1:139084	55.5	16410506	295684
[PB Info 2022-Mar-17 13:16:31] chrUn_GL000195v1:153531	55.7	16410618	294801
[PB Info 2022-Mar-17 13:16:41] chrUn_GL000219v1:67175	55.8	16411192	293931
[PB Info 2022-Mar-17 13:16:51] chrUn_GL000219v1:177529	56.0	16412172	293074
[PB Info 2022-Mar-17 13:17:01] chrUn_GL000219v1:177529	56.2	16412172	292204
[PB Info 2022-Mar-17 13:17:11] chrUn_GL000220v1:23940	56.3	16412344	291343
[PB Info 2022-Mar-17 13:17:21] chrUn_GL000224v1:177466	56.5	16414147	290515
[PB Info 2022-Mar-17 13:17:31] chrUn_KI270746v1:62353	56.7	16418636	289740
[PB Info 2022-Mar-17 13:17:41] chrUn_KI270749v1:86190	56.8	16420880	288930
[PB Info 2022-Mar-17 13:17:51] chrUn_KI270754v1:19128	57.0	16423682	288134
[PB Info 2022-Mar-17 13:18:01] chrUn_KI270755v1:4801	57.2	16423859	287297
[PB Info 2022-Mar-17 13:18:11] chrUn_GL000218v1:23951	57.3	16428126	286537
[PB Info 2022-Mar-17 13:18:21] chr1_KI270766v1_alt:19167	57.5	16429773	285735
[PB Info 2022-Mar-17 13:18:31] chr3_KI270783v1_alt:42961	57.7	16435123	285002
[PB Info 2022-Mar-17 13:18:41] chr6_GL000254v2_alt:595046	57.8	16554002	286236
[PB Info 2022-Mar-17 13:18:51] chrUn_KN707896v1_decoy:28709	58.0	16575402	285782
[PB Info 2022-Mar-17 13:19:01] chrUn_KN707957v1_decoy:9599	58.2	16576362	284980
[PB Info 2022-Mar-17 13:19:11] chrUn_KN707958v1_decoy:2424	58.3	16576397	284166
[PB Info 2022-Mar-17 13:19:21] chrUn_KN707963v1_decoy:4610	58.5	16576468	283358
[PB Info 2022-Mar-17 13:19:31] chrUn_JTFH01000010v1_decoy:4681	58.7	16578303	282584
[PB Info 2022-Mar-17 13:19:41] chrUn_JTFH01000023v1_decoy:7015	58.8	16578720	281791
[PB Info 2022-Mar-17 13:19:51] chrUn_JTFH01000096v1_decoy:3557	59.0	16579882	281014
[PB Info 2022-Mar-17 13:20:01] chrUn_JTFH01000097v1_decoy:3569	59.2	16579900	280223
[PB Info 2022-Mar-17 13:20:11] chrUn_JTFH01000098v1_decoy:3534	59.3	16579930	279437
[PB Info 2022-Mar-17 13:20:21] chrUn_JTFH01000098v1_decoy:3534	59.5	16579930	278654
[PB Info 2022-Mar-17 13:20:31] chrUn_JTFH01000101v1_decoy:3267	59.7	16579964	277876
[PB Info 2022-Mar-17 13:20:41] chrUn_JTFH01000102v1_decoy:3490	59.8	16579985	277102
[PB Info 2022-Mar-17 13:20:51] chrUn_JTFH01000102v1_decoy:3490	60.0	16579985	276333
[PB Info 2022-Mar-17 13:21:01] chrUn_JTFH01000108v1_decoy:3324	60.2	16580082	275569
[PB Info 2022-Mar-17 13:21:11] chrUn_JTFH01000112v1_decoy:3206	60.3	16580155	274809
[PB Info 2022-Mar-17 13:21:21] chrUn_JTFH01000112v1_decoy:3206	60.5	16580155	274052
[PB Info 2022-Mar-17 13:21:31] chrUn_JTFH01000113v1_decoy:3201	60.7	16580177	273299
[PB Info 2022-Mar-17 13:21:41] chrUn_JTFH01000114v1_decoy:3179	60.8	16580212	272551
[PB Info 2022-Mar-17 13:21:51] chrUn_JTFH01000202v1_decoy:2578	61.0	16581456	271827
[PB Info 2022-Mar-17 13:22:01] chrUn_JTFH01000299v1_decoy:2099	61.2	16582442	271102
[PB Info 2022-Mar-17 13:22:11] chrUn_JTFH01000891v1_decoy:901	61.3	16586395	270430
[PB Info 2022-Mar-17 13:22:21] chrUn_JTFH01001345v1_decoy:2000	61.5	16590498	269764
[PB Info 2022-Mar-17 13:22:31] chrUn_JTFH01001851v1_decoy:3102	61.7	16596169	269127
[PB Info 2022-Mar-17 13:22:41] HLA-DRB1*15:02:01:10276	61.8	16598422	268438
[PB Info 2022-Mar-17 13:22:52] Total time taken: 3722.013167
[PB Info 2022-Mar-17 13:22:52] ------------------------------------------------------------------------------
[PB Info 2022-Mar-17 13:22:52] ||        Program:                         GPU-GATK4 HaplotypeCaller        ||
[PB Info 2022-Mar-17 13:22:52] ||        Version:                                           3.7.0-1        ||
[PB Info 2022-Mar-17 13:22:52] ||        Start Time:                       Thu Mar 17 12:20:23 2022        ||
[PB Info 2022-Mar-17 13:22:52] ||        End Time:                         Thu Mar 17 13:22:52 2022        ||
[PB Info 2022-Mar-17 13:22:52] ||        Total Time:                          62 minutes 29 seconds        ||
[PB Info 2022-Mar-17 13:22:52] ------------------------------------------------------------------------------

exit code:
Done!
Thu Mar 17 13:22:53 UTC 2022
end_time: 1647523373
Total time: 6h 33m 31s

[1]+  Done   sh germline.sh 2>&1 | tee germline.sh.log
```
- monitor: program1
    [![](https://i.imgur.com/tYIu9Yi.png)](https://i.imgur.com/tYIu9Yi.png)
    - CPU 使用 11/16


<br>

<hr>

<br>

## 執行評估
> **g4dn.4xlarge (T4 + 16vCPU + 64GB)**
> - A instance: 3000IOPS, 125MiB (理想) (預設值)
> - B instance: 14600IOPS, 571MiB (理想)

- meminfo
    - MemTotal: 62.03 Gib
    - MemAvailable: 60.89 Gib
- parms
    - `--memory-limit`: ml
    - `--num-cpu-threads`: nct
- P: Program
    - P1: GPU-BWA mem, Sorting Phase-I
    - P2: Sorting Phase-II
    - P3: Marking Dmuplicates, BQSR	
    - P4: GPU-GATK4 HaplotypeCaller	

| # | ml | nct | P1 | P2 | P3 | P4 | Total|
|---|----|-----|---:|---:|---:|---:|-----:|
| **A.1** | - | - | 4h55m30s | 5m40s | 29m30s | 62m29s | 6h33m31s |
| **A.2** | - | - | 4h55m45s | 5m41s | 29m46s | 62m48s | 6h34m08s |
||
| **B.1** | - | - | 5h01m28s | 3m31s | 14m31s | 65m09s | 6h24m48s |
| **B.2** | 62 | 16 | 4h16m12s | 3m31s | <span style="color: red;">killed<br>(OOM)</span> | | ~~4h20m57s~~ |
| **B.3** | - | 16 | 4h17m30s | 3m30 | 14m21s | 63m30s | 5h39m00s |
| **B.4** | 60 | 16 | 4h19m05s | 3m40s | <span style="color: red;">killed<br>(OOM)</span> | | ~~4h24m17s~~ |
| **B.5** | 50 | 16 | 4h20m06s | 3m40s | 15m11s | 65m19s | 5h44m25s |
| **B.6** | 40 | 16 | 4h21m20s| 3m41s | 15m21s| 63m49s | 5h44m20s |
| **B.7** | 50 | 8 | 4h26m08s | 3m30s | 14m30s | 63m30s | 5h47m48s |
| **B.8** | - | 8 | 4h23m55s | 3m40s | 15m21s | 65m40s | 5h48m46s |
| **B.9** | - | 32 | 4h38m37s| 3m40s | 15m22s | 65m50s | 6h03m37s |
| **B.10** | - | - | 5h00m58s | 3m31s | 14m31s | 64m10s | 6h23m18s |
- **B.1** 和 **B.10** 是兩個相同參數，在不同時間點跑得結果，看起來沒有差異
