---
# System prepended metadata

title: '[log] container 內部的細節'
tags: [Clara, 基因體, Parabricks-v3.5, 二級分析, NVIDIA, Parabricks]

---

[log] container 內部的細節
===
###### tags: `Parabricks-v3.5`
###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `二級分析`

<br>

[TOC]

<br>

:::warning
:bulb: **啟發**
- 原本是由 pbrun 啟動容器
- 拿這個容器的 image 當作 base
- 再把 pbrun 複製到 image
- 原本 pbrun 是執行 docker run, 改成 python3 /parabricks/run_pipeline.py
:::


## 建置 image
:::warning
:bulb: 需先前查看 [container 的組態](#inspect)
:::

### 準備 Dockerfile
```dockerfile=
FROM parabricks/release:v3.5.0
ENTRYPOINT ["/bin/bash"]
```

<br>

### 執行 docker build
```bash=
docker build -t parabricks/release:v3.5.0-by-tj .
```
- `.` 表示當前目錄，從當前目錄尋找 `Dockerfile`

<br>

### 執行 docker run
```bash=
docker run --rm -it parabricks/release:v3.5.0-by-tj
```
- v3.5 版本 :+1: 
    - 沒有拔掉 pwd, ls, find, grep, ...
        [![](https://i.imgur.com/VOeIOxj.png)](https://i.imgur.com/VOeIOxj.png)
    - user 是 root
        ```
        # id
        uid=0(root) gid=0(root) groups=0(root)
        ```
    - 可以自行安裝套件
        ```bash=
        apt install nano
        ```
- 舊版
    - pwd, ls, find, grep, ... 等所有的指令，都是空殼
    - 要執行這些指令，必須從外部掛進去

<br>

## 掛載資料測試
### 進入到容器
> 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 18:50)
```bash=
# 切到含有 parabricks_sample 子目錄的目錄

rm -rf tmp-dir
mkdir tmp-dir
WORKSPACE=`pwd`

docker run \
  --gpus all \
  --rm \
  -it \
  --net=host \
  -w=$WORKSPACE \
  -v $WORKSPACE:$WORKSPACE \
  -v /opt/parabricks:/INSTALL/ \
  parabricks/release:v3.5.0-by-tj

```
- 這邊要移除掉 `-u=1000:1000`參數
    因為 user 是 `uid=0(root) gid=0(root) groups=0(root)
`
    否則會有底下的 error：
    `groups: cannot find name for group ID 1000`
- `--gpus all`
    - 主機若沒有 CPU，會遇到下列錯誤：
        > docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
        - 即使有安裝 Nvidia runtime (CUDA Toolkit)、nvidia-container-toolkit 仍然一樣
        - `nvidia-smi`
            ```
            $ nvidia-smi
            NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
            ```
    - 解決方式
        - 暫時不要使用 `--gpus all` 參數 

<br>

### [在容器中] 執行 fq2bam (簡單版)
> 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 19:15)
```bash=
python3 /parabricks/run_pipeline.py fq2bam \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-fq parabricks_sample/Data/sample_1.fq.gz \
          parabricks_sample/Data/sample_2.fq.gz \
  --out-bam output.bam \
  --tmp-dir tmp-dir \
  --x3
```
- 是 `python3` 程式，不是 `python`
    否則會執行到 Python 2.7.17
- 指令同 [`pbrun fq2bam`](/on6QkW93SKyxvV2codV7GA##執行-pbrun-fq2bam)，但要加上 `--tmp-dir` 參數
    否則會有底下的 error：
    ![](https://i.imgur.com/a8c4NF7.png)
    `ValueError: '--tmp-dir' is not in list`
- 曾經遇到：
    - 硬碟空間不足，但實際上空間很大(1TB)，目前無法重製
        ```
        ...
        progressMeter - Percentage
        [08:10:31]	0.0	 0.00 GB
        Received signal: 11
        Please make sure you have enough disk space in the output and temp directory
        Contact Parabricks-Support@nvidia.com for support
        Please contact Parabricks-Support@nvidia.com for any questions
        There is a forum for Q&A as well at https://forums.developer.nvidia.com/c/healthcare/Parabricks/290
        Exiting...
        ```
        - Please make sure you have enough disk space.

<br>

### [在容器中] 執行 fq2bam (複雜版)
> 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 19:30)
```bash=
python3 /parabricks/run_pipeline.py fq2bam \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-fq parabricks_sample/Data/sample_1.fq.gz \
          parabricks_sample/Data/sample_2.fq.gz \
          "@RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1" \
  --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
  --out-bam output.bam \
  --out-recal-file report.txt \
  --tmp-dir tmp-dir \
  --x3
  
```

<br>

### [在容器中] 執行 germline

:::warning
:warning: **底下指令不存在** :no_entry_sign: 
```bash=
python3 /parabricks/run_pipeline.py germline \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz \
  --knownSites parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
  --out-bam output.bam \
  --out-variants output.vcf \
  --out-recal-file report.txt \
  --x3
```
![](https://i.imgur.com/HSnvoCl.png)
- germline 指令是由 pbrun 控制
- 容器內部並無 germline 指令
:::

> 精簡過後，確認可以跑得 docker run 指令 (2021/06/11 19:43)
```
python3 /parabricks/run_pipeline.py haplotypecaller \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-bam output.bam \
  --in-recal-file report.txt \
  --out-variants output.vcf \
  --ploidy 2 \
  --num-htvc-threads 5 \
  --num-gpus 2 \
  --x3 \
  --tmp-dir tmp-dir
```

<br>

<hr>

<br>


## inspect
### [`pbrun fq2bam`](/on6QkW93SKyxvV2codV7GA#pbrun-fq2bam)
```bash=
pbrun fq2bam \
  --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
  --in-fq parabricks_sample/Data/sample_1.fq.gz \
          parabricks_sample/Data/sample_2.fq.gz \
  --out-bam output.bam \
  --x3
```
```bash=
# 透過 pbrun 正常啟動的容器
$ docker inspect db3d02a38ec0
```
- Path
    `"/parabricks/run_pipeline.py"`
- Args
    ```json=
    [
        "fq2bam",
        "--ref",
        "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta",
        "--in-fq",
        "parabricks_sample/Data/sample_1.fq.gz",
        "parabricks_sample/Data/sample_2.fq.gz",
        "--out-bam",
        "output.bam",
        "--x3",
        "--tmp-dir",
        "/uploads/workspace/9DBIL9ME"
    ]
    ```
- Cmd
    ```bash=
    [
        "fq2bam",
        "--ref",
        "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta",
        "--in-fq",
        "parabricks_sample/Data/sample_1.fq.gz",
        "parabricks_sample/Data/sample_2.fq.gz",
        "--out-bam",
        "output.bam",
        "--x3",
        "--tmp-dir",
        "/uploads/workspace/9DBIL9ME"
    ]
    ```
- Entrypoint
    ```json=
    "/parabricks/run_pipeline.py"
    ```
- 執行結果
    :::spoiler
    ```json
    [
        {
            "Id": "db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf",
            "Created": "2021-06-10T03:14:57.398476975Z",
            "Path": "/parabricks/run_pipeline.py",
            "Args": [
                "fq2bam",
                "--ref",
                "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta",
                "--in-fq",
                "parabricks_sample/Data/sample_1.fq.gz",
                "parabricks_sample/Data/sample_2.fq.gz",
                "--out-bam",
                "output.bam",
                "--x3",
                "--tmp-dir",
                "/uploads/workspace/9DBIL9ME"
            ],
            "State": {
                "Status": "running",
                "Running": true,
                "Paused": false,
                "Restarting": false,
                "OOMKilled": false,
                "Dead": false,
                "Pid": 24839,
                "ExitCode": 0,
                "Error": "",
                "StartedAt": "2021-06-10T03:14:59.631886014Z",
                "FinishedAt": "0001-01-01T00:00:00Z"
            },
            "Image": "sha256:da882a2f26c9e3858c92e2dc8ac4c05d3dfbab69702dc75e4d218369e6c3cfe6",
            "ResolvConfPath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/resolv.conf",
            "HostnamePath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/hostname",
            "HostsPath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/hosts",
            "LogPath": "/var/lib/docker/containers/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf/db3d02a38ec04e44d35f8b3d2ddd9c73dad8bb5add966b9701c1769098f3cbcf-json.log",
            "Name": "/distracted_mendeleev",
            "RestartCount": 0,
            "Driver": "overlay2",
            "Platform": "linux",
            "MountLabel": "",
            "ProcessLabel": "",
            "AppArmorProfile": "docker-default",
            "ExecIDs": null,
            "HostConfig": {
                "Binds": [
                    "/uploads/workspace/parabricks_sample/Data:/uploads/workspace/parabricks_sample/Data",
                    "/opt/parabricks:/INSTALL/",
                    "/uploads/workspace/9DBIL9ME:/uploads/workspace/9DBIL9ME",
                    "/uploads/workspace:/uploads/workspace",
                    "/uploads/workspace/parabricks_sample/Ref:/uploads/workspace/parabricks_sample/Ref"
                ],
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "json-file",
                    "Config": {}
                },
                "NetworkMode": "host",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": true,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "host",
                "Dns": [],
                "DnsOptions": [],
                "DnsSearch": [],
                "ExtraHosts": null,
                "GroupAdd": null,
                "IpcMode": "private",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 67108864,
                "Runtime": "runc",
                "ConsoleSize": [
                    0,
                    0
                ],
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": [],
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [],
                "DeviceCgroupRules": null,
                "DeviceRequests": [
                    {
                        "Driver": "",
                        "Count": -1,
                        "DeviceIDs": null,
                        "Capabilities": [
                            [
                                "gpu"
                            ]
                        ],
                        "Options": {}
                    }
                ],
                "KernelMemory": 0,
                "KernelMemoryTCP": 0,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": [
                    "/proc/asound",
                    "/proc/acpi",
                    "/proc/kcore",
                    "/proc/keys",
                    "/proc/latency_stats",
                    "/proc/timer_list",
                    "/proc/timer_stats",
                    "/proc/sched_debug",
                    "/proc/scsi",
                    "/sys/firmware"
                ],
                "ReadonlyPaths": [
                    "/proc/bus",
                    "/proc/fs",
                    "/proc/irq",
                    "/proc/sys",
                    "/proc/sysrq-trigger"
                ]
            },
            "GraphDriver": {
                "Data": {
                    "LowerDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab-init/diff:/var/lib/docker/overlay2/95100646f1ff8316d1c06a7b0b3451439dca5bdf4e33bde6f584065c96866c09/diff:/var/lib/docker/overlay2/c9158a87787badbb60960de32a39cfa18bfc56f2b22e502cbda712f5cb99be42/diff:/var/lib/docker/overlay2/b6f618b9e13e49ab439c08f179dc7ddf2e756ac1cd53bbdc31b3872f01f4d3d4/diff:/var/lib/docker/overlay2/8ce465351f799fd7e0c9284cbeb1619e594b963713b79516fd5f861da11f8c7c/diff:/var/lib/docker/overlay2/5f51dbee605647a571a609ff9f33d411c5d944503356ac3f20ce44a8e6421a65/diff:/var/lib/docker/overlay2/e3a6db43fff265a671c939b6a28574bb0599e96603ef72aa77bdb325e29468da/diff:/var/lib/docker/overlay2/dbf16f87259a59e72ac5edef6594586296916c0c969c928648441d64b4b03b9e/diff:/var/lib/docker/overlay2/93478e0a56e909c6026d2cb020fa33c26633b9661f0be122be6b9b528f3437ee/diff:/var/lib/docker/overlay2/e3209e1ad5328df416136513d36e9bfafd1f28cde75ac54be142d25090185177/diff:/var/lib/docker/overlay2/417b5ab64d2485fddc0a6369434e9acdde7d40a360aab00b8a9087ed7803ce00/diff:/var/lib/docker/overlay2/aff613731f0d6d16380efe3573d38ae44985c4adcf303de9af822ace571ab5e3/diff:/var/lib/docker/overlay2/204381c326d056c06c381c9e3f80d6ff753999aeecd90fbea0c3e32f2aa3018d/diff:/var/lib/docker/overlay2/1f53eb8731ff13cf6389da0f0b91cb7ae75dfae974d727baf718167c685ec90c/diff:/var/lib/docker/overlay2/2cbce77b8816ba28ca718757c351060a5fc31d3e4e01f90fe685a4745fbface8/diff:/var/lib/docker/overlay2/adb4dc92c8c2a038a2eb3aefdb59eaae2ca6a9f13dbe46efad3b1c8e79c35456/diff:/var/lib/docker/overlay2/98513893aa6968b97d1b0df0937407e76e41b92accad4b6f8a87093eaaaa125d/diff:/var/lib/docker/overlay2/b82cc3de5486d714f0babf1ba64caba690dede00c10d60adcb736635edc3eaf3/diff:/var/lib/docker/overlay2/187bea67669cd8d2de88f6ba252751249243b6a3a3db21a1f78e12e7182b8558/diff:/var/lib/docker/overlay2/44efbdf16447b790efaf613ced36a7801203006beba5d53f7728d76bf9a1a723/diff",
                    "MergedDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab/merged",
                    "UpperDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab/diff",
                    "WorkDir": "/var/lib/docker/overlay2/5ecdbc7c1100fc3590882a2d58d1c71020defdb5cecd855427485250d61012ab/work"
                },
                "Name": "overlay2"
            },
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/uploads/workspace/parabricks_sample/Data",
                    "Destination": "/uploads/workspace/parabricks_sample/Data",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/opt/parabricks",
                    "Destination": "/INSTALL",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/uploads/workspace/9DBIL9ME",
                    "Destination": "/uploads/workspace/9DBIL9ME",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/uploads/workspace",
                    "Destination": "/uploads/workspace",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "bind",
                    "Source": "/uploads/workspace/parabricks_sample/Ref",
                    "Destination": "/uploads/workspace/parabricks_sample/Ref",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                }
            ],
            "Config": {
                "Hostname": "GPU-VM-NC12s-v2",
                "Domainname": "",
                "User": "1000:1000",
                "AttachStdin": false,
                "AttachStdout": true,
                "AttachStderr": true,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/miniconda/bin",
                    "CUDA_VERSION=10.1.243",
                    "CUDA_PKG_VERSION=10-1=10.1.243-1",
                    "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                    "NVIDIA_VISIBLE_DEVICES=all",
                    "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                    "NVIDIA_REQUIRE_CUDA=cuda>=10.1 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419"
                ],
                "Cmd": [
                    "fq2bam",
                    "--ref",
                    "parabricks_sample/Ref/Homo_sapiens_assembly38.fasta",
                    "--in-fq",
                    "parabricks_sample/Data/sample_1.fq.gz",
                    "parabricks_sample/Data/sample_2.fq.gz",
                    "--out-bam",
                    "output.bam",
                    "--x3",
                    "--tmp-dir",
                    "/uploads/workspace/9DBIL9ME"
                ],
                "Image": "parabricks/release:v3.5.0",
                "Volumes": null,
                "WorkingDir": "/uploads/workspace",
                "Entrypoint": [
                    "/parabricks/run_pipeline.py"
                ],
                "OnBuild": null,
                "Labels": {
                    "maintainer": "NVIDIA CORPORATION <cudatools@nvidia.com>"
                }
            },
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "3ccfc9126c93fb12933d89f28314c3719180f8c6169cf4adecd8b59da4a6a549",
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "Ports": {},
                "SandboxKey": "/var/run/docker/netns/default",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "",
                "Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "MacAddress": "",
                "Networks": {
                    "host": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": null,
                        "NetworkID": "e93977ab9b9b8ab0ba8b127243f8b6d66bca2596428a25857a131f68b5ee2b41",
                        "EndpointID": "0de75fab853cb4be062e1eae031598814599f0030e050c389c0c964644c9e139",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "MacAddress": "",
                        "DriverOpts": null
                    }
                }
            }
        }
    ]
    ```
    :::

<br>

<hr>

<br>


## binary 檔的真正位置在
:::warning
**資料來源：[[log] WGS + Azure:GPU-NC12s-v2(P100-16GB x2)](/fS60joh8TNqroKm2L_0WYg)**
- log 中出現的 binary 檔位置，並非是真正位置
    ```
    /usr/local/cuda/.pb/binaries//bin/bwa
    ```
:::

```
/usr/local/cuda-10.1/.pb/binaries/bin
```
```bash=
usr/local/cuda-10.1/.pb/binaries/bin$ ls -ls
total 467644
   360 -rwxrwxrwx    367256 Feb 23 applyBQSR
  1192 -rwxrwxrwx   1218248 Feb 23 bamreadcount
   296 -rwxrwxrwx    301024 Feb 23 bcftoolscall
   432 -rwxrwxrwx    440368 Feb 23 bcftoolsmpileup
    96 -rwxrwxrwx     96584 Feb 23 bedcov
   500 -rwxrwxrwx    510696 Feb 23 bqsr
  2424 -rwxrwxrwx   2481896 Feb 23 bwa
   708 -rwxrwxrwx    723584 Feb 23 cnnscorevariants
   220 -rwxrwxrwx    223192 Feb 23 cnvkit
   844 -rwxrwxrwx    863712 Feb 23 collectmultiplemetrics
   336 -rwxrwxrwx    342408 Feb 23 coverage
   272 -rwxrwxrwx    276456 Feb 23 dbsnp
  2252 -rwxrwxrwx   2305216 Feb 23 deepvariant
   652 -rwxrwxrwx    664832 Feb 23 deviceQuery
296520 -rwxrwxrwx 303633183 Feb 23 gatk-package-4.1.0.0-local.jar
    20 -rwxrwxrwx     19197 Feb 23 gatk_cpu
   572 -rwxrwxrwx    584152 Feb 23 genotypegvcf
146552 -rwxrwxrwx 150068464 Feb 23 glnexus
  2132 -rwxrwxrwx   2182792 Feb 23 htvc
   252 -rwxrwxrwx    255960 Feb 23 indexgvcf
   220 -rwxrwxrwx    223168 Feb 23 licenseManagerTool
   324 -rwxrwxrwx    329688 Feb 23 licenseinfo
   220 -rwxrwxrwx    223104 Feb 23 licensereturn
    44 -rwxrwxrwx     42968 Feb 23 markQueryName
    32 -rwxrwxrwx     31848 Feb 23 mergegvcf_humanpar
  1824 -rwxrwxrwx   1867488 Feb 23 mutect
    12 -rwxrwxrwx     10120 Feb 23 pb_driver
  1000 -rwxrwxrwx   1022960 Feb 23 postsort
   316 -rwxrwxrwx    321512 Feb 23 samtoolsmpileup
   340 -rwxrwxrwx    346616 Feb 23 somaticsniper
   432 -rwxrwxrwx    440296 Feb 23 sort
   364 -rwxrwxrwx    370976 Feb 23 splitncigar
  2920 -rwxrwxrwx   2989784 Feb 23 star
   624 -rwxrwxrwx    636896 Feb 23 starfusion
   468 -rwxrwxrwx    477168 Feb 23 trioCombineGVCF
   768 -rwxrwxrwx    784640 Feb 23 variantfiltration
   312 -rwxrwxrwx    317400 Feb 23 varscan
   792 -rwxrwxrwx    809120 Feb 23 vqsr
```