---
# System prepended metadata

title: MEX HGX A100 interactive tests

---

# MEX HGX A100 interactive tests 

## Single GPU test

Working directory  : */nesi/project/nesi99999/Dinindu/ont-basecalling* 

Command 
```bash
$ time guppy_basecaller -i $PWD/input  -s $PWD/output_fastq \
>  --config /opt/nesi/CS400_centos7_bdw/ont-guppy-gpu/6.1.2/data/dna_r9.4.1_450bps_sup.cfg  \
> --device auto  --recursive --records_per_fastq 4000 --min_qscore 7 
```


### 40G (Old) A100 `wbl008`

```
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 31919 ms, Samples called: 225442427, samples/s: 7.06295e+06
Finishing up any open output files.
Basecalling completed successfully.

real	0m37.426s
user	1m10.992s
sys	0m35.118s
```

```
-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     62396      C   guppy_basecaller                35717MiB |
+-----------------------------------------------------------------------------+
```
### wmg002

```bash
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 29901 ms, Samples called: 225442427, samples/s: 7.53963e+06
Finishing up any open output files.
Basecalling completed successfully.

real	0m36.436s
user	0m39.748s
sys	0m23.602s
```


```bash
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     69077      C   guppy_basecaller                65513MiB |
+-----------------------------------------------------------------------------+
```

### wmg003

```bash!
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 29648 ms, Samples called: 225442427, samples/s: 7.60397e+06
Finishing up any open output files.
Basecalling completed successfully.

real	0m38.397s
user	0m39.280s
sys	0m23.421s

```


```bash!
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    132121      C   guppy_basecaller                65513MiB |
+-----------------------------------------------------------------------------+
```

### wmg004

```
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 28824 ms, Samples called: 225442427, samples/s: 7.82134e+06
Finishing up any open output files.
Basecalling completed successfully.

real	0m38.075s
user	0m38.878s
sys	0m23.717s
```


```
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    142884      C   guppy_basecaller                65513MiB |
+-----------------------------------------------------------------------------+
```

### Four processes on a single GPU - `wmg004`

```
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    145150      C   guppy_basecaller                20827MiB |
|    0   N/A  N/A    145162      C   guppy_basecaller                20827MiB |
|    0   N/A  N/A    145174	 C   guppy_basecaller                15863MiB |
|    0   N/A  N/A    145180      C   guppy_basecaller                20827MiB |
+-----------------------------------------------------------------------------+
```

with --gpu_runners_per_device set to **1**

```
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    145765      C   guppy_basecaller                10725MiB |
|    0   N/A  N/A    145777      C   guppy_basecaller                10725MiB |
|    0   N/A  N/A    145784	 C   guppy_basecaller                10725MiB |
|    0   N/A  N/A    145788      C   guppy_basecaller                10725MiB |
+-----------------------------------------------------------------------------+
```

## `gpu-burn` test

Working directory : `/nesi/project/nesi99999/Dinindu/m-ex_bench/gpu-burn`

```bash!
./gpu-burn -tc -d ..

```


60 seconds test on `wmg004`

Command 

```
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    149601      C   ./gpu_burn                      73049MiB |
|    1   N/A  N/A    149613      C   ./gpu_burn                      73049MiB |
|    2   N/A  N/A    149614	 C   ./gpu_burn                      73049MiB |
|    3   N/A  N/A    149615	 C   ./gpu_burn                      73049MiB |
+-----------------------------------------------------------------------------+
```
```
Tested 4 GPUs:
	GPU 0: OK
	GPU 1: OK
	GPU 2: OK
	GPU 3: OK
```


|        | Temp (Celsius |
--------------------------
| NodeID | 0 | 1 | 2 | 3 |
| -------|---|---|---|---|
| wmg002 |57 |66 |58 |66 |
| wmg003 |67 |73 |66 |77 |
| wmg004 |69 |74 |65 |77 |

Two different cooling profiles on wmg002 vs wmg003+wmg004, `Thermal ConfigurationMaximum Cooling` vs `Thermal ConfigurationOptimal Cooling`, respectively. 

All changed to `Thermal ConfigurationMaximum` now


|        | Temp (Celsius)|
--------------------------
| NodeID | 0 | 1 | 2 | 3 |
| -------|---|---|---|---|
| wmg002 |57 |66 |58 |65 |
| wmg003 |59 |61 |59 |65 |
| wmg004 |59 |61 |57 |63 |


## AlphaFold2 runs

* Working Directory : `/nesi/project/nesi99999/Dinindu/alphafold`
```
module purge
module unload XALT
module load cuDNN/8.1.1.33-CUDA-11.2.0 Singularity/3.9.8

DOWNLOAD_DIR=/opt/nesi/db/alphafold_db
INPUT=/nesi/project/nesi99999/Dinindu/alphafold/input_data
OUTPUT=/nesi/project/nesi99999/Dinindu/alphafold/results

singularity exec -B /nesi/project/nesi99999/Dinindu/alphafold/ -B $DOWNLOAD_DIR --nv /opt/nesi/containers/AlphaFold/alphafold_2.2.0.simg python /app/alphafold/run_alphafold.py  \
 --use_gpu_relax \
 --data_dir=$DOWNLOAD_DIR  \
 --uniref90_database_path=$DOWNLOAD_DIR/uniref90/uniref90.fasta  \
 --mgnify_database_path=$DOWNLOAD_DIR/mgnify/mgy_clusters_2018_12.fa  \
 --bfd_database_path=$DOWNLOAD_DIR/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt  \
 --uniclust30_database_path=$DOWNLOAD_DIR/uniclust30/uniclust30_2018_08/uniclust30_2018_08 \
 --pdb_seqres_database_path=$DOWNLOAD_DIR/pdb_seqres/pdb_seqres.txt \
 --template_mmcif_dir=$DOWNLOAD_DIR/pdb_mmcif/mmcif_files  \
 --obsolete_pdbs_path=$DOWNLOAD_DIR/pdb_mmcif/obsolete.dat \
 --uniprot_database_path=$DOWNLOAD_DIR/uniprot/uniprot.fasta \
 --model_preset=multimer \
 --max_template_date=2022-1-1 \
 --db_preset=full_dbs \
 --output_dir=$OUTPUT \
 --fasta_paths=/nesi/project/nesi99999/Dinindu/alphafold/input_data/test_multimer.fasta

```

```
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     83982	 C   /opt/conda/bin/python           73039MiB |
|    1   N/A  N/A     83982	 C   /opt/conda/bin/python             413MiB |
|    2   N/A  N/A     83982	 C   /opt/conda/bin/python             413MiB |
|    3   N/A  N/A     83982	 C   /opt/conda/bin/python             413MiB |
+-----------------------------------------------------------------------------+

```