# MEX HGX A100 interactive tests ## Single GPU test Working directory : */nesi/project/nesi99999/Dinindu/ont-basecalling* Command ```bash $ time guppy_basecaller -i $PWD/input -s $PWD/output_fastq \ > --config /opt/nesi/CS400_centos7_bdw/ont-guppy-gpu/6.1.2/data/dna_r9.4.1_450bps_sup.cfg \ > --device auto --recursive --records_per_fastq 4000 --min_qscore 7 ``` ### 40G (Old) A100 `wbl008` ``` 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Caller time: 31919 ms, Samples called: 225442427, samples/s: 7.06295e+06 Finishing up any open output files. Basecalling completed successfully. real 0m37.426s user 1m10.992s sys 0m35.118s ``` ``` -----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 62396 C guppy_basecaller 35717MiB | +-----------------------------------------------------------------------------+ ``` ### wmg002 ```bash 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Caller time: 29901 ms, Samples called: 225442427, samples/s: 7.53963e+06 Finishing up any open output files. Basecalling completed successfully. real 0m36.436s user 0m39.748s sys 0m23.602s ``` ```bash +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 69077 C guppy_basecaller 65513MiB | +-----------------------------------------------------------------------------+ ``` ### wmg003 ```bash! 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Caller time: 29648 ms, Samples called: 225442427, samples/s: 7.60397e+06 Finishing up any open output files. Basecalling completed successfully. real 0m38.397s user 0m39.280s sys 0m23.421s ``` ```bash! +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 132121 C guppy_basecaller 65513MiB | +-----------------------------------------------------------------------------+ ``` ### wmg004 ``` 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Caller time: 28824 ms, Samples called: 225442427, samples/s: 7.82134e+06 Finishing up any open output files. Basecalling completed successfully. real 0m38.075s user 0m38.878s sys 0m23.717s ``` ``` +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 142884 C guppy_basecaller 65513MiB | +-----------------------------------------------------------------------------+ ``` ### Four processes on a single GPU - `wmg004` ``` +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 145150 C guppy_basecaller 20827MiB | | 0 N/A N/A 145162 C guppy_basecaller 20827MiB | | 0 N/A N/A 145174 C guppy_basecaller 15863MiB | | 0 N/A N/A 145180 C guppy_basecaller 20827MiB | +-----------------------------------------------------------------------------+ ``` with --gpu_runners_per_device set to **1** ``` +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 145765 C guppy_basecaller 10725MiB | | 0 N/A N/A 145777 C guppy_basecaller 10725MiB | | 0 N/A N/A 145784 C guppy_basecaller 10725MiB | | 0 N/A N/A 145788 C guppy_basecaller 10725MiB | +-----------------------------------------------------------------------------+ ``` ## `gpu-burn` test Working directory : `/nesi/project/nesi99999/Dinindu/m-ex_bench/gpu-burn` ```bash! ./gpu-burn -tc -d .. ``` 60 seconds test on `wmg004` Command ``` +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 149601 C ./gpu_burn 73049MiB | | 1 N/A N/A 149613 C ./gpu_burn 73049MiB | | 2 N/A N/A 149614 C ./gpu_burn 73049MiB | | 3 N/A N/A 149615 C ./gpu_burn 73049MiB | +-----------------------------------------------------------------------------+ ``` ``` Tested 4 GPUs: GPU 0: OK GPU 1: OK GPU 2: OK GPU 3: OK ``` | | Temp (Celsius | -------------------------- | NodeID | 0 | 1 | 2 | 3 | | -------|---|---|---|---| | wmg002 |57 |66 |58 |66 | | wmg003 |67 |73 |66 |77 | | wmg004 |69 |74 |65 |77 | Two different cooling profiles on wmg002 vs wmg003+wmg004, `Thermal ConfigurationMaximum Cooling` vs `Thermal ConfigurationOptimal Cooling`, respectively. All changed to `Thermal ConfigurationMaximum` now | | Temp (Celsius)| -------------------------- | NodeID | 0 | 1 | 2 | 3 | | -------|---|---|---|---| | wmg002 |57 |66 |58 |65 | | wmg003 |59 |61 |59 |65 | | wmg004 |59 |61 |57 |63 | ## AlphaFold2 runs * Working Directory : `/nesi/project/nesi99999/Dinindu/alphafold` ``` module purge module unload XALT module load cuDNN/8.1.1.33-CUDA-11.2.0 Singularity/3.9.8 DOWNLOAD_DIR=/opt/nesi/db/alphafold_db INPUT=/nesi/project/nesi99999/Dinindu/alphafold/input_data OUTPUT=/nesi/project/nesi99999/Dinindu/alphafold/results singularity exec -B /nesi/project/nesi99999/Dinindu/alphafold/ -B $DOWNLOAD_DIR --nv /opt/nesi/containers/AlphaFold/alphafold_2.2.0.simg python /app/alphafold/run_alphafold.py \ --use_gpu_relax \ --data_dir=$DOWNLOAD_DIR \ --uniref90_database_path=$DOWNLOAD_DIR/uniref90/uniref90.fasta \ --mgnify_database_path=$DOWNLOAD_DIR/mgnify/mgy_clusters_2018_12.fa \ --bfd_database_path=$DOWNLOAD_DIR/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \ --uniclust30_database_path=$DOWNLOAD_DIR/uniclust30/uniclust30_2018_08/uniclust30_2018_08 \ --pdb_seqres_database_path=$DOWNLOAD_DIR/pdb_seqres/pdb_seqres.txt \ --template_mmcif_dir=$DOWNLOAD_DIR/pdb_mmcif/mmcif_files \ --obsolete_pdbs_path=$DOWNLOAD_DIR/pdb_mmcif/obsolete.dat \ --uniprot_database_path=$DOWNLOAD_DIR/uniprot/uniprot.fasta \ --model_preset=multimer \ --max_template_date=2022-1-1 \ --db_preset=full_dbs \ --output_dir=$OUTPUT \ --fasta_paths=/nesi/project/nesi99999/Dinindu/alphafold/input_data/test_multimer.fasta ``` ``` +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 83982 C /opt/conda/bin/python 73039MiB | | 1 N/A N/A 83982 C /opt/conda/bin/python 413MiB | | 2 N/A N/A 83982 C /opt/conda/bin/python 413MiB | | 3 N/A N/A 83982 C /opt/conda/bin/python 413MiB | +-----------------------------------------------------------------------------+ ```