###### tags: `software` `usage guide`
# GROMACS quick usage guide

## Description
GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles and is a community-driven project.

## Installation Status

| 功能 | Taiwania2 |
| :-------------: | :-------------: |
| native gpu | 2023.4<br>$$\surd$$ |
| GPU container | 2023.2<br>$$\surd$$ |

$$\surd \text{ : Tested} $$

$$\triangle \text{ : Not Ready} $$

$$\star \text{ : Untested} $$

$$\times \text{ : Not support} $$

## Installation Path
### Taiwania2

## Basic usage
### native app (Will not be updated anymore!!)

source setgromacs_2018
mpirun gmx_mpi mdrun -v ${YOUR_INPUT}

### container
Use singularity to run application in container on HPC environment.

module load singularity
singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${GROMACS_SIF} gmx mdrun -v ${YOUR_INPUT}

## Step by Step Usage : 
### Single Node Single GPU (Taiwania2)

* #### STEP0a 
    Pull the image in your own folder

module load singularity
singularity build GROMACS_2022.1.sif docker://nvcr.io/hpc/gromacs:2022.1
export GROMACS_SIF=${PWD}/GROMACS_2022.1.sif

* #### STEP0b (Alternate)
    Use the image in pkg folder   

export GROMACS_SIF=/opt/ohpc/pkg/gromacs/container/gromacs-2022.1.sif

* #### STEP1 Download the submission script

export CWD=`pwd`

cp /opt/ohpc/pkg/gromacs/container/gromacs_snsg.sh $CWD

* #### STEP2 Modify the submission script and change your wallet account

sed 's/JOB_ACCOUNT/XXXXXX/' gromacs_snsg.sh

        
    XXXXXX 填入計畫帳號
    
    or you can edit `gromacs_snsg.sh` in your favirote editor and modify it.
    
* #### STEP3 Prepare your gromacs input files - Run a simple testcase
     * ##### STEP3-1 Download the testcase from gromacs homepage

wget -c ftp://ftp.gromacs.org/pub/benchmarks/water_GMX50_bare.tar.gz

            
    * ##### STEP3-2 Decompression the file 

* ##### STEP3-3 Copy a simple example to current folder

            cp -r /opt/ohpc/pkg/gromacs/example/water-cut1.0_GMX50_bare ./

* ##### STEP3-4 Choose one testcase

    * ##### STEP3-5 Convert file format for the benchmark data

* #### STEP4 Submit the job and execute the benchmark
* Prepare your submission script, modify the execute part.

        singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${GROMACS_SIF} \
                     gmx mdrun \
                     -ntmpi ${GPU_COUNT} \
                     -nb gpu \
                     -ntomp ${OMP_NUM_THREADS} \
                     -pin on \
                     -v \
                     -noconfout \
                     -nsteps 5000 \
                     -s topol.tpr

* ```singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${GROMACS_SIF} ``` : don't modify it
* ```gmx mdrun``` : modify if if you want to run different task
* ```-ntmpi ${GPU_COUNT}``` : don't modify it, it depends on --gres=gpu:1
* ```-nb gpu``` : tell singularity that you want to use gpu
* ```-ntomp ${OMP_NUM_THREADS}``` : don't modify it, it depends on --cpus-per-task=4
* ```-pin on``` : tell slurm to enable CPU Pinning
* ```-noconfout``` : tell mdrun not to output
* ```-nsteps 5000 ``` : tell mdrun the max iter runs
* ```-s topol.tpr ``` : specify the tpr file (it should be generated in step3-5)

the you can submit your job via

    
    Here are the submition script 

    #### gromacs_snsg.sh

#!/bin/bash

#SBATCH -J GROMACS_Singlenode_SingleGPU_Job
#SBATCH -A JOB_ACCOUNT
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
#SBATCH --job-name=gromacs_job
#SBATCH --output=gromacs_output.txt

#module load
module load singularity

# Initialize the path
export APP_ROOT=$PWD
cd $APP_ROOT

# Script arguments
GPU_COUNT=`echo $SLURM_JOB_GPUS | tr "," " " | wc -w`
GROMACS_SIF=/opt/ohpc/pkg/gromacs/container/gromacs-2022.1.sif

# Set number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}

# Singularity will mount the host PWD to /host_pwd in the container
SINGULARITY="singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${GROMACS_SIF}"

# Prepare benchmark data
${SINGULARITY} gmx grompp -f pme.mdp

# Run benchmark
${SINGULARITY} gmx mdrun \
-ntmpi ${GPU_COUNT} \
-nb gpu \
-ntomp ${OMP_NUM_THREADS} \
-pin on \
-v \
-noconfout \
-nsteps 5000 \
-s topol.tpr

    
### Single Node multiple gpus (Taiwania2)
    
The SLURM directives for multi gpu requests are the --ntasks=# and --gres=gpu:# .

Replace # to the number of GPUs. (7>#>1). The rest parts are same with single gpu version.
    
Here are the submition script 
    
#### gromacs_snmg.sh

#!/bin/bash

#SBATCH -J GROMACS_Singlenode_MultipleGPU_Job
#SBATCH -A JOB_ACCOUNT
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:2
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
#SBATCH --job-name=gromacs_job
#SBATCH --output=gromacs_output.txt

#module load
module load singularity

# Initialize the path
export APP_ROOT=$PWD
cd $APP_ROOT

# Script arguments
GPU_COUNT=`echo $SLURM_JOB_GPUS | tr "," " " | wc -w`
GROMACS_SIF=/opt/ohpc/pkg/gromacs/container/gromacs-2022.1.sif

# Set number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}

# Singularity will mount the host PWD to /host_pwd in the container
SINGULARITY="singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${GROMACS_SIF}"

# Prepare benchmark data
${SINGULARITY} gmx grompp -f pme.mdp

# Run benchmark
${SINGULARITY} gmx mdrun \
-ntmpi ${GPU_COUNT} \
-nb gpu \
-ntomp ${OMP_NUM_THREADS} \
-pin on \
-v \
-noconfout \
-nsteps 5000 \
-s topol.tpr

    
Note: If you want to use specific version, you can download by the following instruction.

## Fetch image from NGC (optional)
##### SELECT TAG
###### Several GROMACS images are available, depending on your needs.

export GROMACS_TAG={TAG}

###### For example:

export GROMACS_TAG=2023.2

##### PULL THE IMAGE

module load singularity
export GROMACS_SIF=${PWD}/${GROMACS_TAG}.sif
singularity build ${GROMACS_SIF} docker://nvcr.io/hpc/gromacs:${GROMACS_TAG}

#### Contributor: YI-CHENG HSIAO
## Container usage on naive gpu (2023.2)

cp -r /opt/ohpc/pkg/gromacs/container/gromacs_2023.2 $HOME/gromacs_job
cd $HOME/gromacs_job

### run single node single gpu

sbatch gromacs_single_node_single_gpu.sh

#### gromacs_single_node_single_gpu.sh

#!/bin/bash

#SBATCH --job-name=GromacsSNSG
#SBATCH --account=<JobAccount>
#SBATCH --partition=gp1d
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
#SBATCH --output=gromacs_output.txt

module load singularity

# Initialize the path
export APP_ROOT=$HOME/gromacs_job
cd $APP_ROOT

# Script arguments
GPU_COUNT=`echo $SLURM_JOB_GPUS | tr "," " " | wc -w`
SIMG=${2:-"${PWD}/GROMACS_2023.2.sif"}

# Set number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}

# Singularity will mount the host PWD to /host_pwd in the container
SINGULARITY="singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${SIMG}"

# Prepare benchmark data
${SINGULARITY} gmx grompp -f pme.mdp

# Run benchmark
${SINGULARITY} gmx mdrun \
-ntmpi 1 \
-nb gpu \
-ntomp ${OMP_NUM_THREADS} \
-v \
-nsteps 5000 \
-s stmv.tpr

#### view result

cat gromacs_output.txt

### run single node multiple gpu

sbatch gromacs_single_node_single_gpu.sh

#### gromacs_single_node_multi_gpus.sh

#!/bin/bash

#SBATCH --job-name=GromacsSNMG
#SBATCH --account=<JobAccount>
#SBATCH --partition=gp1d
#SBATCH --nodes=1
#SBATCH --gres=gpu:2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
#SBATCH --output=gromacs_output.txt

#module load
module load singularity

# Initialize the path
export APP_ROOT=$HOME/gromacs_job
cd $APP_ROOT

# Script arguments
GPU_COUNT=`echo $SLURM_JOB_GPUS | tr "," " " | wc -w`
SIMG=${2:-"${PWD}/GROMACS_2023.2.sif"}

# Set number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}

# Singularity will mount the host PWD to /host_pwd in the container
SINGULARITY="singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${SIMG}"

# Run benchmark
${SINGULARITY} gmx mdrun \
-ntmpi ${GPU_COUNT} \
-nb gpu \
-ntomp ${OMP_NUM_THREADS} \
-v \
-nsteps 5000 \
-s stmv.tpr

#### view result

cat gromacs_output.txt

### run multiple node multiple gpu

sbatch gromacs_multi_nodes_multi_gpus.sh

#### gromacs_multi_nodes_multi_gpus.sh

#!/bin/bash

#SBATCH --job-name=GromacsMNMG
#SBATCH --account=<JobAccount>
#SBATCH --partition=gp1d
#SBATCH --nodes=2
#SBATCH --gres=gpu:4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
#SBATCH --job-name=gromacs_job
#SBATCH --output=gromacs_output.txt

#module load
module load singularity

# Initialize the path
export APP_ROOT=$HOME/gromacs_job
cd $APP_ROOT

# Script arguments
GPU_COUNT=`echo $SLURM_JOB_GPUS | tr "," " " | wc -w`
SIMG=${2:-"${PWD}/GROMACS_2023.2.sif"}

# Set number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}

# Singularity will mount the host PWD to /host_pwd in the container
SINGULARITY="singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${SIMG}"

# Prepare benchmark data
${SINGULARITY} gmx grompp -f pme.mdp

# Run benchmark
${SINGULARITY} gmx mdrun \
-ntmpi ${GPU_COUNT} \
-nb gpu \
-ntomp ${OMP_NUM_THREADS} \
-pin on \
-v \
-nsteps 5000 \
-s stmv.tpr

#### view result

cat gromacs_output.txt

## Module usage on naive gpu (2023.4)
### run task on GPU

cp -r /opt/ohpc/pkg/gromacs/2023.4/example $HOME/gromacs_job/
cd $HOME/gromacs_job
sbatch gromacs_gpu_example.sh

#### gromacs_gpu_example.sh

#!/bin/bash

#SBATCH --job-name=GromacsGPUTask
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --account=<JobAccount>
#SBATCH --partition=gp1d
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1
#SBATCH --output=output.txt

source /opt/ohpc/pkg/gromacs/2023.4/gromacs/GromacsUsage.sh

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}

cd $HOME/gromacs_job
srun --mpi=openmpi gmx_mpi mdrun \
-v -s stmv.tpr \
-ntomp ${OMP_NUM_THREADS} \
-nb gpu -nsteps 5000

#### view result

cat $HOME/gromacs_job/output.txt

Contributor: SHIH-HSUN WEI