<a href="https://hackmd.io/@teoroo-cluster" alt="Teoroo Cluster">
<img src="https://img.shields.io/badge/Teoroo--CMC-All%20Public%20Notes-black?style=flat-square" /></a>

# LUMI

Some random experience building and running CP2K on the LUMI HPC

## Building CP2K with EasyBuild

# General slurm settings
export SBATCH_ACCOUNT='project_xxxxxxxxx'
export SBATCH_PARTITION='standard'

# Setup EasyBuild
module load LUMI/21.12 partition/C
module load EasyBuild-user/LUMI
export EASYBUILD_JOB_BACKEND='Slurm'
# those two lines are for compiling with SLURM, if you compile locally, the defaults will be OK
export EASYBUILD_BUILDPATH='/tmp'
export EASYBUILD_TMPDIR='/tmp'

# The LUMI contributed repo of build scripts
git clone https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib.git
cd LUMI-EasyBuild-contrib
eb easybuild/easyconfigs/c/CP2K/CP2K-9.1-cpeGNU-21.08.eb -r . --job --job-cores 32 --ignore-osdeps
# the last bit is to get around the osdep of Libint in the contrib repo, usually it's not necessary

# Finally this should work
module load CP2K/9.1-cpeGNU-21.08

:::info
The above script works if you replace the CP2K version 9.1 with 8.2 (adjust the file names in the CP2K directory, and the version number in `CP2K-8.2-cpeGNU-21.08.eb` file accordingly). But the dependency somehow breaks with 8.1.
:::

## Jobs scripts

### CP2K, with EasyBuild

#!/bin/bash -l
#SBATCH --job-name=cp2k-eb-91
#SBATCH --output=cp2k-eb-91.out
#SBATCH --account=project_xxxxxxxxx
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=8
#SBATCH --partition=standard

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
ml LUMI/21.08 partition/C
ml EasyBuild-user/LUMI
ml CP2K/9.1-cpeGNU-21.08

srun cp2k.psmp cp2k-nvt.inp

### CP2K bechmark scripts

**Timing for $\mathbf{64 \times [C_1Mim][OAc]}$ system**

| Nodes | Task per node | cpu per task | Binary | per MD | per OT |
| ----- | ------------- | ------------ | -------- | -------- | -------- |
| 2 | 128 | 1 | popt | 6.65 | 0.79 |
| 2 | 128 | 1 | psmp | 6.67 | 0.79 |
| 2 | 64 | 2 | psmp | 5.93 | 0.69 |
| **2** | **32** | **4** | **psmp** | **5.18** | **0.59** |
| 4 | 128 | 1 | popt | 6.17 | 0.77 |
| 4 | 128 | 1 | psmp | 6.19 | 0.76 |
| 4 | 64 | 2 | psmp | 3.65 | 0.45 |
| **4** | **32** | **4** | **psmp** | **3.33** | **0.37** |
| 8 | 128 | 1 | popt | 6.63 | 0.75 |
| 8 | 128 | 1 | psmp | 6.59 | 0.75 |
| 8 | 64 | 2 | psmp | 3.45 | 0.45 |
| **8** | **32** | **4** | **psmp** | **2.33** | **0.25** |

Stability issue with `ntask-per-node<=4`.

**nextflow script for benchmark**

#!/usr/bin/env nextflow

configs = [
  [2, 32, 4, 'psmp'],
  [4, 32, 4, 'psmp'],
  [8, 32, 4, 'psmp'],
  [2, 64, 2, 'psmp'],
  [4, 64, 2, 'psmp'],
  [8, 64, 2, 'psmp'],
  [2, 128, 1, 'psmp'],
  [4, 128, 1, 'psmp'],
  [8, 128, 1, 'psmp'],
  [2, 128, 1, 'popt'],
  [4, 128, 1, 'popt'],
  [8, 128, 1, 'popt'],
]
  
process cp2k {
    publishDir "benchmark/N$n-T$t-C$c-$binary", mode: 'link'
    clusterOptions "-A SNIC2022-1-27 --nodes $n --ntasks-per-node $t --cpus-per-task ${c*2} -p main"
    time '1d'        // for dardel, set cpus to twice the actual cpu --------------------^
                     // see: https://www.pdc.kth.se/support/documents/run_jobs/job_scripts_dardel.html

    input:
      path input
      path aux
      tuple (val(n), val(t), val(c), val(binary))

    output:
      path 'cp2k*', optional:true, emit:log

    """
    #!/bin/bash
    export OMP_NUM_THREADS=$c
    export OMP_PLACES=cores
    srun cp2k.$binary -i $input | tee cp2k.log
    """
}

workflow {
  ch_input = file('inputs/bench_cp2k.inp')
  ch_aux = file('inputs/cp2k/*')
  ch_config = Channel.fromList(configs)
  cp2k(ch_input, ch_aux, ch_config)
}

**nextflow config**

profiles {
  dardel {
    process {
      errorStrategy='ignore'
      executor='slurm'
      withLabel: cp2k {
        module='PDC/21.11 CP2K/9.1-cpeGNU-21.11'
        clusterOptions='-A SNIC2021-1-32 -N 8 -n 224'
        time='5d'
      }
    }
    executor {
      name = 'slurm'
      queueSize = 50
      queueStatInterval = '2min'
      submitRateLimit = '120 min'
    }
  }
}

**script for report**

#!/usr/bin/env bash
for dir in */
do 
  printf '%-20s  %.2f  %.2f \n' $dir\
  `grep per\ MD $dir/cp2k.log | tail -n10 | awk '{total += $NF; count++} END { print total/count}'`\
  `grep OT\ DIIS $dir/cp2k.log | tail -n100 | awk '{total += $5F; count++} END { print total/count}'`
done