# Delft RaspberryPi HPC workshop part 2 notes, 19 November 2021

# Workshop 2: Parallel Computing with a RaspberryPi cluster

:::info
First part of this workshop can be found here:
[Workshop 1: Building a RaspberryPi cluster](https://hackmd.io/Vemik7JuS_Cep_fd1WwiHA)
:::


Notes prepared by: Dennis Palagin, Jose Urra Llanusa, Jerry de Vos

Special thanks go to Kees Lemmens and Joffrey Wallaart for their excellent [MPI course](https://www.tudelft.nl/cse/education/courses/mpi-programming) we used as an inspiration for slides.

**Event description: https://www.eventbrite.co.uk/e/delft-open-hardware-hack-and-playhpc-with-a-raspberry-pi-cluster-ii-tickets-189735513027**

<iframe src="https://docs.google.com/presentation/d/17s51k8n0BmiMxXaJ5BMUUV-pSX6Ze_TH/embed?start=false&loop=false&delayms=3000" frameborder="0" width="960" height="749" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>

# Recap from the [previous workshop](https://hackmd.io/Vemik7JuS_Cep_fd1WwiHA)

1. The [previous workshop](https://hackmd.io/Vemik7JuS_Cep_fd1WwiHA) ended with us having a functional RaspberryPi cluster, which should be ready to go. Login to the controller node and test slurm to make sure it works. Run the `sinfo` command and you should get the following output (in this example, for the RaspberryPi 3 with a ClusterHAT and 4 Pi Zero as nodes):

PARTITION  AVAIL TIMELIMIT NODES STATE NODELIST
mycluster*    up  infinite     5  idle  <cluster-name>,p[1-4]

:::warning
<details>
<summary>Got an error (e.g. STATE = "down")?</summary>

If the nodes are reporting as being "down", e.g. after a reboot of the node, then a simple re-initialisation will most likely help:

sudo scontrol update nodename=p1 state=idle

</details>
:::

2. The `scontrol show nodes` command gives you the information about available nodes in your cluster:

NodeName=cnat Arch=armv7l CoresPerSocket=1
   CPUAlloc=0 CPUTot=2 CPULoad=0.08
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=172.19.181.254 NodeHostName=cnat Version=18.08
   OS=Linux 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020
   RealMemory=1 AllocMem=0 FreeMem=176 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=2 Owner=N/A MCS_label=N/A
   Partitions=mycluster
   BootTime=2021-10-14T09:13:12 SlurmdStartTime=2021-10-14T09:13:26
   CfgTRES=cpu=2,mem=1M,billing=2
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s


NodeName=p1 Arch=armv6l CoresPerSocket=1
   CPUAlloc=0 CPUTot=1 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=172.19.181.1 NodeHostName=p1 Version=18.08
   OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020
   RealMemory=1 AllocMem=0 FreeMem=137 Sockets=1 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=mycluster
   BootTime=2021-10-14T09:15:42 SlurmdStartTime=2021-10-14T09:16:38
   CfgTRES=cpu=1,mem=1M,billing=1
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s


NodeName=p2 Arch=armv6l CoresPerSocket=1
   CPUAlloc=0 CPUTot=1 CPULoad=0.06
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=172.19.181.2 NodeHostName=p2 Version=18.08
   OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020
   RealMemory=1 AllocMem=0 FreeMem=141 Sockets=1 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=mycluster
   BootTime=2021-10-14T09:15:43 SlurmdStartTime=2021-10-14T09:16:43
   CfgTRES=cpu=1,mem=1M,billing=1
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s


NodeName=p3 Arch=armv6l CoresPerSocket=1
   CPUAlloc=0 CPUTot=1 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=172.19.181.3 NodeHostName=p3 Version=18.08
   OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020
   RealMemory=1 AllocMem=0 FreeMem=140 Sockets=1 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=mycluster
   BootTime=2021-10-14T09:15:44 SlurmdStartTime=2021-10-14T09:16:41
   CfgTRES=cpu=1,mem=1M,billing=1
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s


NodeName=p4 Arch=armv6l CoresPerSocket=1
   CPUAlloc=0 CPUTot=1 CPULoad=0.02
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=172.19.181.4 NodeHostName=p4 Version=18.08
   OS=Linux 5.4.79+ #1373 Mon Nov 23 13:18:15 GMT 2020
   RealMemory=1 AllocMem=0 FreeMem=144 Sockets=1 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=mycluster
   BootTime=2021-10-14T09:15:45 SlurmdStartTime=2021-10-14T09:16:40
   CfgTRES=cpu=1,mem=1M,billing=1
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

Similarly, `scontrol show partitions` displays which partitions you configured:

PartitionName=mycluster
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=cnat,p1,p2,p3,p4
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=6 TotalNodes=5 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

3. You can print the hostname on all of the nodes with the following command:

srun --nodes=5 hostname

Which should produce a similar result to the following:

<cluster-name>
p1
p2
p3
p4

To see how many concurrent tasks you can run, you can use the following command:

srun --ntasks=6 hostname

4. Which should produce a similar result to the following:

<cluster-name>
<cluster-name>
p1
p2
p3
p4

5. Main Slurm commands to remember are:

`squeue` — view scheduled jobs

When you start running longer and longer jobs, it is useful to check their status. To do this, run the `squeue` command. By default, it displays all jobs submitted by all users, and their states:

$ squeue
JOBID PARTITION  NAME     USER ST    TIME  NODES NODELIST(REASON)
  609 glmdev     24.sub.s pi   R     10:16     1 node2

Most of this info is pretty self-explanatory. The only thing I'll note is the ST column, which is the state of the job. R means that the job is running.

`scancel` — cancel a scheduled job

Once a job has been scheduled, it can be cancelled using the `scancel` command:

$ scancel 609

(where 609 is the JOBID that you want to cancel). Note that you can only cancel jobs started by your user.

`sbatch` — schedule a batch script

`sbatch` is really the meat and potatoes of the Slurm scheduler. It's what we use most often when we want to schedule a job to run on the cluster. This command takes a number of flags and configuration, as well as a shell file. That shell file is then executed using whatever requested resources (nodes/cores/etc) are made available to it.

# Running jobs in parallel
## Part 1: Let's go through the tutorials by Garrett Mills:

As a warm-up, let's just follow parts 2 and 3 of the original tutorial and select a couple of nice examples:

[Building a Raspberry Pi cluster tutorial Part 2 by Garrett Mills](https://glmdev.medium.com/building-a-raspberry-pi-cluster-aaa8d1f3d2ca)
[Building a Raspberry Pi cluster tutorial Part 3 by Garrett Mills](https://glmdev.medium.com/building-a-raspberry-pi-cluster-f5f2446702e8)

Special thank goes to the createor of the original tutorials [Garrett Mills](https://glmdev.medium.com/).

**6. Example 1: Hello, World!**

Our job begins with the definition of a batch file. This batch file is usually a bash script that runs our job, however it looks a bit different. We’ll create the file `helloworld.sh`:

#!/bin/bash
#SBATCH --ntasks=6
#SBATCH --partition=mycluster

cd $SLURM_SUBMIT_DIR

echo "Hello, World!" >> helloworld.txt
echo "The following nodes are reporting for duty:" >> helloworld.txt
srun hostname >> helloworld.txt
echo "Have a great day!" >> helloworld.txt

The file begins with a "[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix))". This is required, as it tells Slurm how to execute your job. This is followed by a number of flags that take the following form:

#SBATCH <flag>

These flags are simply any parameters that can be passed to the sbatch command. Here, we specify the number of CPUs and the partition of the cluster that we want to use.

The `cd $SLURM_SUBMIT_DIR` guarantees that our job is running in whatever directory it was submitted from.

Now, we can tell Slurm to schedule and run our job:

$ sbatch helloworld.sh
Submitted batch job 639

Since our job is very simple, it should be done basically immediately. If everything has gone according to plan, we should see the `helloworld.txt` file that we created:

Hello, World!
The following nodes are reporting for duty:
cnat
cnat
p3
p1
p2
p4
Have a great day!

You'll notice that the job doesn't output anything to the shell, which makes sense. If you had a job running for 4 hours, it's not very useful to have to have a terminal open the whole time to get output. Instead, Slurm outputs standard error and standard out to a file in the format `slurm-XXX.out` where `XXX` is the Job's ID number.

**7. Example 2: Data processing with R**

For our first project on the cluster, we're going to do some statistics! Data processing is a big part of what HPC clusters are used for. So, we're going to build a simple R program that generates some random values following a normal distribution, then creates a histogram and graph of those values and outputs them to an image file. Then, we're going to create a script to generate 50 of them using the scheduler.

**Goal:** Output 50 randomly-generated normal distribution graphs and histograms to a folder, as images using R.

**Setup:**

Before we can start building our project, we need to install R. For those unfamiliar, [R is a programming language for statistical programming](https://www.r-project.org/). This makes it very well-suited for our task.

Install R on each node with the following command:

sudo apt install r-base -y

When the installation completes, you should be able to use R on any of the nodes:

pi@node2 ~$ R --version
R version 3.3.3 (2017-03-06) -- "Another Canoe"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: arm-unknown-linux-gnueabihf (32-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.

**Theory:**
We need to run a large number of relatively small jobs. So, what we will do is create a script that, when executed, runs the sbatch command to schedule the same job over and over again.

We will use a Slurm job array to do this. Basically, we give the scheduler a script to run, and tell it an array of numbers to run said script, it will run the script once for each number in the array, and the script can access its index during each job. This is how we will generate 50 random normal curves.

**The R program:**

Before we can schedule our program, we need to write a quick R script to generate the normal data-sets. So, we'll create the file `generate.R`:

arg = commandArgs(TRUE)

samples = rep(NA, 100000)

for ( i in 1:100000 ){ samples[i] = mean(rexp(40, 0.2)) }

jpeg(paste('plots/', arg, '.jpg', sep=""))
hist(samples, main="", prob=T, color="darkred")
lines(density(samples), col="darkblue", lwd=3)
dev.off()

**Submissions script:**

Now that we have our R program, we will create a submission script to run our jobs. Create the file `R_submit.sh`:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --partition=mycluster

cd $SLURM_SUBMIT_DIR

mkdir plots
R --vanilla -f generate.R --args "plot$SLURM_ARRAY_TASK_ID"

Here, we tell Slurm to run the job on 1 node, with 1 core on whatever partition you specified. Then, we change directories to the current folder where we will submit the job from.

Then we run the R program. The name of the plot file is set to `plot$SLURM_ARRAY_TASK_ID`. This will name the image file after whatever index of the array we tell Slurm to run our job against. For example, when this job is run for the 23rd time, it will output the file: `plots/plot23.jpg`.

**Run the job:**

We now have everything we need to run our job. From the login node, you can run the job like so:

$ sbatch --array=[1-50] R_submit.sh
Submitted batch job 910

Now, if we run `squeue`, we should see something like so:


:::info
Tip: open a separate terminal window and make it "monitor" the queue with the `watch squeue` command.
:::

When the jobs complete, the `plots` folder should have 50 of our randomly generated histograms, looking similar to this:


**8. Example 3: Hello, World! in C with MPI**

In this part, we're going to set up OpenMPI, and take a look at running some jobs in parallel using MPI to make use of the multiple cluster nodes.

OpenMPI is an open-source implementation of the Message Passing Interface concept. An MPI is a software that connects processes running across multiple computers and allows them to communicate as they run. This is what allows a single script to run a job spread across multiple cluster nodes.

**Setup:**

Install OpenMPI on every node:

$ sudo apt install openmpi-bin openmpi-common libopenmpi3 libopenmpi-dev -y

**C code:**

We're going to create a C program that creates an MPI cluster with the resources SLURM allocates to our job. Then, it's going to call a simple print command on each process.

Create the file `hello_mpi.c` with the following contents:

#include <stdio.h>
#include <mpi.h>
int main(int argc, char** argv){
    int node;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &node);
    printf("Hello World from Node %d!\n", node);
    MPI_Finalize();
}

Here, we include the `mpi.h` library provided by OpenMPI. Then, in the main function, we initialize the MPI cluster, get the number of the node that the current process will be running on, print a message, and close the MPI cluster.

We need to compile our C program to run it on the cluster. However, unlike with a normal C program, we won't just use `gcc` like you might expect. Instead, OpenMPI provides a compiler that will automatically link the MPI libraries:

$ mpicc hello_mpi.c

The `a.out` file is the compiled program that will be run by the cluster.

**Submission script:**

Now, we will create the submission script that runs our program on the cluster. Create the file `sub_mpi.sh`:

#!/bin/bash
#SBATCH --ntasks=6

cd $SLURM_SUBMIT_DIR

# Print the node that starts the process
echo "Master node: $(hostname)"

# Run our program using OpenMPI.
# OpenMPI will automatically discover resources from SLURM.
mpirun a.out

**Run the job:**

Run the job by submitting it to SLURM:

$ sbatch sub_mpi.sh
Submitted batch job 1211

If we have everything working properly, this should create an MPI cluster with 6 nodes. Assuming this works, we should see some output in our `slurm-XXX.out` file:

Master node: cnat
Hello World from CPU 1!
Hello World from CPU 0!
Hello World from CPU 3!
Hello World from CPU 5!
Hello World from CPU 4!
Hello World from CPU 2!

**9. Example 4: Python with MPI: mpi4py**

We're going to throw together a quick Python job that uses OpenMPI. To interface with OpenMPI in Python, we're going to be using a fantastic library called [`mpi4py`](https://mpi4py.github.io/usrman/).

For our demo, we're going to use one of the demo programs in the `mpi4py` repo. We're going to calculate the value of pi (the number) in parallel.

Before we can write our script, we need to install a few libraries. Namely, we will install the `mpi4py` library, and `numpy`. NumPy is a package that contains many useful structures and operations used for scientific computing in Python. We can install these libraries through `pip` on every node:

python3 -m pip install numpy mpi4py

**Create the Python program**

As mentioned above, we're going to use one of the demo programs provided in the `mpi4py` repo. However, because we'll be running it through the scheduler, we need to modify it to not require any user input. Create the file `calculate_pi.py`:

from mpi4py import MPI
from math   import pi as PI
from numpy  import array

def comp_pi(n, myrank=0, nprocs=1):
    h = 1.0 / n
    s = 0.0
    for i in range(myrank + 1, n + 1, nprocs):
        x = h * (i - 0.5)
        s += 4.0 / (1.0 + x**2)
    return s * h

def prn_pi(pi, PI):
    message = "pi is approximately %.16f, error is %.16f"
    print  (message % (pi, abs(pi - PI)))

comm = MPI.COMM_WORLD
nprocs = comm.Get_size()
myrank = comm.Get_rank()

n    = array(0, dtype=int)
pi   = array(0, dtype=float)
mypi = array(0, dtype=float)

if myrank == 0:
    _n = 20 # Enter the number of intervals
    n.fill(_n)
comm.Bcast([n, MPI.INT], root=0)
_mypi = comp_pi(n, myrank, nprocs)
mypi.fill(_mypi)
comm.Reduce([mypi, MPI.DOUBLE], [pi, MPI.DOUBLE],
            op=MPI.SUM, root=0)
if myrank == 0:
    prn_pi(pi, PI)

This program will split the work of computing our approximation of pi out to however many processes we provide it. Then, it will print the computed value of pi, as well as the error from the stored value of pi.

**Create and submit the job**

We can run our job using the scheduler. We will request some number of cores from the cluster, and SLURM will pre-configure the MPI environment with those cores. Then, we just run our Python program using OpenMPI. Let's create the submission file `sub_calc_pi.sh`:

#!/bin/bash
#SBATCH --ntasks=6

cd $SLURM_SUBMIT_DIR

mpiexec -n 6 python3 calculate_pi.py

Here, we use the `--ntasks` flag, which requests a specific number of cores total. Because we are using MPI, we can have cores across machines. Therefore, we can just request the number of cores that we want. In this case, we ask for 6 cores.

To run the actual program, we use `mpiexec` and tell it we have 6 cores. We tell OpenMPI to execute our Python program using the version of Python we installed.

Note that you can adjust the number of cores to be higher/lower as you want. Just make sure you change the `mpiexec -n ##` flag to match.

Finally, we can run the job:

$ sbatch sub_calc_pi.sh
Submitted batch job 1215

The calculation should only take a couple seconds on the cluster. When the job completes (remember, you can monitor it with `squeue`), we should see some output in the `slurm-####.out` file:

$ cat slurm-1215.out
pi is approximately 3.1418009868930934, error is 0.0002083333033003

## Part 2: Specific application examples:
### Computational materials science example: Surface diffusion energy barriers using the Nudged Elastic Band (NEB) method

This excercise is based on the ASE tutrial: https://wiki.fysik.dtu.dk/ase/tutorials/neb/diffusion.html

Learn more about `ASE` here: https://wiki.fysik.dtu.dk/ase/about.html

More tutorials can be found here: https://wiki.fysik.dtu.dk/ase/tutorials/tutorials.html

**Theory**

The Nudged Elastic Band (NEB) method is a technique for finding transition paths (and corresponding energy barriers) between given initial and final states of the atomic (molecular) system. The method involves constructing a "chain" of "replicas" or "images" of the system and relaxing their geometries in a certain way.

**Figure**: Pictorial representation of a reaction path computed with NEB. Taken from https://www.scm.com/doc/AMS/Tasks/NEB.html

At the beginning of a NEB calculation, the geometry of the initial and final systems are optimized to minimize their energy.

Then, a rough approximation of the reaction path is built: a set of images is created by performing a linear interpolation between the initial and final systems. Optionally, an intermediate system can be provided, in which case the interpolation is performed between the initial and intermediate systems, and then between the intermediate and final systems.

Finally, a reaction path is found by performing a simultaneous optimization of all the images. In the NEB method the images are not independent from each other. The force on each image depend on its neighboring images: at each step the forces parallel to the reaction path are eliminated and a so-called spring force is added that tries to keep each image in the middle between its neighbors. This does not let images slide to the initial or final reaction state and ensures that they are evenly distributed along the reaction path.

#### 10. First, install the prerequisites: `ase` and `gpaw` modules:

python3 -m pip install --upgrade --user ase
sudo apt install libxc-dev
python3 -m pip install gpaw

:::info
Tip: install `ase` on your laptop, and you will be able to access the `ase gui` command. This will let you use the graphical atoms/molecules editor and a way to easily visualize the results of your calculations. As it is written in python, it is easily installed on any operating system.
:::

#### 11. Second, set up the initial and final states of the NEB calculation:

For this, we will be using the following python script:

from ase.build import fcc100, add_adsorbate
from ase.constraints import FixAtoms
from ase.calculators.emt import EMT
from ase.optimize import QuasiNewton

# 2x2-Al(001) surface with 3 layers and an
# Au atom adsorbed in a hollow site:
slab = fcc100('Al', size=(2, 2, 3))
add_adsorbate(slab, 'Au', 1.7, 'hollow')
slab.center(axis=2, vacuum=4.0)

# Make sure the structure is correct:
# view(slab)

# Fix second and third layers:
mask = [atom.tag > 1 for atom in slab]
# print(mask)
slab.set_constraint(FixAtoms(mask=mask))

# Use EMT potential:
slab.calc = EMT()

# Initial state:
qn = QuasiNewton(slab, trajectory='initial.traj')
qn.run(fmax=0.05)

# Final state:
slab[-1].x += slab.get_cell()[0, 0] / 2
qn = QuasiNewton(slab, trajectory='final.traj')
qn.run(fmax=0.05)

Run this script, with:

python3 generate_traj.py

This script produces two files, "inital.traj" and "final.traj", which we will use in the actual NEB calculation.

#### 12. Finally, run the parallel calculation:

Instead of having one process do the calculations for all three internal images in turn, it will be faster to have three processes do one image each. In order to be able to run python with MPI you need a special parallel python interpreter, for example `gpaw`.

from ase.io import read
from ase.constraints import FixAtoms
from ase.calculators.emt import EMT
from ase.neb import NEB
from ase.optimize import BFGS
from ase.parallel import world

from gpaw import GPAW

initial = read('initial.traj')
final = read('final.traj')

constraint = FixAtoms(mask=[atom.tag > 1 for atom in initial])

images = [initial]
j = world.rank * 3 // world.size  # my image number
for i in range(3):
    image = initial.copy()
    if i == j:
        image.calc = EMT()
    image.set_constraint(constraint)
    images.append(image)
images.append(final)

neb = NEB(images, parallel=True)
neb.interpolate()
qn = BFGS(neb, trajectory='neb.traj')
qn.run(fmax=0.05)

This example can then be run with `mpiexec -np 6 python3 gpaw_neb.py`. The submission script will look as follows:

#!/bin/bash
#SBATCH --ntasks=6
#SBATCH --partition=mycluster

cd $SLURM_SUBMIT_DIR

mpiexec -np 6 python3 gpaw_neb.py

Now, simply submit your job to the queue:

sbatch sub_ase.sh

Your slurm output file will log the energies of the optimization iterations and will look something like this:

      Step     Time          Energy         fmax
BFGS:    0 13:23:21        4.219952        3.5208
BFGS:    1 13:23:22        3.937039        2.1765
BFGS:    2 13:23:23        3.719814        0.4351
BFGS:    3 13:23:24        3.709652        0.2301
BFGS:    4 13:23:26        3.708879        0.2441
BFGS:    5 13:23:27        3.706088        0.2577
BFGS:    6 13:23:28        3.698532        0.2134
BFGS:    7 13:23:29        3.692121        0.2462
BFGS:    8 13:23:30        3.692274        0.1873
BFGS:    9 13:23:31        3.693484        0.1727
BFGS:   10 13:23:32        3.692659        0.1514
BFGS:   11 13:23:33        3.690809        0.0736
BFGS:   12 13:23:34        3.690202        0.0708
BFGS:   13 13:23:35        3.690382        0.0782
BFGS:   14 13:23:36        3.690426        0.1034
BFGS:   15 13:23:37        3.689890        0.0998
BFGS:   16 13:23:39        3.689029        0.0543
BFGS:   17 13:23:40        3.688737        0.0289

:::info
Tip: you can see the slurm output file being written in real time with the `tail -f slurm-xxx.out` command.
:::

The program will also generate the file called `neb.traj`, containing the final coordinates trajectory.

To visualize the results, you can create a series of plots, that show the progression of the NEB relaxation, directly at the command line:

ase nebplot --share-x --share-y neb.traj

This will generate a PDF file called `nebplots.pdf`, containing the images looking like this:


# Troubleshooting
## This will not work in our current setup

$ sbatch --nodes=3 --ntasks-per-node=2 sub_mpi.sh

This is because we are requesting more cores than what we actually have. We have 3 nodes, but we can only run one task per node, because the raspberrypi zeroes have only one.

Pizero processor: Broadcom BCM2835,1GHz ARM11 Single-core processor,512MB RAM.

Raspberry pi 3B (The controller node): Quad core Cortex-A72 (ARM v8). Quad-core means four cores.

# Questions:
Yehor: How to configure slurm task file, so it takes one node (with multiple CPU) and uses all CPU on it, but only on this node?

Dennis:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=N

Martijn: Can slurm help to first run a number of tasks (N2), and based on the outcome start another job to run on other number (N2) tasks?

First helpful links (Yehor):

https://stackoverflow.com/questions/40004378/slurm-how-to-qsub-a-task-when-another-task-is-finished

https://stackoverflow.com/questions/46427148/how-to-hold-up-a-script-until-a-slurm-job-start-with-srun-is-completely-finish