# Introduction to HPC

## Ice Breakers

### Let us know who you are

- Peter Steinbach, FWCC, narrow machine learning is what I do for a day job
- Lokamani, FWCC, Online-Workshops is what I do
- Guido Juckeland, FWCC (coming a little later - in a meeting until 10 a.m.), working on connecting the right people with one another
- Oleksandr Pylypovskyi, FWID, theory in nanomagnetism (analytics + computer simulations)
- Cecilia Contreras, FWGE, PhD Candidate, I work with hyperspectral data for mineral exploration using machine learning.
- Pia Hanfeld, FWU, I'm a cybersecurity master's student currently taking deep learning courses and want to know how I can utilize hemera for DL.
- Friedrich Zahn, FWIN, lowering entry barrier to micromagnetic simulations through web-based frontends
- Lukas Körber, FWIN, PhD student, micromagnetic simulations
- Emil Fridman, FWOR
- Ashish Mishra, FWDH, I work on magnetorotational instability theory and simulations.
- Theodore Papapetrou, FWDF, I am working on granular particle mixing.
- Attila Kákay, FWIN, micromagnetic simulations
- Sibel Tas-Köhler, FWDF, I am working on multiphase flow hydrodynamics.
- Florian Ristau, FWDF, I´m working on research for waste water treatment plants
- Tom Weier, FWDH, working on liquid metal batteries
- Anna Willmann, FWKT, working on master thesis on surrogate modelling of Shadowgraphy
- Tobias Förster, HLD, pulsing to high magnetic fields looking for quantum oscillations and calculating bandstructures
- Norbert Jordan, FWO, chemist working on thermodynamics optimization procedures
- Atiqa Arshad, PhD candidate, I am working on THZ deriven phenomena.
- Michael Nimtz, FWDH, working on liquid metal batteries

## Share your expectations for today with us

Discuss in a group what your expectations for today are. After 3 minutes, share a summary of your Brexpectations below. Limit yourself to max. 2 expectations, please. I close the rooms at 9:32am!

- example group: feel the joy and pain working with zoom all day

- Breakout room 5: we are expecting to learn how to use the hemera cluster efficiently and to learn tricks in parallel programming using python and in general

- Breakout room 1: We are expecting to learn about The Clusters and HPC, how to start working with them, and all the possibilities we have with them. What is the difference between a "cluster" and "the servers".
-
- Breakout room 3: Job scripting basics, specifics of the HZDR cluster infrastructure (partitions, resources, quotas...), utilizing multiple GPUs

- Breakout room 4: getting more experience with HPC, how to shift from hypnos to hemera (new scheduling system), overview of HPC infrastructure, HPC basics for people who are not so familiar with it

- Breakout room 6: Get an instruction for doing Simulations and scripts with the hemera cluster and get to know the HPC basics

# Connecting to the cluster

## for those working from home

$ ssh [e-mail]
#you are asked for your password
...
$ ssh [e-mail]
#you are asked for your password

alternatively:

$ ssh [e-mail]
#you are asked for your password
...
$ ssh [e-mail]
#you are asked for your password

## for those from the HZDR internal network, i.e. your office

$ ssh [e-mail]
#you are asked for your password

(eduroam is home)
Try both options, first one will work always

## Please drop a +1 and your initials if you logged into hemera

- MN: +1
- PS: +1
- L: +1
- LK: +1
- AK: +1
- NJ: +1
- PH: +1
- FZ: +1
- TW: +1
- OP: +1
- AM: +1
- FR: +1
- +1
- TP: +1
- CC: +1
- STK: 1
- GJ: +1
- TF: +1

## Comments from participants

-

inverse the arguments
scp -r ("-r" stands for "recursive")

## Exercise: download a full folder to your local machine

PS
AK
TF
FZ
TW
TWPH

OP
LK
CC-> scp -r contre63@hemera5:./writing-from-home ./writing-from-hemera

NJ
AM
MN
STK

## Exercise: All mixed up

Lola needs to obtain a file called `results.data` from a remote machine that is called `safe-store-1`. This machine is hidden behind the login node `cray-1`. However she mixed up the commands somehow that are needed to get the file onto her laptop. Help her and rearrange the following commands into the right order!

1 $ ssh lola@cray-1
2 $ logout
3 $ scp lola@cray-1:results.data .
4 $ scp lola@safe-store-1:results.data .

Answer:

1 $ ssh lola@cray-1
4 $ scp lola@safe-store-1:results.data .
2 $ logout
3 $ scp lola@cray-1:results.data .

# Using the batch system

- in order to share the common resource (the cluster), we need someone to organize all our wishes/requests
- this is done via a software called a "batch system" (or batch scheduler)

## Hemera uses SLURM
https://slurm.schedmd.com/documentation.html

- `srun` -> execute a command (or script) on a compute node and wait for it
- `sbatch` -> submit work to the batch system, but do not wait for its completion
- `squeue` -> tell me things about the jobs that are currently in the batch system
- `scancel` -> abort a job prematurely (while it is waiting to be started or while it is running)

## Comments
- Why there is a variation in the output of the ```srun```?
- once is with a job, once like the simple echo command.

kakay24@hemera5:~/hpc_course$ srun hostname
csk007.cluster
kakay24@hemera5:~/hpc_course$ srun hostname
srun: job 2417219 queued and waiting for resources
srun: job 2417219 has been allocated resources
csk014.cluster

## multiple commands script

#!/bin/bash

date
hostname 
sleep 10
date

- using a reservation with `sbatch`

$ sbatch --reservation=hpc multiple_commands.sh

-NJ I see " Batch job submission failed: Access denied to requested reservation", jordan

-no still not working same message

-LK everything worked fineu
TF: worked fine here

- AK: FYI: for me worked with the reservation too.
-STK: works for me
- TW: ok

## Exercise: How long has this node been running?
Write a batch script that prints the `hostname` and the time the node has been running so far (using the `uptime` command). Advice the scheduler to store the output in a log file. Submit this script multiple times and see, if you find a node that has been running the longest or the shortest. Compare with your neighbor!has this node been running?

## uptime script

#!/bin/bash

hostname
uptime

Please report the uptimes that you find below, like so:

- MN: csk094.cluster up 91 days, 21:55,

- PS, hemera4 `141 days, 19:40`
- L, csk002.cluster 105 days, 50 min
- TW, csk007.cluster 105 days, 50 min
- STK, csk007.cluster
- PH, csk008.cluster 105 days, 49 min
- OP, csk006.cluster 11:16:24 up 105 days, 49 min, 0 users, load average: 29,56, 29,15, 28,84

- AK: csk007.cluster `105 days, 11:16`
- FZ, csk083.cluster, 91 days
- TF: csk007.cluster 11:17:26 up 105 days, 52 min,
- AM, csk083.cluster 11:17:30 up 91 days, 21:55, 0 users, load average: 26.35, 26.44, 24.10

- LK: csk007.cluster, 105 days (5 times)
- TP: csk083.cluster 11:18:37 up 91 days, 21:56, 0 users, load average: 26.27, 26.39, 24.26
- EF: csk007.cluster, 105 days, 55 min, 0 users, load average: 23.10, 22.16,
- FR: csk007.cluster 11:18:54 up 105 days, 53 min, 0 users, load average: 22,84, 21,73, 19,54
- NJ: csk028.cluster 11:26:39 up 105 days, 59 min, 0 users, load average: 24,57, 24,99, 25,13

## 5 min break?
majority is for continuing

## Exercise: Errors and Outputs

Submit [this script](https://psteinb.github.io/hpc-in-a-day/downloads/errors_and_outputs.sh) to your cluster and tell the scheduler to split the output in stdout and stderr. Check the contents of the log files that were created.

Submit the same job again, but this time make the scheduler send both stdout and stderr to the same output file. Use the manpage of the scheduler commands to find out how.

- AK: +1
- MN: +1
- TW: +1
- TF: +1
- PH: +1
- FZ: +1
- TP: +1
- STK: +1
- AM: +1
- NJ: +1

# Wrap-up Morning Session

Please provide feedback on how you liked the morning session. For this, use the space below:

## Something that you want us to improve :exclamation:

Note down anonymously what you were confused about or something that you want us to improve.

- I expected the `ssh` and the `scp` to be already covered with the Unix shell introduction.
- I couldn't use `scp` at all (Windows, Putty)
- an alternative: https://winscp.net/eng/index.php
- Of course you couldn't if you were connected from home.
- You can but you need at the same time to be conneced to your HZDR PC and use a shared directory for files transfer, then it works at least for me (Windows user) that's the case
- So this was normal? You need to make a jump over the uts or mup. This was not shown. Well, a kind of an example was shown how you could do scp in two steps.
-

## Something that you liked or what you enjoyed :heavy_check_mark:

Note down anonymously what you liked or what you enjoyed learning.

- I like very much the teaching style with hands on learning, the matter is well explained and easy to follow, thank you! +1
- Very friendly atmosphere, clear and helpful explanations! I liked it!

# HPC Intro Afternoon Session

**Starting 1pm**

## Question about file transfers from home to `hemera5`

- two-hop approach:

$ scp polar.pdf [e-mail]:
$ ssh [e-mail]
$ scp polar.pdf [e-mail]:

- using a network file system

$ scp [e-mail]:/net/gssnfs/bigdata/share/hello_from_steinb95 .

[local machine] -- [mup/uts] -- [hemera4/hemera5]
link to `bigdata` `/bigdata/share/`
`/net/gssnfs/bigdata/share`

Community Documentation for `hemera`
https://fwcc.pages.hzdr.de/infohub/#

@pia: please create an issue related to big data transfers through mup/uts here https://fwcc.pages.hzdr.de/infohub/#

on project spaces for bigdata:
write a message to `[e-mail]` with the following details
- for how long
- how much disk space you need
- suggest a name

## Environment

https://psteinb.github.io/hpc-in-a-day/code/02_parallel_jobs/fdate.zip

download on the command line on `hemera5`:

$ wget https://psteinb.github.io/hpc-in-a-day/code/02_parallel_jobs/fdate.zip
$ unzip ./fdate.zip

Lmod environment modules:

https://lmod.readthedocs.io/en/latest/

# Parallel Coding with Python

## installing the profiler

$ python -m pip help
$ python -m pip install --user line_profiler

--user

Make line_profiler available

 export PATH=$HOME/.local/bin:$PATH

 Collecting line_profiler
  Downloading https://files.pythonhosted.org/packages/51/b3/225fa6249fcd4e6dc258c8105d15eeef9a00f88cc7df6ce6a1b19bfd59ff/line_profiler-3.0.2-cp36-cp36m-manylinux1_x86_64.whl (68kB)
    100% |████████████████████████████████| 71kB 2.0MB/s 
Requirement already satisfied: IPython in /trinity/shared/pkg/compiler/intel/19.0/intelpython3/lib/python3.6/site-packages (from line_profiler)
Installing collected packages: line-profiler
Successfully installed line-profiler-3.0.2
You are using pip version 9.0.3, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
foerst06@hemera4:~/hpc-intro$ python --version
Python 3.6.5 :: Intel Corporation

foerst06@hemera4:~/hpc-intro$ module list

Derzeit geladene Module:
1) python/3.6.5

- Use `module unload intel`, `module unload python` and the again `module load python/3.6.5`
Did it, now I have:
foerst06@hemera4:~/hpc-intro$ python --version
Python 3.6.5
- Did you now try the `$ python -m pip install --user line_profiler`?
Yes, but I got:

foerst06@hemera4:~/hpc-intro$ python -m pip install --user line_profiler
Requirement already satisfied: line_profiler in /home/foerst06/.local/lib/python3.6/site-packages (3.0.2)
Requirement already satisfied: IPython in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from line_profiler) (6.4.0)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.15 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (1.0.15)
Requirement already satisfied: pickleshare in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (0.7.4)
Requirement already satisfied: backcall in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (0.1.0)
Requirement already satisfied: simplegeneric>0.8 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (0.8.1)
Requirement already satisfied: traitlets>=4.2 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (4.3.2)
Requirement already satisfied: jedi>=0.10 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (0.12.0)
Requirement already satisfied: setuptools>=18.5 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (40.6.2)
Requirement already satisfied: pexpect; sys_platform != "win32" in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (4.5.0)
Requirement already satisfied: pygments in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (2.2.0)
Requirement already satisfied: decorator in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from IPython->line_profiler) (4.3.0)
Requirement already satisfied: wcwidth in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from prompt-toolkit<2.0.0,>=1.0.15->IPython->line_profiler) (0.1.7)
Requirement already satisfied: six>=1.9.0 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from prompt-toolkit<2.0.0,>=1.0.15->IPython->line_profiler) (1.12.0)
Requirement already satisfied: ipython-genutils in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from traitlets>=4.2->IPython->line_profiler) (0.2.0)
Requirement already satisfied: parso>=0.2.0 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from jedi>=0.10->IPython->line_profiler) (0.2.1)
Requirement already satisfied: ptyprocess>=0.5 in /trinity/shared/pkg/devel/python/3.6.5/lib/python3.6/site-packages (from pexpect; sys_platform != "win32"->IPython->line_profiler) (0.5.2)

- Can't call kernprof, LK: same for me, with regular python
- Can Lokamani help? Lets
-
- AK: please try again this: Use `module unload intel`, `module unload python` and the again `module load gcc`, `module load python/3.6.5`

LK:
$ python --version
$ Python 3.6.5
$ kernprof
$ kernprof: command not found
(did you add .local/bin to your path?
`export PATH=$HOME/.local/bin:$PATH`

## profile with `line_profiler`

### Don't Forget ...

$ export PATH=$HOME/.local/bin:$PATH

to make the kernprof application available to your path

### Profile the application

- add `@profile` before the `def main()` line into our script
- call kernprof

$ kernprof -l ./serial_estimate_pi_profile.py 50000000

- visualize the profile results

$ python -m line_profiler serial_estimate_pi_profile.py.lprof

$

## Exercise: Data parallel in Reality

What of the following is a task, that can be parallelized in real life:

1 Manually copying a book and producing a clone
2 Clearing the table after dinner
3 Rinsing the dishes with one sink
4 A family getting dressed to leave the appartment for a birthday party

Think about what the inputs are to the task at hand. Can individual items of the inputs be processed independent of each other?

MN: 2,4

## Exercise: Data parallel Code 1
Does this code expose data independence?

my_data = [ 0, 1, 2, 3, 4, ... , 20000 ]

for i in range(len(my_data)):
  my_data[i] = pi*my_data[i]

TW: should be possible (no interdependence)
LK: yes
AK: yes can be, unless you change i to i-1 at one of the instances. Sorry, didn't know about this.
FZ: yes

## Exercise: Data parallel Code 3
Does this code expose data independence?

from random import randint
my_data = [ 0, 1, 2, 3, 4, ... ]

for i in range(1,len(my_data)):
  my_data[i] = 42*my_data[i-1]

## Amdahl's law

https://psteinb.github.io/hpc-in-a-day/02-02-parallel-estimate/

partitioning the random number generation

index_x = [0,1,2,3, ... ,100000000]

index_x = [0,1,2,3,4,5,6,7,8,9,10,11, .... ,100000000]
^^^^^^^ core 1
^^^^^^^ core 2
^^^^^^^^^ core 3

## Parallel Runtimes with 100.000.000 samples

- PS, 8 cores: 9.0s in parallel, 32.2s serial runtime -> S_exp = 32.2/9 = 3.577
- TW, ditto: 7.979s parallel, 37.279s serial, speed-up about 4.7
-
- AM, 8 cores; 8.849s parallel, 33.769s serial
- LK: 8 cores, 7.973s parallel, 33.236s serial, (was copying the wrong numbers -.-)
- TF: 8 cores 8.4s in parallel, 34.7s serial
- PH, 8 cores, 8 s in parallel, 34 s in serial runtime
- MN, 8 cores, 9 s in parallel, 36 s in serial runtime
- FZ, 8 cores, 8.25s in parallel, 34s in serial

## Want to continue with Message Passing? (30-40min)

TF: +1
TW: +1
AM: +1

LK: +1

MN:+1
PH: +1
FZ: +1

## Using MPI

### Loading OpenMPI

module load gcc/9.2.0
module load openmpi
module load python
python -m pip install --user mpi4py

#### Force pip to compile and link mpi4py from scratch

python -m pip install --user -I mpi4py

# Wrap-up Afternoon Session

Please provide feedback on how you liked the morning session. For this, use the space below:

## Something that you want us to improve :exclamation:

Note down anonymously what you were confused about or something that you want us to improve.

- Not your fault, but using an editor instead of nano especially one with syntax highlighting would greatly facilitate to grasp the code. +1
- hard to realize, but maybe would be better to split into groups of different experience levels (in the beginning a lot had to be explained that maybe a portion of the attendands already knew), was a little bit of the problem especially of the morning session
- difference in speed/difficulty of the two sessions, hard to follow fast coding and closing nano, commands in the shell are not visible after outputs
- I would certainly advice you to split the group into different levels, beginners and experienced. Thus you could show more from the options of the slurm, which I believe could be important.

## Something that you liked or what you enjoyed learning :heavy_check_mark:

Note down anonymously what you liked or what you enjoyed learning.

- Very usesful session, really learned a lot, especially getting to know the multiprocessing package
- The speed increased in the afternoon session and that was quite nice. Parallelisation concepts were well explained and the example was easy to understand. Well done!
- fixing the python MPI problem in real time was very impressive!
- learning about mpi and some backgrounds
- learned a lot about UNIX shell, HPC and python parallelisation. Thank You :)
- Again thanks. It was very helpfull.
- A very nice and easy to follow step-by-step course. Even if problems occured, the instructors were eager to solve them quickly. I never felt like asking dumb questions, very friendly atmosphere.

### Troubleshooting with Tobias

foerst06@hemera4:~/hpc-intro$ module list

Derzeit geladene Module:
1) gcc/7.3.0 2) python/3.6.5

THis is PATH. I used the intel compiler before:

Afte logout and login
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
I will now load the modules
foerst06@hemera4:~$ module list

Derzeit geladene Module:
1) python/3.6.5 2) gcc/7.3.0

now path is:
/home/foerst06/.local/bin:/trinity/shared/pkg/compiler/gcc/7.3.0/bin:/trinity/shared/pkg/devel/python/3.6.5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

____try

export PATH=/trinity/shared/pkg/compiler/gcc/7.3.0/bin:/trinity/shared/pkg/devel/python/3.6.5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
rm -r /home/foerst06/.local/bin
python -m pip install --user line_profiler
export PATH=$HOME/.local/bin:$PATH