# Intro to High Performance Computing

:::danger
## Infos and important links

* To watch: https://aaltoscicomp.github.io/scip/
* To ask questions and interact (this document): https://hackmd.io/@AaltoSciComp/IntroWinter2021
* *click on the "eye" icon on the top right corner and write at the bottom, above the ending line. If you experience lags, switch back to "view mode"*
* Questions that require one-to-one chat with our helpers (e.g. an installation issue):
- Finland: ZOOM LINK SENT TO REGISTERED PARTICIPANTS
* Program (Mon-Tue) https://scicomp.aalto.fi/training/scip/winter-kickstart/
* Previous session HackMD archived at: https://hackmd.io/OcTFy7sTQ4uOjuMWtkU3Ow
* If you have just one screen (e.g. a laptop), we recommend arranging your windows like this:

╔═══════════╗ ╔═════════════╗
║   WEB     ║ ║  TERMINAL   ║
║  BROWSER  ║ ║   WINDOW    ║
║  WINDOW   ║ ╚═════════════╝
║   WITH    ║ ╔═════════════╗
║   THE     ║ ║   BROWSER   ║
║  STREAM   ║ ║   W/HACKMD  ║
╚═══════════╝ ╚═════════════╝

:::

:::info
Aalto: if you still experience this error

srun: error: Unable to allocate resources: Invalid account or account/partition combination specified

send your Aalto username to Enrico to fix it
:::

# 02/02 - High Performance Computing with SLURM

## Icebreaker - Reflection on parallelisation: How can your analysis workflow be parallelised?
*Please write your answer here below*
- I process health data from multiple participants, so my analysis pipeline could work on participants' data in parallel. Every job can correspond to one participant. At the end I gather everything (group statistics).
- I have some simulation code that takes as input parameter a location in the brain. In the end, I want the results for 4000 locations. Each location can be run in parallel. Is creating an array job of size 4000 too much?
- Train image data scenes for semantic segmentation.. using ML tools,
- Which tools do you use?
- for e.g. using python Tensorflow.

- Imaging data from ~40k individuals that each need processing that isn't super intensive. I'm trying to figure out now what the optimal division is (i.e. 400x100 jobs, or 80x500 jobs etc or something in the middle?).
- That's a lot!
- UK Biobank 😎 :100:
- Mostly deep learning related stuff. It would be great to know how to efficiently parallelise data I/O
- which library do you use?
- PyTorch and Tensorflow both

# Serial jobs
*Ask anything, write always at the bottom* (*please include your organization to the question if it is something specific to Aalto or University of Helsinki*)
- This is a demo question or is it?
- This is an answer/comment
- This is a question
- Someone will answer
- You can even have a follow-up question
- Also comments can be written, not just questions
- With double ## it is actually not interpreted right? that way we could comment the code
- Good question! Checking. It recognises #SBATCH as a special case, so everything else will be just treated as a shell script comment https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html#creating-a-job
- Yes. The SLURM batch script is nothing else than a normal SHELL script, those #SBATCH are directives for the SLURM itself, you can even omit them actually. Anything after # is a comment, and thus double # is a comment for ##SBATCH.
- Can I specify list of nodes for allocating jobs? (UH)
- You can with -w <nodename|nodelist>
- You can also exclude the nodes with `-x <nodelist>`. By nodelist in SLURM is meant a comma separated `node1,node2,node67` or `node[1-5],node8` etc. The list of nodes `sinfo -N`.
- Can I wrap any module (python, R, ect.) inside the slurm script?
- indeed! I also like to start with "cd folder..." to make sure I am in the right spot
- Yes, with commands like: `srun python my_python_script.py` and `srun Rscript my_r_script.R`. Make sure you have the proper `module load` commands in your SBATCH script though.
- *Depends what you mean on wrapping?*
- about the first job script you are demoing: what is the difference of running hello.sh WITH or WITHOUT "srun"?
- This is an important one and I will make sure Simo and Richard explain it. TL;DR: with srun you can track on `slurm h` resource usage for that step. So you don't need to do "srun cd $WRKDIR", but you want to do

    srun matlab -r myscript1.m
    srun matlab -r myscript2.m

- `srun ... ` is a *step* from the SLURM point of view, one can set different requirements per step, and there can be as many steps in the SLURM script as you wish. Steps may have different parameters, like you allocate an amount of of CPUs and meory in total and a particular step will use only part of the allocated memory and only limited number of the allocated CPUs, i.e. a way to share resources inside the SBATCH script (`srun --mem 1G --ntasks 4 ...`). Steps can be run in parallel (`srun ... &`). Step are registred and seen as a separate task within the job run, (`slurm history`) thus one can control memory usage etc. `srun ...` also replaces mpirun for the MPI jobs and makes sure your app gets all the needed parallel job related parameters. As mentioned previously, SLURM script is nothing else than a SHELL script, so common best practice: commands like `cd`, `module load ...` or BASH programming logic should use just normal bash syntax, while all the core applications that you run, or anything that uses resources heavily like data copying/extracting makes sense to start with `srun`, thus you can analyze the timing, memory usage etc later.
- Is there a way to know cluster usage from user side?
- `slurm h` show recent jobs. For more advanced querying on your usage there is a command called `sacct`. Another useful one is `seff JOBID` to give you efficiency about the job you just ran. We also have an utility called "`jobstats`" to monitor the I/O of your job (it can be important for data intensive things, e.g. you have lots of data or you use GPUs which want lots of data at once, i.e. I/O can become a bottleneck). Some link to docs:
- https://scicomp.aalto.fi/triton/usage/storage/?highlight=sacct#checking-your-stats
- https://scicomp.aalto.fi/triton/tut/parallel/?highlight=seff#monitoring-performance
- https://scicomp.aalto.fi/triton/usage/jobstats/?highlight=jobstats
- In case you are looking for the info, like how many CPUs are idling, or what percentage of the cluster is free, the easiest option is `slurm features`, see *NODES(A/I/O/T)* column. There A stands for Allocated Nodes, I is Idling, O is some Other status (failed, drained etc) and T is Total. That is the best estimate of the cluster usage at the moment.

### Exercises
:::success
https://scicomp.aalto.fi/triton/tut/serial/#exercises
- Exercises until xx:55, then break until xx:05, then we demo together.
- Do exercises 1-4 by yourself
- Ask in Zoom if you have questions
- We will demo them together afterwards
- You can also use HackMD to ask questions to us
:::

- question
- answer
- Where is the shell script saved to?
- In principle you can save it anywhere. It is good practice to keep it with your data, so that you or someone else can reproduce your workflow.
- On Triton you put it into your *$WRKDIR*, which is a Lustre filesystem, also known as /scratch. The directory is cross-mounted around the cluster nodes. The best practice overall, right after login, `cd $WRKDIR` and organize your data overthere.
- but also try to keep it organized by project
- Same principle applies to UH.
- What will be the difference between `slurm watch q` and `watch slurm q`? I hope it doesn't cause any issues?
- Good point! `watch` does basically the same thing for any program.
- Advanced users may want to take a look at `/usr/bin/slurm` you'll see that `slurm watch ...` is exactly the `watch -n 15 --no-title` that does the job. So, no diff.
- How to end a job that has been submitted?
- scancel JOBID https://scicomp.aalto.fi/triton/usage/jobs/?highlight=scancel#cancelling-jobs
- %A, %a, %j .. what is the difference?
- https://slurm.schedmd.com/sbatch.html scroll down to "filename pattern"

%A
Job array's master job allocation number.
%a
Job array ID (index) number.
%J
jobid.stepid of the running job. (e.g. "128.0")
%j
jobid of the running job.
%N
short hostname. This will create a separate IO file per node.
...

- Will each of the array jobs have the specified time and mem or will they share the recourses?
- yes each will have the same resources allocated

- How to specify the machine we want to run sbatch ? For example not all of them support vector operations.
- There is --constraint= which can set limits on allowed machines, and then Slurm automatically finds machines supporting that.
- See "Arch (constraint)" column here: https://scicomp.aalto.fi/triton/ref/#hardware

- srun will automatically run parallel for different samples?
- It depends :) If your code uses libraries that have parallelization, otherwise you need to implement the paralleization strategy yourself (e.g. array jobs). The general answer is "no" basically.
- Will you have a course for programming with GPUs (CUDA)? :)
- We had one in December https://scicomp.aalto.fi/training/scip/ we will do more. Next coming will be a re-run of MPI course, but we have not yet agreed the dates with the teacher (not necessarily GPU, but yeah..). See last year page: https://scicomp.aalto.fi/training/scip/mpi-introduction/
- Does local computer impact the parallel job?
- Your own local computer will not impact it. (neither will the Triton login machine), however different classes of nodes have different hardware and speed. https://scicomp.aalto.fi/triton/ref/#hardware E.g. one of your array jobs end up in a little bit slower node and you need to wait a little bit longer.
- .
- .
- .
-

## Feedback for the day and the whole course
*Before you leave, please write down something that was useful/positive, and something that could be improved for future courses. Thank you!* (if you are not logged in on hackmd, then feedback is fully anonymous)

- Thank you, it was very useful! Materials is really well done and I will surely go back to it. I liked that the teacher in the zoom exercises was slowly showing what to do. I think there was not enough time for exercises, so I feel I have to do them by myself later.
- +1
- I hope I'm able to run some of my Julia code on Triton now. Thank you!
- Great sharing, thanks a lot! I want to implement training with pytorch on Triton.
- Thank you very much, very useful course. I will start using TRITON for my simulations - I might have visit you at the Garage for some help :)
- Thanks for the quick but useful introduction to working with the Triton cluster! I'm not sure if I will be able to make my codes 100% efficient in both CPU and GPU parts but I look forward to your help when I'm stuck in the future! :)
- Compressed handbook of Triton commands "Triton Cookbook".
- +1 A cheatsheet or _"Triton for Dummies"_ would be welcome!
- https://aaltoscicomp.github.io/cheatsheets/triton-cheatsheet.pdf
- https://scicomp.aalto.fi/triton/ref/

^that is the cheatsheet (search for cheatsheet in our documentation)
-Thanks!

- .
- Overall the sessions were awesome, the last part of GPU and parallelization might deserve longer timeslot IMO since they are also more difficult than the rest.
- Session was awesome. Is it possible to share the email contact for help in future here.
- Aalto? You can reach us at [e-mail] and other channels listed at: https://scicomp.aalto.fi/help/

- The session progressed with intense flow. It was easy to dive in, follow the dialogue and forget else. On the other hand, losing focus riskes one to get dropped on the topic. I liked really much the dialogue between the instructors, as they were explaining to each other the concepts and naturally asked and answered all those obvious questions.
- I wish there would have been some more time to try out the examples, I didn't have the command line routine so I was struggling to even copy paste and run the examples during the exercise breaks. Perhaps make longer day two or move something to day one?
- Thanks so much. It's very helpful.

:::success
Recordings of everything will be (and are already) at https://www.youtube.com/playlist?list=PLZLVmS9rf3nN_tMPgqoUQac9bTjZw8JYc

Daily garage at: https://scicomp.aalto.fi/help/garage/
:::

## Extra session about workflows

- Ask questions here.

----
:::info
^^^ Please write above this line ^^^
:::