owned this note   changed 4 years ago  
Published Linked with GitHub

Intro to High Performance Computing

╔═══════════╗ ╔═════════════╗
║   WEB     ║ ║  TERMINAL   ║
║  BROWSER  ║ ║   WINDOW    ║
║  WINDOW   ║ ╚═════════════╝
║   WITH    ║ ╔═════════════╗
║   THE     ║ ║   BROWSER   ║
║  STREAM   ║ ║   W/HACKMD  ║
╚═══════════╝ ╚═════════════╝

Aalto: if you still experience this error

srun: error: Unable to allocate resources: Invalid account or account/partition combination specified

send your Aalto username to Enrico to fix it

02/02 - High Performance Computing with SLURM

Icebreaker - Reflection on parallelisation: How can your analysis workflow be parallelised?

Please write your answer here below

  • I process health data from multiple participants, so my analysis pipeline could work on participants' data in parallel. Every job can correspond to one participant. At the end I gather everything (group statistics).

  • I have some simulation code that takes as input parameter a location in the brain. In the end, I want the results for 4000 locations. Each location can be run in parallel. Is creating an array job of size 4000 too much?

  • Train image data scenes for semantic segmentation.. using ML tools,

    • Which tools do you use?
      • for e.g. using python Tensorflow.

  • Imaging data from ~40k individuals that each need processing that isn't super intensive. I'm trying to figure out now what the optimal division is (i.e. 400x100 jobs, or 80x500 jobs etc or something in the middle?).

    • That's a lot!
      • UK Biobank 😎
        Image Not Showing Possible Reasons
        • The image file may be corrupted
        • The server hosting the image is unavailable
        • The image path is incorrect
        • The image format is not supported
        Learn More →

  • Mostly deep learning related stuff. It would be great to know how to efficiently parallelise data I/O

    • which library do you use?
      • PyTorch and Tensorflow both

Serial jobs

Ask anything, write always at the bottom (please include your organization to the question if it is something specific to Aalto or University of Helsinki)

  • This is a demo question or is it?
    • This is an answer/comment
  • This is a question
    • Someone will answer
      • You can even have a follow-up question
  • Also comments can be written, not just questions
  • With double ## it is actually not interpreted right? that way we could comment the code
  • Can I specify list of nodes for allocating jobs? (UH)
    • You can with -w <nodename|nodelist>
    • You can also exclude the nodes with -x <nodelist>. By nodelist in SLURM is meant a comma separated node1,node2,node67 or node[1-5],node8 etc. The list of nodes sinfo -N.
  • Can I wrap any module (python, R, ect.) inside the slurm script?
    • indeed! I also like to start with "cd folder…" to make sure I am in the right spot
    • Yes, with commands like: srun python my_python_script.py and srun Rscript my_r_script.R. Make sure you have the proper module load commands in your SBATCH script though.
    • Depends what you mean on wrapping?
  • about the first job script you are demoing: what is the difference of running hello.sh WITH or WITHOUT "srun"?
    • This is an important one and I will make sure Simo and Richard explain it. TL;DR: with srun you can track on slurm h resource usage for that step. So you don't need to do "srun cd $WRKDIR", but you want to do
    ​​​​srun matlab -r myscript1.m
​​​​srun matlab -r myscript2.m
    • srun ... is a step from the SLURM point of view, one can set different requirements per step, and there can be as many steps in the SLURM script as you wish. Steps may have different parameters, like you allocate an amount of of CPUs and meory in total and a particular step will use only part of the allocated memory and only limited number of the allocated CPUs, i.e. a way to share resources inside the SBATCH script (srun --mem 1G --ntasks 4 ...). Steps can be run in parallel (srun ... &). Step are registred and seen as a separate task within the job run, (slurm history) thus one can control memory usage etc. srun ... also replaces mpirun for the MPI jobs and makes sure your app gets all the needed parallel job related parameters. As mentioned previously, SLURM script is nothing else than a SHELL script, so common best practice: commands like cd, module load ... or BASH programming logic should use just normal bash syntax, while all the core applications that you run, or anything that uses resources heavily like data copying/extracting makes sense to start with srun, thus you can analyze the timing, memory usage etc later.
  • Is there a way to know cluster usage from user side?

Exercises

https://scicomp.aalto.fi/triton/tut/serial/#exercises

  • Exercises until xx:55, then break until xx:05, then we demo together.
  • Do exercises 1-4 by yourself
  • Ask in Zoom if you have questions
  • We will demo them together afterwards
  • You can also use HackMD to ask questions to us
  • question
    • answer
  • Where is the shell script saved to?
    • In principle you can save it anywhere. It is good practice to keep it with your data, so that you or someone else can reproduce your workflow.
    • On Triton you put it into your $WRKDIR, which is a Lustre filesystem, also known as /scratch. The directory is cross-mounted around the cluster nodes. The best practice overall, right after login, cd $WRKDIR and organize your data overthere.
    • but also try to keep it organized by project
    • Same principle applies to UH.
  • What will be the difference between slurm watch q and watch slurm q? I hope it doesn't cause any issues?
    • Good point! watch does basically the same thing for any program.
    • Advanced users may want to take a look at /usr/bin/slurm you'll see that slurm watch ... is exactly the watch -n 15 --no-title that does the job. So, no diff.
  • How to end a job that has been submitted?
  • %A, %a, %j .. what is the difference? 
%A
Job array's master job allocation number.
%a
Job array ID (index) number.
%J
jobid.stepid of the running job. (e.g. "128.0")
%j
jobid of the running job.
%N
short hostname. This will create a separate IO file per node.
...

  • Will each of the array jobs have the specified time and mem or will they share the recourses?

    • yes each will have the same resources allocated

  • How to specify the machine we want to run sbatch ? For example not all of them support vector operations.

  • srun will automatically run parallel for different samples?

    • It depends :) If your code uses libraries that have parallelization, otherwise you need to implement the paralleization strategy yourself (e.g. array jobs). The general answer is "no" basically.

  • Will you have a course for programming with GPUs (CUDA)? :)

  • Does local computer impact the parallel job?

    • Your own local computer will not impact it. (neither will the Triton login machine), however different classes of nodes have different hardware and speed. https://scicomp.aalto.fi/triton/ref/#hardware E.g. one of your array jobs end up in a little bit slower node and you need to wait a little bit longer.

  • .

  • .

  • .

Feedback for the day and the whole course

Before you leave, please write down something that was useful/positive, and something that could be improved for future courses. Thank you! (if you are not logged in on hackmd, then feedback is fully anonymous)

  • Thank you, it was very useful! Materials is really well done and I will surely go back to it. I liked that the teacher in the zoom exercises was slowly showing what to do. I think there was not enough time for exercises, so I feel I have to do them by myself later.

    • +1

  • I hope I'm able to run some of my Julia code on Triton now. Thank you!

  • Great sharing, thanks a lot! I want to implement training with pytorch on Triton.

  • Thank you very much, very useful course. I will start using TRITON for my simulations - I might have visit you at the Garage for some help :)

  • Thanks for the quick but useful introduction to working with the Triton cluster! I'm not sure if I will be able to make my codes 100% efficient in both CPU and GPU parts but I look forward to your help when I'm stuck in the future! :)

  • Compressed handbook of Triton commands "Triton Cookbook".

    ^that is the cheatsheet (search for cheatsheet in our documentation)
    -Thanks!

  • .

  • Overall the sessions were awesome, the last part of GPU and parallelization might deserve longer timeslot IMO since they are also more difficult than the rest.

  • Session was awesome. Is it possible to share the email contact for help in future here.

  • The session progressed with intense flow. It was easy to dive in, follow the dialogue and forget else. On the other hand, losing focus riskes one to get dropped on the topic. I liked really much the dialogue between the instructors, as they were explaining to each other the concepts and naturally asked and answered all those obvious questions.

  • I wish there would have been some more time to try out the examples, I didn't have the command line routine so I was struggling to even copy paste and run the examples during the exercise breaks. Perhaps make longer day two or move something to day one?

  • Thanks so much. It's very helpful.

Recordings of everything will be (and are already) at https://www.youtube.com/playlist?list=PLZLVmS9rf3nN_tMPgqoUQac9bTjZw8JYc

Daily garage at: https://scicomp.aalto.fi/help/garage/

Extra session about workflows

  • Ask questions here.

^^^ Please write above this line ^^^

Select a repo