Try   HackMD

Introduction to Triton and HPC / Winter 2025 (Archived)

  • To watch: https://www.twitch.tv/coderefinery
  • To ask questions and interact (this document): [link for those registered, check email]
    • to write on this document, click on the
      Image Not Showing Possible Reasons
      • The image file may be corrupted
      • The server hosting the image is unavailable
      • The image path is incorrect
      • The image format is not supported
      Learn More →
      (Edit) icon on the top left corner and write at the bottom, above the ending line. If you experience lags, switch back to "view mode" ("eye" icon)
  • Previous of this document are archived at: n/a - later
  • Zoom room for afternoon exercises:
    • Link sent to registered participants
  • Program and materials:
  • Prerequisites:
  • Suggestion: if you have just one screen (e.g. a laptop), we recommend arranging your windows like this:
╔═══════════╗ ╔═════════════╗
║   WEB     ║ ║   PYTHON    ║
║  BROWSER  ║ ║   WINDOW    ║
║  WINDOW   ║ ╚═════════════╝
║   WITH    ║ ╔═════════════╗
║  TWITCH   ║ ║   BROWSER   ║
║  STREAM   ║ ║    W/Q&A    ║
╚═══════════╝ ╚═════════════╝
  • Do not put names or identifying information on this page

Please do not edit above this

Introduction to Triton and HPC - 26/2/2025

Schedule

Please see the course page at: https://scicomp.aalto.fi/training/scip/winter-kickstart-2025/

Icebreakers

How familiar are you with the command line / shell

  • very: ooooo
  • use it well:oooooooo
  • somewhat familiar: oooooo
  • what is this?: oo

Have you used version control (for example git) before?:

  • yes, plenty: ooooooooo
  • I can but don't use often:oooo
  • a few times:oooooo
  • what is this?: o

What types of computers have you used for work before:

  • Own laptop/desktop: ooooooooooooooooo
  • Remote desktop servers: ooooooooooo
  • Group server connected to remotely with ssh: oooooooo
  • High-performance computing cluster: oooooo
  • Cloud computing: oo

What types of programming do you do:

  • Python, small things: ooooooooooo
  • Python, scientific work: oooooooooooooo
  • C/Fortran low-level: o
  • Shell scripting: ooooo
  • Matlab,ooooo
  • C++: oooo
  • Specific softwares for Neuroscience (FSL, fmriprep) : o
  • First principle software
  • Labview: oo
  • R: oooo

What type of scientific work do you do, or think you will do:

  • Pytorch DL models: oo
  • "medical" type of research e.g. many study participants and same processing pipeline for each participant and then statistics across participants : oo
  • Large Language Models: fa
  • Laboratory work and measurement system building: o
  • Tensor decomposition of large feature data: o
  • Dictionary learning: o
  • FEM simulation (Comsol)
  • .

Introduction

https://scicomp.aalto.fi/training/scip/winter-kickstart-2025/

You can ask questions here at any time, there are people always watching and answering live. We bring up important questions on the stream.

A short intro to scientific computing

https://hackmd.io/@AaltoSciComp/SciCompIntro

  • A question
    • An answer
  • In the kitchen metaphor, what are the nodes we define for SLURM ?
    • Good question! So a cluster is really big: each node is like a separate apartment (or commercial kitchen) connected by a hallway. Each node looks like a pretty normal kitchen, the benefit is that you have many of them that can communicate
    • How independent are these kitchens/nodes ? How can we know if distributing the work across nodes is a good idea?
    • Very independent: they really are like stand-alone OSs. The network connecting them is very fast (tens or hundreds of Gbit) and low latency.
      • The network is still noticably slower than communication inside the node however, so "is it worth" mostly depends how often the nodes would need to communicate with each other. This will then heavily depend on your program.
      • Programs such as MPI programs are designed to work across multiple nodes, but this depends on the program design. If the program is not designed to work across multiple nodes, it might not know how to utilize the fast network.
    • The question "is it worth distributing" is very good: it's why this is so interesting work. You have to try and put work into optimizing it.
  • Is there a smart way to move data between Triton scratch and the department workstation ? I normally have the department connected through sshfs
    • Depends on the needs. For small visualization, the network drive mount is probably suitable. For bigger analysis a copy is probably worth it. (Also consider the benefits of having a backup)
    • Some possible options for doing that: https://scicomp.aalto.fi/triton/tut/remotedata/ For example the section "Remote mounting using SMB" shows how you can "see" scratch with the file browser of your operating system (Windows/Linux/Mac). You will need to be in the Aalto network / VPN. Try it!
  • Does the triton terminal from ondemand have the same settings as a terminal I would get via ssh?
    • Basically yes
  • Do Triton nodes have \local_scratch?
    • /scratch is mounted on all compute nodes. there is a /tmp folder for temporary files. Some nodes have actual hard disks, which is used for /tmp, other will use memory for data stored there (which is part of the memory you requested)
      • So, if the nodes happen to have their own memory they will use it automatically right? This is for very write/read intensive codes I use.
        • Again, this depends on the node. On nodes with discs (mainly the GPU nodes which have SSD drives), those will be used. On nodes without, it will go into memory if you use /tmp. But as mentioned, note, that the memory is part of what you requested, which means that if you load too much data onto /tmp (on nodes without discs), you might face out of memory issues (i.e. jobs getting killed).
          • You can also tell slurm how much /tmp space you want with --tmp=1G (same syntax as ram requests). I'm not sure how this behaves on nodes without an actual disc however. (Whether it adds to your --mem request or just uses the requested RAM)
    • Different nodes have different local disks: https://scicomp.aalto.fi/triton/overview/
  • I have a stand-alone computer in my department from which I want to access Triton from, it is connected to Aalto network but not possible to connect to VPN, how to use triton from such stand-alone computers. It has Linux, ubuntu.
    • If you are in the Aalto network, you are in the VPN so the connection to VPN should not work. If you are in the "Aalto Open network" then you are not in the VPN and you should be able to start the VPN program like you would do from home.
    • Also: You can always use ondemand (which is accessible from anywhere), or you can set up ssh keys on the jump hosts (like kosh) and use ssh via those hosts.
  • I loved the kitchen metaphor with cat!!!
    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    • Come for the HPC, stay for the cat :)
    • CATS often visits during courses but I'm not sure if this one is long enough for it to wake up and drop by.

Gutenberg example

General tutorials: https://scicomp.aalto.fi/triton/#tutorials
Exercises: https://scicomp.aalto.fi/triton/tut/exercises-ngrams/
This is a demo: we will talk fast without time to type along

Example data: https://users.aalto.fi/~darstr1/public/Gutenberg-Fiction-first100.zip

  • Can anyone get a group project? I am not sure if we have one

    • You should ask your group leader, but in essence yes anyone can get one. We can help preparing to talk to the group leader. (Most groups know of this and have something).

  • Is it that the terminal that it is shown now is not inside ondemand, right?

    • Correct. We started a terminal from theb operating system, and then "ssh" to the cluster. This is what we usually do.
    • You could start a terminal through OOD as well and the commands would mostly be the same
    • Correct! Also the terminal shows the name of the "node" where the instructor is at right now "login4" is the name of the triton login node. If this terminal would be inside an OOD instance that you have started, the name of the node would be something different "e.g. pe20"

  • If we see that some user is "abusing" the login node, what should we do?

    • Nowadays every user is limited to 2 cores and 20GB of ram on login node, so you don't really need to do anything. Anyone using login node too much are mostly making things harder for themselves.
      • It is still best practice not to overuse login node, since if enough users did that at the same time it could still cause issues.

  • Is the priority calculated on the time/memory i put in my command or bash script or the time/memory actually used?

    • Your overall priority is based on the actual resource usage. It also has a half-life of two weeks if I recall correctly, so running huge jobs won't kill your priority permanently
    • The queue time of your job depends on the resources requests and that is then scaled by your priority.
    • Blog post on how busy is the cluster that also talks about priorities: https://aaltoscicomp.github.io/blog/2024/how-busy-is-the-cluster/

  • How do you usually estimate the memory and time needed for the task?

    • Yes this is the question! We have this page in our docs https://scicomp.aalto.fi/triton/usage/program-size/
    • Basically you start small and with monitoring tools estimate if you are requesting for the right amount of resources and how long it takes (slurm history, seff, sacct https://scicomp.aalto.fi/triton/tut/monitoring/). Then, scale up. (scaling won't be linear)
    • Another quick take home messages: with lots of jobs that are only a few seconds long you spend more time in the queue, affect your priority, have more resources spent for the "overhead" of allocating resources rather than actual computing. Having a too long job without any "checkpoint" increases the risk that if the job dies (e.g. it goes over memory and is killed) you will lose your computations. So the best is to have jobs that are about a few hours.

  • Is the time requested or the time consumed that matters? (For the fair share)

    • Actually used time (if the job is shorter, you don't "pay" for the longer time), but requestet mem/CPU/GPUs since no one else can use that
    • The time request is used by the queue to determine the size of the job and the finished priority. So basically "if this user runs this job with this size, what will be their priority after the job finishes". This will affect your

Array jobs: embarassingly parallel execution

Exercise that is being demoed https://scicomp.aalto.fi/triton/tut/exercises-ngrams/#array-jobs-embarassingly-parallel-execution

  • So the script is a shell script, but the lines that start with #SBATCH are like instructions for slurm to request what I need?

    • Yes, essentially slurm looks into th script and takes any comment line starting with SBATCH and converts it into a command line parameter (i.e. everything in there can also be given as a command line parameter)
      • It's also good to note that #SBATCH directives need to be before any actual commands. Slurm stops reading the directives when it gets to the first executable line in the script.

  • And if a line starts with ##SBATCH is it ignored by the "sbatch" interpreter? I meant with two comments

    • It's read by the sbatch interperter but ignored by the shell (bash).'
    • Then sbatch will see a line that starts with #SBATCH (i.e. not SBATCH), so it's ignored.
      • This means we can "comment out" the #SBATCH lines which is useful
        • Yes. You can also overwrite parameters from the command line, if you want to try out different parameters. E.g. sbatch --time=01:00:00 my_script.sh would overwrite a #SBATCH --time=00:05:00 parameter in the script.

  • Can I write from many different tasks to the same csv file or will this lead to conflicts? (Thanks for the answers :)

    • Short answer: Dont do this. long answers: below
    • If you open the file in "append" mode it might work (append mode should only add new data to the end of a file): in theory the "append" operations should work (be atomic: each append happens all or nothing), but there might be problems the way it's distributed. You need to carefully check. Also writes may need to be smaller than the buffer size.
      • And even if it works with append, you will likely end up blocking any job that is trying to write to the file, while another one is writing, wasting runtime…
    • It will cause issues. There are some file formats so that you can tell "write exactly on this portion of a file" (like imagine a matrix and one of the processes knows to write exactly at location (i,j) of the matrix and no other scripts are going to write in that location)
    • Best case, your write come in at different times and it works fine. Worst case, you write nonsense into the file.
    • Sometimes databases are useful for this becaues they take care of the concurrent writing requests
    • Often people just write to separate files… and sometimes they end up with millions of tiny files in a folder. This is very inefficient from the cluster storage point of view, so if this is your case, get in touch with us. :)
      • Relatively simple solution for cases like this is to write the files to /tmp. Then at the end of the job you create a single zip/tar archive of them and move that to scratch
    • Each triton user actually has a "quota" limit on the number of files, so sometimes you are not out of disk space, you are out of file numbers.
    • One more comment: I would say the common workflow is to write the parallel task outputs to separate files and then combine these into a single file afterwards if a single file feels more comfortable to process.
      • And overall, since you should commonly aim for job sizes of at least 1 hour, this normally doesn't cause an issue since you won't normally have too many jobs (i.e. run against the file limit wall).

  • Is there an alternative to nano you would recommend to quickly edit code to be sent to SLURM?

    • From the command line you can also use vi, vim or emacs, but you can also try running e.g. vscode with a remote plugin and connect that o triton to edit your files. Or, you could mount the folders to your local machine and modify them with your favorite tool (although we recommand not using tools like vscode in this way, since it puts a lot of stress on the file servers - as they try to index the files.)
    • Thanks for th clarification! Could you tell me more about the remote plugin/ share some link on how to set it up?

  • .

Multiprocessing

https://scicomp.aalto.fi/triton/tut/exercises-ngrams/#shared-memory-parallelism-multithreading–multiprocessing

  • How do I know that my program supports multiprocessing? (I use python)

    • A quick and simple way for doing that is to submit a job with for example 4 cpus, and then when the job ends you can check using "seff JOBID" if the percentage of usage is more than 100/4. If the percentage is less than 100/number-of-cpus then it is using only one cpu even though it could use more.
      • +1 Often it is easiest to try it and see what happens
    • It needs to not just support multiprocessing but know how many procesrers Slurm is giving it. Not many codes do this: you need to read and see. (it's fustrating that people don't often document this)
    • In practice this heavily depends on the packages you are using. Python doesn't support multiprocessing on its own, but lot of the popular packages have different implementations of it.

  • should i do changement in my code for multiprocessing it?

    • If your code runs fast enough for your purposes, then there is no need. If you need to run bigger jobs where a single process isn't enough, then you want to look into implementing some kind of multiprocessing.
    • If you are a Python user, you can have a look at our training materials from "Python for Scientific Computing" https://aaltoscicomp.github.io/python-for-scicomp/ there is a section dedicated to Parallel programming
    • In the example that is used right now https://github.com/AaltoSciComp/hpc-examples/blob/master/ngrams/count-multi.py the script uses "import multiprocessing" and calls to that library.
    • If there is an option to run it as an array job, that is highly preferable to actual multiprocessing, since the latter comes with a lot more complexity than just "running with different parameters".
      • +1 Helping people to run Python scripts on Triton which use "import multiprocessing" and just aren't running fast, the easiest solution has always been to remove the multiprocessing and use array jobs

  • I don't understand why it would be slower with multiprocessing: is it the I/O?

    • One reason would be that the processes generally need to communicate with each other. In some cases (depending on the structure of the program) this can get out of hand and lose all the benefit you would get.
    • I/O could also be an issue if it can't be parallelized. In some cases each thread/process could waste most of their time waiting to actually access the data instead of doing anything useful.
    • Standard issue: You use some code with e.g. numpy. Numpy internally already uses multi-processing to speed up some calculations. Let's assume we reserve 10 cores for the job. Numpy will try to use those 10 jobs (out of the box). Before changing your code you had a simple for loop going over 10 values. Now you change this into multi-processing, and run the 10 calcluations simultaneously. Most likely, all single instances would try to use those 10 cores, so now the system needs to schedule a lot of resources, which slows all processes down a lot. And even if you have single processor code for each of the 10 iterations, you now have startup costs for the additional processes.

  • imagine i have a system with n interactions, and i'm doing loop for each interactions and doing "huge" computation calculation on each part of (interaction) loop, what should i do there with python to parallelize and multiprocess the tasks so my computation is way more faster (i already optimized as much as i can my code in the normal way) ?

    • First thing I would check: Are there any libraries, that already use multiple CPUs for the calculations you are doing (what kind of calculations are those? Matrix multiplications or similar things often already support these kind of things)
      Then, it depends a lot on how much data those calculations need and whether they depend on each other. if they are independent, you can probably use something like multiprocessing to calculate them in parallel.
      Assuming that the amount of data needed for each isn't too big, it makes sense to do this, assuming that you have e.g. a nested for loop or while if it's just an "outer" for loop, I would recommend array jobs.
      but if you have something like: 
      ​​​​while condition_not_met:
​​​​    new_result = []
​​​​   for x in previous_result:
​​​​      new_result.append(function(x))
​​​​   previous_result = new_result
      and function(x) is the expensive function, multiprocessing this can make sense.
      • ex numpy calculation (numpy.linalg.pinv)
      • i did it in a way normally everything is independant between each interactions
      • i guess i only need for the moment to do it for my : "for interaction in interactions", so like if i have 100 interactions i can split it and have my result faster
        • Than thats an array job. Where you select the interaction based on the array job id.
          • ok thqnks i'll check this
      • For in-code improvements you can also try e.g. Numba, but it works best if your code is written as smaller functions that are easier to optimize.
        • I checked numba before but some part of my code don't allow me to use numba (yes i should do smaller function for it but kinda complicated with my current code OOP logical)

  • Can you please tell how to install Python libraries that are under testing/development?

Looking at outputs graphically (e.g. browsing the files, or doing plots)

Try it out at https://ondemand.triton.aalto.fi

Feedback for the morning part

This is the first time we try this! We will keep on using this document for the afternoon exercises.

We resume after lunch (13 EET) in Zoom (check emails) for hands-on help, more questions, and exercises. You can follow up with any questions from the morning there.

Today was:

  • too fast: o
  • too slow: o
  • right speed: ooooooo
  • too advanced: o
  • too basic: ooo
  • right level: ooo
  • I will use what I learned today: oooooo
  • I would recommend today to others: oooooo
  • I would not recommend today to others: o

One good thing about today:

  • We can use slurm queue instead of squeue -u username . Gamechanger
  • I didn't know of On Demand
  • There is a way to use VS for editing code in Triton. I don't have to cry over nano anymore

One thing to improve for next time:

  • .
  • .
  • .

Any other feedback?

  • The kitchen metaphor is awesome
  • Read through the notes right after this session. The notes are valuable not only for quick recaps but also a good way to learn these terms that I am not familiar with before.
  • .

Any other questions:

  • .
  • .
  • .

Afternoon exercises

We have reserved 3 hours, let's use what we need (i.e. you can leave earlier if you need to leave).
Let's remember to have 10 minutes breaks at the beginning of each hour.

Room 1 Exercises from Triton Tutorials

We have our standard exercises:

  1. https://scicomp.aalto.fi/triton/tut/connecting/#exercises

ssh USERNAME@triton.aalto.fi

Were you able to do the exercises?
Yes, all done: oooo
Still doing:
Not doing, just watching: o
Not able, I need help:

  • Hi, what if I want to save the output to an output file and also want to see it on the screen? What should I use? tee?
    • It depends, on the terminal ">" redirects the standard output (but not the errors) to a local file
    • some_command | tee output.txt
    • some_command 2>&1|tee output_with_errors.txt
  1. https://scicomp.aalto.fi/triton/tut/cluster-shell/#exercises-directories

Were you able to do the exercises?
Yes, all done: ooo
Still doing: o
Not doing, just watching:
Not able, I need help:

  1. https://scicomp.aalto.fi/triton/tut/modules/#exercises

To check which modules we have, see https://scicomp.aalto.fi/triton/#applications
List which "module load NAME" you have tried by pasting the output of "module list"

  • module load matlab
    module list
    Currently Loaded Modules:
    1. matlab/r2024b
  • module load scicomp-python-env
    module list
    Currently Loaded Modules:
    1. scicomp-python-env/2024-01

Were you able to do the exercises?
Yes, all done: ooo
Still doing:
Not doing, just watching: o
Not able, I need help:

  1. https://scicomp.aalto.fi/triton/tut/storage/
  2. https://scicomp.aalto.fi/triton/tut/remotedata/#exercises

Download the small gutenberg dataset from https://users.aalto.fi/~darstr1/public/Gutenberg-Fiction-first100.zip and move it to triton using what you prefer (rsync command from the shell, graphical interface from the operating system, ondemand)

Were you able to do the exercises?
Yes, all done: oo
Still doing:
Not doing, just watching:
Not able, I need help:

Please mention if you used 1) terminal, 2) the graphical file browser, or 3) ondemand.triton.aalto.fi

  • terminal: sshfs & rsync
  1. https://scicomp.aalto.fi/triton/tut/interactive/#exercises

Go to login node and run this command:

srun –time=2:00:00 –mem=600M –pty bash

hostname

  1. https://scicomp.aalto.fi/triton/tut/serial/#exercises

Let's first clone the exercise repository

Your first job can be

#!/bin/bash
#SBATCH --time=00:05:00
#SBATCH --mem=100M
#SBATCH --output=pi.out

echo "Hello $USER! You are on node $HOSTNAME.  The time is $(date)."

# For the next line to work, you need to be in the
# hpc-examples directory.
srun python3 slurm/pi.py 10000

hello_result = pipe({"role" : "user", "content" : "hello"})

system_message = [
    {"role": "system", "content": system}
]

user_messages = [
    {"role": "user", "content": answer} for answer in answer_data
]

messages = system_message + user_messages

Hello eglerean! You are on node pe79. The time is Wed Feb 26 15:00:27 EET 2025.
Calculating pi via 100000000 stochastic trials
{"pi_estimate": 3.14147488, "iterations": 100000000, "successes": 78536872}

Hello [redacted] You are on node pe78. The time is Wed Feb 26 14:57:44 EET 2025.
Calculating pi via 10000 stochastic trials
{"pi_estimate": 3.126, "iterations": 10000, "successes": 7815}

Were you able to do the exercises?
Yes, all done: ooo
Still doing:
Not doing, just watching:
Not able, I need help:

  1. https://scicomp.aalto.fi/triton/tut/monitoring/#exercises
  2. https://scicomp.aalto.fi/triton/tut/array/#exercises
#!/bin/bash
#SBATCH --time=00:05:00
#SBATCH --mem=100M
#SBATCH --output=array_example_%A_%a.out
#SBATCH --array=1-10


echo "Hello $USER! You are on node $HOSTNAME.  The time is $(date). This is array ID $SLURM_ARRAY_TASK_ID"

# For the next line to work, you need to be in the
# hpc-examples directory.
srun python3 slurm/pi.py 100000

And we can also combine them with the demo shown in the streaming
https://scicomp.aalto.fi/triton/tut/exercises-ngrams/

https://scicomp.aalto.fi/triton/usage/smallfiles/

Room 2 Connecting to Triton + other troubleshooting

Room 3 GPUs and other advanced cases

Room 4 Local LLMs on Triton

Room 5 speech2text tool

Other rooms

Other rooms can be used ad-hoc, e.g. to solve issues specific to a single user (account issues, permissions, etc)

–
Note: please always write at the end of the document, above this line. If this document feels slow, switch to the "view" mode (eye icon on top left).

Last changed by 
 
AaltoSciComp
0
73

Read more

Kickstart-2025-{2,3} (archive)

Programming langages you use? (poll, add o to vote):

Jun 5, 2025
Kickstart-2025-1 (archive)

To watch: https://www.twitch.tv/coderefinery

Jun 5, 2025
LLM Examples

materials:

Jun 5, 2025
2025 summer kickstart - triton tutorial notes

This is the notes from the

Jun 5, 2025
Read more from AaltoSciComp
Published on HackMD