# Using CSC HPC environment efficiently, Part 1: Basics 1.-2.10.2025

###### tags: `puhti` `mahti` `allas`

> This is the collaborative "notebook" for the "Using CSC HPC environment efficiently" basics course organised in October 2025 by CSC -IT center for Science.
> [Prerequisite course page in eLena platform](https://e-learn.csc.fi/course/view.php?id=75)
> [Course page in eLena platform](https://e-learn.csc.fi/course/view.php?id=76)
> [Collection of slides and material](https://csc-training.github.io/csc-env-eff/)
> [Zoom-link for COURSE sessions](https://cscfi.zoom.us/j/64937965010?pwd=2bYjwrdEbJocLxkZlYRlUeG1kRSXLi.1)
> [Zoom-link for SUPPORT sessions / weekly user meetings](https://cscfi.zoom.us/j/65059161807)

[ToC]

## How we work
- [Code of Conduct](https://hackmd.io/1Yz0r5hWQXqWSpgJi7O4Og#Code-of-conduct)
- [Zoom instructions](https://hackmd.io/1Yz0r5hWQXqWSpgJi7O4Og#General-Zoom-instructions)
- [FAQs on HackMD, eLena and general course matters](https://hackmd.io/1Yz0r5hWQXqWSpgJi7O4Og#Tool-tips-amp-FAQs)

## 📅 Agenda

### Prerequisite / support session 24.9.
14:00-15:00 Join the pre-course support session, which is also part of [CSC's weekly user meetings](https://cscfi.zoom.us/j/65059161807)

### Day 1: Wednesday 1.10.

| Time | Content |
|-------|---------|
| 8:45 | Login, check connections work, test microphone |
| 9:00 | Welcome, course practical info, learning targets, ice-breaker, feedback/expectations (15 min) |
| 9:15 | **Topic 1: [Prerequisites & Connecting](https://csc-training.github.io/csc-env-eff/part-1/prerequisites/)** (15 min) |
| 9:30 | **Topic 2: [Introduction to HPC Environment](https://csc-training.github.io/csc-env-eff/part-1/hpc-intro/)** (20 min) |
| 9:50 | _Break_ (10 min)|
| 10:00 | **Topic 3: [Disk Systems](https://csc-training.github.io/csc-env-eff/part-1/disk-areas/)** (15 min) |
| 10:15 | **Intro to breakout rooms & hosts, joining the breakout rooms, accessing materials** (10 min) |
| 10:25 | **Topic 3: [Hands on 3.1](https://csc-training.github.io/csc-env-eff/hands-on/disk-areas/disk-areas-tutorial-maindisks.html)** (20 min) |
| 10:45 | **Going through the hands-on(s) together** (15 min)|
| 11:00 | _Break_ (10 min) |
| 11:10 | **Topic 4: [Module system](https://csc-training.github.io/csc-env-eff/part-1/modules/)** (15 min)|
| 11:25 | **Topic 4: [Hands on 4.1](https://csc-training.github.io/csc-env-eff/hands-on/modules/modules-puhti.html)** (20 min)|
| 11:45 | **Going through the hands-on(s) together** (10 min)|
| 12:00 | Finish |

### Day 2: Thursday 2.10.

| Time | Content |
|-------|---------|
| 8:45 | Channel open, "morning coffee" |
| 9:00 | Re-cap, feedback/expectations + some Qs from HedgeDoc (5 min) |
| 9:05 | **Topic 5: [Batch queue system and interactive use](https://csc-training.github.io/csc-env-eff/part-1/batch-jobs/)** (15 min)|
| 9:20 | **Topic 5: [Hands on 5.1](https://csc-training.github.io/csc-env-eff/hands-on/batch_jobs/serial.html)** (20 min) |
| 9:45 | **Going through the hands-on(s) together** (10 min)|
| 9:55 | _Break_ (10 min)|
| 10:05 | **Topic 6: [Understanding resource usage](https://csc-training.github.io/csc-env-eff/part-1/batch-resources/)** (20 min)|
| 10:25 | **Topic 6: [Hands on 6.1](https://csc-training.github.io/csc-env-eff/hands-on/batch_resources/tutorial_sacct_and_seff.html)** (20 min) |
| 10:45 | **Going through the hands-on(s) together** (10 min)|
| 10:55 | _Break_ (10 min) |
| 11:05 | **Topic 7: [Allas and where to keep your data](https://csc-training.github.io/csc-env-eff/part-1/allas/)** (25 min)|
| 11:30 | **Topic 7: [Hands on 7.1](https://csc-training.github.io/csc-env-eff/hands-on/allas/tutorial_allas-file-transfer.html)** (15 min) |
| 11:45 | **Going through the hands-on(s) together** (10 min)|
| 11:55 | Recap (5 min)|
| 12:00 | Finish |

---

## ☃️ ICE BREAKER (HackMD -practice)

### Ice breakers
Let's learn how to use this HackMD document by answering an ice breaker question!

**Q1: What topic of this course are you most excited about?**

**Answers:**

* Type here :)
* something
* yay!
* Topic 7! +1
* hello
* Topic 6
* Hello!*
* I am ready :D
* Topic 6
* Topic 5

## ✏️ Q&A: Add your questions here

Please type your questions ON THE BOTTOM OF THIS SECTION. We will answer them, and organise the document topically.

:::info
Scroll :arrow_down: to the bottom of the page to submit a question
:::

- [x] **Q1: As a postdoc at a Finnish uni, should I create a new project for my work on CSC with myself as the leader? Or do I need to find a project already set up within my university group/unit? (i.e. are single user projects frowned upon?)**
- A: It depends on your need. If you have some big project with computational needs, it is perhaps better to create a project for yourself. If your computing needs are very synergetic with your group's project, it may be just better to join with existing project. But yes, as a postdoc at a Finnish university, you can be the PI of your own project.

- [x] **Q2: When I login to to Puhti using command line on terminal, the system asks for phasephrase every time. It shouldn't, right? I'm using virtual Linux in Uni of Helsinki**
- A: It depends how you have set up your ssh keys. You can set ssh login with and without passphrase. Setting up ssh login with passphrase is more secure. Without a passphrase, anyone who gains access to your computer has the potential to copy your private key.
- A: It is also possible to have a SSH key with a passphrase (so that if someone steals the file from your hard drive, they can't use it without guessing the password) and have your system store the password so that when you log in, your SSH key is also automatically unlocked. It would be great if someone can share the exact steps for that, but if not, the University IT Helpdesk can surely give system-specific intructions!

- [x] **Q3: I can login to Puhti through the web interface and using the app PuTTY, but when I try to login using Powershell I get an error, why is that so? I have set up my key pair and have the private key in my local computer and the public key in MyCSC.**
- A: Powershell has some known issues, which have been outlined in this FAQ entry https://docs.csc.fi/support/faq/i-cannot-login/#why-is-my-ssh-client-saying-corrupted-mac-on-input does this help?

- [x] **Q4: I have the same problem as Q3: logging in via Powershell results in the error "Corrupted MAC on input". I tried to exclude/replace the broken algorithms as suggested by the FAQ entry but get only "Permission denied (publickey, hostbased)". Is this a problem of Powershell not finding my private key or maybe something else?**
- A: It could be, but it could also be something else. Have you been able to log in from something else than Powershell (e.g. MobaXterm or PuTTy)? If so, then we know that the MyCSC/Puhti side of the SSH key setup is ok. If not, and it's easy for you to try, you could. If you cannot log in from them either, then it's likely the [add public key to MyCSC part](https://csc-training.github.io/csc-env-eff/hands-on/connecting/ssh-keys.html#add-public-key-to-mycsc) that has problems. If you can log in elsewhere, then it's likely PowerShell not finding your key, and you could perhaps start by verifying that the private key can be found at `C:\Users\WINUSER/.ssh/id_rsa` (or other file name in the same directory) where `WINUSER` is your username.

- [x] **Q5: If I used a lot of resources and I am low in Fairshare, waiting for the job in queue list for long time, how to fix it? how to recover the Fairshare?**
- A: Fairshare is meant to give an equal opportunity for all users. Within that fairshare quota, one has to unfortnately sometimes wait in the queue before jobs are executed. That said, one can always write some efficient batch scripts where appropriate to avoid longer queueing. if you have too many jobs to submit, you can think of using some e.g.,subjob schedulers like HyperQUEUE(https://docs.csc.fi/apps/hyperqueue/) or array jobs (https://docs.csc.fi/computing/running/array-jobs/). You can also think of using workflows (https://docs.csc.fi/computing/running/throughput/) if it fits to your needs. Bottomline is to submit fewer slurm jobs to accomplish the same goal. Also check if you are unnecessarily requesting more resources (e.g.,, precious resources such as NVMe or GPUs resources).
- A: See also: https://docs.csc.fi/support/faq/why-is-my-job-queueing-so-long/

- [x] **Q6: Can I set nicer defaults for sacct?**
- A: Yes, if you write a custom function in eg. your `$HOME/.bashrc`. I have the following magic to also accomplish auto-width output:

sac() {
    if [[ "$*" != *"--format"* ]]; then
	sacct --parsable2 --format="JobID,State,ExitCode,Start,Elapsed,JobName" "$@" | column -t -s'|'
    else
	sacct --parsable2 "$@" | column -t -s'|'
    fi
}

This way "sac" runs it like that.

- [x] **Q7: I didn't quite understand why the error file was specified separately, and what did the %A %a notation stand for?**
- A: Programs usually produce output in "standard out", which is information about their normal processing. They can also produce output in "standard error", a different stream of information, which often contains nothing, but might have information about errors or something urgent. Slurm allows you to define separate templates for the files where those streams of information go. %A is job id number and %a is array index (you don't have to use it if you don't have many of those).
- A: By default all output goes to single file (slurm-<jobid>.txt). In case of single job this is usually ok. If for some reason you want to separate output and error messages you can specify `--output` and `--error` with a file name. You can use `%j` for job id to get a unique file name. In case of an array job there is an additional reason to specify separate files. By default the output from all the subjobs would be written into a single file. It would be difficult or impossible to know which message comes from which subjob. If we specify both `%A` and `%a`, output from each sub job goes to a separate file.

- [x] **Q8: Ok thanks - follow up: in which contexts can one use the %A %a -shortcuts to speficy the job id and array index - is it just in the SBATCH speficication? Is there any similar shortcut in the command line (to avoid typing in the job number every time...)?**
- A: Well, inside a Slurm batch script you have a job number and array index, and those can be accessed by the programs you are running as environment variables. For example `$SLURM_JOB_ID`. But when you are using eg. seff to ask about a job, the system doesn't know which job you're interested in, so you have to specify. Are you thinking about automation for showing information about the only task you have submitted, if there is only one? **That makes a lot of sense :D - I think rather a shortcut to the "last job", but this is maybe just very lazy.:)** Definitely doable with scripting, but I haven't run into this! Interesting idea.
- A: For scripting purposes it is sometimes handy to include option `--parsable`: `sbatch --parsable myjob.bash` returns only the jobid.

---
**:arrow_up: Write your questions above this line :arrow_up:**

---

## 📝 Feedback

### Feedback - Day 1

One thing you liked:

- I discovered how to see an overview of my projects, thanks! :)
- Knowing how to check what dependencies an older module version needs, so that all runs smoothly
- Introduction on disk systems was useful esp. for someone who's just getting into HPC.
- Lue -tool!
-

### Feedback - Day 2

One thing you liked:

- Batch job resource usage was extremely useful!
-
-

One thing that could be improved:

-
-
-
-

:::info
When you have time, please give us feedback also through the Official Feedback Form:
https://ssl.eventilla.com/feedback/101H7PYx9JlYbDR

(This is required to get the course certificate, in that case, give the feedback thru e-Lena!)

:::

## Useful links
- [CSC Computing Environment Part2 : Next steps](https://csc.fi/en/training-calendar/csc-computing-environment-part-2-next-steps/) Registration now open!!!
- [Elements of Supercomputing](https://edukamu.fi/elements-of-supercomputing) online course
- [Training links](https://docs.csc.fi/support/training-material/#training-materials-and-sources-from-csc-and-partners) by us and our friends
- [Weekly research user meetings](https://ssl.eventilla.com/usersupportcoffee) (come and ask anything! Also course related questions are welcome!)
- [Data management self-learning course](https://e-learn.csc.fi/course/view.php?id=63)
- [Sensitive Data (SD) services webpage](https://research.csc.fi/sensitive-data)
- [Sensitive Data (SD) services user guide](https://docs.csc.fi/data/sensitive-data/)