Using CSC HPC environment efficiently, Part 2: Next steps 12.-13.11.2025

# Using CSC HPC environment efficiently, Part 2: Next steps 12.-13.11.2025 ###### tags: `puhti` `mahti` `allas` > This is the collaborative "notebook" for the "Using CSC HPC environment efficiently, Part 2: Next steps" course organised online in November 2025 by CSC -IT center for Science. **LINKS** > [Prerequisite course page in eLena platform](https://e-learn.csc.fi/course/view.php?id=75) > [Course page in eLena platform](https://e-learn.csc.fi/course/view.php?id=76) > [Collection of slides and material](https://csc-training.github.io/csc-env-eff/) > [Zoom-link for COURSE sessions](https://cscfi.zoom.us/j/63421430507?pwd=Wlsizj0jYaUxwuWV9IAmFORJDXIfkz.1) **Table of contents:** [ToC] ## How we work - [Code of Conduct](https://hackmd.io/1Yz0r5hWQXqWSpgJi7O4Og#Code-of-conduct) - [Zoom instructions](https://hackmd.io/1Yz0r5hWQXqWSpgJi7O4Og#General-Zoom-instructions) - [FAQs on HackMD, eLena and general course matters](https://hackmd.io/1Yz0r5hWQXqWSpgJi7O4Og#Tool-tips-amp-FAQs) ## 📅 Agenda ### Prerequisite / support sessions On Wednesdays 14:00-15:00, feel free to join the [CSC's weekly user meetings](https://cscfi.zoom.us/j/65059161807) to get extra help and discuss with us, also about the course matters! ### Day 1: Wednesday 12.11. | Time | Content | |-------|---------| | 8:45 | Login, check connections work, test microphone | | 9:00 | Welcome, course practical info (15 min) | | 9:15 | **Topic 8: [Working efficiently with data](https://csc-training.github.io/csc-env-eff/part-2/data-io/)** (30 min) | | 9:45 | _Break_ (10 min)| | 9:55 |**Intro to breakout rooms & hosts, joining the breakout rooms, accessing materials** (5 min)| | 10:00 | **Topic 8: [Hands on 8.1](https://csc-training.github.io/csc-env-eff/hands-on/data-io/tutorial-fastdisks.html)** (20 min)| | 10:25 | **Going through the hands-on(s) together** (10 min)| | 10:35 | **Topic 9: [Installing applications](https://csc-training.github.io/csc-env-eff/part-2/installing/)** (25 min) | | 11:00 | _Break_ (10 min) | | 11:10 | **Topic 9: Hands on [9.1](https://csc-training.github.io/csc-env-eff/hands-on/installing/installing_hands-on_binary.html) and [9.2](https://csc-training.github.io/csc-env-eff/hands-on/installing/installing_hands-on_python.html)** (30 min)| | 11:40 | **Going through the hands-on(s) together** (15 min)| | 11:55 | Recap + preparations for the next day (5 min) | | 12:00 | Finish | | 14:00 | [Weekly research user meeting](https://ssl.eventilla.com/usersupportcoffee) | ### Day 2: Thursday 13.11. | Time | Content | |-------|---------| | 8:45 | Channel open, "morning coffee" | | 9:00 | Re-cap, feedback/expectations + some Qs from HedgeDoc (5 min) | | 9:05 | **Topic 10: [Containers & Apptainer](https://csc-training.github.io/csc-env-eff/part-2/containers/)** (25 min)| | 9:30 | **Topic 10: [Hands on 10.1](https://csc-training.github.io/csc-env-eff/hands-on/singularity/singularity-tutorial_part1.html)** (30 min) | | 10:05 | _Break_ (10 min)| | 10:15 | **Going through the hands-on(s) together** (15 min)| | 10:30 | **Topic 11: [How to speed up jobs](https://csc-training.github.io/csc-env-eff/part-2/workflows/)** (35 min)| | 11:05 | _Break_ (10 min) | | 11:15 | **Topic 11: [Hands ons 11.1](https://csc-training.github.io/csc-env-eff/hands-on/throughput/scaling.html)** (35 min) | | 11:45 | **Going through the hands-on(s) together** (15 min)| | 11:55 | Recap (5 min)| | 12:00 | Finish | --- ## ☃️ ICE BREAKER (HackMD -practice) Let's learn how to use this HackMD document by answering an ice breaker question! **What topic of this course are you most excited about?** **Answers:** * Excited to do some hands-on learning! * * * ## ✏️ Q&A: Add your questions here Please type your questions **ON THE BOTTOM OF THIS SECTION**. We will answer them, and organise the document topically. :::info Scroll :arrow_down: to the bottom of the page to submit a question ::: - [x] **Q1: Was it so that the S3 protocol will be removed in the future and only Swift remains?** - A: The other way around: New Allas coming in 2026 will only support S3, Swift support will be removed. This is good to keep in mind when starting new projects: Prefer S3 instead of Swift. - [x] **Q2: To follow up on the question on individual file size in local scratch: If let's say I need to do I/O work on large raster files, can this be done in $LOCAL_SCRATCH? Is there a GB limit to a single file?** - A: There is no limit on the file size as such. if you have lot of such files, make sure to reserve enough local disk space. However, there is a limit on max local storage (NVMe) per node (please check here: https://docs.csc.fi/computing/running/batch-job-partitions/) - [x] **Can we use uv instead of pip in CSC ?** - A: Uv is not available in CSC managed Python environments. If you build your own containers you have more choise on packet managers. - [x] **How do we know which one should we use - process or threads? let's say we have two sorts of jobs, one is looping through a series of files and do the exact same thing on each, and they are independent, so I guess this should be used as a process? and then another sort of job that first runs a code, takes the output of this and does another thing on the output and then this output will be used for the input of the next code. This should be thread as it needs memory sharing?** - A: (from a non-MPI/openMP expert) The first case is more like a farming job (Looping through a series of files, independently), there is no need to have a shared memory (across file tasks), one prefers to use processes. In the later case (more like workflows for Chained tasks), you can't parallelise across the task and may get benefitted from shared memory for each task, we can prefer threads here. EDITING CLOSED. --- **:arrow_up: Write your questions above this line :arrow_up:** --- ## 📝 Feedback ### Feedback day 1 One thing you liked: - - - ### Feedback day 2 One thing you liked: - - One thing that could be improved: - - - ### Course feedback Please give us also "official" feedback at: https://ssl.eventilla.com/feedback/101HBzGxpVmY4lk (This will be used to develop our courses further. Note, that if you need the course certificate, please access this same form through the e-learning platform :) ) ## Useful links - [Elements of Supercomputing](https://edukamu.fi/elements-of-supercomputing) online course - [Training links](https://docs.csc.fi/support/training-material/#training-materials-and-sources-from-csc-and-partners) by us and our friends - [Weekly research user meetings](https://ssl.eventilla.com/usersupportcoffee) (come and ask anything!) - [Data management self-learning course](https://e-learn.csc.fi/course/view.php?id=63) - [Sensitive Data (SD) services webpage](https://research.csc.fi/sensitive-data) - [Sensitive Data (SD) services user guide](https://docs.csc.fi/data/sensitive-data/)