owned this note changed 3 years ago
Published Linked with GitHub

CSC Computing environment Q&A session 26.4.2022

tags: puhti mahti allas

This is the common "notebook" for the CSC Computing Services Q&A session" webinar organised in April 2022 at CSC -IT center for Science. Course page in e-Lena platform

This is the place to ask questions about the workshop content! We use the Zoom chat only for posting links, reporting Zoom problems and such.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
Hint: HackMD is great for sharing information in this kind of courses, as the code formatting is nice & easy with MarkDown! Just add 3 ticks (`) for thecode blocks. Otherwise, it's like a Google doc: it allows simultaneous editing.

Code of conduct

We strive to follow the Code of Conduct developed by The Carpentries organisation to foster a welcoming environment for everyone. In short:

  • Use welcoming and inclusive language
  • Be respectful of different viewpoints and experiences
  • Gracefully accept constructive criticism
  • Focus on what is best for the community
  • Show courtesy and respect towards other community members

TO DO before the course

  • Please make sure you have a CSC user account, project with Puhti and Allas as services, and have accepted the terms of use for Puhti and Allas in my.cs

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
Q & A

You can type your questions here. We will answer them, and this document will store the answers for you for later use!

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

General and practical matters

  • Q1: I have difficulty pasting my questions into HackMD (here). Do you have some instructions on how to write here?

    • A: Can you see these three icons on top left corner, next to HackMD text? There’s pencil, this side-by-side symbol, and an eye. In eye view, you can’t edit, you are just viewing. The other two reveal the markdown (MD) version of the page, which you can edit. I find it easiest to edit with the side-by-side view.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
Hint: You can also apply styling from the toolbar at the top
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
of the editing area.

  • Q2: I cannot access most of the slides from the course page. These are marked with an earth logo.

    • A: Try to use the arrow keys or space bar to switch the slide!
  • Q5: Do we get ECTS credits from this course for our university transcripts?

    • A: After you have completed the course including all the quizzes, you can download a certificate where we recommend 1 credit for this course. We are not allowed to give credits, but we can recommend them, and by taking the certificate to your university, they usually accept :)
  • Q6: Should I get my "own" project for the course, can't I just use a project you provide?

    • A: The course project will be available only during the course, and before and after that you need to do the exercises on your own projects anyway :) Also, we wish you to help you get past this first obstacle of setting up everything in the beginning! So yeah, better get your own project!
  • Q7: So many credendials Which ones do I need and where?!?

    • A: You need CSC credentials to login to Puhti, whereas for my.csc.fi and e-Lena you can nowadays login also with Haka credentials (=your university credentials). In my.csc.fi you can check the CSC username. If you have forgotten your password, check this link: https://docs.csc.fi/accounts/how-to-change-password/ -Note that it takes some time for the password to update to Puhti, if you need to change that :)
  • Q: How long are the course materials available online? Tutorials, slides etc.?

  • Q: To whom (what level student) should the CSC services be recommended?

    • Well, the level is not so critical but the need. Already now it's possible to set up an account for yourself (as a teacher/lecturer) and ask for training accounts for the participants (available for the duration of the course). To access the supercomputers the participant needs to be a member of a CSC project and setting up a project needs a Project manager with some requirements (a bit longer term position in a University). If you're interested send email to servicedesk@csc.fi. A new setup is also in the pipeline to enable "student accounts" which don't need a "supervisor" which will make this easier.
  • Q: Tykky container service. How to install python / local packages?

  • Q: I registered as a student but I can't found COMSOL multiphysics software. are available softwares are limited for student?

    • No such limit. Where did you look for comsol? How?
  • Q: How to access huge data (very big! :) from ALLAS to work with Matlab from cPouta? Are there examples online (I couldn't find them)?

  • Q: Hi, I have a basic question. I used to get email notif when my batch job status is failed. But now I only get notified when it successfully ran. Is there some changes in csc with this?

#!/bin/bash
#SBATCH --job-name=Jacque
#SBATCH --account=project_2000967
#SBATCH --time=06:00:00
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --output=Jacque
#SBATCH --error=Jacque-err
#SBATCH --mem=64G
#SBATCH --partition=small
#SBATCH --gres=nvme:300
#SBATCH --cpus-per-task=8
#SBATCH --mail-type=END
#SBATCH --mail-user=jacqueline.malazarte@oulu.fi
  • Great! You're asking --mail-type=END i.e. send email when the job ends. If you want mail also when it starts > --mail-type=BEGIN,END. If you also want notification when job fails, also specify FAIL > --mail-type=BEGIN,END,FAIL

  • Q: I am trying to run a batch job, basically doing a diamond blast to one database (non-redundant, nr). Previously I do it with swiss-prot, which took around 16min/sample and between 8-10G of memory. But with nr database the job is cancelled after 10h because exceeds the maximum memory (16G), which cannot be increased. I checked and any output file of the 40 desired was created before the job was cancelled. How can I run the job? How can I by-pass the memory limitation? I send this question by email hpcplatforms #538239

#!/bin/bash
#SBATCH --job-name=myTestdiamond2
#SBATCH --account=project_2005233
#SBATCH -o output_%j.txt
#SBATCH -e errors_%j.txt
#SBATCH --time=15:00:00
#SBATCH --mem-per-cpu=16G
#SBATCH --partition=small
##SBATCH --mail-type=BEGIN #uncomment to enable mail
  • Try sinfo command to learn about partitions, and about max mem

  • Check chapters 5 and 6 from the self-learning course, there we are talking about this! It's a common difficulty!

  • A good tutorial, where the memory runs out, and how to figure that out: https://csc-training.github.io/csc-env-eff/hands-on/batch_resources/tutorial_sacct_and_seff.html

  • How much memory is available? List of Puhti's partitions (queues) https://docs.csc.fi/computing/running/batch-job-partitions/

  • Try: #SBATCH --mem-per-cpu=32G

  • Don't use max mem, if you don't need that much -the "why" for this is explained in the training material

  • Q: Does the '#SBATCH time' starts when the job starts or when you submit the batch job?

    • When the job actually starts. So that's the limit of the actual time of running the job, not the queuing time
  • Q:

Action points:

  • add a quiz/FAQ for how much memory I can ask / is available?
Select a repo