HackMD
New archive: https://hackmd.io/uVI5gLKDQoWZgAVcufYg2A?view
Contents of this documents and quicklinks
Recordings, slides etc material from short talks
7.9. Puhti web interface (recording)
14.9. LUMI (slides, recording)
21.9. CSC Notebooks (slides, recording)
28.9. Puhti OS update (slides)
5.10. Tykky (slides, recording)
12.10. SD services (slides, recording )
19.10. No short talk (autumn holidays)
26.10. Machine Learning guide (no slides, all information in docs, recording)
2.11. CSC cloud services Pouta and Rahti (slides, recording)
9.11. Workflows and Hyperqueue (recording)
16.11. CSC Training materials (recording)
23.11. RStudio/R in different platforms (slides, recording)
30.11. Introduction to the Language Bank of Finland (Kielipankki) (slides_part1, slides_part2, recording )
7.12. MyCSC (recording)
14.12. mini crash course in HPC
11.1. Upcoming CSC Computing Environment course (Self-learning course page)
18.1.Data retention policy updates (Slides, recording, data clean up in docs.csc.fi)
25.1.: No short talk, just Q&A session
1.2.: No short talk, just Q&A session
8.2.: Mini crash course in HPC 2
22.2.: SD Desktop (recording)
1.3.: LUMI (slides, recording)
8.3.: Copying scratch dir to Allas (recording)
15.3.: R/python-based web tools at CSC for reproducible research and teaching (slides)
22.3. No short talk
29.3. ssh keys and login safely (recording)
5.4.: Mini course as a first step to CSCs services (mini course)
2022-09-07 session
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Juha Lento
- Maria Dimitrova - computational/quantum chemistry, general HPC
- Anni Järvenpää - Kielipankki
- Laxman Yetukuri - Bioinformatics
- Robin Karlsson - Puhti web interface
- Mats Sjöberg - Machine learning
- Francesca Morello - Sensitive Data services
- Alvaro Gonzalez - Cloud support (Pouta/Rahti)
- Päivi Rauste - RDM training
- Kimmo Mattila - Allas, Sensitive Data services
- Juha Fagerholm - Resources, accounts, projects
-
Q1: Are there other platforms other than Puhti where to run R scripts? Would like to have more than 1534GB of RAM
- A: Laxman - Main room: Only other alternative we have at CSC is to use CSC notebooks. If you have not heard it before, please explore here: https://notebooks.rahtiapp.fi/welcome. It is probably possible to request more memory but that kind of large memory requests are difficult to entertain on cloud environment where CSC notebooks are deployed. For heavy memory usage, Puhti is probably the better choice.
- Juha - Main room: Asking more than 1.5TB memory for R-script… you are probably doing something wrong :) A rule of thumb (in current machines) is to use less than 4GB per cpu core. What is the science area? What are you trying to calculate?
- OP: Social Sciences. I am running Latent Class Growth modelling with multinomial outcome variable with 1-12 classes. Each class will have to be run around 40-50 times to avoid local maxima so will need to run an analysis that requireds around 11-12GB of ram around 500 times.
- Juha: Sounds like our data analysists could help here
- Maria D.: Note that a compute node has an upper limit of the amount of RAM. The CPUs in puhti can handle up to 768 GB each. One node contains two CPUs, so this makes a maximum of 1.5 TB of memory per node. This is a hardware limitation in typical HPC systems. There is a special Intel CPU model that can take both RAM and Intel Optane which add up to a maximum of 4 TB.
-
Q2: (same op as Q1) Can you share a computation between tasks in R-singularity container? In other words, if task n does not have enough memory, can it borrow memory from another task n+1?
- A: Laxman: When a container is launched, it is usually meant to perform a task. If two containers are lauched in different nodes, it is bit difficult to have shared memory. In HPC environment, each job has predefined memory and dedicated to only that task. So it is difficult share memory between different containers.
- Maybe useful tutorial: https://docs.csc.fi/support/tutorials/many/
-
Q3: How to give access to others for a ocean simulation code NEMO that I have compiled (FMI)? Could you make it available for others to use?
- A: Juha F - Accounts, projects You should adjust the access rights to the directory and files. Please contact CSC if you need assistance with the compilation and the use of the codes.
-
Q4: I am providing support to researchers working in our unit. How should I instruct the users to get access to CSC?
- A: Kimmo M - Accounts, projects Chipster can be accessed immediately with Haka. Other services require that there is a PI that sets up a project. You should not be shy to set up projects as you will not be charged for billing units. However try to scale your resource requests so that you are not waisting resources.
2022-09-14 session
Short talk: LUMI (slides)
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Kimmo Mattila - Allas, Sensitive Data services
- Mats Sjöberg - Machine learning
- Jesse Harrison - R, Data Analysis
- Martin Matthiesen - Language Technology / Data management
- Rasmus Kronberg - Chemistry
- Tua Hindersson-Söderholm - Research Data Management
- Alvaro Gonzalez - Cloud computing
- Francesca Morello _ Sensitive Data services
- Juha Fagerholm - Resources, accounts, projects
- Maria Dimitrova - computational/quantum chemistry, general HPC
Questions :
-
Q5: I am a PhD researcher who has never worked with RNA seq (bulk and single cell) data before, and I now need to decide where to store my data and where to analyse it. I don't really know how much data I will generate since I haven't worked with these methods before, but we're doing both bulk and scRNA seq from a patient cohort (40-100 patients). No one in my group has done seq studies and those who have just store their data on external hard drives. We don't really have a data management plan either. I suppose I would use ePouta for storing the data (involves patient information), but I'm also going to do analysis on Chipster and/or R in Puhti, and don't quite understand how exporting data from the cloud works. The university's research data "helpdesk" isn't very helpful, so I'm asking basically for guidelines on where I should store the data and mostly how do these platforms ePouta and Puhti work
-
Edit: 15.9. Sorry for not being available at the zoom on Wednesday, bu thank you for the tips, I will try to show up next week and then get some further advice :)
- A: Kimmo M - "RNAseq data management" room As you are working with human sequence data you should use either ePouta or CSC Sensitive data services: SD Connect and SD Desktop. Lets chat about using these services in the break out room.
- A: Minna A - answering in HackMD I's good that you are seeking for advice to plan your data management! I strongly suggest that you start by drafting a DMP at this point and update it when needed. This way you will solve data managament issues timely and avoid unnecessary risks. Your university propably has it's own guidance for drafting a DMP and the basic national guidance is available eg. in the www.DMPTuuli.fi or http://doi.org/10.5281/zenodo.5242629 Basic guidelines for Puhti: https://docs.csc.fi/support/tutorials/puhti_quick/ Sensitive data services user guide: https://docs.csc.fi/data/sensitive-data/
- A: I add here the general webpage about Sensitive Data service: https://research.csc.fi/sensitive-data-services-for-research You don't have to read the entire user guide to start using the services, we can point out the essential documentation. You can contact us at servicedesk@csc.fi (subject:Sensitive Data) and we can help you figure out how to use the services for your project.
- Martin M We discussed key management a bit and it became even clearer to me that we need better solutions for that.
-
Q6: I'm sometimes having trouble accessing the Puhti Webinterface. Trying to access puhti.csc.fi just times out. It worked just fine this morning (13.09.) but suddenly stopped working around 11 am. This has happened previously as well. What could be the problem?
- A: Mats & Sebu - Puhti Web Interface room It can be a local networking issue, if it works for your colleagues. But it can also sometimes be the load on the Puhti server. You can check the load situation on the dashboard. Link to the dashboard stats is on the front page of www.puhti.csc.fi . Just log in and scroll down, then you should see the stats :)
-
Q7 : I'm planning to use CSC notebooks for course work with the notebook-workspace that we are planning to create. Since there are some EU students who do not have any Haka accounts, how CSC is handling their processes, do we need to pay something on behalf of those students? Also, for the memory allocation, I have an estimate memory usage per session but can those numbers change for concurrent user. For ex: If paraller jupyter instances share most of the code in memory, total usage might be smaller. And what is the limit of memory than 1 student can have per session.
- A: Samantha -Notebooks room You can ask for separate "only notebooks" accounts which for the course would then use "special login" under the "login" button on Notebooks page; this happens via an email to notebooks@csc.fi
- International participants: Please send an email to notebooks@csc.fi about this case and explain your setup
-
Q8 : We would like to use the autograding possibilities in Jupyter Notebooks server created for students. It looks like there are almost all the components available already, jupyterhub, autograder-labextension. Administrator need to run only
pip install grader_labextensionand then everything is ok. Is this correct? If not we could run nbgrader manually, but what would be the optimal way to distribute the feedback to the students? Can workspace owner write to students permanent directories? Or perhaps the feedback could be stored in the shared directory with permissions allowing only the certain user to read it? But these are not necessary if autograder already works.- A: Samantha - Notebooks room
- CSC Notebooks: Sharing a read-write directory with students can't be done currently, but we have a feature request for it.
- partly resolved; contact with Notebooks team initiated to fully solve
-
Q9: Anyone from LUMI-G EAP? Multi-gpu pytorch issue
- Simppa asked this question. Ticket just sent (#853).
- A: [Emmanuel & Orian & Mats - room 4]
- export NCCL_SOCKET_IFNAME=hsn0,hsn1,hsn2,hsn3
- export NCCL_NET_GDR_LEVEL=3
[Mats] in the near future (within a week or so), we'll be updating our ML guide, which should have some LUMI examples for running multi-node PyTorch: https://docs.csc.fi/support/tutorials/ml-multi/
2022-09-21 session
Short talk: CSC Notebooks
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Alvaro Gonzalez (CSC) - Cloud support
- Rasmus Kronberg - Chemistry
- Anni Järvenpää - Kielipankki
- Mats Sjöberg - Machine learning, supercomputers
- Jake Muff - Quantum Computing, LUMI
- Laura Kalliokoski - Sensitive Data services, secondary use data
- Kimmo Mattila - Allas, senitive data, biosciences
- Maria Dimitrova - quantum chemistry and HPC
- Katri Tegel - Manager for Tools for dataintensive sciences, Product owner - CSC Notebooks (+ Olli + Ai)
- Jemal Tahir - Cloud support
- Tewodros Deneke - Cloud
From previous session: this customer joining the session today!
-
Q5: I am a PhD researcher who has never worked with RNA seq (bulk and single cell) data before, and I now need to decide where to store my data and where to analyse it. I don't really know how much data I will generate since I haven't worked with these methods before, but we're doing both bulk and scRNA seq from a patient cohort (40-100 patients). No one in my group has done seq studies and those who have just store their data on external hard drives. We don't really have a data management plan either. I suppose I would use ePouta for storing the data (involves patient information), but I'm also going to do analysis on Chipster and/or R in Puhti, and don't quite understand how exporting data from the cloud works. The university's research data "helpdesk" isn't very helpful, so I'm asking basically for guidelines on where I should store the data and mostly how do these platforms ePouta and Puhti work
- Edit: 15.9. Sorry for not being available at the zoom on Wednesday, bu thank you for the tips, I will try to show up next week and then get some further advice :)
- A: Kimmo M - "RNAseq data management" room As you are working with human sequence data you should use either ePouta or CSC Sensitive data services: SD Connect and SD Desktop. Lets chat about using these services in the break out room.
- A: Minna A - answering in HackMD I's good that you are seeking for advice to plan your data management! I strongly suggest that you start by drafting a DMP at this point and update it when needed. This way you will solve data managament issues timely and avoid unnecessary risks. Your university propably has it's own guidance for drafting a DMP and the basic national guidance is available eg. in the www.DMPTuuli.fi or http://doi.org/10.5281/zenodo.5242629 Basic guidelines for Puhti: https://docs.csc.fi/support/tutorials/puhti_quick/ Sensitive data services user guide: https://docs.csc.fi/data/sensitive-data/
- A: I add here the general webpage about Sensitive Data service: https://research.csc.fi/sensitive-data-services-for-research You don't have to read the entire user guide to start using the services, we can point out the essential documentation. You can contact us at servicedesk@csc.fi (subject:Sensitive Data) and we can help you figure out how to use the services for your project.
- Martin M We discussed key management a bit and it became even clearer to me that we need better solutions for that.
-
Q10: I have a large social media dataset and am thinking about the best tools to analyze the data. The data are potentially sensitive and thus might require SD-services. However certain tools such as Spark are not available by default there. So I'm wondering whether I could somehow get deploy those tools using SD-services at CSC, or whether I should instead analyze the data for instance at Rahti.
- A: Alvaro Gonzalez - answering in HackMD If the data is sensitive you cannot use Rahti, it is not considered secure enough. Asyou probavbly know, Rahti provides a Spark template, at the moment Spark's version is at v2.4, but we are working in making availabe a newer version.
- A: Mats Sjöberg Best tools for large-scale data analysis can be found in the supercomputers, but Spark isn't really suitable there. Are you sure you need to use Spark specifically? How sensitive is the data?
- The data contain usernames and e.g. political views of users, so under GDPR they might be classified as sensitive. However they are also publicly available on Twitter. The data are 80GB and we need to do computational analysis on the text contents. Do you think that this could be feasible using just default tools.
- A. Even if the information is publicly available, it still contains sensitive data, thus you might need to discuss with your organisation's legal service what they think about it. Also, the info where shared for one purpose but you will use it for research, thus it is good to double check.
- A. There are of course different levels of sensitivity. Supercomputers do have files protected
-
Q11: RHEL8 update
- A: Mats & Juha & Rasmus General discussion concerning RHEL8 update (what will change, what actions are expected from the users, how to communicate). In short, along with the update we will recompile the most used software and libraries and remove old ones. New modules and versions will be accordingly updated in docs.csc.fi. A preview of the updated docs is available at https://csc-guide-preview.rahtiapp.fi/origin/puhti-rhel8/. Now is a good time to test the new environment and modules, recompile your own codes and notify CSC if something is missing that is essential to the user. You can send a ticket to Service Desk (servicedesk@csc.fi) or ping us at chat.csc.fi in case you notice any issues. Further instructions available in Sebastian von Alfthan's email on 19.9 to the puhti-users list.
-
Q12: Rahti and "glusterfs" issues
- A: Alvaro The current "glusterfs" issue is that it is not possible to create new gluster volumes and the general performance on launching new Pods/containers is degraded. We believe the two issues are linked. This has been going on for some days already. We don't know for how muh long this is going to last, the Rahti administrators have a good lead on a solution, but nothing is for sure at the moment. We discussed alternatives, like using cinder volumes, a different type of volume with other characteristics and use cases that is not affected by this problem. It was also mentioned that the new version of Rahti, based on OpenShift OKD v4, will drop support for glusterfs altogether. Redhat is the company behind the OpenShift development and does not consider Glusterfs stable and mature enough to be used in OpenShift. A recommendation is to switch to Cinder now when possible, as this will ease the transition to the new Rahti in the future. Other filesystem solutions that cover the glusterfs volume use cases are being studied at the moment.
-
Q13: Large VMs requested by univ Oulu IT customers. ePouta running our resources. Sensitive data.
- A: JKimmo @ specialists: please fill in few notes here
-
Q14: Notebooks :)
- A: Katri, Olli, Ai @ specialists: please fill in few notes here
-
Q15: Containerizing non public python packages
- A: You can install local packages with pip install -e (if they are provided with setup.py). Running your own code should work as normal just by launching the Python-interpreter from the container. For installing other packages see: https://docs.csc.fi/apps/python/#installing-python-packages-to-existing-modules
2022-09-28 session
Short talk: Puhti Operating System update
Slides
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Sami Ilvonen - Puhti OS update
- Francesca Morello - Sensitive Data services
- Rasmus Kronberg - Chemistry
- Mats Sjöberg - Machine learning
- Anni Järvenpää - Kielipankki
- Alvaro Gonzalez - Cloud support
- Laura Kalliokoski - Sensitive Data services, secondary use
- Martin Matthiesen - Kielipankki
- Maria Dimitrova - Quantum chemistry, HPC
Room and topic
Room 1 -
Room 2 - Q18 (Henrik)
Room 3 - Conda & Tykky (Ari-Matti, Kimmo)
Room 4 -
Questions:
-
Q16: Do the Slurm development headers exist somewhere in Puhti? I've been trying to compile the PySlurm library, and it does not seem to find them.
- A: Mats As far as I know there are no Slurm headers present as it has been installed directly from rpm package. Is there a good reason to use PySlurm? The recommended way is to use Slurm directly, as using it via a library can sometimes cause undesirable effects, such as launching a huge amounts of jobs due to programming error, which could lock up the entire system.
- I think it would be nice (and more efficient) to use the Slurm API for e.g. custom monitoring scripts, but I guess the same thing can be done using the output from Slurm commands, so PySlurm isn't absolutely necessary.
- OK, I would recommend sending an email to servicedesk@csc.fi and ask for the headers to be installed. Then our sysadmins can answer that.
-
Q17: I am OpenFOAM user. I run my simulations on Mahti. I do not want to use multi-threading in CFD simulations; as it doesn't bring any speed-up. In the summary emails that I got from Mahti at the end of simulations show that I utilize "Cores per node: 256" even though as I know this value is 128 for Mahti without multi-threading (Also, my Slurm output file shows that SLURM_TASKS_PER_NODE= 128). Am I doing something wrong or is this info safe to ignore? In addition to that, during OpenFOAM simulations, if I use collated write output, my simulation hangs during write time and it is like that till I cancel it. However, same run works well in non-collated output. How may I solve this issue? Thanks a lot in advance!
- A: Esko -room “Cores per node: 256” is just Mahti's seff commands way to compare the performance of your batch job against the full capacity available on the servers nodes. On Mahti you may fill up the node using 128 cores, BUT as an extra, you have some additional capacity available, total of 256 threads. Some applications might use also that "extra" capacity. But, as you wrote, OpenFOAM does not gain anything from those extras, as it is extremely well scaling MPI application, but dont benefit of threading.That "256" causes some confusion when you look at the output of the seff command, for example, if you have ran perfectly scaling job with 128 cores, but you still get output like this:
Cores per node: 256
CPU Efficiency: 50%
50% means actually that your job has used 100% of the core capacity - an excellent MPI application! That is because the comparison is made against 256 threads.
That collated method should be a robust method. Could you send me your batch jobs script, and some description of you OpenFOAM model, such the size of the mesh, and the equations (sub-models) being solved? You may send that to servicedesk@csc.fi. The message will finally end up to me.
-
Q18: Please let me ask a question, I want to import my libraries in my program on the puhti.csc.fi site, but I get the following error: ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/projappl/project_123' Check the permissions. I tried the following command: pip install – user gym, but it is not resolved. Could you please advise me?
- A: Mats - The solution was to replace project_123 with the correct project for the user.
-
Q19 Conda and Tykky tips needed, could we have breakout
- Are tykky containers portable -> Not between completely incompatible OS:versions as the container startup process won't work. But
-
Q20: Why does CSC not support MPIrun anymore?
- A: Discussed in main room; was removed over a year ago; srun is misisng some MPI paramter (MPI_SET_NUM), so it does not work the same way it used to work before -> SLURM PMIx could be used or …(please fill in details :) )
- request to push srun people to implement the argument
- communicating via harddisk is bad idea? yes, filesystem synicing is terrible idea
- interprocess communication
2022-10-05 session
Short talk: Tykky container wrapper
Slides
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Anni Järvenpää - Kielipankki
- Rasmus Kronberg - Chemistry
- Alvaro Gonzalez - Cloud computing
- Mats Sjöberg - Machine learning
- Francesca Morello - Sensitive Data services
-
- many more
26 in total online
Room and topic
Tykky - Henrik & questions about Tykky
Room 2 -
Room 3 -
Room 4 -
Questions:
-
Q22: I'm using Gfortran in a project I just started to use in CSC. For my project I have made a precompiler which generates necessary use statement to acces variables or subroutines stored in modules. The names of these variables start with prefix j_ or p_. The precompiler also makes indentations which I have bee lazy to do. When making indentations the precompiler keeps track of if()then …endif and do i=1, … end do -structures. When these structures are mixed up, the precompiler writes an error message which is much more useful than the error message of Gfortran. When Gfortran writes at the end of a subroutine 'expecting endif' my precompiler writes at what lines all open if-then and do-structures have started. It makes the life much easier. I think that it would be quite easy to put these error messages into Gfortran. Is there anyone in CSC who would be interested to do this?
- A: We don't do Fortran development at CSC. It might be a good idea to contact Fortran developers.
-
Q22: How can Apptainer containers work with GUI programs?
- A: -main room: For accelerated viz, use base image that is tested with OOD: RHEL8 rockylinux , then test locally
2022-10-12 session
Short talk: SD services
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Laura Kalliokoski - Sensitive Data services, secondary use
- Rasmus Kronberg - Chemistry
- Mats Sjöberg - Machine learning
- Kalle Happonen - (e)Pouta services
-
- more
Room and topic
Room 1 - Questions regarding the short talk (sensitive data services, Francesca & Laura)
Room 2 - ePouta (KalleH)
Room 3 - Machine Learning / CUDA (Mats)
Room 4 -
Questions:
- Q22: I am doing a research where very large RAM and parallel computing are needed using MATLAB. However, there is limited introduction about this in the CSC document. I have followed the instruction provided there but failed to running codes on the server. The following is my configuration:
When I try submit the batch job, error arises as follows:
There are a few questions here.
- I am not clear what are the problems and how to fix them in the above codes.
- How to setup the identity file? When I try submit the task, it requires either identify file or password. With the identity file, I don’t have to input the password every time I submit tasks.
- What are the difference and relationship between ntasks-per-core and cpus-per-task? Is any rules to set those two values? I thought core==cpu. I am confusing about the meaning of ntasks-per-core and cpus-per-task.
- How can I specify the value of parameters that in the ‘additionalSubmitArgs’ as shown above, such as ntasks, cpus-per-task, mem-per-cpu, ntasks-per-core.
- Is it feasible to run matlab script with .sh file similar to
and to config parallel computing mode? I have tired that but didn’t work.
-
A: Unfortunately we did not have a MATLAB specialist on call today (12.11), but we will ask them to fill in an answer here. The very best way would be to send these questions also to servicedesk@csc.fi, and then ask to discuss the topic further in the weekly support session if needed :)
-
Q23: We are going to analyze SLAM-seq data, which unfortunately there is no option for it at CHIPSTER. So, we'll need a place to store our data set and analyze it. It appears to be possible at ePouta. I was wondering if I could discuss it at the coffee break today at 14:00.
- A: KalleH-room 2 This seems to be a suitable use case for epouta. As the use case is from the University of Helsinki, the University IT support can help with the access to ePouta, and this is usually the easiest way to get started.
-
Q24: What's the current status of LUMI-G, and can you provide any estimate (even if rough) when pilots might be able to start?
- A: Emmanuel (not in call today) -room Pilots have already started since the end of July on a dedicated partition called pilot, however unofficially. The official start of the 2nd pilot phase will hopefully begin at the end of October/beginning of November (very rough estimate!). Benchmark performance tests are still ongoing and it's difficult to foresee at the moment when they will end.
- Follow-up: thank you, much appreciated! Is there a possibility that more than the currently available 100 nodes could be added to the pilot partition before the official pilot period starts?
- A: I'm afraid not. It is even possible that HPE will need all the GPU nodes for their final benchmark performance tests during a short period. Users will be notified in advance.
-
Q25: Training with the ROCm port of Megatron-DeepSpeed appears to be numerically unstable in a way that does not impact the original CUDA version. Where to start debugging this?
- A: Mats I guess the first point would be trying to isolate the problem and submit a bug to the GitHub repository, those who did the initial conversion should be in the best position to fix it. I can also ping ROCm people inside CSC, but unfortunately none of them are machine learning experts.
-
Q26: Last week some Scipion/Relion jobs at Mahti were being killed by CSC staff due to cluster overheating. Could we look into the reasons with someone from the Mahti end here, as I haven't heard back from CSC staff about the issue so far?
- A: This question is being discussed among CSC admin, unfortunately no solution has been found yet, follow up via servicedesk
-
Q27: Is there GDPR training for researchers available in Finland? In the UK, the Office of National Statistics runs a "safe researcher" training course, which often is compulsory before accessing data.
- A: Francesca -room 1 I think that each organisation might have specific tranings.
-
Q28: I need to combine meteorogical informations with Coronavirus data from Oulu University.Some of the information needed is aggregated data (e.g. https://www.worldometers.info/coronavirus/), software needed e.g. QGIS
- A: Francesca -room 1 If you are processing aggregated and anonymised data you don't need to use SD services. However, if the data are pseudonymised, they are still sensitive. You can start testing or familiarise with the service here:
https://docs.csc.fi/data/sensitive-data/
Important things to take in account
- A: Francesca -room 1 If you are processing aggregated and anonymised data you don't need to use SD services. However, if the data are pseudonymised, they are still sensitive. You can start testing or familiarise with the service here:
-
Who is the data controller? Agreements between CSC and the Data Controller should be in place.
-
What software do you need to analyse the data? (https://docs.csc.fi/data/sensitive-data/sd_desktop/#pre-installed-software)
We can then have a meeting to discuss the details.
2022-10-19 session
No short talk (autumn holidays)
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Mats Sjöberg - Machine learning
- Laura Kalliokoski - Sensitive Data services, secondary use
- Alvaro Gonzalez - Cloud Computing
- Rasmus Kronberg - Chemistry
- and few more…
Room and topic
Room 1 - Q29 and Q30 with Rasmus - chemistry
Room 2 - Q31 - Kimmo - bash scripts on Puhti
Room 3 - Q32 - Francesca and Laura - Sensitive Data Services
Room 4 -
Questions:
-
Q29: Since new update on Puhti, Schrödinger maestro is not working on interactive session through desktop application
- A: Rasmus, room 1 Not sure what "not working" means, but one issue that has come up multiple times is related to the renaming of the login nodes. Please make sure that your
schrodinger.hostsfile does not contain any references to old login nodes, e.g.puhti-login1. If it does, please rename these aspuhti-login11. You can either do this by hand, or then delete the old hosts file and create a new one by running the/appl/soft/chem/schrodinger/set-hosts-file.bashscript.
- A: Rasmus, room 1 Not sure what "not working" means, but one issue that has come up multiple times is related to the renaming of the login nodes. Please make sure that your
-
Q30: Since new update on Puhti, MMPBSA.py of AmberTools requires additional setup (e.g. Python 2)
- A: Rasmus, room 1 Some of the Python scripts distributed with AmberTools are very old and may indeed require Python 2. Since Python 2 has been deprecated on CSC supercomputers, you would need to install Python 2 yourself within a containerized environment. This can be done quite easily with Tykky, see docs.csc.fi/computing/containers/tykky/). We will add a note highlighting this to docs.csc.fi/apps/amber/
-
Q31: How could one create a bash script for Puhti to run X number of different scripts that have resources, time etc allocatec?
- A: Kimmo, room 2 The answer/summary of the answer here
-
Q32: Interested in Sensitive Data
- A: Francesca, room 3 General discussion about sensitive data, from what is sensitive data, what service should be used, differences between SD service or ePouta, legal agreements needed for processing and data transfers. Feedback: difficult to find link to the page, documentation is comprehensive but needs definition of sensitive data to understand if there is a need or not. Suggestions: contact directly to servicedesk@csc.fi
-
Q33: How many years can you keep data in IDA? How about Etsin?
- A: Kimmo, room x Several years, we will find out more details; it can be there for long time but should not be forgotten, there has to always be someone responsible
Feedback/Note: Webinar and other event registration, sometimes difficult to find link to join, suggestion to send links via email before the event
Newsletter about webinars etc: Training pages: https://www.csc.fi/en/training#training-calendar , sign-up to training newsletter: https://www.csc.fi/newsletter
Number of participants in courses from Uni Helsinki?
ML guide link: https://docs.csc.fi/support/tutorials/ml-guide/
Feedback/Note on Puhti OS update: some panic about ssh key update that came with Puhti OS update, email got overread as usually there is mostly general info in them; after re-reading the email everything was fine again :)
short summary on LUMI talk discussed: https://www.lumi-supercomputer.eu ; early access platform available for everyone for testing
Feedback: Docs have everything people need, but it is not organized in a way that makes it easy to find the information; also it can be hard to decide if a certain page is something one needs to be reading or not
Relevant pages are often sent from servicedesk
2022-10-26 session
No short talk (autumn holidays)
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- and more…
Room and topic
Room 1 -
Room 2 - Schrödinger Maestro
Room 3 -
Room 4 -
Questions:
Note: Puhti currently not available: "Lustre metadata server is overloaded by requests."
Short Discussion about when to choose Puhti, Mahti or LUMI.
-
Q34: Regarding my question in the previous session "Since the new update on Puhti, Schrödinger maestro is not working on interactive sessions through desktop applications". Here "not working" means when I book an interactive session on https://www.puhti.csc.fi/ and uses a desktop app to have a maestro GUI session for analysing the results of my simulations, I get the following error message.
-
This happend after updating my host files so I think the issue is not due to host file but something else. Thanks
- A: It seems that you might have some python environment activated that conflicts with the one of Schrödinger. Also, you are launching maestro from a compute node, meaning that it might still be a host issue. You have probably defined
login11orlogin12as the host, but currently you are launching fromr18c02. These are mostly guesses, however, as I cannot try to reproduce this with Puhti being down due to Lustre issues. I'll get back to this and meanwhile would suggest you to send an email to servicedesk@csc.fi so that we can better document this. - Update: Launched Maestro as a
- A: It seems that you might have some python environment activated that conflicts with the one of Schrödinger. Also, you are launching maestro from a compute node, meaning that it might still be a host issue. You have probably defined
-
Q35: How to start using eDuuni-workspaces? Also interested in API support.
- A: Please send an email to servicedesk@csc.fi regarding this topic
-
Q36: Are there login problems to Puhti? I cant seem to login via SSH or the webfront.
- A: There seems to be a problem at the moment, and our guys are fixing it, so they are not currently present on the call :) We don't know right now what is causing it.
-
Q37: LUMI, AMD vs NVIDIA? Will user run into problem, when they are used to NVIDIA?
- A: Pytorch & Tensorflow for example exists for both, so user shouldn't even notice. Some very low-level CUDA-specific stuff of course might be effected, but when staying in Python level live should stay easy :)
2022-11-02 session
Short talk: CSC Cloud services, Pouta and Rahti (intro)
CSC Specialists available
- Maria Lehtivaara - Bio stuff :)
- Samantha Wittke - Geoinformatics
- Mats Sjöberg - Machine learning
Room and topic
Room 1 - Pouta & Rahti
Room 2 - SLURM & Nextflow
Room 3 - RStudio & Puhti web interface
Room 4 - Machine learning & HPC, getting started
Questions:
-
Q38: SLURM and Nextflow, how to submit to squeue?
- A: csc docs related to nextflow and SLURM here https://docs.csc.fi/support/tutorials/nextflow-puhti/
Also with hyperqueue https://docs.csc.fi/support/tutorials/nextflow-hq/
- A: csc docs related to nextflow and SLURM here https://docs.csc.fi/support/tutorials/nextflow-puhti/
-
Q39: Suppose you have a Docker container that you have developed and tested locally on your own machine. What is the procedure to move this container to Rahti and run it? Do you need to pay any mind to concepts such as kubernetes, pods, orchestration…?
- A: Room 1, Alvaro, Jemal
- You can use Rahti to deploy your application using the already created Docker image ( using the command line directly) or using a remote image registry (for example Docker hub). The other option is that you can deploy to Rahti using UI (source-2-image) directly from the project's git repo.
-
Q40: How do you know what Rahti template to choose?
- A: Room 1, Alvaro, Jemal
- Rahti has a collection of templates for different programming languages, so it is suitable to choose which programming language you build.
-
Q41: Puhti R Studio interface
- A: Room 3, Heli, Mats
Getting started with Puhti web interface and RStudio: go to https://www.puhti.csc.fi/ and log in. Click on the RStudio logo to start a session (you can start with the default resources). Note that you can set a time for how long the session lasts (the system does not warn when the time is up). RStudio should be running now in your browser. The files shown are your normal Puhti directory.
- A: Room 3, Heli, Mats
-
Q42: AI on supercomputer
- A:Room 4, Mats, Samantha
- machine learning guide resolved all questions: https://docs.csc.fi/support/tutorials/ml-guide/
- … and in particular the Getting started guide: https://docs.csc.fi/support/tutorials/ml-starting/
Internal discussion: Getting RStudio and R MD to work together (container + browser bridge)
Note / action point: Advertise the ML guide in training newsletter
2022-11-09 session
Short talk: Workflows and hyperqueue
CSC Specialists available
Maria Lehtivaara - Host, Bio stuff :)
Maria Dimitrova - Co-host
Rasmus Kronberg - Short talk about workflows & hyperqueue
Mats Sjöberg - Machine learning
Anni Järvenpää - Language bank
Room and topic
Room 1 - Tykky & LUMI, Rasmus
Room 2 - VSCode & ssh connection to Puhti, AnniJ, MariaD, Laxman & Mats
Room 3 -
Questions:
Q38: Where are we to find the other questions (<Q38)? Thanks!
A: In the archive! Link is in the box at the top of this document, and here: https://hackmd.io/@CSC-research-support/weekly_session_archive
Q39: How can I download NDVI data from NASA MODIS database? https://lpdaac.usgs.gov/products/mod13q1v006/ I have gps coordinates of my study sites and need to extract NDVI values from a given time period for these for further analyses. I have trouble getting the data downloaded in raster format that I could use. Or perhaps there is some R code ready that can be used for this, as I will be continuing analyses in R (package LEA)?
A: There seems to be some instructions for R here: https://rspatial.org/terra/modis/2-download.html and here: https://flograttarola.com/post/modis-downloads/ (no experience with them though). In the latter there is some possibility to define the output format which might help?
Q40: Do you have container wrapper tykky on Lumi? & which workflow tools do you have here?
A: Yes, https://docs.lumi-supercomputer.eu/software/installing/container-wrapper/. Although it is not named Tykky, it is basically the same tool. Regarding workflow tools, HyperQueue is projected to be on LUMI at some point. Latest versions of the program also support dependencies, although only through the Python API.
Q41: I used Visual Studio Code to connect to puhti.csc.fi but since last week I am not able to connect anymore. I haven't been able to solve the solution myself. (x2)
A: After the RHEL8 operating system update, the public key of the server has changed, making the ssh client suspect that you are not connecting to the intended machine but to a possibly malicious agent instead. To remove the outdated key from your computer, remove "puhti.csc.fi" lines from .ssh/known_hosts file. On unix machines the .ssh directory is usually found in your home directory, on Windows machines it typically is in C:\Users\<username>\.ssh unless a program has saved it to a specific location.
Q42: One RT ticket discussed in main room :) Hyperqueue might be the solution there.
Q43: It would be helpful for the newbies like me if there was an example of how to write a batch job to execute a Jupyter notebook (using Tensorflow for high intensive deep learning). Thanks in advance!
A: Good idea! Unfortunately I cannot join the call today, but you could take a look at the papermill tool. I can try to create some examples on how to use it on Puhti when I have some more free time. If you send a request to servicedesk@csc.fi I can reply to you directly there once I have some examples ready.
NOTE: 16.11. notes in HedgeDoc!!!
2022-11-16 session
Short talk: Training materials from CSC and partners
| Source | Example Content |
|---|---|
| CSC training calendar | Upcoming courses, Link to Elements of Supercomputing |
| eLena (Moodle environment) | Research data management, CSC computing environment |
| Tutorials at docs.csc.fi | Getting started with Puhti, Machine learning guide |
| Training material collection in docs.csc.fi (this page) | Links to thematically collected material |
| CSC Notebooks | Introduction to Python, Introduction to R, 2h intro to Quantum computing |
| VideoCSC | Introductions to topics, Recordings of weekly user zoom talks |
| CSC YouTube-channel | See also Chipster channel, Elmer channel |
| CSC Training GitHub | Repositories to course materials ( |
| LUMI Training events | Workshop on quantum computing, LUMI-C environment |
| CodeRefinery Lessons | Version control, open and collaborative coding |
| Aalto Python for Scientific Computing | pandas, plotting, binder, parallel programming |
| Source | URL | Example Content |
|---|---|---|
| CSC training calendar | Link to csc.fi | Upcoming courses, Link to Elements of Supercomputing |
| eLena (Moodle environment) | eLena | Research data management, CSC computing environment |
| Tutorials at docs.csc.fi | Docs: Tutorials | Getting started with Puhti, Machine learning guide |
| Training material collection in docs.csc.fi | Docs: Training | Links to thematically collected material |
| CSC Notebooks | CSC Notebooks | Introduction to Python, Introduction to R, 2h intro to Quantum computing |
| Video CSC | VideoCSC | Introductions to topics, Recordings of weekly user zoom talks |
| Youtube | CSC YouTube-channel | Chipster channel, Elmer channel |
| CSC Training GitHub | CSC Training repos | Repositories to course materials ( |
| LUMI events and training | LUMI Training events | Workshop on quantum computing, LUMI-C environment |
| CodeRefinery lessons | CodeRefinery Lessons | Version control, open and collaborative coding |
| Aalto Python for Scientific Computing | Material and registration | pandas, plotting, binder, parallel programming |
CSC Specialists available
Maria Lehtivaara - Bio stuff :)
Matias Jääskeläinen - Training Coordinator
Maria Dimitrova - quantum chemistry, HPC environment
Devaraju Narayanappa (CSC)- Climate stuff
Alvaro Gonzalez (CSC) - Cloud support
Room and topic
Room 1 - Python script in Puhti
Room 2 - Big dataset in Allas
Room 3 -
Room 4 -
Questions:
Q&As from previous sessions in the archive!
Q43: It would be helpful for the newbies like me if there was an example of how to write a batch job to execute a Jupyter notebook (using Tensorflow for high intensive deep learning). Thanks in advance!
- Good idea! Unfortunately I cannot join the call today, but you could take a look at the papermill tool. I can try to create some examples on how to use it on Puhti when I have some more free time. If you send a request to servicedesk@csc.fi I can reply to you directly there once I have some examples ready.
Q44: May be you can send a summary e-mail to CSC users with these links you just explained/ provided?
A: Super good idea! We will take a look into this. Tasks:
- Add to training newsletter
- Add table to Training material page in Docs
- Send to user-announcements -list
Q45: For my large model, even the longrun on Puhti with 14 days limit cannot satisfy the computation. I'm trying to use Abaqus on CSC with GPU. but I only used CPU before, how should I set the configuration accordingly with GPU assignment? Could you please help me to take a look with the batch file to change from CPU to GPU?
- Thanks for mentioning this! The standard recipe (in general, if running time is an issue) is to configure the simulation software (Abaqus, in this case) to write a checkpoint file, that can be used to restart the job in a new batch job - if indeed the work is so heavy that the 14 days is not enough. The other tricks are to reduce the problem size (so, there's less to compute), parallelize more (i.e. use more resources to make the job proceed faster - but adding more resources not always speeds up the work: this needs to be checked! (https://csc-training.github.io/csc-env-eff/#101-slides and also chapters 5 and 6 at https://csc-training.github.io/csc-env-eff/), or switch to another code that is more efficient (or which can use more resources in parallel to complete faster). In this particular case (Abaqus related) I'd encourage you to send an email to servicedesk and explain what you're aiming at, a copy of your batch script and an output file (and a description how far your job completed) and our Abaqus specialist can take a look if there's something concrete to try!
- About GPUs: some software can make use of GPUs, but typically not all functionality is available. Often just some tasks can make use of GPU. Did you check that running a small job with GPU (with Abaqus) is faster than CPU? If not, then likely it doesn't make sense to try with the bigger job either. What does the documentation say? Can your task make use of GPU? The links above that talk about efficient resource usage instruct to compare how fast the job goes with more or less resources (or CPU or GPU in this case). Always test with a small system: it completes quick and you don't need to wait long and at the same time only little resources are used on testing. If the test doesn't suggest GPUs are fast for your use case, they should not be used. Please have a look at the usage policy page which outlines when to use GPUs depending on how fast they are for your use case: https://docs.csc.fi/computing/usage-policy/
- Documentation about creating job scripts:
https://docs.csc.fi/computing/running/creating-job-scripts-puhti/
2022-11-23 session
Short talk: R & RStudio in CSC environment
CSC Specialists available
Maria Lehtivaara - Bio stuff :)
Maria Dimitrova - quantum chemistry, HPC environment
Juha Fagerholm - resource applications
Mats Sjöberg - machine learning
Alvaro Gonzalez - Cloud - Rahti and Pouta
Room and topic
Room 1 - R / RStudio
Room 2 - Nexflow
Room 3 - Moving data in Puhti / exporting to a database
Room 4 -
Questions:
Q&As from previous sessions in the archive!
Q46: I’m trying to use both Wandb and Tensorboard with my Python code. Could you give an example of how to add both in Python code? I'm confused as for the syntax. Thanks! I tried to look at https://docs.csc.fi/support/tutorials/gpu-ml/#profilers, where it says "See also how to launch TensorBoard using the Puhti web interface". However, once you click it doesn't take you to an explanation of this, but just to a listings of the apps (in that sense it's a bit misleading!).
A: The person asking the question was not in the call. Please send it as a servicedesk ticket instead. See: https://docs.csc.fi/support/contact/
Q47: I would like to discuss the possibility of executing Nextflow based pipelines from https://nf-co.re/ in puhti. More details here https://nf-co.re/docs/usage/installation
(Laxman): Yes. It is possible to run nf-core pipelines in CSC environment. You can check (tutorial 4 as a toy example) on how to run a nextflow pipeline here: https://yetulaxman.github.io/Biocontainer/tutorials/nextflow_tutorial.html
If you need to run heavy computing we might want to change differnt executors (local, slurm or HyperQueue)
Q48: Maybe irrelevant, but what is the difference between HackMD and HedgeDoc? This forum used to be in HackMD. Just curious :)
A: Yes, we used HackMD previously, but it was getting a bit buggy, so now we are trying HedgeDoc. They work very similarly, but HedgeDoc is installed on our own Rahti service.
Q49: Hi, my question is about OpenCL driver for a virtual desktop on Puhti, and how we can install it so we could be able to interface with the processing units via OpenCL in python for parallel computing.
A: There was a RT ticket sent by the customer, Henrik & Sebastian are replying to that.
Q50: Hello, Can you R just avalibe in Puhti?
A: Heli discussed this in the short talk and in the breakoutroom number 1
Q51: Type your question and (nick) name here!
Q52: Type your question and (nick) name here!
Q53: Type your question and (nick) name here!
Note: Add link to Zoom to HedgeDoc
Note: Tutorial for getting started when you have a fucntional environment in your own computer, and you need to move to Puhti? (You're not a root, there's SLURM, which packages you had installed (<- RDM stuff)! etc…) Lowering the bar
2022-11-30 session
Short talk: Introduction to the Language Bank of Finland (Kielipankki)
Slideset part 1 coming soon
Slideset part 2
CSC Specialists available
Maria Lehtivaara - Bio stuff :)
Maria Dimitrova - quantum chemistry
Samantha Wittke - Geoinformatics
Martin Matthiesen - Language bank
Sam Hardwick - Language bank
Mats Sjöberg - Machine learning
Laxmana Yetkuri - Bioinformatics
Jemal Tahir - cloud computing
Heli Juottonen - R
Anni Järvenpää - Language bank
Kimmo Mattila - Allas, sensitive data services
Questions:
Q51: frankier: How can we access /appl/data/kielipankki on Mahti?
A: Having the data on Mahti has not been requested before; ..
Note: Discussion about listing datasets available (staged) in Puhti, and extending services to Mahti and LUMI in future.
2022-12-07 session
Short talk: myCSC
CSC Specialists available
Maria Dimitrova - quantum chemistry
Samantha Wittke - Geoinformatics
Ville - myCSC
Juha Fagerholm
Juha Lento
Kimmo Mattila
Alvaro Gonzales
Heli J - R
Mats Sjöberg - ML
and some more
Room and topic
8 rooms used
Questions:
Q&As from previous sessions in the archive!
Type your question below if you are joining the next call -this is a way for us to trying to make sure we have the correct specialist(s) on the call. Add your (nick)name so we can then more easily connect you with the specialist. This is NOT to replace our servicedesk (servicedesk@csc.fi)! You can also just join the call and ask your question there.
Q52: In Puhti, I am trying to install an R package from a github repository using the "remotes" library, but I receive errors. The package requires a C++ compiler setup that is appropriate for the version of R installed. I am not sure if the error I receive is due to this, or due to some other issue. So, my questions are: 1) How can I install an R package from github to my libpath (/projappl/…)? 2) How can I set up the C++ compiler? [Hande Topa/NC]
-> Room 2 in Zoom
- C++ compiler should not be a problem, because there is one in the r-env container environment. Based on the discussion, the problem could be in package dependencies. The asker will send package details through service desk and we will try to install the package from the CSC end.
Q53: I noticed there's a mismatch between spatial data available in Puhti and documentation regarding Sentinel-1 mosaics. In documentation it reads these mosaics are available since October 2019, while that's not the case as the earliest mosaics in Puhti are from the start of 2018. However, there's a gap in the mosaic data between the end of 2018 and October 2019. Why is this and is there something to be done about it in near future? [Arttu Kivimäki/FGI]
- great question! Because of the gaps, we do not mention the pre 10/19 data in the docs. We can discuss this in todays session, but we will also invite soon to have a broader discussion with known users of this datasets; to figure out what is the best way forward for everyone Samantha
Q54: In Puhti, I'm having problems using Stan in r-env via "cmdstanr"-backend. I'm following the CSC r-env guides about using Stan. According to the guide the installation should be found at "cmdstanr::set_cmdstan_path("/appl/soft/math/r-env/421-stan/cmdstan-2.30.1")", but it doesn't seem to be there as i get an error message: "Path not set. Can't find directory: /appl/soft/math/r-env/421-stan/cmdstan-2.30.1". I'm using R through interactive RStudio -session. [Henri Wallen/LaY]
-> Room 2 in Zoom
- For some reason, the rstan folder is visible in the r-env folder on Puhti terminal but not when you access RStudio through the Puhti web interface. There is already a ticket for the same problem so we will continue solving the problem through there.
Q55: Narges - Room 4 in Zoom
Bioinformatics, RNA velocity, Python based tools
I couldnt find any of the python tools for RNA velocity. I was wondering if they can be available on CSC and if not what is the best way to install several packages.
- RNA velocity tool, scvelo, needs to be insatlled using Tykky wrapper as it is not readily available as module on Puhti. The documentation for Tyyky tool is available in CSC docs here: https://docs.csc.fi/computing/containers/tykky/.
As scvelo is available as a conda package, the user can install it easily using Tykky.
Q56: Is there any progress with sensitive data services regarding saving work independently of virtual machine (i.e., not losing it when crashing)? This refers to the secondary data use as in Findata. Tom R
-> Room 5 in Zoom
- Reusable data volumes will become available in the next SD Desktop version.
Q57: Is the ntasks setting consistent for Spark setting? And what will happen if I call CUDA program from cpu nodes? [name=Haining]
-> Room 6 in Zoom
- User advised to get in touch with Service Desk for further information at servicedesk@csc.fi
Q58: Pytorch distributed computing issues. How to debug, how to run? General advice on distributed computing [name=Maxim]
-> Room 7 in Zoom
Q59: Rahti seems to have an old version of Kubernentes v1.11. Is there any roadmap to upgrade Rahti?
- Yes, the Rahti team is currently working on deploying a new version of Rahti with a more recent Kubernetes version. For reference the current Rahti kubernetes version is:
v1.11.0+d4cacc0, and the internal BETA version of Rahti has kubernetesv1.23.5. Among other improvements that will come with new Rahti, we will get new versions of Kubernetes deployed in a more "agile" way. There is yet no date for the new Rahti.
-> room 8
Spyder availability on Puhti
-> geoconda and python-data modules include Spyder
https://docs.csc.fi/apps/python/#spyder
2022-12-14 session
Short talk: Mini crash course in HPC
CSC Specialists available
- Mats Sjöberg, Machine learning
- Heli Juottonen, R
- Maria Dimitrova
- and more
Questions:
- Q60: Type your question and (nick) name here!
- …
- Q61: Rostislav ([CSC #588823] Bug in "system" call from MPI task at puhti)
Just copy-paste of the OP
I have a simple tool to make large arrays of small job as a single MPI
task pexes.c (heavily reworked source from Juha Lento, attached). It gets a
text from stdin to the MPI member 0, and uses other tasks to execute each line
of the text via system (3). The program can be
compiled with enabled debug, so it shows actual status of 'system'
calls.
My use case normally is some long bash script with several loops
producing lists of commands that are fed to the srun … pexes command
via pipe after done. Each list needs different memory/cpu/no_tasks,
so separate job is needed for them, and there might be some lightweight
commands in between the loop. Some lists might be empty because needed results
have been done already.
For some reason, when called over more than one node on puhti the
system calls return zero status immediately without doing anything.
I have made a simple test case to illustrate the issue:
creates only first 19 files, while
works perfectly well. The above scripts used to work at puhti before the
recent upgrade, and still works perfectly well (with different project
and partition) at mahti, fmi cray systems and several other
environments.
Juha Lento from CSC suggested that it is likely a bug in slurm. Can that
be fixed?
Juha also suggested to use sbatch-hq, but I would need something that
reads stdin rather than a file, and can specify per-task memory.
In any case, it would be great to figure out the reason for silent
failure of 'system', since pexes is not the only MPI program we use,
where the call is used.
For now the suggestions have been to change the workflow and not to use system from MPI processes. I wonder if the issue can be fixed?
- More details will be given via the existing service desk ticket.
-
Q62: Type your question and (nick) name here!
I have question regarding running jupyter notebook installed via tykky.
Thanks in advance!
Fatemeh- Currently tykky-based installations do not work with Jupyter started from Puhti web UI. However the "old fashioned" method described here should still work: https://docs.csc.fi/support/tutorials/rstudio-or-jupyter-notebooks/#2-load-module-and-start-rstudio-or-jupyter-notebook-server
-
Q63: Rahti openshift kubernetes cluster (Joaquin Rives)
I have question regarding Rahti service. We would like to deploy kubeflow pipelines, however
the instalation requires the creation of some cluster-scoped resources and configuration.
In Rahti you only have acccess to your project namespace, so the deployment fails.
Is there any work around this problem?
E.g. could someone with admin privileages do the deployment for us?
Is there any other CSC service we could use?- Hi. Rahti is a shared and CSC-maintained platform which means projects get permissions to their namespaces only. This is because of isolation between projects. If projects need cluster-wide access, they can build and manage custom kubernetes installation on cPouta. In the next Rahti platform version we try to bring more developer tools available for projects.
-
Q64: Lumi eap question. I can't run pytorch multinode training without setting environment variable NCCL_SOCKET_IFNAME="lo". If this is set, scripts run nicely on one node, but not on multiple nodes. Any idea how to fix this?
-
Q65 Lumi eap question. What comparative performance should one expect for nvidia tuned pytorch models compared to lumi's 250x?
-
Q66 When LUMI's ROCm version will be updated? Currently LUMI is running ROCm 5.0 when 5.4 is already out. PyTorch's official PIP packages are compiled against ROCm 5.2 and they do not work as they need a higher ROCm to be installed on the machine.
-
Q67 LUMI pytorch packages
- Very experimental modules available:
2022-12-21 session
No short talk today
CSC Specialists available
- a few
Questions:
Click the pencil icon in the top left to switch to edit mode to start typing.
Note that if you are typing a question in here, we expect you to join the Zoom session
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Otherwise, it's probably best to send a support request to servicedesk@csc.fi!
This is not replacing our service desk and ticketing system!
- Q68 Puhti being slow
- a general phenomenon at the moment
- Q69 FastAQC (?) related to ticket <link>
- ticket is assigned and will be answered soon
- Q70 File removal from scratch
- …
- Q70 GPU on SD desktop
- medical imagery for ML
- not possible to use supercomputer for sensitive data currently, LUMI tests ongoing
2023-01-11 session
Short talk: Upcoming CSC Computing Environment online courses
- Next online course 7.-9.2. , registration now open!
- Self-learning course based on same materials -work on your own pace!
- Prerequisites:
- Basic Unix skills
- CSC credentials & project
- Laptop with Zoom (and an SSH client)
- Watch the lecture videos beforehand
- To help you, we have created this minicourse & also organise a separete support session!
- All material available for you!
- e-Lena e-learning platform with the materials and quizzes to test your learning
- Slides, tutorials & exercises, videos
CSC Specialists available
- Maria Lehtivaara (bio, CSC Computing Environment course)
- Ari-Matti Sarén (bio)
- Heli Juottonen (R )
- Samantha Wittke (Geoinformatics)
Room and topic
Room 1 - Matti & Ari-Matti, Q71
Room 2 - Sakshi & Samantha, Puhti issue with running jobs
Room 3 -
Room 4 -
Questions:
Click the pencil icon in the top left to switch to edit mode to start typing.
- Q71 Matti Javanainen: I assume a lot of cryo-EM data is processed on CSC clusters, and those are often deposited to EMPIAR (https://www.ebi.ac.uk/empiar/). Would it be possible to have the Aspera tool available on the clusters, so data could be deposited directly from CSC from the command line? We're about to deposit a 14TB dataset :)
- What (research)field does this relate to? Bio?
- ascp is available as part of biokit module
- Room1
- Q72 Sakshi: Problem running job (srun) in Puhti, error13?**
- VSCode Remote SSH: https://docs.csc.fi/support/tutorials/remote-dev/#visual-studio-code-with-remote-ssh-plugin
- Room2
- Q73 Eetu: MATLAB problem.
- Sampo not present on the call
- Q74 Type your question and nickname here
- ..
2023-01-18 session
Short talk: Supercomputer storage and updated policies
CSC Specialists available
- Sebastian von Alfthan, data removal policies
- Henrik Nortamo, Tykky, containers
- Mats Sjöberg, machine learning
- Ari-Matti Sarén, bio
- Alvaro Gonzalez, cloud support
- Heli Juottonen, R
Questions:
-
Q74 How to use Comsol on CSC? (Max)
- ..I want to use Comsol on CSC, how does it work?
- A: Unfortunately, COMSOL support belongs to Sampo S, who is at the moment in another meeting at the same time. Sampo suggests to first take a look at COMSOL docs page, and then contact us via servicedesk@csc.fi , if there are problems!
-
Q75 Alphafold, own copy of db for each? (JVL)
-
GPU node on Puhti vs in ePouta; do they differ in practice for alphafold (if one can use either)?
- At least one thing to consider when choosing between the two options is the availability of computing resouces in epouta. if I remember correctly, memory (at least 32 GB) and disk space (around 2-3 TB) requiremnets are high for Alphafold. And also SSD drives are good when working with huge databses as is the case here.
- Using Puhti or Mahti uses much less billing units, as you are paying only for the actual computation.
- At least at the moment you will need to download your own copy of the DB. We are looking into this, but there are performance issues to consider.
-
Q76 Jupyter notebooks for courses, best practices for using Jupyter notebooks for bioinformatic courses?
- CSC notebooks is the first option: https://docs.csc.fi/cloud/csc_notebooks/
- Teacher needs CSC account, students need only HAKA
- If exceptionally a lot of computing resources needed, or Puhti existing installations / data useful: https://docs.csc.fi/computing/webinterface/jupyter-for-courses/
- Everybody needs CSC accounts.
- Are modules available when using "Jupyter for courses" approach thru Puhti web interface?
- yes, you can either use existing modules or build your own
- Two tutorials: https://docs.csc.fi/support/tutorials/#csc-resources-for-courses
- Great, thanks!
- CSC notebooks is the first option: https://docs.csc.fi/cloud/csc_notebooks/
-
Q77 Hi, can we ask help for puhti-related problems during the session? (rosannahu)
- R: Why it gives a message "Project xxx is not accessible" when trying to get access to a project folder?
- Mats: Where exactly do you see this message? Puhti web user interface or terminal (ssh)?
- R: Terminal.
- Mats: can we continue in breakout room 3? My Zoom just crashed, but I'm back now :-)
- Answer: some strange thing with project not accessible on Puhti. Contact servicedesk@csc.fi.
-
Q78 When will new OpenStack volume quota be available in ePouta? (Juhana K)
- …
- About the GPU flavors the documentation is https://docs.csc.fi/cloud/pouta/vm-flavors-and-billing/#gpu-flavors_2
-
Q79 Applying billing units: In the application there is question about "program codes". What does this exactly refer to?
- It refers to what you plan to do with the resources: which applications (or programs, or codes) you plan to use. Maybe we could rephrase it, thanks!
- Ah, thank you! Maybe a practical guide for the application form would be good? :D Thank you!
- Yes. Sometimes knowing a bit more on what is being planned helps us support / plan ahead, but those texts don't need to be funding application level documents ;) Mentioning results and publications (and acknowledging CSC there) helps us convince the Ministry that the capacity is needed and used to do science. Also, for big requests in particular, we need to able to evaluate if granting resources makes sense (good plan, CV/history suggests the users know what they're doing).
2023-01-25 session
- There is also LUMI support session: https://lumi-supercomputer.eu/events/usercoffeebreaks/
Short talk: no short talk today, just Q&A
- Q80 How to setup within-chain parallellization for brms/Stan in R-env (Puhti)?
- This requires a project-specific installation of cmdstan (needs about 5 GB of memory and generates around 30K files). Sending instructions to the asker (available by emailing servicedesk@csc.fi).
2023-02-01 session
Short talk: no short talk today, just Q&A
-
Q81
We would like to use an internal python library in a conda environment within Puhti, so we are wondering what is the best way to do so. Cloning the github repository directly to the SCRATCH folder does not seem to work. The error that is raised when trying cloning it using SSH is:ssh: connect to host gitlab.iqm.fi port 22: Connection timed out- Answer: It looks like you are not using GitHub, but some company-internal GitLab instance which is not accessible from the outside world. Can you copy the files directly with other means? See our documentation on copying files. BTW, we strongly discourage the use of conda in our Supercomputers. See this page for alternatives (and explanation why it's discouraged): https://docs.csc.fi/support/deprecate-conda/
-
Q82 ePouta: are there io.haswell.32core flavors currently available? (Juhana K)
- Answer: no ePouta specialists available in zoom, check via servicedesk
2023-02-08 session
Short talk: Guide to CSC webpages for researchers
-
Q83 I am trying to run several analyses in r-package LEA for my RAD snp data (exported from STACKS and modified to .geno format with vcftogeno -r-code). However, I need to impute missing snp values with a custom r-code, which seems to have issues not functioning as it should. My formatting of the snp data also loses names of the snps, and I can not quite figure out how I should format the snp data to keep the ID information and get the code running. So would need help from someone familiar with R functions and preferably also with LEA r-package. (Finnish also ok!)
- Answer: We don't have anyone very familiar with this specific R package, but I'm happy to try to solve this in a breakout room. :)
Great I am in Zoom, but missed if you already looked out for me…
- Answer: We don't have anyone very familiar with this specific R package, but I'm happy to try to solve this in a breakout room. :)
-
Q84 I noticed that the
python-datamodules on Puhti set the environment variableOMP_NUM_THREADS=1. The obvious workaround to this (when using applications that actually use OpenMP and like threads) is to set the variable myself after loading the modules, but I'd still like to understand why these modules behave like that.- Python/numpy uses Intel MKL, which is threaded. It's easy to launch too many threads per cpu core. MKL uses OMP_NUM_THREADS if MKL_NUM_THREADS is not defined (or defaults to nproc if neither is set!?!). You can use MKL_NUM_THREADS to control MKL threads and OMP_NUM_THREADS to control application threads, but it's better for users' to control it "manually."
- We discussed in the break out room that
OMP_NUM_THREADS=1setting should be better documented (at least mentioned in docs.csc.fi). We (CSC) could also consider some logic that if it's already set the module doesn't change that setting. We'll consider that kind of setting.
-
Q85 Since tuesday's maintenance break, I lost some 80% of performance on 200 nodes w/ Vlasiator, OpenMPI&GCC. ~no differenec between running a single node - two node test.
- A: Try setting
export SRUN_CPUS_PER_TASK="${SLURM_CPUS_PER_TASK:-}"in your batch script before runningsrun. The slurm version was updated and this affects how certain settings are inherited from the #SBATCH directives (notably, the number of threads per task). We are currently investigating if we could make this environment variable export global so that the old behavior would be restored.- AA: Alright, let's try that!
- AA: helped with small testcase, about the needed improvement
- Resolved at scale!
- Very good!
- Remember to check FMI Slack fmi-computing channel, and Mahti motd when you log in :)
- hah, ofc :). Need to check out if UH Vlasiateurs can get to FMI Slack as well ^^'
- I posted the info there (INAR Slack channel computing_discussion), too :)
- Very good if you can warn the UH people directly, not everyone is reading the Slack ;)
- A: Try setting
-
Q86 I am new to the CSC swithcing from UH's cluster. I have two questions: a) I would need xvfb installed, it is not found on the current available modules, how can I get it? b) I have problems setting up the ssh keys, although I think I have done everything correctly, I would need someone to go through the process with me to spot the error.
- We found the culprit in the breakout room: The fix was: chmod 700 ~/.ssh on Puhti (It must not have group read rights, even though the group has only the user in it.)
-
Q87 Is there any advances in using SD-desktop with single registry data?
- Updated the status of single register use cases on SD Desktop (after a pilot ready for use, needs contracts between data controller and CSC) and discussed the availability of GPU's on SD Desktop (not yet available and no estimate about the availability in future), continued the discussion from previous weeks.
-
Q88 I am a member of a project which is shared between my supervisor (project manager), me, and my Master's student. Me and my student are working on the same things and using the same conda environments that I originally created. We are having some problems with reading/writing/executing permissions. For example, he can install new packages on my environments, but then I'll be unable to use them and also the packages that were overwritten by him. However, I am able to deinstall those packages by deleting them manually. What should we do to be able to both use the same environment? Should we both run something like 'chmod -R g+rwX /path/to/env/dir'? The environments are created in accordance with these instructions: https://docs.csc.fi/support/tutorials/conda/
- Default umask is 0007, isn't it? It should allow the g+w. Perhaps group sticky bit on project directory?
- Using conda directly is bad on the parallel file system. We recommend to use containerized version of conda environments with tykky tool
https://docs.csc.fi/computing/containers/tykky/ .
I suggest creating a common GitHub repository and host a list of python packages in a YAML file, that tykky tool can directyly use. And all members of the project can create the same environment (or modify if needed).
2023-02-15 session
Short talk: Mini crash course in HPC: Writing batch job scripts: Introduction and allocating CPU resources
See also more information on batch job scripts in CSC Documentation
To be continued 8.3: Mini crash course in HPC - Batch job scripts: Memory, disks and GPUs
-
Q89 I might have misunderstood, but did you say that srun is used for parallel processing. Isn’t it also used for any other program (e.g. serial)?
- discussed in main room
- JL: cluster jobs are primarily "not now", "not here", and non-interactive – all input from files, all output to files. That feels different from the "now, here, GUI" use of programs on PC
- JL: hyperthreading is typically not enabled on HPC clusters; it benefits (some) non-HPC workloads
- JL: the 'srun' executes process within allocation and gathers statistics from the run. If run outside allocation (i.e. on login-node), the srun creates allocation and then executes process within that allocation. The srun also initializes MPI communication for a MPI application
- A specific allocation before calling srun is also possible via the command salloc, but generally, the recommended workflow is to write a batch job script with the resources listed inside and submitted as
sbatch jobscript.sh - The allocation parameters can be given as arguments to the command sbatch, though, it may be hard to remember what you requested at submission time, so you should list the requests within the batch script. The mentioned (and not recommended) way of giving parameters to sbatch can look like
sbatch -N 1 -n 1 -c 40 --time 4:00:00 --mem 4G jobscript.sh
- A specific allocation before calling srun is also possible via the command salloc, but generally, the recommended workflow is to write a batch job script with the resources listed inside and submitted as
- JL: threaded applications tend to detect how many cores the system has, but do usually offer command-line parameter to specify number of threads. (The detection does not always match allocation. Hence explicit parameter is safer.) The section "OUTPUT ENVIRONMENT VARIABLES" in
man sbatchlists environment variables that the job script can use. For example, the 'SLURM_CPUS_PER_TASK'.- A common way of setting the number of threads for OpenMP codes is by
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK - Some programs run in parallel even without srun (you have to check the manual of the program you are using and/or the CSC Docs if it's a pre-installed one on the CSC computers)
- A common way of setting the number of threads for OpenMP codes is by
- JL: a job script (or sbatch/srun command-line parameters) does request resources (cores, memory, time). SLURM allocates resources for the job and commands run within the allocation. If commands try to exceed allocation, then SLURM kills the job:
- If you do not request enough resources, then your program quits prematurely
- If you do request too much resources, then some remain unused. From other side: when others request too much requests, your job has to wait unnecessarily. You don't want to be that "other"
- Jobs can be run in parallel also by manually splitting the calculation into meaningful chunks and executing them separately, e.g., via array job. Make sure you combine a series of jobs within the same job script! Do not submit 50 jobs individually to SLURM!
-
Q90 Problem with own script not running on Puhti, Anna -> room 1, Juha
- Juha: We looked into how to see the actual resource usage of the program in the compute node using
htopcommand. This is kind of easy, just check withsqueuein which compute node the job is running, and while the job is running, ssh to the node with something likessh r07c01, and in the compute node givemodule load htopandhtopto see how many threads/processes the program has launched and which cpu cores are working with the threads. We saw that for some input files the program was using quite lot of memory (probably the cause of the original problem), and also that most of the time only a few cpu cores had anything to do (short burst of parallel processing). Now, the tricky part is how to write a better program that uses memory and cpu cores more efficiently, but that is another story. The first step there is to profile the existing program, to see in which parts are slow and which parts use a lot of memory.- JL: one "trick" for tricky part is to split the job into multiple steps and run them in "appropriate" resources/partitions. Alas, that works only if different applications do the steps – not when one executable does all (hi and lo memory parts) in one go. ('sbatch' has option
--dependencyfor such "split workflow".)- Juha: This works only if each step is long enough, in the order of ten minutes+, ir so. Here most of the time was used in sequential processing, so the program itself needs some attention, if more performance is required.
- JL: one "trick" for tricky part is to split the job into multiple steps and run them in "appropriate" resources/partitions. Alas, that works only if different applications do the steps – not when one executable does all (hi and lo memory parts) in one go. ('sbatch' has option
- Juha: We looked into how to see the actual resource usage of the program in the compute node using
-
Q91 What to do if memory runs out, Pekka -> room 2, Ari-Matti
- A: There are some things to try:
- First try reserving more memory. Puhti nodes have up to 1,5 TB available. For reservations over 382 GB you need to specify "hugemem" (or "hugemem_longrun") partition. More in Docs: https://docs.csc.fi/computing/running/batch-job-partitions/
- For Java based software it is often not enough to just increase memory reservation, you also need to adjust Java settings, usually maximum heap size (
-Xmx) See software documentation for details. - If running a threaded application, try reducing the number of threads. In many cases this also lowers memory requirements.
- Check software documentation. It is typical that some parameters have large effect on memory usage. (For bio software "usual suspects" are things like Kmer size or allowed N of mismatches that can increase the search space.) Some software even have a command line option to limit memory usage.
- Check if the data could be processed in parts.
- Check if other software is available for same kind of task.
- A: There are some things to try:
2023-02-22 session
Short talk: SD Desktop, what is it and what is new
-
Q92 Can CSC sensitive data storage be used as a backup environment for university servers? If that's the case, does CSC itself offer a backup tool to its cloud storage, or can https://restic.net/ that helps to backup to Amazon Cloud, for example, be used alongside SD Desktop? Thank you in advance – Mastaneh
-
Q93 General question: what would be the best way to convert two docker containers (the second inherits from the first) to use within Puhti? I know about Tikky, but I am sure if this is the best way to go about. Since the first container is based on python 3.9.12 and I add a personal package from a repository, I thought maybe I could instead directly build both containers already as Apptainers. Initials: Emiliano Godinez. Will join at 13:30
- You can also convert a Docker container to App
- tainer (with
apptainer buildcommand), or create it as an Apptainer container from scratch: https://docs.csc.fi/computing/containers/creating/ - Also in Apptainer, you can refer to another container (also Docker one) as starting point, with
From:
-
Q94 Restart write performance (~3.3TB, romio hints, striping as before) seems to have gone down
- question for sys admins, contact servicedesk@csc.fi
- I did see few successfully fast writes over the weekend, so may it indeed is "normal fluctuations", but a bit harsh variability
- question for sys admins, contact servicedesk@csc.fi
-
Q95 Mahti BU usage not being applied to project total, only to per-user totals at my.csc.fi?
- no clear answer, possibly a bug?
- Some documentation on BUs on CSC docs: https://docs.csc.fi/accounts/billing/
-
Q96 JCD UH: Is there still a problem with the docker image not working with the external drive?
-
Q97 JCD UH: I really love the new software library tool. Can we ask for new tools to be added after they have been tested locally?
-
Q98 JCD UH: Do you know yet when the move from CentOS7 to Ubuntu might happen?
- no date yet
-
Q99 I have questions on the purging of files planed to March
- Allas or making an archive as a solution for storing intermediate files
- You can move important files to projappl, symbolic links can be used to keep the path same, (e.g. /scratch/project_123/my_file -> /projappl/project_123/my_file)
- Can one use 'rclone' to access Allas? The rclone can convert symlinks to files on-the-fly …
- Yes, this would be one option. https://docs.csc.fi/data/Allas/using_allas/rclone/
- In the long run, one should try to plan the data management of the project keeping in mind that scratch will be regularly cleaned up.
-
Q100 Do we have a list of which breakout rooms cover what topics?
- Room 1: SD Desktop with Kimmo
- Room 2: basic usage
- Room 3: ???
- Room 4: ???
- Room 5: ???
- Yes, the list is just above the questions :-)
-
Q101 Is there going to be another course/seminar on how to create singularity containers? We are starting a new project and have to bring in quite a bit of software.
- Yes, currently planned for early June but no official time yet.
-
Q102 What is the advantage of using same virtual machines within project members?
- saving billing units and resources (?)
- only having to set up additional software once
-
Comment1 Great news, that the external drive can now be used accross serval VM'S!
2023-03-01 session
Short talk: LUMI
- Q103 Differences in using Pouta between Windows and Mac (or a problem with ssh keys?)
- please fill in summary here
- Q104 Is there as web interface like puhti in Lumi?
- Yes, there will be.
2023-03-08 session
Short talk: Copying scratch dir to Allas
Questions
-
Q105 I would like to ask if it is possible to fetch specific files from my bucket in Allas. I recently moved my scrtach directory there and I need to access some files which I don't want to upload the directory again because of limited space for other users.
- A:
-
Q106 Hi, I need to create a PostgreSQL database and populate it with an existing SQL dump. I already created the empty databse (??) on Rahti but I'm clueless about how to load the SQL dump into it.Can I get some help with it please? Also, on Rahti I see a Deployments section with an error under Postgresql but not under websocat and something about failing to allocate a pod.
- A: Today we (cloud solutions) are all busy preparing tomorow's course. We could help the customer next week, if he/she sends us a ticket (service desk). ~~But in essence the dump has to be copied using "oc cp" (https://docs.csc.fi/cloud/rahti/usage/cli/#how-to-install-the-oc-tool)~.
- 'psql' is a PostgreSQL client. It can connect to database and read/write data.
Isn't it already installed on Rahti together with PostgreSQL? Do I hav to install it on Puhti as well? It seems to me it comes toegther with PostgreSQL which I rather not install myslef.
-
I just need to run 'psql pharmacodb < pharmacodb-1.1.1.sql' to load the dump into an empty daabse but I don't know how/where to run this command.
I'm really confused about oc cp! it says "Requires that the 'tar' binary is present in your container
image. If 'tar' is not present, 'oc cp' will fail".
What container and what tar? All the examples given for oc cp are about copying folders to a pod and nothing about populating a database.
Please read: https://docs.csc.fi/support/faq/database/
- The database should be created to cPouta, but it requires some understanding of databases and virtual machines.
- In Puhti is needed the client for contacting the database, from RT ticket is seemed to be psycopg2 Python package. Which Python installation in Puhti are you using? Does it have it?
I have already installed psycopg2 using pip in my tykky container but I need to populate the database first as I mentionned.
- A: Yesterday and today there is an issue in Rahti with storage. It might me source of the problem
- A: Best to send the question still to servicedesk@csc.fi! Rahti people will reply to you as soon as they can, once their course is over. (There was already one ticket sent to research@csc.fi, but that email doesn't reach the Rahti specialists!)
Okay, I just emailed servicedesk with my question!Thank you.
-
Q107 Before backup to Allas, do you archive/tar-files (of smaller size) into a a larger size file - filename.tar (size of 1?2?3? GB)
- A: It is up to the use case, not necessarily. AFAIK the performance is the same.
-
Q108 How do we get to Allas directory from myCSC login?
- A: You can browse/upload/download Allas files from Pouta's web ui (https://pouta.csc.fi/dashboard/project/containers/)
-
Q109 SD Desktop related question
- ?
