Collaborative notes and questions for HPC/NIRD course Spring 2021

Please ask questions always at the bottom of this document (edit pen top left of this page to edit).

Links related to the workshop

• Link to this page: https://hackmd.io/@hpc/course-spring-2021
• Course page: https://wiki.uib.no/hpcdoc/index.php/HPC_and_NIRD_toolkit_user_course_March_2021
• Code of conduct: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html (contacts: [e-mail] and [e-mail])
• GitHub repository for exercise https://github.com/trz42/HPC-NTK-user-course-2021.git
• Docker Training GitHub: https://github.com/abdulrahmanazab/docker-training-neic
• User Feedback form https://skjemaker.app.uib.no/view.php?id=10083968

Presentation material

Day 1:

• Folder with presentation material: https://drive.google.com/drive/u/1/folders/10VuU7REr4xlijXzg7R7wxfzTsamXHqJw
• EasyBuild slides: https://cicero.xyz/v2/remark/github/stigrj/easybuild-demo/main/easybuild-demo.mkd
• Pip and conda commands and info can be found in our documentation: https://documentation.sigma2.no/software/userinstallsw/python.html#installing-python-packages

Day 2:

• How to improve job scripts for better resource usage
• NIRD Archive
  • Archive Screencast

Day 3:

• Folder with presentation material: https://drive.google.com/drive/u/1/folders/10VuU7REr4xlijXzg7R7wxfzTsamXHqJw
• NIRD Toolkit : https://apps.sigma2.no
• Asking for help with supercomputers
• Using SSH with public-key authentication

Notes from previous days

Copied to here: https://hackmd.io/0KojPjA1RGCY7WlKwFbZsw

Day 4

Experience with containers

• Jon Lærdahl: Very little, mainly from seminars/brief courses - no practical experience
• Lorenz Michel: No experience with containers (excited to learn more)
• Alex Paul: little experience, had some short introduction to Docker and Singularity
• Sunniva Indrehus: Currently learning and playing with Docker. It is AWSESOME! I do not have experience with Singularity.
• Erik Hjerde: Some experience
• Hermann Luyt: None.
• Emily: My experience with containers is only from individual programs I have installed using Docker.
• Lars Snipen: I have used singularity on our local cluster, never on sigma2
• Emiliano Gelati: I have used Docker (basic user).
• Tomasz Kopec - SysAdmin background but not much experience with containers, worked mainly with virtual machines.
• Raphael Tabase-none
• Zhihong: none

Docker Training GitHub: https://github.com/abdulrahmanazab/docker-training-neic

• Abdulrahman's email:

Dear All,


Welcome to the container tutorial!

Practical info:

1- Here is the github repo: 
https://github.com/abdulrahmanazab/docker-training-neic

2 - The key file for accessing your VM is attached​ (CentOS 7)


3 - If your laptop is Linux/Mac then relax, you are good :) If you have a Windows, then you need to use PUTTY for ssh (https://help.dreamhost.com/hc/en-us/articles/115001764232-Connect-to-your-instance-with-SSH-keys-in-Windows)


See you in at 09:00 ! :)


BR,

Abdulrahman Azab

Questions

• In the Matlab + R example: If there are shared dependencies, can they be shared and thus would you get a smaller size than 300 MiB?

• Could you comment on the difference between an image, a Dockerfile and a layer?

  • source : https://phoenixnap.com/kb/docker-image-vs-container
  • A Docker image is an immutable (unchangeable) file that contains the source code, libraries, dependencies, tools, and other files needed for an application to run. This you should treat read only and not supposed to modify
  • Container is a virtualized run-time environment based on the image
  • The Dockerfile is the recipe for creating the image, basically a text file with commands
  • I have found this useful: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008316

• I have heard someone say "containers" about .yml files with Python instructions/versions. Is this not technically correct?

  • you mean like environment.yml files in Conda environments? or is this something else?
    • Yes!
      • OK I would not call this a container but I would call it a recipe to create an environment. It provides some isolation but not the same isolation as a container. A container is an isolated operating system whereas environment.yml creates an envionment in the same operating system. This can also mean that precisely the same environment.yml can produce slightly different results on different machines if it links to system libraries "outside" of that environment.

• Also the "clean way" looks like an imperative way to build an image. Is it possible to do it declaritively?

• Using a Dockerfile, you'd get the newest software, right? Since you don't specify versions, this approach seems to give unpredictable results, so the exact docker image won't be reproducible. Is that correct?

• On the other hand, the "dirty way" would give something static, and thus reproducible but not really transparent (not known how it was created).

• If possible, could you show a simple implementation how to build or use the Docker to benefit for data analyzing?

• In order to register at training.play-with-docker i need to respond to a verfication email. I get no such email. I have looked into all spam folders. I am stuck here… OK got the email now, but this took a loooong time…

  • I was able to logon without confirming from an e-mail.

• Why does sigma2 use quay.io instead of docker hub as a default for publishing the images?

  • I find quay has faster builds (in automatic builds when coupled with a recipe file stored on GitHub)

• Can we run out of memory on our computer if we run a lot of containers?

  • I think so but I also think that it is not typical to run several containers at the same time but rather to run one at a time but to have lots of containers/images on the disk. So a more typical issue is to fill up the disk but there are cleanup commands to get rid of old/unused layers/images.
    • How do we 'cleanup' properly after a session?
      • For me this worked: docker image rm -f ourfiglet, but maybe there are more subtle ways.

• Can you please show an example on SAGA or FRAM. Do you have singularity on LUMI ?

• Why do we need these virtual machines for the exercices I mean.

  • Mee too, what is the point of this demo ?

• Do we have to install Singularity ourselves on SAGA before we use it, I do not see it as a module ?
  - Info here https://documentation.sigma2.no/software/containers.html?highlight=singularity
  - Thank you. This would have been more usefull, VM is just a waste of time

• Saga it is not working what you are showing, can you please show atleast the pull step on saga

• How do I access project directory from container

• Why would we like to compile something on saga in a contrainer? The point with having all the modules precompiled on Saga is that the developer does not have to compile herself? Is the motivation reprodusability?

---

Please ask questions always at the bottom of this document, right above this line.