# TTT4HPC - Singularity episode

###### tags: `Training` `TTT4HPC`

## Notes from 26/4 meeting
- Simo drafted 4 sections https://coderefinery.github.io/ttt4hpc_containers/
- there's many flags and there might be obscure to most. We need to explain the commands inside apptainer
- apptainer syntax (we can also point to apptainer documentation pages rather than making our own list. It can be a "box" which points to the docs https://apptainer.org/docs/user/main/cli.html )
- maybe somebody did an apptainer cheatsheet that we can link. Even collecting all the commands we show might be interesting as a cheatsheet
- we need to make sure to cover the basics and no assumptions (shipping applications -> shipping containers as used by boats (standard boxes) -> lots of shipping metaphors (e.g. dock) -> container and outside world do not interfer)

### todo list:
- making or finding a cheatsheet
- adding a box about apptanier syntax from the apptainer docs (see above)
- a brief intro on what is a "container image" already on page 1. This can be done during the talk "but what is an image simo?"
- Tasks for materials
- ...
- ...
- Other tasks to do:
- try the commands on various clusters
- FAQs/typical-uses: should we add them?
- e.g. root did the installation and you cannot access it when running (do not be root when isntalling)
- existing container with existing code how do I edit (copy out, edit, re-bind)
- do we want an example for binding ports? mysql example? a web app, e.g. tensorboard or rstudio

### timings morning
- general intro of ttt4hpc: 05m
- Enrico and Richard: about the course in general, credits, exercises, etc
- intro page: 15m
- Simo leads and OTHER(MP?EG?) responds to predefined questions and/or asks predefined questions
- basics of running containers 20m
- Simo leads and OTHER(MP?EG?) responds
- container images: 20m
- ???
- break: 10m
- discussion from the notes doc: 10m
- Enrico hosts and filters questions
- maybe we need to continue with images page?: 10m
- xxx leads
- advanced page: 30m
- xxx leads
- more discussions: 10m
- Enrico can host/filter questions

### timings afternoon
- Exercises can be in the second table of content of the page (now called examples)
- test container
- build a simple cowsay with lolcat
- containerised conda? build and run something small
- test a gpu? (pytorch?)

- Other more optional exercises later?
-

## Notes from 18/4 meeting

- Demos and exercises is confusing
- Intro
- Simo drafted the intro https://coderefinery.github.io/ttt4hpc_containers/intro_and_motivation/
- discussion on use cases/approaches
- maybe you just want to be a user (reuse existing containers)
- maybe you need to build
- and here what i need if I want to
- main learning outcome
- you need to install something where you don't have permissions and containers are the only way (e.g. LUMI supercomputer)
- some uusual cases: pytorch container (e.g. pulled from their repo)
- we could ask Radovan and DIana
- the containerised conda (very important for CSC cluster) https://github.com/simo-tuomisto/micromamba-apptainer/
- things to point out
- in the conda example, you can of course install ubuntu, add conda, etc etc... or get an existing container (like the micromamba) that already has tools
- we can show where we are with an icon or textprompt (am I on the host, am I on the container)

### TODOs

- Simo adds placeholders for the 2h morning
- MP and/or EG can help expand on placeholders or iterate/review
- We are all free to add more "demos" stand alone examples following the template "https://raw.githubusercontent.com/coderefinery/ttt4hpc_containers/main/content/verify_installation.rst" (the solutions do not have to be there)

### Touching base:

# timeline
- 13/02 meeting: notes at the end. Simo and Maiken do a first write hre on hackmd. Enrico gives it a second pass next week
- 14/02

---

# What's out there / references to cite
- https://carpentries-incubator.github.io/singularity-introduction/
- ...?

### Use stories / use cases
- think of cases for the people to identify themselves with

### learning goals
- expand on the goals (e.g. based on the use cases above)

---

# Structure for the day
### Stream + Lunch + Zoom-hands-on

### 0) prerequisites
- run some singularity command before? (e.g. get an image)
- check what you have in your cluster (singularity-ce or apptainer?)
- install (but you are on your own)
- basic of linux shell + how to connect to your cluster

### 1) Live stream on TwitchTV 2h
- 00:00 - 00:05 Intro on the day (Enrico + Richard)
- 00:5 - 00:30 First steps + first commands (SIMO?)
- Basic concepts added concept by concept
- Why singularity? When is it useful?
- Mention the use cases we outline
- Some notes at the end of this doc: start simple and add concepts+commands
- first command
- very simple exercise singularity pull e.g.
- Introduce the concept of the container image (why is it named image? because of the layers)
- ssh into the image (singularity shell...) you are now on a different OS...
- not everything that is on the materials has to be on streaming
- image is read only, how do we do stuff with it? mounting / interacting with local filesystem
- definition files
- 01:00 - 01:15 Break
- 01:10 - 01:30 More complex stuff
- env vars
- mounting
- volumes
- converting from docker to singularity
- showing several container services communicating with eachother (?) At least a demo would be good.
- docker compose is not availble on singularity, so communication between containers can happen on the machine (e.g. a singularity container running mysql db and another container running the client) https://github.com/AaltoSciComp/secure-workflows
- when you develop stuff, the read-only version of the singularity image is annoying. But you can keep the dependencies fixed in the image, and mount the "live code". If you need to add packages all the time it is not the best solution.
- sandbox images ... to advanced?
- minimizing image size?
- 01:50 - 02:00 Discussion, future directions, what we will do in the exercises

### 2) Lunch 1h
- 02:00 - 03:00 Go eat

### 3) Zoom 1.5h
TO BE CHANGED (ENRICO)
- 00:00 - 00:10 Intro to Exercise 1
- 00:10 - 00:30 Exercise 1
- 00:30 - 00:45 Exercise 1 solution
- 00:45 - 00:50 Intro to Exercise 2
- 00:50 - 01:10 Break + Exercise 2
- 01:10 - 01:30 Exercise 2 Solution + q&a + where to go from here

# What could we cover?
(brainstorming list)
- singularity vs apptainer
- containerize a conda environment
- build a container from dockerhub
- build a contaner from local docker
- build a container from recipe
- shell in a container
- run app from the container
- graphical app in the container
- (useful but maybe too difficult for this workshop with various setups?)
- singularity on nvidia/AMD gpu (singularity on lumi)
- fakeroot stuff
- container file format
- other useful commands with singularity
- singularity slurm example
- env variables by apptainer / singularity
- cache: how to use it, empty it
- tempdir and cachedir and how to avoid going overquota
- sharing singularity images

## Notes from 13/2 meeting
- getting it to work on various systems might be tricky
- no hope for running it on mac/windows laptops
- preprequisites / preparation for our audience is necessary (on HPC or on a VDI system?)
- installation instructions if you want to run it on your laptop
- make clear that it works on our clusters, others are welcome to watch but things might not work in their cluster
- imporatnt to stress the reason on using singularity: is to move thinngs to other systems rather than developing on own computer
- also showcase features that are beneficial for the admins: lack of many files (e.g. one file for a whole conda environment)
- exercises:
- solutions that singularity offers, e.g. a custom conda setup to put in a container (this is how LUMI uses it for example)
- if something comes via ubuntu (e.g. apt get install something) it might be faster to put it inside of a container rather than trying to port it to the operating system of the cluster (redhat? centos? different ubuntu version?)
- reusing ready made images (from docker also)
- you can share your code with e.g. zenodo
- users are not usuallay familiar to with container terms, clear intro is important independent of the solution e.g. from the CodeRefinery workshop https://coderefinery.github.io/reproducible-research/environments/
- it would be best to go from the simple metaphores to the more complex things (e.g. file system mounting cannot be the first thing). Make it clear "this command is inside the container, this command is outside".
- Metaphor of connecting to a remote computer
- More advanced stuff: containers cannot be changed, modify things outside and mount them inside
- metaphor of graphical tools where you work with layers and then squash them into a single picture
- metaphors are nice but not too many metaphors
- building 10 different images with similar structures is not helpful
- in lumi containers are recommended ways of working for conda env. In practice people need to change things often and rebuild image often. We should stress this.
- we can implicitly say why containers, why I want to wrap evertyhing into a single file and run it somewhere else
- good to mention + outline usecases
- use case for publications: put image on zenodo
- graphical applications are tricky
- the use of sudo in installations scripts of packages, it fails in singularity, so we should be clear that sudo is not needed. Containerize something
- would be good to find an example?
- or at least in the materials we can have a list of these "commonly encoutnered problems"