<style> .reveal { font-size: 18px; } .reveal pre { 2 font-size: 20px; } .reveal section p { text-align: left; font-size: 18px; line-height: 1.2em; vertical-align: top; } .reveal section figcaption { text-align: center; font-size: 20px; line-height: 1.2em; vertical-align: top; } .reveal section h1 { font-size: 26pxem; vertical-align: top; } .reveal section h2 { font-size: 24px; line-height: 1.2em; vertical-align: top; } .reveal section h3 { font-size: 22px; line-height: 1.2em; vertical-align: top; } .reveal ul { display: block; } .reveal ol { display: block; } </style>  # A Practical Introduction to Apptainer on Bridges-2 Ivan E. Cao-Berg Research Software Specialist [Pittsburgh Supercomputing Center](https://www.psc.edu/) [Carnegie Mellon University](https://www.cmu.edu/) --- ## Before we begin - :warning: Have an issue or question? - Feel free to ask during the presentation, on chat or Slack - Send an email to the Help Desk `help@psc.edu` after the workshop - :computer: What is the project charge ID? - `see240002p` - :computer: What is the reservation name? - `RMapptainer` - :computer: Where can I find the code and data? - The code and data is located in `/ocean/projects/see240002p/shared` - The code that will be used in this workshop will be be found in this [repo](git@github.com:hubmapconsortium/hubmap-data-exploration-workshop.git) --- ## Resources available during this experience * 20 regular-memory compute nodes that can be accessed using SLURM from the partition named `RM-shared` and reservation `RMhubmap`. * Use OpenOnDemand to connect to Bridges 2 using the link `http://ondemand.bridges2.psc.edu` * Bridges 2 official [documentation](https://www.psc.edu/resources/bridges-2/). * --- ## What to expect * A gentle but practical introduction to containers and containerization. * Inspect and run some simple examples to get you started with Apptainer. * Pull containers from public registries and build your own on Sylabs. * This presentation is in the context of a basic power user (take some of my statements with a grain of salt since some things might be doable with the support of PSC engineers). * We will monitor the Slack workspace for a week after the workshop for any questions or concerns. * The presentations, documentation and video recording will be made available online soon after the workshop. --- ## Motivation  --- ## Containerization in Computing A **container** is a lightweight, standalone software package that encapsulates everything needed to run an application, including code, runtime, libraries, and settings. --- ## Why is container technology popular? **1. Isolation** - *Lightweight:* Containers are lighter than virtual machines. - *Isolation:* Each container isolates its application and dependencies. **2. Portability** - *Consistency:* Containers run consistently across environments. - *Platform-agnostic:* Containers run on various platforms. **3. Efficiency** - *Resource Efficiency:* Containers share the host OS kernel. - *Fast Start-up and Scaling:* Containers start quickly and scale easily. **4. Flexibility** - *Polyglot Environments:* Supports multiple programming languages. --- **5. Resource Utilization** - *Optimized Resource Utilization:* Containers efficiently use resources. - *Density:* Many containers can run on a single host. **6. Security** - *Isolation:* Containers limit the impact of security breaches. - *Immutable Infrastructure:* Containers, with immutable infrastructure, enhance security. **7. Community and Ecosystem** - *Open Source Ecosystem:* Strong open-source communities. - *Standardization:* Containers are a standard unit of deployment. --- ## My (biased) opinion about containers * Users do not have to wait for an engineer to install a tools system-wide * Users can install in their users space non-traditional applications, such as editors, utilities and more. * Users can deploy applications that may not be built using the toolkits available on Bridges 2 * Users can easily deploy applications that are no longer supported, outdated or deprecated --- ## Ok so I heard about Docker. What is Docker? * [Docker](https://www.docker.com/) is a popular **containerization platform** that simplifies the process of creating, deploying, and managing containers. * While Docker is very popular, most HPC clusters do not support Docker out of the box :no_entry:. * [Docker Hub](https://hub.docker.com/) is a **cloud-based registry** provided by Docker that serves as a centralized platform for managing and distributing Docker containers. * **uDocker** is a user-level tool designed to enable the execution of Docker containers without requiring escalated privileges. It serves as a user-space replacement for Docker in scenarios where running Docker itself is not possible due to limitations such as the lack of root access (does not work with every container). --- ## What is Singularity? * [Singularity](https://sylabs.io/singularity/) is an open-source container platform designed for high-performance computing (HPC) and scientific workloads. * Singularity is designed for high compatibility with various Linux distributions and HPC environments. * Singularity is relatively easy to use, especially for users familiar with containerization concepts. * Singularity containers generally introduce minimal overhead, making them suitable for high-performance computing tasks. * Singularity facilitates reproducibility by encapsulating the entire software stack and dependencies within containers. * Singularity can convert Docker images, enhancing the usability of existing containerized applications. * Singularity is well-suited for scientific workflows, particularly in research and data analysis. --- ## What is Apptainer? * Sylabs provides licensing, enterprise-level support, professional services, cloud services, and value-added tooling for performance-intensive, mission-critical compute environments and edge deployments. * Apptainer is an open-source project with a friendly community of developers and users. The user base continues to expand, with Apptainer/Singularity now used across industry and academia in many areas of work. --- ## Limitations * Even though most software can be containerized, there are many pieces of software that will not work properly due to their implementation. * For example, this includes software that may require temp files in the container. * Some microservices can be deployed in Singularity, however orchestration using Singularity can be challenging. --- ## Docker vs Singularity | Feature | Singularity | Docker | | ------------------------------ | ----------------------------------------------- | ------------------------------------------------ | | **Use Case** | High-performance computing (HPC), Scientific workloads | General-purpose containerization | | **Compatibility** | Optimized for HPC environments | Versatile, used in various environments and platforms | | **User Privileges** | User-friendly, runs with user privileges | Typically requires administrative privileges | | **Container Format** | Single-file format (.sif) | Multi-layer image format | | **Daemon Requirement** | No daemon required | Requires a background daemon for running containers | | **Security** | Emphasizes security, user namespace feature | Strong security features, with namespaces and cgroups | | **Transport and Sharing** | Single-file container, easy to transport and share | Images can be shared via registries like Docker Hub | | **Integration with Docker** | Can run Docker containers | Natively supports Docker container execution | | **Popularity** | Commonly used in HPC and scientific communities | Widely adopted in the software development community | *Note: This table provides a general comparison based on common characteristics, and specific use cases may influence the choice between Singularity and Docker.* --- ## What do I need to build a container? 1. **Base Operating System Image** 2. **Definition File (a recipe)** 3. **Bootstrap Process** 4. **Environment Setup** Remember that Singularity simplifies many aspects of containerization, making it user-friendly and particularly suitable for high-performance computing environments. --- ## Getting ready! Connect to OnDemand Connect to [Bridges-2](https://ondemand.bridges2.psc.edu) and use your Bridges-2 login to connect to the system (not your ACCESS ID)  --- and it should look like  --- click `JupyterLab`  --- and complete the form  --- click and wait  --- Once you are ready to start the session, click `Connect to Jupyter`  --- Once it starts it should look similar to  --- Click `Terminal`  --- In Terminal type ```bash= change_primary_group see240002p cd /ocean/projects/see240002p/$(whoami) git clone https://github.com/pscedu/practical-intro-to-apptainer.git cd practical-intro-to-apptainer ``` --- ## Enabling the remote builder on Sylabs.io. To enable the remote builder on Sylabs.io we need to follow these steps * Create an account on SyLabs.io. Click login  and `Sign up` --- <img src="https://hackmd.io/_uploads/ry_SFTKgT.png" width="50%"/> * Click `Access Tokens` on the left menu ---  * Click Create a `New Access Token` --- <img src="https://hackmd.io/_uploads/Hk70tpFla.png" width="100%" /> * Add a label and click `Create Access Token` ---  * Click `Copy token to Clipboard` ---  In the Terminal type ```bash= module load SingularityCE/4.1.2 singularity remote login ``` --- * Paste the token and click `Enter`. Now you are ready to build containers remotely. --- ## Let's look at a definition file The definition file includes **similar instructions to installing the software in your local system**. Generally speaking, just follow the developers' instructions. ```bash= Bootstrap: docker From: debian:stretch %environment export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ %post apt update apt install -y libblosc1 wget unzip openjdk-8-jdk cd /opt/ wget -nc https://github.com/glencoesoftware/bioformats2raw/releases/download/v0.3.0/bioformats2raw-0.3.0.zip unzip bioformats2raw-0.3.0.zip && rm -f bioformats2raw-0.3.0.zip ln -s /opt/bioformats2raw-0.3.0/bin/bioformats2raw /usr/local/bin/bioformats2raw apt remove -y wget unzip apt clean %runscript /usr/local/bin/bioformats2raw ``` --- ## And another... ``` Bootstrap: docker From: alpine:edge %labels MAINTAINER icaoberg EMAIL icaoberg@psc.edu SUPPORT help@psc.edu REPOSITORY http://gitub.com/pscedu/singularity-ncdu COPYRIGHT Copyright © 2021 Pittsburgh Supercomputing Center. All Rights Reserved. VERSION 1.16 %post apk update apk add ncdu ``` --- ## Let's look at a definition file A Dockefile that can be used to print `Hello, World!` looks like ``` FROM alpine CMD ["echo", "Hello, World!"] ``` An Apptainer definition file that can be used to print `Hello, World!` looks like ``` Bootstrap: docker From: ubuntu:20.04 %runscript exec echo "Hello, World!" ``` --- ## Wait, what is bootstrap? * The only keyword that is required for every type of build is Bootstrap. * It determines the bootstrap agent that will be used to create the base operating system you want to use. --- ## Preferred bootstrap agents - **library** (images hosted on the Container Library) - **docker** (images hosted on Docker Hub) - **shub** (images hosted on Singularity Hub) - **oras** (images from supporting OCI registries) - **scratch** (a flexible option for building a container from scratch) In this workshop we will only use `docker`. --- ## But what is a base image? | Rank | Default Package Manager | Distribution | Description | |------|-------------------------------------------------------|-------------------|---------------------------------------------------------------------------------------------------------| | 1 | [APT](https://manpages.debian.org/stretch/apt/apt.8.en.html) | Ubuntu | A popular desktop and server distribution known for its ease of use and large community support. | | 2 | [DNF (Dandified YUM)](https://dnf.readthedocs.io/en/latest/command_ref.html) | Fedora | Developed by the Fedora Project, it's known for its focus on innovation and use of cutting-edge software.| | 3 | [APT](https://manpages.debian.org/stretch/apt/apt.8.en.html) | Debian | A stable and versatile distribution that serves as the foundation for many other Linux distributions. | | 4 | [YUM](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum) | CentOS | A free and open-source Linux distribution based on RHEL (Red Hat Enterprise Linux). | | 5 | [Pacman](https://man.archlinux.org/man/pacman.8) | Arch Linux | A lightweight and flexible distribution targeted at experienced users who want a customizable system. | | 6 | [APT](https://manpages.debian.org/stretch/apt/apt.8.en.html) | Linux Mint | Based on Ubuntu, Linux Mint provides an elegant and user-friendly desktop experience out of the box. | | 7 | [Zypper](https://en.opensuse.org/SDB:Zypper_manual) | openSUSE | Known for its stability and flexibility, openSUSE offers both a community-supported and enterprise version.| | 8 | [Pacman](https://man.archlinux.org/man/pacman.8) | Manjaro | Based on Arch Linux, Manjaro aims to provide a user-friendly experience while still offering flexibility. | | 9 | [APT](https://manpages.debian.org/stretch/apt/apt.8.en.html) | Elementary OS | A distribution with a focus on simplicity and aesthetics, resembling macOS in its design philosophy. | | 10 | [YUM](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-yum) | RHEL (Red Hat Enterprise Linux) | A leading enterprise Linux distribution known for its stability and long-term support. | --- ## Tips and Tricks * Select the base image that is supported by the software you want to install. * For example if the software has a `deb` installation package choose Debian or Ubuntu. * For example if the software has instructions for `yum`, then select a base image consistent with the package installer. --- ## In a nutshell * If you know how to install the software of interest locally then you should be able to construct a definition file * Installations work out of the most cases, but there are some edge cases or complexities to consider --- ## In a nutshell (cont.)  --- ## Apptainers build options (for today)  --- ## ...and! [singularity_remote_build](https://hackmd.io/_uploads/HJj-7-PRa.png)