<style> .reveal { font-size: 18px; } .reveal pre { 2 font-size: 20px; } .reveal section p { text-align: left; font-size: 18px; line-height: 1.2em; vertical-align: top; } .reveal section figcaption { text-align: center; font-size: 20px; line-height: 1.2em; vertical-align: top; } .reveal section h1 { font-size: 26pxem; vertical-align: top; } .reveal section h2 { font-size: 24px; line-height: 1.2em; vertical-align: top; } .reveal section h3 { font-size: 22px; line-height: 1.2em; vertical-align: top; } .reveal ul { display: block; } .reveal ol { display: block; } </style> <img align="center" width="25%" src="https://hackmd.io/_uploads/Syhyrl9uT.png" /> # Spatial Transcriptomics Working Group ## Computational Resources Ivan E. Cao-Berg Research Software Specialist Brain Image Library Biomedical Applications Group Jan 9, 2024 1-4 pm ET --- ## Before we begin - :question: Have a question during the presentation? <a href="https://www.lifewire.com/raise-hand-in-zoom-5100882"><img src="https://hackmd.io/_uploads/r1PWX3YOa.png" width="50%" /></a> - :warning: Have an issue or a question after the workshop? - Send an email to the Help Desk `bil-support@psc.edu` --- ## Resources available during this workshop * A SLURM reservation named `workshop` that lasts 24 hours. * Access to the large memory nodes using the `compute` partition that is shared among all users. --- ## GPU-enabled scientific computing * Number of GPUs: 8 * Model: [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) * Memory per GPU: 80GB * SLURM Partition: `gpu` * Max walltime: 4 days --- ## Bridges 2  * For more powerful generic computating. * Mnanaged through [ACCESS CI](https://access-ci.org/). * Submit a proposal for access to Bridges2. --- ## Bridges 2 (cont.)  * The Brain Image Library (BIL) file system is mounted on Bridges 2 so all data is directly accessible on these resources. --- ## Neocortex * Neocortex is a resource that targets AI-powered scientific discovery and provides hardware for the development of efficient algorithms for artificial intelligence and graph analytics. * For more info about the proposals contact the AI and Big Data group [here](https://www.cmu.edu/psc/aibd/neocortex/2023-03-cfp-spring-2023.html). --- <img src="https://lmod.readthedocs.io/en/latest/_static/Lmod-4color@2x.png" width="25%" /> Environment Modules provide a convenient way to dynamically change the users' environment through modulefiles. ```bash= module avail #to list all available modules module load <module-name> #to load module <module-name> module unload <module-name> #to unload module <module-name> ``` These are the most commonly used options you will be more than likely be using. Full documentation on how to use LMOD can be found [here](https://lmod.readthedocs.io/en/latest/index.html). --- ### LMOD - Example 1 - List all available modules ```bash= module avail ------------------------- /bil/modulefiles ------------------------- ANARI-SDK/anari-sdk gnu_parallel/20210522 ITK/5.2.1 gotop/3.3.0 ImageMagick/7.1.0 graphviz/2.44.0 ImageMagick/7.1.0-2 (D) htslib/1.9 R/3.5.1 ilastik/1.3.3 R/3.6.3 (D) imagej-fiji/1.52p Rust/1.58.1 itksnap/3.8.0 Scala/2.13.5 java/jdk8u201 TeraStitcher/1.10.18 java/jdk8u211 VisRTX/0.2.0 java/jdk8u241 (D) anaconda/3.2019.7 julia/1.0.5 anaconda3/4.9.2 knime/4.3.2 anaconda3/4.11.0 (D) lazygit/0.22.9 aspera/3.9.6 matlab/2019a aws-cli/2.4.17 matlab/2021a (D) bcftools/1.9 md5deep/4.4 bioformats/6.0.1 ncdu/1.16 bioformats/6.1.1 nextflow/21.10.6 --More-- ``` --- ### LMOD - Example 2 - Module load `Matlab 2021a` ```bash= module load matlab/2021a matlab -nodesktop -nosplash MATLAB is selecting SOFTWARE OPENGL rendering. < M A T L A B (R) > Copyright 1984-2021 The MathWorks, Inc. R2021a Update 5 (9.10.0.1739362) 64-bit (glnxa64) August 9, 2021 To get started, type doc. For product information, visit www.mathworks.com. >> ``` Every user needs to request access to Matlab. To request access, click [here](https://www.psc.edu/resources/software/matlab/permission-form/). --- ### LMOD - Example 3 - Module load `Anaconda3` ```bash= module load anaconda3 ipython Python 3.9.7 (default, Sep 16 2021, 13:09:58) Type 'copyright', 'credits' or 'license' for more information IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: ``` --- ### LMOD :::info :bulb: Useful Tips and Tricks When building scripts, add as many calls to LMOD as needed. For example, ```bash= #!/bin/bash module load bioformats/6.8.0 module load bioformats2raw/0.3.0 module load raw2ometiff/0.3.0 ``` loads Bio-Formats as well as some other Glencoe tools. ::: --- ## Containers * A **container** is a lightweight, standalone software package that encapsulates everything needed to run an application, including code, runtime, libraries, and settings. * They encapsulate applications and their dependencies, enabling portability and reproducibility across diverse computing environments. * For containers maintained by the Biomed group, click [here](https://github.com/pscedu/singularity). --- ## Why is container technology popular? **1. Isolation** - *Lightweight:* Containers are lighter than virtual machines. - *Isolation:* Each container isolates its application and dependencies. **2. Portability** - *Consistency:* Containers run consistently across environments. - *Platform-agnostic:* Containers run on various platforms. **3. Efficiency** - *Resource Efficiency:* Containers share the host OS kernel. - *Fast Start-up and Scaling:* Containers start quickly and scale easily. **4. Flexibility** - *Polyglot Environments:* Supports multiple programming languages. --- **5. Resource Utilization** - *Optimized Resource Utilization:* Containers efficiently use resources. - *Density:* Many containers can run on a single host. **6. Security** - *Isolation:* Containers limit the impact of security breaches. - *Immutable Infrastructure:* Containers, with immutable infrastructure, enhance security. **7. Community and Ecosystem** - *Open Source Ecosystem:* Strong open-source communities. - *Standardization:* Containers are a standard unit of deployment. --- ## What is Docker? * [Docker](https://www.docker.com/) is a popular **containerization platform** that simplifies the process of creating, deploying, and managing containers. * While Docker is very popular, most HPC clusters do not support Docker out of the box :no_entry:. * [Docker Hub](https://hub.docker.com/) is a **cloud-based registry** provided by Docker that serves as a centralized platform for managing and distributing Docker containers. --- <img src="https://hackmd.io/_uploads/BykH2KcOa.png" width="25%" /> * **uDocker** is a user-level tool designed to enable the execution of Docker containers without requiring escalated privileges. It serves as a user-space replacement for Docker in scenarios where running Docker itself is not possible due to limitations such as the lack of root access (does not work with every container). * Enables download and execution of docker containers by non-privileged users in Linux systems where docker is not available. --- ## What is Singularity? * [Singularity](https://sylabs.io/singularity/) is an open-source container platform designed for high-performance computing (HPC) and scientific workloads. * Singularity is designed for high compatibility with various Linux distributions and HPC environments. * Singularity is relatively easy to use, especially for users familiar with containerization concepts. * Singularity containers generally introduce minimal overhead, making them suitable for high-performance computing tasks. * Singularity facilitates reproducibility by encapsulating the entire software stack and dependencies within containers. * Singularity can convert Docker images, enhancing the usability of existing containerized applications. * Singularity is well-suited for scientific workflows, particularly in research and data analysis. --- ## Docker vs Singularity | Feature | Singularity | Docker | | ------------------------------ | ----------------------------------------------- | ------------------------------------------------ | | **Use Case** | High-performance computing (HPC), Scientific workloads | General-purpose containerization | | **Compatibility** | Optimized for HPC environments | Versatile, used in various environments and platforms | | **User Privileges** | User-friendly, runs with user privileges | Typically requires administrative privileges | | **Container Format** | Single-file format (.sif) | Multi-layer image format | | **Daemon Requirement** | No daemon required | Requires a background daemon for running containers | | **Security** | Emphasizes security, user namespace feature | Strong security features, with namespaces and cgroups | | **Transport and Sharing** | Single-file container, easy to transport and share | Images can be shared via registries like Docker Hub | | **Integration with Docker** | Can run Docker containers | Natively supports Docker container execution | | **Popularity** | Commonly used in HPC and scientific communities | Widely adopted in the software development community | *Note: This table provides a general comparison based on common characteristics, and specific use cases may influence the choice between Singularity and Docker.* --- ## Discussion: Desired Software * Send an email to the Help Desk `bil-support@psc.edu` ---