Introduction About Bridges-2 Bridges-2, PSC’s flagship supercomputer, began production operations in March 2021. It is funded by a $10-million grant from the National Science Foundation. Bridges-2 provides transformative capability for rapidly evolving computation- and data-intensive research, and creates opportunities for collaboration and convergence research. It supports both traditional and non-traditional research communities and applications. Bridges-2 integrates new technologies for converged, scalable HPC, machine learning and data; prioritizes researcher productivity and ease of use; and provides an extensible architecture for interoperation with complementary data-intensive projects, campus resources, and clouds. Bridges-2 is available at no cost for research and education, and at cost-recovery rates for other purposes. Bridges-2 User Guide

In the era of big data, computational pipelines have become indispensable for efficiently processing and analyzing vast amounts of data. With the advent of high-performance computing systems like Bridges-2, researchers now have access to unprecedented computing power and resources. However, designing and executing data-driven computational pipelines on such systems can be challenging. This presentation aims to explore the advantages and some use cases of three popular workflow management systems: NextFlow, Snakemake, and cwltool, all within the context of Bridges-2. These systems provide a streamlined approach to building scalable and reproducible computational pipelines for processing biological data. Additionally, we will discuss best practices for deploying these systems on Bridges-2, including resource management, job scheduling, and data management strategies. We will also address the challenges and potential solutions encountered when integrating these workflow management systems with Bridges-2’s unique features and constraints. By the end of this presentation, attendees will have a generic understanding of NextFlow, Snakemake, and cwltool, and how these frameworks can empower researchers to build robust and scalable data-driven computational pipelines on Bridges-2. Speaker Biography: Ivan Cao-Berg is a research software specialist in the Biomedical Applications Group tinkering with technology in scientific related projects. At the moment, Ivan is involved in several projects HuBMAP, The Brain Image Library, SenNet and on occasion, with Bridges 2. What are workflows? In computational workflows, individual tasks or steps are organized in a logical order, where the output of one task serves as the input for the subsequent task. This allows for the creation of reproducible process that can be executed reliably and efficiently. Workflows can be designed to handle a wide range of tasks, including data processing, analysis, simulation, modeling, and decision-making. There are different types of computational workflows, including procedural workflows, data-driven workflows, and model-driven workflows Procedural workflows: These workflows follow a predefined sequence of steps or procedures. Each step specifies the input requirements, the processing to be performed, and the output produced. Procedural workflows are often used in scientific simulations or data processing tasks.

This document is meant to be a gentle introduction to virtualization and containerization. The key concepts that will be discussed in the lecture along this document are virtualization containerization virtualization vs containerization orchestration :::info

Docker Singularity mkdir -p $HOME/code/postgres/data mkdir -p $HOME/code/postgres/run cd -p $HOME/code/postgres/ singularity build postgres.sif docker://postgres:14.0 You will need to setup a password before you proceed

Singularity is container software written at Lawrence Berkeley Labs. If you need a specialized computing environment, you can use a Singularity container on Bridges. Your Singularity container will execute on Bridges-2 compute nodes and can use other Bridges-2 resources, including filesystems. Within your container you can use a different Unix operating system and any software you need. You can set up your Singularity container without any intervention from PSC staff. If you have questions about using Singularity containers on Bridges-2 send email to help@psc.edu. Documentation Singularity documentation and examples at Sylabs.io. Usage

:::warning :warning: This document is not finished, I will more than likely finish it before Friday. ::: This document is meant to be a gentle introduction to virtualization and containerization. The key concepts that will be discussed in the lecture along this document are virtualization containerization

This document explains how to run a Jupyter notebook on Bridges-2 using a Jupyter client on your local machine in a web browser. Login to Bridges-2 Connect to Bridges-2 via ssh, gissh, or XSEDE Single Sign-On. I generally use SSH. In the commands described below replace <username> with your XSEDE username. $ ssh <username>@bridges2.psc.edu For example, on MacOSX on Terminal I can type