# CBS SLURM server guide

This guide introduces how to use the CBS SLURM cluster for your computational research. The cluster uses the Simple Linux Utility for Resource Management (SLURM) that allows multiple users to **share computing resources fairly and efficiently**. Work is submitted to SLURM in the form of **batch jobs** or **interactive sessions**, which are "smartly" scheduled and distributed to the compute nodes without interfering with other users.

The CBS SLURM cluster consists of **7 high-performance compute nodes**, each with **128 CPU cores, 500+ GB RAM, 7TB local SSD**, and **2× NVIDIA L40S GPUs** plus **3 nodes**, each with **64 CPU cores, 300 GB RAM, and 1 TB local storage**, and **2× NVIDIA A100 GPUs** per node, optimized for compute-intensive and data-heavy workloads (See details at General information section).

| Node Type | Nodes | CPU Cores | RAM | Local Storage | GPUs per Node | Total GPUs | Nodes label |
|------------------|---------|-----------|---------|------------|--------------------|------------|--------|
| L40 GPU Nodes | 7 | 128 | 500+ GB | 7 TB | 2 × NVIDIA L40 | 14 | h |
| A100 GPU Nodes | 3 | 64 | 300 GB | 1 TB | 2 × NVIDIA A100 | 6 | v |

This guide explains how to:
- Connect to the cluster and understand SLURM.
- Request CPU, GPU, and memory resources.
- Store and manage data across `/nfs/<pi_last_name>`, `/tmp`, and `/nfs/scratch`.
- Submit batch and interactive jobs.
- Use modules to manage software.
- Use kslurm as a wrapper for SLURM commands.

## 1. What is SLURM?

- **SLURM** (Simple Linux Utility for Resource Management) is a **job scheduler**.
- A job scheduler manages **when, where, and how jobs run** on a shared computing cluster.
- Instead of running programs directly, you **submit jobs to SLURM**, which allocates resources (CPU, memory, GPU, time) and executes your workload on the compute nodes.

This ensures **fairness**, **efficiency**, and **stability** in a multi-user research environment.

## 2. Connecting to the Cluster

To log in to the **master node**:

ssh [e-mail]

__Note__: An automatic setup script runs every hour to configure new users. If this is your first time connecting to the SLURM server, please:
- Connect and disconnect once.
- Wait one hour.
- Then reconnect.

> [!Caution]
> **NEVER** run computations directly on the **master node**.
> The master node is only for **submitting jobs, file management, and light setup**. All computational work should be submitted via SLURM.

> [!Important]
> - Direct ssh access is only possible through the CBS VDI servers or when you are connected to the UWO network.
> - For access off-campus you should concect through the CBS VDI servers.

> [!Tip]
> You can use `-X` to enable **X11 forwarding** if you require GUI applications in interactive sessions (see [section 9](#9.-Interactive-Jobs-(with-X11-Forwarding))).
---

## 3. Submitting a Job

Jobs are submitted using:

sbatch my_job_script.sh

Where `my_job_script.sh` is a shell script that describes:
1. The resources you need (CPUs, memory, GPUs, runtime).
2. The commands to run.

See [section 7](#7.-Job-Script-Template) for an example template of a job script.

> [!TIP] Best practice
> Jobs cannot exceed **48 hours**. If your workflow is longer, design your code with **checkpoints** so you can restart from intermediate results.

## 4. Common SLURM Flags for Resource Requests

Add these to your job script as `#SBATCH` directives:

| Flag | Purpose | Example |
|------|----------|---------|
| `--job-name` | Job name | `#SBATCH --job-name=myjob` |
| `--time` | Runtime limit (HH:MM:SS, max = 48:00:00) | `#SBATCH --time=12:00:00` |
| `--cpus-per-task` | CPUs per task | `#SBATCH --cpus-per-task=2` |
| `--mem` | Memory per node | `#SBATCH --mem=16G` |
| `--gpus-per-node` | Request GPU(s) | `#SBATCH --gpus-per-node=1` |
| `--gpus-per-node=name:number` | Request specific type of GPU(s) | `#SBATCH --gpus-per-node=l40s:2` (Note: l40s and a100 GPUs are available.|
| `--output` | Save output log | `#SBATCH --output=slurm-%j.out` |

## 5. Managing Software with **Modules**

The cluster uses **modules** (Lmod) to manage software environments.

- See available software:

  module avail

- Load a module:

  module load freesurfer/7.4.1

- Check loaded modules:

  module list

- Remove a module:

  module unload freesurfer/7.4.1

Load modules in your script or interactive session to have access to the software (See example in [section 7](#7.-Job-Script-Template)).

## 6. Data Storage: `/nfs/<pi_last_name>`, `/nfs/scratch`, and `/tmp`

> [!Caution] Home folder
> - Do not save data in your home folder.
> - Home folder is meant to be used for scripts or small python enviroments.
> - Data or big python enviroments should be saved in a persistant lab share.

### 🔒 Lab Shares: `/nfs/<pi_last_name>`

- Labs can request a **dedicated fileshare** for storing and processing project data:

  /nfs/<pi_last_name>

- These shares are provisioned on the **2025 OneFS fileserver** and are accessible from all **compute nodes**.
- **Note:** Older lab shares mounted via `/cifs/...` from the legacy **2018 fileserver** are **not accessible from the compute nodes**.
All compute jobs should use the new `/nfs/<pi_last_name>` shares or `/nfs/scratch`.

> 💡 For details on the CBS servers storage, see the [CBS Server Storage Document](https://hackmd.io/@CompCore/cbs_storage).

### 📂 Shared Temporary Space: `/nfs/scratch`

- `/nfs/scratch` is a **shared, high-capacity (25 TB) space** accessible to all users on the compute nodes.
- Use it to **stage input/output data** before and after job runs — especially if your lab doesn’t yet have a `/nfs/<pi_last_name>` share.

> [!Caution] Scratch Policy
> - Files older than **30 days** are **automatically deleted**.
> - This storage is **not backed up** and is intended **only for temporary use**.

### ⚡ Fast Local Storage: `/tmp` on Compute Nodes

- Each compute node has a **7TB fast SSD** local disk.
- When your job starts, a **personal `/tmp` folder** is automatically created **just for that job**.
- This folder is:
- **Private to your job**
- **Faster than network storage** (/nfs/...)
- **Deleted at job completion/termination** (all contents are wiped)

> [!Tip] Best practice
> Copy your data to `/tmp`, run your job there, and move outputs back to `/nfs/<pi_last_name>` or `/nfs/scratch` **before the job finishes**.

### 💬 Requesting Storage

- Contact [[e-mail]](mailto:[e-mail]) to request a **new lab share** on `/nfs/<pi_last_name>`.
- For billing and quota details, refer to the [CBS Servers Document](https://hackmd.io/@CompCore/cbs_servers) or email: [e-mail]

## 7. Job Script Template

📋 **Example Workflow**

A typical script might:
1. **Copy data** from `/nfs/<pi_last_name>` or `/nfs/scratch` to `/tmp`.
2. **Proccess data** using data in `/tmp` and write logs locally.
3. **Checkpoint** outputs periodically (e.g., after each step in a pipeline) to `/nfs/scratch`. (Optional but recomended).
4. **Copy final results** from `/tmp` to `/nfs/<pi_last_name>` or `/nfs/scratch` before the job exits.

Example: `my_job_script.sh`

#!/bin/bash
#SBATCH --job-name=myanalysis
#SBATCH --time=12:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --gpus-per-node=0
#SBATCH --output=slurm-%j.out

# Load software
module load python/3.13.2

# Copy input data to local scratch (/tmp)
cp /nfs/lab/myproject/data/input.csv /tmp

# Run computation
python myscript.py /tmp/input.csv > /tmp/results.txt

# Copy results back to lab share
cp /tmp/results.txt /nfs/lab/myproject

Submit it with:

sbatch my_job_script.sh

## 8. Monitoring and Managing Jobs

- List your jobs:

  squeue -u <your_uwo_username>

- Cancel a job:

  scancel <jobID>

- Show job history:

  sacct -u <your_uwo_username>

- Detailed info on a running or completed job:

  scontrol show job <jobID>

- List nodes on the cluster, what is up and down:

  sinfo

## 9. Interactive Jobs

Sometimes you need an **interactive session** (e.g., for debugging or set-up).

> [!Caution]
> Interactive session are limited to **5 hours**.

- Request an interactive session:

    salloc --time=05:00:00  --cpus-per-task=2 --mem=8G

- With GPUs:

    salloc --time=05:00:00 --cpus-per-task=2 --mem=8G --gpus-per-node=1

### GUI (X fowarding)
In case you need GUI or windows you need to start your interactive session with X fowarding enabled (using --x11 flag). The above example would become:

salloc --time=05:00:00  --cpus-per-task=2 --mem=8G --x11 

Make sure you connect to the SLURM cluster with X forwarding enabled:

ssh -X [e-mail]

# Best Practices / Notes

### **Avoid running heavy jobs on the login node**

The login (master) node is intended for editing files, compiling code, and submitting jobs. Do not run computationally intensive workloads directly on the login node. Always submit jobs through SLURM.

### Use the `/tmp` folder for computation

Reading and writing large amounts of data directly to the NFS filesystem during job execution can generate heavy network traffic and degrade performance for other users. Instead, **perform computations locally on the compute node using the `/tmp` directory**, as described in the *Data Storage* section. Local disk access is typically much faster than NFS.

A common workflow is to copy input data to `/tmp` at the start of the job, run all computations there, and copy the results back to `/nfs/<pi_last_name>` or `/nfs/scratch` before the job finishes. The `/tmp` directory is automatically cleared after the job ends, so ensure all required outputs are copied before termination.

### Request realistic resources
Request only the resources required by your job. Over-requesting CPUs or memory can increase queue time and reduce cluster efficiency.

(ToDo: Give users options/ideas in how to monitor their jobs used resources.)

### Internet access and initial setup

Compute jobs should **not require internet access**. Only the master node and the `v1–v3` nodes have internet connectivity.

It is good practice to prepare your working environment beforehand (e.g., downloading datasets, building containers, or creating Python environments) so that jobs can run without external dependencies.

If a setup step requires significant compute resources and internet access, it may be preferable to run it in an interactive session on one of the **V nodes** instead of the master node, which is a small virtual machine. You can select a specific node using the `--nodelist` option.

Example: start an interactive session on `v1`:

salloc --time=05:00:00 --cpus-per-task=2 --mem=8G --nodelist=rri-cbs-v1.schulich.uwo.ca

Available V nodes:

* `rri-cbs-v1.schulich.uwo.ca`
* `rri-cbs-v2.schulich.uwo.ca`
* `rri-cbs-v3.schulich.uwo.ca`

### GUI and graphical interfaces

Although SLURM supports GUI applications through **X forwarding**, we recommend using the **CBS VDI** for visualization tasks or graphical applications. The VDI environment is generally more stable and provides a better user experience for GUI workflows.

### Containers

The CBS SLURM cluster uses **Apptainer** (formerly **Singularity**) for containerized workflows. Apptainer is available through the module system.

Commonly used containers are stored in:

/srv/containers

You are encouraged to reuse these images when possible to avoid unnecessary downloads. If a container is outdated or a commonly used tool is missing, please contact the administrators so it can be added for the community.

*(TODO: document recommended cache location for container builds.)*

### Python

For recommended Python practices on the cluster, see the **Python guide**:

[https://hackmd.io/@CompCore/python_cbs](https://hackmd.io/@CompCore/python_cbs)
*(Work in progress)*

### kslurm (Simplified SLURM Wrapper)

**kslurm** is a wrapper that simplifies SLURM commands.

It provides two main commands:

* **kbatch** — submit batch jobs (non-interactive)
* **krun** — request interactive sessions

These commands use simplified argument syntax, allowing you to request resources without writing a full SLURM script or specifying multiple `--options`.

Examples:

Schedule a **12-hour job** with **16 cores** and **24 GB of memory**:

kbatch 12:00 16 24G recon-all <recon-all-args>

Request an **interactive session** with **4 cores**, **3 hours**, **15 GB of memory**, and **a GPU**:

krun 4 3:00 15G gpu

`kslurm` will be available as a module *(TODO)*.
See the full documentation for details:

[https://kslurm.readthedocs.io/en/latest/](https://kslurm.readthedocs.io/en/latest/)

# General information

## Billing Rates

CBS SLURM server is only available to Power users. Note that CBS SLURM only support the OneFS Storage datashare (/nfs). For further details and fees please check the [CBS servers](https://hackmd.io/@CompCore/cbs_servers) wiki entry.

## Cluster Architecture

The CBS SLURM cluster is designed to support high-throughput, data-intensive research with a combination of powerful compute nodes and a lightweight master node for job submission.

### 🖥️ Master Node (`rri-cbs-slurm.fmd.uwo.pri`)

Used for **SSH access**, **file management**, and **SLURM job submission**.
It is not intended for any computation or heavy processing.

- **CPU:** 2× Intel® Xeon® Gold 6230R @ 2.10GHz (2 logical CPUs total)
- **Memory:** 8 GB
- **Storage:**
- 30 GB system disk (`/`)
- 100 GB scratch (`/localscratch`)
- **Virtualization:** Hosted on VMware

### 🚀 Compute Nodes
These are the nodes where all batch and interactive compute jobs run. There are two categories of nodes (**h** and **v**). Each node in each category is identical and designed for high-performance computing (HPC) and GPU-accelerated workflows.

#### 🖥 `rri-cbs-h1` → `rri-cbs-h7.schuich.uwo.ca`

- **CPU:** 7× Intel® Xeon® Gold 6448Y
- 64 physical cores × 2 threads = **128 logical CPUs**
- Clock Speed: 800 MHz (min) – 4.1 GHz (max)
- 120 MB L3 cache
- **Memory:** ~504 GB usable RAM
- **GPUs:** 2× NVIDIA L40S
- 46 GB memory each
- CUDA 12.8, Driver version 570.133.20
- **Local Scratch Storage:**
- 7 TB SSD (RAID-0) mounted at `/tmp`, `/localscratch`
- Fast I/O for temporary job data; auto-deleted at job completion

> [!Caution]
> There is no internet access on these nodes. Only the Master node and v1-v3 nodes have internet access.

#### 🖥 `rri-cbs-v1` → `rri-cbs-v3.schuich.uwo.ca`

- **CPU:** Intel® Xeon® Gold 6142
- 32 physical cores × 2 threads = **64 logical CPUs**
- Clock Speed: 3.70 GHz (max)
- 44 MB L3 cache
- **Memory:** ~300 GB usable RAM
- **GPUs:** 2× NVIDIA A100
- 40 GB memory each
- CUDA 12.8, Driver version 570.172.08
- **Local Scratch Storage:**
- 1 TB SSD mounted at `/tmp`, `/localscratch`
- Fast I/O for temporary job data; auto-deleted at job completion

# Additional resources

For detailed explanations and further reading we highly recommend Canada Alliance documentation on SLURM.

https://docs.alliancecan.ca/wiki/What_is_a_scheduler

https://docs.alliancecan.ca/wiki/Running_jobs

https://docs.alliancecan.ca/wiki/Using_GPUs_with_Slurm

https://docs.alliancecan.ca/wiki/MATLAB

# Need help?
📧 Contact: [e-mail]