# SLURM and LMOD ###### tags: `Fall 2021` `Brain Image Library` `workshop` ## Resources As a member of the Brain Image Library project you have access to * The virtual machine `workshop.bil.psc.edu` * 8 large-memory compute nodes set in a SLURM from within the virtual machine ## LMOD ![](https://i.imgur.com/TiNg8y8.png) Lmod is a Lua based module system that easily handles the MODULEPATH Hierarchical problem. Environment Modules provide a convenient way to dynamically change the users’ environment through modulefiles. This document only lists a few commands. For complete documentation click [here](https://lmod.readthedocs.io/en/latest/010_user.html). :::info :bulb: Remember to submit software install requests to `bil-support@psc.edu`. ::: ### Listing available modules To list all available modules use the command ``` module avail ``` For example ``` module avail -------------- /bil/modulefiles --------------- anaconda/3.2019.7 anaconda3/4.10.1 aspera/3.9.6(default) bcftools/1.9(default) bioformats/6.0.1 bioformats/6.1.1 bioformats/6.4.0 bioformats/6.5.1 bioformats/6.6.1(default) bioformats2raw/0.2.4(default) c-blosc/1.19.0(default) dust/0.5.4 ffmpeg/20210611 ``` ### Listing specific modules To list specific modules use the command ``` module avail <package-name> ``` for example ``` module avail matlab -------------- /bil/modulefiles --------------- matlab/2019a matlab/2021a ``` ### Load module To load a module use the command ``` module load <package-name> ``` for example ``` module load matlab/2021a which matlab /bil/packages/matlab/R2021a/bin/matlab → matlab -nodesktop MATLAB is selecting SOFTWARE OPENGL rendering. < M A T L A B (R) > Copyright 1984-2021 The MathWorks, Inc. R2021a Update 5 (9.10.0.1739362) 64-bit (glnxa64) August 9, 2021 To get started, type doc. For product information, visit www.mathworks.com. >> ``` ### Listing loaded modules To list the loaded modules use the command ``` module list ``` for example ``` module list Currently Loaded Modulefiles: 1) matlab/2021a ``` ### Unload module To load a module use the command ``` module unload <package-name> ``` for example ``` module unload matlab/2021a ``` ### Using modules on scripts When building scripts that are using more than one tool available as modules, simply type the module command for each tool ``` #!/bin/bash module load matlab/2021a module load bioformats ``` ## SLURM ![](https://i.imgur.com/3gPmu1r.png) Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. This document only lists a few commands. For complete documentation click [here](https://slurm.schedmd.com/documentation.html). ### sinfo ``` sinfo - View information about Slurm nodes and partitions. SYNOPSIS sinfo [OPTIONS...] ``` For example ``` sinfo -p compute ``` ### squeue ``` squeue - view information about jobs located in the Slurm scheduling queue. SYNOPSIS squeue [OPTIONS...] ``` For example ``` squeue -u icaoberg JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 14243 compute script.s icaoberg R 15:34 1 l001 ``` ### scontrol ``` scontrol - view or modify Slurm configuration and state. SYNOPSIS scontrol [OPTIONS...] [COMMAND...] ``` As a regular user you can view information about the nodes and jobs but won't be able to modify them. The view information about the nodes use the command ``` scontrol show nodes ``` To view information about a specific node, use the node name to print this information. For example ``` scontrol show nodes l002 NodeName=l002 Arch=x86_64 CoresPerSocket=20 CPUAlloc=0 CPUTot=80 CPULoad=0.03 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=l002 NodeHostName=l002 Version=18.08 OS=Linux 4.18.0-305.7.1.el8_4.x86_64 #1 SMP Tue Jun 29 21:55:12 UTC 2021 RealMemory=3000000 AllocMem=0 FreeMem=3090695 Sockets=4 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=compute BootTime=2021-07-16T15:47:48 SlurmdStartTime=2021-08-03T20:58:25 CfgTRES=cpu=80,mem=3000000M,billing=80 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s ``` Because there exists one partition, the you can run `sinfo` or `sinfo -p compute` to gather basic information about this partition. For example ``` → sinfo -p compute PARTITION AVAIL TIMELIMIT NODES STATE NODELIST compute* up infinite 8 idle l[001-008] ``` ### sbatch ``` sbatch - Submit a batch script to Slurm. SYNOPSIS sbatch [OPTIONS(0)...] [ : [OPTIONS(N)...]] script(0) [args(0)...] ``` ### srun ``` srun - Run parallel jobs SYNOPSIS srun [OPTIONS(0)... [executable(0) [args(0)...]]] [ : [OPTIONS(N)...]] executable(N) [args(N)...] ``` For example, to allocate an 8 hour debugging session you can type ``` srun -p compute --mem=16Gb --time=08:00:00 --pty /bin/bash ``` ##### interact The interact command is an in-house script for starting interactive sessions ``` → interact -h Usage: interact [OPTIONS] -d Turn on debugging information --debug --noconfig Do not process config files -gpu Allocate 1 gpu in the GPU-shared partition --gpu --gres=<list> Specifies a comma delimited list of generic consumable resources. e.g.: --gres=gpu:1 --mem=<MB> Real memory required per node in MegaBytes -N Nodes Number of nodes --nodes -n NTasks Number of tasks (spread over all Nodes) --ntasks-per-node=<ntasks> Number of tasks, 1 per core per node. -p Partition Partition/queue you would like to run on --partition -R Reservation Reservation you would like to run on --reservation -t Time Set a limit on the total run time. Format include mins, mins:secs, hours:mins:secs. e.g. 1:30:00 --time -h Print this help message -? ``` * At the moment, there only exists one partition named `compute`, so running ``` interact ``` or ``` interact -p compute ``` * To specify the amount of memory use the option `--mem=<MB>`. For example `interact --mem=1Tb` * This is a shared partition, if you wish to get all the resources in a compute node, use the option `--nodes`. For example `interact -N 1`. Since this is a shared resource, please be considerate using this resource. ### scancel ``` scancel - Used to signal jobs or job steps that are under the control of Slurm. SYNOPSIS scancel [OPTIONS...] [job_id[_array_id][.step_id]] [job_id[_array_id][.step_id]...] ``` The is no need to * To cancel a specific job use the command `scancel <job_id>`. For example `scancel 00001` * To cancel all your running jobs use the command `scancel -u <username>`. For example `scancel -u icaoberg`. ## Exercise 1. Contrast-stretching with ImageMagick This exercise is trying to tie up together all the concepts discussed above. Imagine I am interested in the collection `84c11fe5e4550ca0` that I found in the portal ![](https://i.imgur.com/pTX96xE.jpg) There is no need to download the data locally because the data is available when you our resources. ![](https://i.imgur.com/TMhoZTw.png) *I can navigate to `/bil/data/84/c1/84c11fe5e4550ca0/` to see the contents of the collection.* Unbfortuntaly it is difficult to visually inspect the images because these are not contrast stretched. ![](https://i.imgur.com/WzYR67M.png) *The images are not contrast stretched and cannot be visually inspected.* Fortunately there are tools like Fiji that can contrast stretch the images. However I want to do this in batch mode as a job since this process can be automated. ![](https://i.imgur.com/Z6AcrjZ.jpg) [ImageMagick](https://imagemagick.org/index.php) is a robust library for image manipulation. The `convert` tool in this library has an option to [contrast-stretching](https://imagemagick.org/script/command-line-options.php#contrast-stretch). The format is ``` convert <input-file> -contrast-stretch <output-file> ``` Next I will create a file called `script.sh` and will place it in a folder in my Desktop. ``` #!/bin/bash #this line is needed to be able to use modules on the compute nodes source /etc/profile.d/modules.sh #this command loads the ImageMagick library module load ImageMagick #this for loop finds all the images in the sample folder and contrast-stretch for FILE in /bil/data/84/c1/84c11fe5e4550ca0/SW170711-04A/*tif do convert $FILE -contrast-stretch 15% $(basename $FILE) done ``` :::info :bulb: For simplicity, you can find the script above in ``` /bil/workshops/2021/data_submission2 ``` to copy the script to your Desktop run the command in terminal ``` cp /bil/workshops/2021/data_submission2/script.sh ~/Desktop/ ``` ::: Next I can submit my script using the command ``` sbatch -p compute --mem=64Gb script.sh ``` Since I am doing serially I don't need much memory but if I were to do this in parallel I might. To monitor your job progress use the command `squeue -u <username>`. For example, ``` squeue -u icaoberg JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 14243 compute script.s icaoberg R 15:34 1 l001` ``` This leads to ![](https://i.imgur.com/uHB0Vr6.png)