Workshop: scaling up gene regulatory network simulations
===

:::info
- **Location:** Online
- **Date:** 1-2 December, 2021 | 10am - 4:00pm
- **Contact:**
- **Resources:**
- [Click here for the workshop material](https://genomicsaotearoa.github.io/Gene_Regulatory_Networks_Simulation_Workshop/)
-
-
- **Code of conduct** ([example](https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html)) - If you at any point feel uncomfortable please direct message an instructor or helper
:::
# **Agenda**

| Day | Time | Topic |
|----------|----------------|----------------------------------------------------------------|
| 1st day: | 10am-12:30pm | 1. Introduction |
| | 12:30pm-1:30pm | *Lunch break* |
| | 1:30pm-2:30pm | 2. Getting started with sismonr |
| | 2:30pm-4pm | 3. Scaling up your work<br>4. Working with job scheduler |
| 2nd day: | 10am-12pm | 5. Automating large number of tasks |
| | 12pm-1pm | *Lunch break* |
| | 1pm-3pm | 5. Post-processing |
| | 3pm-4pm | Question time |

## General questions and comments ❓🤔🆘
*If you need help troubleshooting an error please copy/ paste the error message below. All questions and comments welcome*

- paste your question here!
- Question!!!!
-
-
-
-
-
-
-
-

## Notes

Here is an example of what my home directory looks like

# change to home directory
[hradxj@wbn003 ~]$ cd
# Where am I?
[hradxj@wbn003 ~]$ pwd
/home/hradxj
# What is here?
[hradxj@wbn003 ~]$ ls -l
total 1
lrwxrwxrwx 1 hradxj hradxj 58 Nov 30 14:02 sism_2021 -> /nesi/project/nesi02659/sismonr_workshop/workingdir/hradxj

You can see the symbolic link pointing to the correct place

Then click on top left File > New > Notebook, then select the sismonr-4.1.0 kernel from the list.
This should open a new empty notebook.

Try running

library(sismonr)

You should get the result

Checking that the required Julia modules are installed...

Done.

-------

To copy preprepared anthocyanin GRN, open terminal (File > New > Terminal), if you don't have one open already) and

cp /nesi/project/nesi02659/sismonr_workshop/sismonr_anthocyanin_system.RData ~/sism_2021/

-----

Here is what I see when doing **Exercise 4.2**

# change to correct directory
[hradxj@wbn003 sism_2021]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/$USER
# where am I?
[hradxj@wbn003 hradxj]$ pwd
/nesi/project/nesi02659/sismonr_workshop/workingdir/hradxj
# make new directory and change directory to it
[hradxj@wbn003 hradxj]$ mkdir 4_wwscheduler && cd 4_wwscheduler
# where am I?
[hradxj@wbn003 4_wwscheduler]$ pwd
/nesi/project/nesi02659/sismonr_workshop/workingdir/hradxj/4_wwscheduler 

-----
Here is what I see when doing **Exercise 4.3**

[hradxj@wbn005 ~]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/$USER/Exercise_4.3
[hradxj@wbn005 Exercise_4.3]$ ls -F
example1_arraysum.R  example1_arraysum.sl*  slurmout/
[hradxj@wbn005 Exercise_4.3]$ cat example1_arraysum.sl 
#!/bin/bash -e

#SBATCH --account       nesi02659
#SBATCH --job-name      r-arraysum-example
#SBATCH --cpus-per-task 2
#SBATCH --mem           1G
#SBATCH --time          00:03:00
#SBATCH --output        slurmout/r-arraysum.%j.out
#SBATCH --mail-type     END
#SBATCH --mail-user     [e-mail]

module purge
module load sismonr/2.0.0-gimkl-2020a-R-4.1.0

Rscript example1_arraysum.R
echo "Done"

Note that I used `nano example1_arraysum.sl` and changed the notification email address from Dini's address to mine

[hradxj@wbn005 Exercise_4.3]$ sbatch example1_arraysum.sl 
Submitted batch job 23365253
[hradxj@wbn005 Exercise_4.3]$ squeue -u $USER
JOBID         USER     ACCOUNT   NAME        CPUS MIN_MEM PARTITI START_TIME     TIME_LEFT STATE    NODELIST(REASON)    
23365121      hradxj   nesi02659 spawner-jupy   4      8G interac Dec 02 10:02     7:32:57 RUNNING  wbn005              
23365253      hradxj   nesi02659 r-arraysum-e   2      1G large   Dec 02 10:29        2:43 RUNNING  wbn009              

[hradxj@wbn005 Exercise_4.3]$ nn_seff 23365253
Job ID: 23365253
Cluster: mahuika
User/Group: hradxj/hradxj
State: COMPLETED (exit code 0)
Cores: 1
Tasks: 1
Nodes: 1
Job Wall-time:  18.33%  00:00:33 of 00:03:00 time limit
CPU Efficiency: 154.55%  00:00:51 of 00:00:33 core-walltime
Mem Efficiency: 0.00%  0.00 MB of 1.00 GB

-----

Here is what I see when running **Exercise 4.4**.

[hradxj@wbn005 Exercise_4.3]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/$USER/Exercise_4.4
[hradxj@wbn005 Exercise_4.4]$ ls -F
example1_arraysum.R  example1_arraysum.sl*  profile_plot_Jul2020.py  slurmout/
[hradxj@wbn005 Exercise_4.4]$ nano example1_arraysum.sl 
[hradxj@wbn005 Exercise_4.4]$ sbatch example1_arraysum.sl 
Submitted batch job 23365580

Now I am using the job id `23365580`, yours will be different

[hradxj@wbn005 Exercise_4.3]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/$USER/Exercise_4.4
[hradxj@wbn005 Exercise_4.4]$ ls -F
example1_arraysum.R  example1_arraysum.sl*  profile_plot_Jul2020.py  slurmout/
[hradxj@wbn005 Exercise_4.4]$ nano example1_arraysum.sl 
[hradxj@wbn005 Exercise_4.4]$ sbatch example1_arraysum.sl 
Submitted batch job 23365580
[hradxj@wbn005 Exercise_4.4]$ sh5util -j 23365580
sh5util: Merging node-step files into ./job_23365580.h5
[hradxj@wbn005 Exercise_4.4]$ module purge 

The following modules were not unloaded:
   (Use "module --force purge" to unload all):

  1) XALT/minimal   2) slurm   3) NeSI

The above 3 modules are base modules that never get unloaded, which is why you **don't** use `module --force purge` and you'd probably never use it.

[hradxj@wbn005 Exercise_4.4]$ module load Python/3.8.2-gimkl-2020a
[hradxj@wbn005 Exercise_4.4]$ python profile_plot_Jul2020.py job_23365580.h5
profile_plot_Jul2020.py:37: H5pyDeprecationWarning: The default file mode will change to 'r' (read-only) in h5py 3.0. To suppress this warning, pass the mode you need to h5py.File(), or set the global default h5.get_config().default_file_mode, or set the environment variable H5PY_DEFAULT_READONLY=1. Available modes are: 'r', 'r+', 'w', 'w-'/'x', 'a'. See the docs for details.
  f = h5py.File(fn)

Don't worry about the `H5pyDeprecationWarning`!!
You can see that a `.png` file has been created

[hradxj@wbn005 Exercise_4.4]$ ls
example1_arraysum.R   job_23365580.h5           profile_plot_Jul2020.py
example1_arraysum.sl  job_23365580_profile.png  slurmout

You can see the file `job_23365580_profile.png` by going to the correct directory in the file browser (left hand side) and double clicking on it.

---

Here is what I see when running **Exercise 4.5**.

Note that I don't have an "Exercise_4.5" directory so I had to create it.

[hradxj@wbn005 Exercise_4.4]$ mkdir /nesi/project/nesi02659/sismonr_workshop/workingdir/hradxj/Exercise_4.5
[hradxj@wbn005 Exercise_4.4]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/hradxj/Exercise_4.5
[hradxj@wbn005 Exercise_4.5]$ cp /nesi/project/nesi02659/sismonr_workshop/dev/slurm_small_sim/simulate_colsystem_1second.R ./
[hradxj@wbn005 Exercise_4.5]$ ls
simulate_colsystem_1second.R

Now I have the R script I intend to run through the slurm script, but I have not yet created the slurm script. I do this by copying the bits from the previous slurm script

To create the slurm script I use `nano` and cut and paste the text

mkdir -p $TMPDIR

module purge
module load sismonr/2.0.0-gimkl-2020a-R-4.1.0

Rscript --vanilla simulate_colsystem_1second.R

.. into the slurm script. **But I am also putting the correct slurm commands at the start of the script manually.** I do this by borrowing text from a previously supplied slurm script (I use `/home/sism_2021/Exercise_4.4/example1_arraysum.sl`) and modifying to suit.

#!/bin/sh

#SBATCH --account       nesi02659
#SBATCH --job-name      my_random_job
#SBATCH --cpus-per-task 2
#SBATCH --mem           1G
#SBATCH --time 		00:03:00
#SBATCH --output        slurmout/r-arraysum.%j.out

So putting it all together the script looks like this (created with the command `nano firstsim_slurm.sl` then by cutting and pasting and modifying the correct text)

#!/bin/sh

#SBATCH --account       nesi02659
#SBATCH --job-name      my_random_job
#SBATCH --cpus-per-task 1
#SBATCH --mem           1G
#SBATCH --time          00:12:00
#SBATCH --output        slurmout/mylogfile.%j.out

export TMPDIR=/nesi/nobackup/nesi02659/tmp/tmp_$SLURM_JOB_ID
mkdir -p $TMPDIR

module purge
module load sismonr/2.0.0-gimkl-2020a-R-4.1.0

Rscript --vanilla simulate_colsystem_1second.R 

Then I also make sure there is a directory called `slurmout` so that the log file `slurmout/mylogfile.%j.out` is put somewhere that exists.

[hradxj@wbn005 Exercise_4.5]$ pwd
/nesi/project/nesi02659/sismonr_workshop/workingdir/hradxj/Exercise_4.5
[hradxj@wbn005 Exercise_4.5]$ mkdir slurmout

Now I can run the script

[hradxj@wbn005 Exercise_4.5]$ sbatch firstsim_slurm.sl 
Submitted batch job 23366175
[hradxj@wbn005 Exercise_4.5]$ squeue -j 23366175
JOBID         USER     ACCOUNT   NAME        CPUS MIN_MEM PARTITI START_TIME     TIME_LEFT STATE    NODELIST(REASON)    
23366175      hradxj   nesi02659 my_random_jo   2      1G large   Dec 02 11:07       11:47 RUNNING  wbn009   

___

Here is an example of running **Exercise 5.3**

[hradxj@wbn005 Exercise_4.5]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/$USER/Exercise_5.3
[hradxj@wbn005 Exercise_5.3]$ ls -F
firstslurm_array.sl*  slurmout/
[hradxj@wbn005 Exercise_5.3]$ cat firstslurm_array.sl 
#!/bin/bash -e

#SBATCH --account       nesi02659
#SBATCH --job-name      first_slurm_Array
#SBATCH --time          00:02:30
#SBATCH --output        slurmout/sleeparray.%A.%a.out
#SBATCH --cpus-per-task 1
#SBATCH --mem           100
#SBATCH --array         1-5
#SBATCH --export        NONE

###Some Jupyter specific variabes to submit srun commands from Jupyter Terminal
unset SLURM_MEM_PER_NODE
export SLURM_EXPORT_ENV=ALL

##We are calling Slurm srun command to invoke parallel processes 
srun sleep 120

echo "I am a slurm job and I slept for 120 seconds but this time in Parallel"
echo "This is the result for ${SLURM_ARRAY_TASK_ID}"
[hradxj@wbn005 Exercise_5.3]$ sbatch firstslurm_array.sl 
Submitted batch job 23366304

Now I run a `watch` command to watch the queue in real time. You will see not just 1 job but 5, since the `#SBATCH --array` is set to `1-5`. Use `Control-C` to exit. Alternatively just use `squeue -j 23366304` but with your own job ID.

[hradxj@wbn005 Exercise_5.3]$ watch -n 1 -d "squeue -j 23366304"

___

Here is an example of running **Exercise 5.4**

[hradxj@wbn005 Exercise_5.3]$ cd /nesi/project/nesi02659/sismonr_workshop/workingdir/$USER/Exercise_5.4
[hradxj@wbn005 Exercise_5.4]$ ls -F
250sims_2arrayindex.sl*  simulate_colsystem_array_2sim.R  slurmout/
[hradxj@wbn005 Exercise_5.4]$ cat 250sims_2arrayindex.sl
#!/bin/bash -e

#SBATCH --account       nesi02659
#SBATCH --job-name      simulations_250_test             
#SBATCH --time          00:15:00                 
#SBATCH --mem           2GB 
#SBATCH --cpus-per-task 2                    
#SBATCH --array         1-2
#SBATCH --output        slurmout/sims_250_test_%A_%a.out # Include the array ID in the names of
#SBATCH --error         slurmout/sims_250_test_%A_%a.err # the output and error files
#SBATCH --export NONE

###Some Jupyter specific variabes to submit srun commands from Jupyter Terminal
unset SLURM_MEM_PER_NODE
export SLURM_EXPORT_ENV=ALL

###Some processes can generate temporary files which can be redirected from RAM memory to scratch(nobackup) to reduce the Memory footprint
export TMPDIR=/nesi/nobackup/nesi02659/tmp/tmp_${SLURM_JOB_ID}_${SLURM_ARRAY_TASK_ID}
mkdir -p $TMPDIR

#Sismonr specific variable
export GROUP_ID=1.01

module purge
module load sismonr/2.0.0-gimkl-2020a-R-4.1.0

srun Rscript simulate_colsystem_array_2sim.R

[hradxj@wbn005 Exercise_5.4]$ sbatch 250sims_2arrayindex.sl
Submitted batch job 23366394

Now I run a `watch` command to watch the queue in real time. You will see not just 1 job but 2, since the `#SBATCH --array` is set to `1-2`. Use `Control-C` to exit. Alternatively just use `squeue -j 23366394` but with your own job ID.

[hradxj@wbn005 Exercise_5.4]$ watch -n 1 -d "squeue -j 23366394"

---

<!-- Other important details discussed during the meeting can be entered here. -->

Exercise 5.3 - GROUP_ID:

Alexis 1.01
Alyesha 1.02
Ayo 1.03
George 1.04
Jana 1.05
Lamia 1.06
Matthew 1.07
Molly 1.08
Nic 1.09
Poovaiah 1.10
Prudence 1.11
Steffi 1.12
Valter 1.13

Dan 1.20
Dominic 1.21
Matthieu 1.22
Matt 1.23
Dini 1.24
Ngoni 1.25