# New starter @ TuringPatterns info
commands for logging in to baskerville
- ssh : connect to Baskerville
We copied this to your ~/.bash_profile file the shorthand version is bsk
you can copy your ssh public key to Baskerville so that you dont need to type password but just the OTP

- scp : secure copy files
scp -r [e-mail]:/bask/projects/v/vjgo8416-ms-img-pc/marji/bb .

- rsync : This command syncs too folders on your computer

rsync -av -e ssh --exclude='*.pkl' --exclude='*.out' --exclude='*.stats' --exclude='*.yaml' --exclude='*.pt' --exclude='*.html' alphanumeric/ alphanumeric/

---------
sshfs is a good way of viewing your files remotely on your machine

1. Download OSXFuse (dmg) (pkg) from http://osxfuse.github.io
2. Open OSXFuse dmg , run installer
3. Download SSHFS (pkg) from from http://osxfuse.github.io
4. Run SSHFS installer pkg

This requires a stable connection :

To mount :
aliased for Chris to : fsbsk
sshfs [e-mail]:/bask/projects/v/vjgo8416-ms-img-pc/marji/ /Users/mfamili/Baskerville

To unmount :
aliased for Chris to : ufsbsk

umount -f /Users/mfamili/Baskerville

You can put this in an alias in your ~/.bash_profile.sh :

alias fsbsk='sshfs [e-mail]:/bask/projects/v/vjgo8416-ms-img-pc/marji/ /Users/mfamili/Baskerville'

alias ufsbsk='umount -f /Users/mfamili/Baskerville'

### Generating and copying your SSH key to Baskerville + first time login

Visit the Baskerville first time access page by following the link recieved by email in your initial signup to baskerville or visiting here (https://docs.baskerville.ac.uk/logging-on/#first-time-access). By following the steps within you will have generated a one-time password via an authentica app of your choice. This will allow you to login without password prompt.

The following creates a public ssh key

ssh-keygen -t rsa

When prompted there is no need to enter a username or password, so hit enter three times. Copy your ssh key

pbcopy < ~/.ssh/id_rsa.pub

and paste your ssh key into the authentication portal. Return to terminal and run

ssh [e-mail]

where your username was provided to you in the intial setup email and will be the same username in the first field of the page you just put your ssh key into. Enter your one time password from your authenticator app and when prompted add baskerville to your known hosts list.

### Graphical applications on the terminal of Baskerville

ssh with a flag of `-X` or `-Y`

ssh [e-mail] -X

If you are requesting an `srun` sesssion add the `-X11` to your request. Test that graphical display works by running `xeyes` or `xclock`

### terminal software
vim
:wq save and quit
:q quit
:q! force quit without saving
dd delets line

### Baskerville HPC : slurm algorithm
https://docs.baskerville.ac.uk/baskerville-basics/jobs/#monitoring-jobs

see the queue : squeue

There are few ways of running jobs on Baskerville

#### use job files
This method is used for running programs that you are sure of its reliability and output and can be run unsupervised.

job file example. This is a shell script so should be saved in a file name.sh

- To submit this job sbatch name.sh
- To cancel : scancel jobID

#!/bin/bash
#SBATCH --account=vjgo8416-ms-img-pc
#SBATCH --qos=turing
#SBATCH --gpus=1
#SBATCH --time 12:15:00

#Load required modules

module purge
module load baskerville

module restore system
module load bask-apps/test
module load Miniconda3/4.10.3
eval "$(${EBROOTMINICONDA3}/bin/conda shell.bash hook)"

###activate my conda environment
conda activate /bask/projects/v/vjgo8416-ms-img-pc/marji/envmfvne
python3 /bask/homes/h/hdjd5168/vjgo8416-ms-img-pc/marji/full_tomo/shrec_vae/VAEshrec_subtomo.py

#### To use srun

This is used for debugging and developing , creating conda environments , etc. (tasks that you need to continously interact with)

If you need to use GPUs in your srun session you have to specify this. The order of appearence of flags are important.

srun --account=vjgo8416-ms-img-pc --qos=turing --time 120 --nodes 1 --gpus 1 --cpus-per-gpu 36 --export=USER,HOME,PATH,TERM --pty /bin/bash

Any step after #SBATCH commands need to be done by hand

#### Interactive JupiterLab
Benchmark this link :)

https://portal.baskerville.ac.uk/pun/sys/dashboard

The terminal in the JupyterLab session is equivalant of having run an srun command

Once you ahve created the conda environment, if you start a new jupyter session, the conda environment would appear as a kernel to run notebooks with

## Submit high-throughput calculations

This bash script submits a series of calculations
./nameoffile

you dont require to be in a session:

#!/bin/bash
# Basic range in for loop
for i in 0.01 0.1 1 10 100
do
for j in 3 4 5 6
do
for k in 0 1
do
if [ $k -eq 0 ]; then
p=0
else
p=3
fi
folder_name=g${k}_b${i}_lr_${j}_p${p}
echo $folder_name

# Create directory if it doesn't exist
if [ ! -d "$folder_name" ]; then
mkdir "$folder_name"
fi

cp for_calculation.sh test.sh
printf " python3 /bask/homes/e/eixc0651/camila/affinity-vae/run.py --config_file avae-test-config.yml --pose_dims $p --gamma $k --beta $i -lr 1e-$j --epochs 5000" >> test.sh

cd $folder_name
cp ../avae-test-config.yml .
cp ../test.sh .
sbatch -J g${k}b${i}l${j} test.sh
cd ../
done
done
done
echo All done

### Conda environment

##### 1- request a node using srun
For creating a conda environment start an srun session , request enough time so it doesnt die mid installation

` srun --account=vjgo8416-ms-img-pc --qos=turing --time 120 --export=USER,HOME,PATH,TERM --pty /bin/bash
`

Before installing packages export tmp so that the downloads go in a tmp folder and your home directory wouldnt run out of space :

export CONDA_PKGS_DIRS=/tmp

make sure you run the following regularly to clean the cache: conda clean --all

https://docs.baskerville.ac.uk/portal/jupyter-conda/?h=conda

conda env list

conda list

##### 2- load all the modules that allow you to use conda on baskerville

module purge; module load baskerville
module load bask-apps/live
module load Miniconda3/4.10.3
eval "$(${EBROOTMINICONDA3}/bin/conda shell.bash hook)"

##### 3- install required packages

#### For affinity VAE :

conda install --yes python=3.10
conda install --yes numpy
conda install --yes requests
conda install --yes -c anaconda pandas
conda install --yes -c anaconda scikit-image
conda install --yes -c anaconda scikit-learn
conda install --yes -c anaconda scipy
conda install --yes -c anaconda pillow
conda install --yes -c conda-forge mrcfile
conda install --yes -c conda-forge altair
conda install --yes -c conda-forge umap-learn
conda install --yes -c conda-forge matplotlib
conda install --yes pytorch torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
conda install --yes -c anaconda click

# Grace

### This has worked for some people including myself at somepoint! but it creates an error now whilst installing napari(31/aug/2023)
installing a successful grace environment which contains napari, the order of the installation is important as it avoids conflicts:

Make sure you remove any old environments

`conda remove --name grace-env --all`
`conda clean --all`

`conda env create -f environment.yaml`
`conda activate grace-env`

Then install napari using pip only, using conda results in conflicts:

https://napari.org/stable/tutorials/fundamentals/installation.html

`python -m pip install "napari[all]"`

Do not update!

Install grace:
`cd. grace/`
`pip install -e ".[dev]"`

Use conda to install pytables
`conda install pytables`

### New instruction

conda remove --name grace_env --all
conda clean --all
conda env create -f environment.yaml
conda activate grace_env
conda install -c conda-forge napari pyqt
conda install pytables

It takes quite a bit of time to resolve the conflicts but works ,