---
# System prepended metadata

title: New starter @ TuringPatterns info

---

# New starter @ TuringPatterns info 
    commands for logging in to baskerville 
    - ssh : connect to Baskerville 
            We copied this to your ~/.bash_profile file the shorthand version is bsk 
        you can  copy your ssh public key to Baskerville so that you dont need to type password but just the OTP 

- scp :  secure copy files
        scp -r hdjd5168@login.baskerville.ac.uk:/bask/projects/v/vjgo8416-ms-img-pc/marji/bb  .
    
- rsync : This command syncs too folders on your computer 
        
    rsync -av -e ssh --exclude='*.pkl' --exclude='*.out' --exclude='*.stats' --exclude='*.yaml' --exclude='*.pt' --exclude='*.html'  alphanumeric/ alphanumeric/

---------
sshfs is a good way of viewing your files remotely on your machine 
        
1. Download OSXFuse (dmg) (pkg) from http://osxfuse.github.io
2. Open OSXFuse dmg , run installer
3. Download SSHFS (pkg) from from http://osxfuse.github.io
4. Run SSHFS installer pkg

This requires a stable connection :
        
    To mount :
    aliased for Chris to : fsbsk 
    sshfs hdjd5168@login.baskerville.ac.uk:/bask/projects/v/vjgo8416-ms-img-pc/marji/ /Users/mfamili/Baskerville
        
    To unmount : 
    aliased for Chris to : ufsbsk 

    umount -f /Users/mfamili/Baskerville

You can put this in an alias in your ~/.bash_profile.sh :

    alias fsbsk='sshfs hdjd5168@login.baskerville.ac.uk:/bask/projects/v/vjgo8416-ms-img-pc/marji/ /Users/mfamili/Baskerville'

    alias ufsbsk='umount -f /Users/mfamili/Baskerville'

### Generating and copying your SSH key to Baskerville + first time login

Visit the Baskerville first time access page by following the link recieved by email in your initial signup to baskerville or visiting here (https://docs.baskerville.ac.uk/logging-on/#first-time-access). By following the steps within you will have generated a one-time password via an authentica app of your choice. This will allow you to login without password prompt.

The following creates a public ssh key

    ssh-keygen -t rsa

When prompted there is no need to enter a username or password, so hit enter three times. Copy your ssh key

    pbcopy < ~/.ssh/id_rsa.pub

and paste your ssh key into the authentication portal. Return to terminal and run

    ssh username@login.baskerville.ac.uk

where your username was provided to you in the intial setup email and will be the same username in the first field of the page you just put your ssh key into. Enter your one time password from your authenticator app and when prompted add baskerville to your known hosts list. 

### Graphical applications on the terminal of Baskerville 
    
ssh with a flag of `-X` or `-Y`

    ssh username@login.baskerville.ac.uk -X 

If you are requesting an `srun` sesssion add the `-X11` to your request. Test that graphical display works by running `xeyes` or `xclock`




### terminal software 
    vim
    :wq save and quit 
    :q quit 
    :q! force quit without saving 
    dd delets line 

    

### Baskerville HPC : slurm algorithm 
https://docs.baskerville.ac.uk/baskerville-basics/jobs/#monitoring-jobs


see the queue : squeue


There are few ways of running jobs on Baskerville 

#### use job files 
    This method is used for running programs that you are sure of its reliability and output and can be run unsupervised. 
    
    job file example. This is a shell script so should be saved in a file name.sh 

    - To submit this job sbatch name.sh 
    - To cancel : scancel jobID


    #!/bin/bash
    #SBATCH --account=vjgo8416-ms-img-pc 
    #SBATCH --qos=turing
    #SBATCH --gpus=1
    #SBATCH --time 12:15:00

    #Load required modules

    module purge
    module load baskerville

    module restore system
    module load bask-apps/test
    module load Miniconda3/4.10.3
    eval "$(${EBROOTMINICONDA3}/bin/conda shell.bash hook)"

    ###activate my conda environment 
    conda activate /bask/projects/v/vjgo8416-ms-img-pc/marji/envmfvne
    python3 /bask/homes/h/hdjd5168/vjgo8416-ms-img-pc/marji/full_tomo/shrec_vae/VAEshrec_subtomo.py 


#### To use srun 

This is used for debugging and developing , creating conda environments , etc. (tasks that you need to continously interact with)
    
If you need to use GPUs in your srun session you have to specify this. The order of appearence of flags are important. 
    


srun  --account=vjgo8416-ms-img-pc  --qos=turing  --time 120 --nodes 1 --gpus 1 --cpus-per-gpu 36 --export=USER,HOME,PATH,TERM --pty /bin/bash

Any step after #SBATCH commands need to be done by hand 

#### Interactive JupiterLab 
Benchmark this link :)

https://portal.baskerville.ac.uk/pun/sys/dashboard

    The terminal in the JupyterLab session is equivalant of having run an srun command 
    
    Once you ahve created the conda environment, if you start a new jupyter session, the conda environment would appear as a kernel to run notebooks with 

## Submit high-throughput calculations 

This bash script submits a series of calculations 
./nameoffile

you dont require to be in a session:

    #!/bin/bash
    # Basic range in for loop
    for i in 0.01 0.1 1 10 100  
    do
        for j in 3 4 5 6 
        do
            for k in 0 1 
            do
                if [ $k -eq 0 ]; then 
                    p=0
                else
                    p=3
                fi
                folder_name=g${k}_b${i}_lr_${j}_p${p}
                echo $folder_name

                # Create directory if it doesn't exist
                if [ ! -d "$folder_name" ]; then
                    mkdir "$folder_name"
                fi

                cp  for_calculation.sh test.sh
                printf " python3 /bask/homes/e/eixc0651/camila/affinity-vae/run.py --config_file avae-test-config.yml --pose_dims $p --gamma $k --beta  $i -lr 1e-$j  --epochs 5000"  >>  test.sh

                cd  $folder_name
                cp ../avae-test-config.yml .
                cp ../test.sh .
                sbatch -J g${k}b${i}l${j} test.sh   
                cd ../
            done
        done
    done
    echo All done




### Conda environment 

##### 1- request a node using srun 
For creating a conda environment start an srun session , request enough time so it doesnt die mid installation 

` srun  --account=vjgo8416-ms-img-pc  --qos=turing  --time 120 --export=USER,HOME,PATH,TERM --pty /bin/bash
`


Before installing packages export tmp so that the downloads go in a tmp folder and your home directory wouldnt run out of space :

export CONDA_PKGS_DIRS=/tmp

make sure you run the following regularly to clean the cache: conda clean --all

https://docs.baskerville.ac.uk/portal/jupyter-conda/?h=conda

conda env list 

conda list 


##### 2- load all the modules that allow you to use conda on baskerville 

module purge; module load baskerville
module load bask-apps/live
module load Miniconda3/4.10.3
eval "$(${EBROOTMINICONDA3}/bin/conda shell.bash hook)"


##### 3- install required packages 

#### For affinity VAE : 

conda install --yes python=3.10
conda install --yes numpy
conda install --yes requests
conda install --yes -c anaconda pandas
conda install --yes -c anaconda scikit-image
conda install --yes -c anaconda scikit-learn
conda install --yes -c anaconda scipy
conda install --yes -c anaconda pillow
conda install --yes -c conda-forge mrcfile
conda install --yes -c conda-forge altair
conda install --yes -c conda-forge umap-learn
conda install --yes -c conda-forge matplotlib
conda install --yes pytorch torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
conda install --yes -c anaconda click


# Grace



### This has worked for some people including myself at somepoint! but it creates an error now whilst installing napari(31/aug/2023)
installing a successful grace environment which contains napari, the order of the installation is important as it avoids conflicts: 

Make sure you remove any old environments 

`conda remove --name grace-env --all`
`conda clean --all`

`conda env create -f environment.yaml`
`conda activate grace-env`

Then install napari using pip only, using conda results in conflicts:

https://napari.org/stable/tutorials/fundamentals/installation.html

`python -m pip install "napari[all]"`

Do not update!

Install grace:
`cd. grace/`
`pip install -e ".[dev]"`

Use conda to install pytables
`conda install pytables`

### New instruction 

conda remove --name grace_env --all
conda clean --all
conda env create -f environment.yaml
conda activate grace_env
conda install -c conda-forge napari pyqt
conda install pytables

It takes quite a bit of time to resolve the conflicts but works  , 
