# Using Unity cluster By Tanya Lama 1. Sign up for a Unity account here. You need a PI to endorse your account. https://unity.rc.umass.edu/ 2. Linux/Mac Users Most distributions of linux come with the openssh client, which you can use to connect to the cluster. On your *local* terminal, navigate to `/Users/username/.ssh`. If this folder does not exist, create it using `mkdir /Users/tanyalama/.ssh` If the file ~/.ssh/config doesn't exist, create it using `nano config`. Append the following contents: ``` Host unity HostName unity.rc.umass.edu User netid_umass_edu IdentityFile PATH_TO_PRIVATE_KEY #you don't have this yet, we'll come back to this ``` 3. Create a public/private key Open your LOCAL terminal and type `ssh-keygen -t rsa` Enter the file in which to save the key pair `/Users/tanyalama/.ssh/id_rsa` > Your private key will be saved as > `/Users/tanyalama/.ssh/id_rsa` > Your public key will be saved as > `/Users/tanyalama/.ssh/id_rsa.pub` Good job. Now that you have a public and private key, you need to go back to your config file and update it. `vi /Users/tanyalama/.ssh/config` `IdentityFile /Users/tanyalama/.ssh/id_rsa #put in the path to your private key (the one that doesn't end in *.pub) 4. Add your public key to Account Settings in Unity go to https://unity.rc.umass.edu/ again Under Account Settings, click `+ to Add New Key` Select `Paste Key` Copy and paste the Public Key that you generated at `/Users/tanyalama/.ssh/id_rsa.pub` 5. You can now connect to the cluster using the command `ssh unity `in terminal. If you have a PC, you'll have to follow the directions here at https://unity.rc.umass.edu/docs/#connecting/ssh/ Note that this is a SLURM cluster, so the commands you use for things like bqueues will now be sinfo, etc. Annoying! The other important aspect of this cluster is that your /home space now has a capacity of 500Gb and /scratch space is virtually unlimited. You will need to "touch" every file in every folder every 90 days in scratch, or it will be automatically deleted. Some systems will email you before your scratch gets deleted, some will not. Nadia provides this wonderful script that iteratively "touches" every file in every folder in your scratch space. You just need to set a 90-day calendar reminder to go in and execute it. Thanks Nadia! ``` #!/bin/bash #PBS -N touchfiles #PBS -q fnrgenetics #PBS -l nodes=1:ppn=20 #PBS -l walltime=24:00:00 #PBS -m abe #PBS -M nbfernan@purdue.edu ​ cd $PBS_O_WORKDIR ​ find * -exec touch -a {} \; ``` ## Using scp for file transfer with ssh key authentication This is also going to require a little set up. Honestly, this took me at least an hour and I'm not even sure how it came about working properly. I'm going to try my best to walk through what I did. Blair shared this very helpful tutorial as well. https://www.techrepublic.com/article/how-to-use-secure-copy-with-ssh-key-authentication/ 1. From the Unity cluster, make a small file called hello with a few words in it `nano hello` 2. Use scp to send that file to your other cluster (like Shared Cluster for example! Or the cluster at SBU) ``` scp -r ./hello talama@login.seawulf.stonybrook.edu:/gpfs/home/talama ``` > The authenticity of host 'login.seawulf.stonybrook.edu (129.49.83.50)' can't be established. > ECDSA key fingerprint is SHA256:aowOa9cUl6KggKrcZey8o+7fSUNDgDVbx246xoi9zFM. > Are you sure you want to continue connecting (yes/no/[fingerprint])? **yes** > Warning: Permanently added 'login.seawulf.stonybrook.edu,129.49.83.50' (ECDSA) to the list of known hosts. > talama@login.seawulf.stonybrook.edu's password: **enter password** **approve DUO authentication request** Your file should have sent successfully. Login to your other cluster account in a new terminal and check. Then do the reverse (send something from your other cluster account to Unity) to make sure everything is working with the key authentication. ``` scp -r ./myoNig tlama_umass_edu@unity.rc.umass.edu:/home/tlama_umass_edu/project_bat1k_longevity/data/ ``` This was not a straightforward process for me. I got tons of errors and was able to resolve it through a few different attempts. ###### tags: `tools` `How To` `Basic` `Documentation` `Genomics` `Bioinformatics`
Last changed by  
 
Tanya Lama
As a molecular ecologist I use population genomic data to explore the ecology of species of conservation concern.
280

Read more

Configuring Rclone & Backing up to Google Drive/BOX/OneDrive

By Tanya Lama

10/28/2023
Pairwise Sequentially Markovian Coalescent (PSMC) model

By Tanya Lama These scripts were initially run for my dissertation along with other pop gen metrics, but the analysis was eventually abandoned because I didn't have time/support in interpreting the results at the time. This analysis has become much more common among popgen studies so there are likely better resources out there, but if you need to get scripts running in a pinch this should do it for you. Essentially, step one is to take the fasta Canada lynx genome file, align it to a reference (in this case we have Canada lynx but previously had also used Felis Catus) and use the resultant bam file for the PSMC workflow. Only have one genome (e.g., just the reference?) No problem! Just align the raw reads from your assembly to the assembly itself, and use that bam file for the PSMC. I beliee Bentley et al did this for this paper in PNAS and I bet the scripts are decent too: here #This may be easier than converting our .maf to a bam?

6/17/2023
Downloading VGP genomes using awscli

Downloading VGP genomes using awscli By Tanya Lama Table of Contents [TOC] Install install or module load anaconda and awscli/1.16.144 I have installed awscli on Unity following the instructions here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

2/24/2022
SNeP: trends in recent effective population size trajectories using genome-wide SNP data

SNeP: trends in recent effective population size trajectories using genome-wide SNP data By Tanya Lama Read more about SNeP from Mario Barbato, here: https://www.frontiersin.org/articles/10.3389/fgene.2015.00109/full Objective We're interested in looking at effective population size trends in "contemporary" demographic history (most recent ~1000 generations) that would be complementary to PSMC and likely more of interest to our state agency folks than PSMC. SNeP and IBDNe are two methods to do this, but SNeP has proven easier to use because IBDNe requires a full pedigree and linkage map which we don't have at this time. Background

2/3/2022
Read more from Tanya Lama
Published on HackMD