Using Unity cluster

By Tanya Lama

1. Sign up for a Unity account here. You need a PI to endorse your account.
   https://unity.rc.umass.edu/
2. Linux/Mac Users
   Most distributions of linux come with the openssh client, which you can use to connect to the cluster.

On your local terminal, navigate to /Users/username/.ssh. If this folder does not exist, create it using mkdir /Users/tanyalama/.ssh
If the file ~/.ssh/config doesn't exist, create it using nano config.

Append the following contents:

Host unity
     HostName unity.rc.umass.edu
     User netid_umass_edu
     IdentityFile PATH_TO_PRIVATE_KEY #you don't have this yet, we'll come back to this

3. Create a public/private key
   Open your LOCAL terminal and type
   ssh-keygen -t rsa
   Enter the file in which to save the key pair
   /Users/tanyalama/.ssh/id_rsa

Your private key will be saved as
/Users/tanyalama/.ssh/id_rsa
Your public key will be saved as
/Users/tanyalama/.ssh/id_rsa.pub

Good job. Now that you have a public and private key, you need to go back to your config file and update it.
vi /Users/tanyalama/.ssh/config
`IdentityFile /Users/tanyalama/.ssh/id_rsa #put in the path to your private key (the one that doesn't end in *.pub)

4. Add your public key to Account Settings in Unity
   go to https://unity.rc.umass.edu/ again
   Under Account Settings, click + to Add New Key
   Select Paste Key
   Copy and paste the Public Key that you generated at /Users/tanyalama/.ssh/id_rsa.pub
5. You can now connect to the cluster using the command ssh unity in terminal.
   If you have a PC, you'll have to follow the directions here at https://unity.rc.umass.edu/docs/#connecting/ssh/

Note that this is a SLURM cluster, so the commands you use for things like bqueues will now be sinfo, etc. Annoying! The other important aspect of this cluster is that your /home space now has a capacity of 500Gb and /scratch space is virtually unlimited. You will need to "touch" every file in every folder every 90 days in scratch, or it will be automatically deleted. Some systems will email you before your scratch gets deleted, some will not. Nadia provides this wonderful script that iteratively "touches" every file in every folder in your scratch space. You just need to set a 90-day calendar reminder to go in and execute it. Thanks Nadia!

#!/bin/bash
#PBS -N touchfiles
#PBS -q fnrgenetics
#PBS -l nodes=1:ppn=20
#PBS -l walltime=24:00:00
#PBS -m abe
#PBS -M [e-mail]
​
cd $PBS_O_WORKDIR
​
find * -exec touch -a {} \;

Using scp for file transfer with ssh key authentication

This is also going to require a little set up. Honestly, this took me at least an hour and I'm not even sure how it came about working properly. I'm going to try my best to walk through what I did. Blair shared this very helpful tutorial as well.
https://www.techrepublic.com/article/how-to-use-secure-copy-with-ssh-key-authentication/

1. From the Unity cluster, make a small file called hello with a few words in it
   nano hello
2. Use scp to send that file to your other cluster (like Shared Cluster for example! Or the cluster at SBU)

scp -r ./hello [e-mail]:/gpfs/home/talama

The authenticity of host 'login.seawulf.stonybrook.edu (129.49.83.50)' can't be established.
ECDSA key fingerprint is SHA256:aowOa9cUl6KggKrcZey8o+7fSUNDgDVbx246xoi9zFM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'login.seawulf.stonybrook.edu,129.49.83.50' (ECDSA) to the list of known hosts.
[e-mail]'s password: enter password approve DUO authentication request

Your file should have sent successfully. Login to your other cluster account in a new terminal and check. Then do the reverse (send something from your other cluster account to Unity) to make sure everything is working with the key authentication.

scp -r ./myoNig [e-mail]:/home/tlama_umass_edu/project_bat1k_longevity/data/

This was not a straightforward process for me. I got tons of errors and was able to resolve it through a few different attempts.

tags: tools How To Basic Documentation Genomics Bioinformatics