How to backup data on a remote computer

# How to backup data on a remote computer Materials: https://mad.oxli.org/t/82 --- ## Some backup rules - "One backup? zero backups." - A backup is no good if you can't restore it - Try to have backups in different physical locations - "I have two backups, one in each hard drive. Both are in my office" ❌ --- ## What to backup? - Your personal computer: EVERYTHING, FREQUENTLY - buy an extra drive, set up time capsule on mac - (not sure about windows options...) - I use [restic](https://restic.net/) ---- - In a project: - ✔️ raw data - ❗ intermediate data - how long did it take to generate? - how large it is? - ✔️ results --- ## Backup options - code? git is built for it. - data: - AWS S3 and Glacier - Backblaze - Dropbox - ... ---- - "What does UC Davis offer for cloud storage?" https://ucdavisit.service-now.com/servicehub/?id=ucd_kb_article&sys_id=3f20e2b56f712d40bc4f8a20af3ee4b2 --- ## Today: rclone (https://rclone.org) ---- ## Farm login - [Be a good citizen](https://twitter.com/PicardTips/status/1188562183182962689)! Farm asks to do file transfer in another node ```bash ssh -p 2022 irber@farm.cse.ucdavis.edu ``` - rclone is a module! `module load rclone` ---- - easy to install (single binary): ```bash $ wget https://github.com/rclone/rclone/releases/download/v1.50.1/rclone-v1.50.1-linux-amd64.zip $ unzip rclone-v1.50.1-linux-amd64.zip $ cp rclone-v1.50.1-linux-amd64/rclone bin/ ``` (replace `linux` with `osx` or `windows`) Versions: https://github.com/rclone/rclone/releases ---- - ~~but you can also install with conda: `conda install rclone`~~ :angry: - (will open a pull request later to create a conda package...) --- # Configuring rclone - (many, many options...) - `rclone config` - we will use box: https://rclone.org/box/  ---- ``` Storage> 6 (this number might change!) client_id> (empty) client_secret> (empty) box_config_file> (empty) box_sub_type> 1 (user) Edit advanced config? N Use auto config? N ``` - At this point we need access to a computer with a browser. ---- - On a computer with a browser: ```bash $ rclone authorize box ``` - Copy `result>` ---- ## Using rclone ``` rclone lsd box: rclone copy local_file box: rclone sync -P ncbi/ box:ncbi/ rclone delete box:ncbi/ rclone purge box:ncbi/ ``` --- Thanks!