By Tanya Lama
Configure rclone on Unity for the transfer of files to and from cloud storage.
Note that there may be a file size limit on uploads, depending on what platform you are using for storage (e.g., the Box limit is 15GB). Any file larger will return an error. Google Drive does not appear to have an upper limit.
Need to install rclone on your local machine? Follow these instructions:
Download the latest version of rclone.
This will install rclone into your local /usr/bin/
Tanya has already stepped through the install instructions below to set up remote backup with Google Drive.
At this point rclone should direct you back to your local terminal to enter some code that looks something like this:
rclone authorize "drive" "eyJzY29wZsdfsdfyaXZlIn0"
This command will open a web browser, where you will login to GoogleDrive using your netid and password/DUO authentication and authorize rclone for access.
Select Allow
Copy the code from your local terminal and enter it in the Unity terminal
y/n> n
Yes this is OK
y/e/d> y
Current remotes:
Name Type
==== ====
sbugoogledrive drive****
srun --pty -t 120:00 --mem=8G -p cpu bash
rclone copy filetomove.txt "sbugoogledrive:directorytocopyto/"
rclone copy ./folder "sbugoogledrive:unity_backup/folder" --skip-links
The following command backups files directly from onedrive to google drive without downloading them locally
bsub -n 1 -R rusage[mem=2000] -W 120 -q short "rclone copy onedrive:migrated/10-30-2020-HP-backup googledrive:mghpcc_backup/10-30-2020-hp-backup"
This command is optimized to move big folders quickly (72G in 1hr):
bsub -n 8 -R rusage[mem=2000] -W 72:00 -q long "rclone copy onedrive:migrated/project_canada_lynx_wgs/output_canada_lynx_wgs googledrive:mghpcc_backup/project_canada_lynx_wgs/output_canada_lynx_wgs"
Now that you’ve transfered files, we’ll use rclone check
.
rclone check
confirms that the files in the source and destination match by comparing sizes and hashes (MD5 or SHA1) and provides a report. Ensure that you are in the same directory as the copied files.
Output should read:
NOTICE: sbugoogledrive root ‘sbugoogledrive:unity_backup/project_bat1k_longevity/’: **0 differences found**
NOTICE: sbugoogledrive root ‘sbugoogledrive:unity_backup/project_bat1k_longevity/’: **35 matching files**
You can now delete files from Unity that have been backed up successfully :)
I backup my cluster AND my local machine to googledrive using rclone. I then use the google drive desktop application on my local machine (Mac). This gives me full control over my files. NOTHING is stored locally on my Mac. I use external hard drives as secondary backup for raw reads and other important files.
Execute the following on your local machine:
This opens a browser asking you to authorize box.
Rclone on your local machine terminal will give you an authorization code
You will paste the code into the **cluster terminal **. Note that this is super finnicky. Select EVERYTHING between the –-> and <–- arrows
–->
{"access_token":"XXXXXXXXXXXXXXXXXXXXX","token_type":"bearer","refresh_token":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","expiry":"2020-08-14T10:37:06.2877126+08:00"} <–-
Now it should be configured to upload directly to Box
Check Box to see if the file has been uploaded.
Now that you’ve transfered files, we’ll use rclone check (ensure that you are in the same directory as the copied files).
Check confirms that the files in the source and destination match by comparing sizes and hashes (MD5 or SHA1) and provides a report:
#a single file
#a whole folder
Output should read:
Find this document incomplete? Leave a comment!
tools
`scripts'[in an interactive session]
bsub -n 1 -R rusage[mem=2000] -W 120 -q short "rclone copy onedrive:migrated/pictures ./pictures"
bsub -n 1 -R rusage[mem=2000] -W 120 -q short "rclone copy onedrive:migrated/msc-thesis googledrive:mghpcc_backup/msc_thesis"
bsub -n 1 -R rusage[mem=2000] -W 4:00 -q short "rclone copy ./pictures googledrive:mghpcc_backup/pictures"
bad:
rclone check ./fq.gz googledrive:project_red_squirrel_rad/data_red_squirrel_rad/raw_reads_novogene/*.fq.gz>
bsub -n 8 -R rusage[mem=2000] -W 72:00 -q long "rclone copy onedrive:migrated/project_canada_lynx_wgs/output_canada_lynx_wgs googledrive:mghpcc_backup/project_canada_lynx_wgs/output_canada_lynx_wgs"
bsub -n 1 -R rusage[mem=2000] -W 1:00 -q short "rclone copy onedrive:migrated/project_canada_lynx_wgs/project_furbearer_book googledrive:mghpcc_backup/project_canada_lynx_wgs/project_furbearer_book"
bsub -n 1 -R rusage[mem=2000] -W 1:00 -q short "rclone copy onedrive:migrated/project_canada_lynx_wgs/rsconnect googledrive:mghpcc_backup/project_canada_lynx_wgs/rsconnect"
bsub -n 1 -R rusage[mem=2000] -W 1:00 -q short "rclone copy onedrive:migrated/project_canada_lynx_wgs/Eukaryotes_Genomes_from_NCBI_2019.csv googledrive:mghpcc_backup/project_canada_lynx_wgs/"
bsub -n 1 -R rusage[mem=2000] -W 1:00 -q short "rclone copy onedrive:migrated/project_canada_lynx_wgs/SNPs_per_CHR_pop_assignment.xlsx googledrive:mghpcc_backup/project_canada_lynx_wgs/SNPs_per_CHR_pop_assignment.xlsx"
##super fast
bsub -n 8 -R rusage[mem=2000] -W 72:00 -q long "rclone copy onedrive:migrated/project_canada_lynx_wgs/output_canada_lynx_wgs googledrive:mghpcc_backup/project_canada_lynx_wgs/output_canada_lynx_wgs"
#72g in like an hour
googledrive:mghpcc_backup/project_canada_lynx_wgs/output_canada_lynx_wgs
Total objects: 32
Total size: 8.991 GBytes (9654030492 Bytes)
[tl50a@c40b03 ~]$ rclone size onedrive:migrated/project_canada_lynx_wgs/output_canada_lynx_wgs
[tl50a@c40b03 ~]$ rclone size googledrive:mghpcc_backup/project_canada_lynx_wgs/output_canada_lynx_wgs
rclone size onedrive:migrated/box_backups (3T)
rclone size googledrive:mghpcc_backup/box_backups
onedrive:migrated/box_backups/project_canada_lynx_WGS
bsub -n 8 -R rusage[mem=2000] -W 48:00 -q long "rclone copy onedrive:migrated/box_backups/project_red_squirrel_RAD googledrive:mghpcc_backup/box_backups/project_red_squirrel_RAD"
onedrive:migrated/box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/VCFs/mLynCan4_v1.p_lynx
bsub -n 4 -R rusage[mem=2000] -W 0:45 -q short "rclone copy onedrive:migrated/box_backups/project_canada_lynx_WGS/scripts_canada_lynx_wgs/final-backup-01-06-2021 googledrive:mghpcc_backup/box_backups/project_canada_lynx_WGS/scripts_canada_lynx_wgs/final-backup-01-06-2021"
still running: rclone copy onedrive:migrated/box_backups/project_canada_lynx_WGS googledrive:mghpcc_backup/box_backups/project_canada_lynx_WGS
rclone check onedrive:migrated/box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/cleancopy_novogene googledrive:mghpcc_backup/box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/cleancopy_novogene
onedrive:migrated/box_backups/project_canada_lynx_WGS/R_canada_lynx_wgs/outliers_GEA googledrive:mghpcc_backup/box_backups/project_canada_lynx_WGS/R_canada_lynx_wgs/outliers_GEA