---
tags: Data management, cluster
---
# Configuring Rclone to upload data straight from cluster to OneDrive
### **Note** that there is a file size limit on uploading to OneDrive (15GB), any file larger will return an error (to be confirmed on OneDrive).
### Ensure that you also have Rclone downloaded & installed on your local machine: https://rclone.org/downloads/
### I got this working with a Windows install, assuming it will be easier with Linux & Mac.

# In your home directory on the cluster, run:

cd $HOME

module load rclone/1.51.0
# Doesn't seem to work with version 1.55.0

rclone config

### Rclone will run you through the configuration processline-by-line, it will look like this:

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote_onedrive
Type of storage to configure.
Choose a number from below, or type in your own value # Select onedrive; number 23
[deleted]
Storage> 23
** See help for onedrive backend at: https://rclone.org/onedrive/ **

Microsoft App Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Microsoft App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine (same rclone version recommended) :
        rclone authorize "onedrive"
Then paste the result below:
result>

For this to work, you will need rclone available on a machine that has
a web browser available.

For more help and alternate methods see: https://rclone.org/remote_setup/

Execute the following on the machine with the web browser (same rclone
version recommended):

	rclone authorize "onedrive"

Then paste the result below:
result>

### At this point, go to your local machine and fire up Rclone:

rclone authorize "onedrive"

### This opens a browser asking you to authorize box. Once you have clicked authorize it will tell you to go back to Rclone, where there should be an authorization code, e.g.:

{"access_token":"XXXXXXXXXXXXXXXXXXXXX","token_type":"bearer","refresh_token":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","expiry":"2020-08-14T10:37:06.2877126+08:00"}

### Copy all the text between the "{}" to the cluster:

result> SECRET_TOKEN
--------------------
[acd12]
client_id = 
client_secret = 
token = SECRET_TOKEN
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d>

### From there it should be configured to upload directly to Box

module load rclone/1.51.0

rclone copy filetomove.txt "remote_onedrive:directorytocopyto"

### Check OneDrive to see if the file has been uploaded.
### Now that you've transfered files, we'll use rclone check (ensure that you are in the same directory as the copied files).
### Check confirms that the files in the source and destination match by comparing sizes and hashes (MD5 or SHA1) and provides a report:

#a single file
rclone check file.vcf.gz "remote_onedrive:box_backups/project_canada_lynx_WGS/"

#a whole folder
rclone check tbi "remote_onedrive:box_backups/project_canada_lynx_WGS/" --one-way

### Output should read:
#### NOTICE: box root 'box_backups/project_canada_lynx_WGS': 0 differences found
#### NOTICE: box root 'box_backups/project_canada_lynx_WGS': 35 matching files

### You can now delete files from project/uma_lisa_komoroske that have been backed up successfully :)

## Note:

rclone md5sum ./*

## **Does not work** with OneDrive! Recommend navigating to folder on local terminal and running standard md5sum for checksums.
---
## Copying data from the cluster to the MEC Lab folder:
### In a browser, navigate to your main backup folder, e.g. MEC_LAB_DOCS_OneDrive/MEC_lab_projects/BACKUP_DATA/Blair/
### Once in the folder, select "Add shortcut to My files"
### Shortcut to folder should now be present in your home directory
### To transfer from cluster to file:

rclone copy filetomove.txt "remote_onedrive:/Blair/Data/"

### And vice-versa

rclone copy "remote_onedrive:/Blair/Data/filetomove.txt" ./