Tracking Cluster Use & Storage Space

# Tracking Cluster Use & Storage Space By Tanya Lama Project space usage: ``` du -h --max-depth=1 /project/uma_lisa_komoroske/Tanya/ | sort -n -r ``` # Backing up to BOX ``` module load rclone/1.51.0 rclone copy hello.txt "remote_box:box_backups" ``` ##################################################################### # analyses ## step1_fastqc DONE ## step2_FastqToSam DONE ## step3_MarkAdapters DONE ## step4_AlignCleanBam DONE ## step5_MarkDups DONE ## step6_RemoveBadReads DONE Note: backup complete, deleted. This is duplicative of download/bams_mLynCan4_v1.p as well. ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy step6_RemoveBadReads "remote_box:box_backups/project_canada_lynx_WGS/analyses/step6_RemoveBadReads"" bsub -q short -W 2:00 -R rusage[mem=1000] -n 1 "rclone check step6_RemoveBadReads "remote_box:box_backups/project_canada_lynx_WGS/analyses/step6_RemoveBadReads"" ``` ## step7_BQSR FAIL Note: moving everything but the bams, which are too big ``` bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone copy keep "remote_box:box_backups/project_canada_lynx_WGS/analyses/step7_BQSR"" bsub -q short -W 1:00 -R rusage[mem=16000] -n 1 "rclone check keep "remote_box:box_backups/project_canada_lynx_WGS/analyses/step7_BQSR"" ``` ## step8_GATKHG FAIL Note: moved everything but the VCFs, which are too big ``` bsub -q long -W 8:00 -R rusage[mem=16000] -n 1 "rclone copy tbi "remote_box:box_backups/project_canada_lynx_WGS/analyses/step8_GATKHG"" bsub -q long -W 8:00 -R rusage[mem=16000] -n 1 "rclone check tbi "remote_box:box_backups/project_canada_lynx_WGS/analyses/step8_GATKHG"" ``` ## step9_GenotypeGVCFs FAIL ## step10_obtain_high_qual_sites FAIL ## step11_filter_high_qual_variants DONE Note: backup complete, deleted ``` bsub -q long -W 8:00 -R rusage[mem=1000] -n 1 "rclone copy step11_filter_high_qual_variants "remote_box:box_backups/project_canada_lynx_WGS/analyses/step11_filter_high_qual_variants"" ``` ##################################################################### # scripts FAIL Note: backup incomplete, leave in cluster ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy scripts "remote_box:box_backups/project_canada_lynx_WGS/scripts"" bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone check scripts "remote_box:box_backups/project_canada_lynx_WGS/scripts" --one-way" ``` ##################################################################### # download ## bams_mLynCan4_pre-curated DONE Note: backup complete, deleted ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy bams_mLynCan4_pre-curated "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/bams_mLynCan4_pre-curated"" bsub -q long -W 8:00 -R rusage[mem=1000] -n 1 "rclone check bams_mLynCan4_pre-curated "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/bams_mLynCan4_pre-curated"" ``` ## bams_mLynCan4_v1.p_lynx 8995489 Note: backup completed, leave in cluster ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy bams_mLynCan4_v1.p_lynx "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/bams_mLynCan4_v1.p_lynx"" bsub -q long -W 8:00 -R rusage[mem=1000] -n 1 "rclone check bams_mLynCan4_v1.p_lynx "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/bams_mLynCan4_v1.p_lynx"" ``` ## cleancopy_novogene DONE/ backup to hard copy Backup complete. moved to /Morelli_RAD/download/backup_to_hardrive ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy cleancopy_novogene "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/cleancopy_novogene"" bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone check cleancopy_novogene "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/cleancopy_novogene"" ``` ## cleancopy_tcag DONE/ backup to hard copy Note: backup complete, moved to /Morelli_RAD/download/backup_to_hardrive ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy cleancopy_tcag "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/cleancopy_tcag"" bsub -q long -W 8:00 -R rusage[mem=1000] -n 1 "rclone check cleancopy_tcag "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/cleancopy_tcag"" ``` ## fastq DONE Backup complete, deleted ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy fastq "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/fastq"" bsub -q long -W 8:00 -R rusage[mem=1000] -n 1 "rclone check fastq "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/fastq"" ``` ## refs DONE mLynCan4_v1.p stays in cluster ``` bsub -q short -W 4:00 -R rusage[mem=1000] -n 1 "rclone copy refs "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/refs"" bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone check refs "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/refs"" ``` ## VCFs DONE VCFs stay in cluster ``` bsub -q long -W 9:00 -R rusage[mem=1000] -n 1 "rclone copy VCFs "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/VCFs"" bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone check VCFs "remote_box:box_backups/project_canada_lynx_WGS/download_canada_lynx_wgs/VCFs"" ``` ##################################################################### # Morelli_RAD ## analyses DONE ``` bsub -q short -W 4:00 -R rusage[mem=16000] -n 1 "rclone copy analyses "remote_box:box_backups/project_red_squirrel_RAD/analyses"" ``` ## download DONE Note: leaving VCFs because I'm still using them ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy download "remote_box:box_backups/project_red_squirrel_RAD/download"" bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone check download "remote_box:box_backups/project_red_squirrel_RAD/download"" ``` ## ipyrad_squirrel DONE Note: backup complete, deleted ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy ipyrad_squirrel "remote_box:box_backups/project_red_squirrel_RAD/ipyrad_squirrel"" bsub -q short -W 1:00 -R rusage[mem=16000] -n 1 "rclone check ipyrad_squirrel "remote_box:box_backups/project_red_squirrel_RAD/ipyrad_squirrel"" ``` ## STACKS_L7 DONE Note: backup completed, deleted ``` bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone copy STACKS_L7 "remote_box:box_backups/project_red_squirrel_RAD/STACKS_L7"" bsub -q short -W 1:00 -R rusage[mem=16000] -n 1 "rclone check 02_process_out "remote_box:box_backups/project_red_squirrel_RAD/STACKS_L7/02_process_out"" ``` ## STACKS_L8 DONE Note: backup complete, deleted ``` bsub -q long -W 24:00 -R rusage[mem=16000] -n 1 "rclone copy STACKS_L8 "remote_box:box_backups/project_red_squirrel_RAD/STACKS_L8"" bsub -q short -W 1:00 -R rusage[mem=1000] -n 1 "rclone check STACKS_L8 "remote_box:box_backups/project_red_squirrel_RAD/STACKS_L8"" ``` ##################################################################### # TO DO 1. steps 7, 8, 9, 10 aren't able to be backed up because file sizes are too large. Try backing them up using Filezilla on a remote desktop? 2. what products would we definitely need to rerun GATK GenotypePosteriors? 3. cleancopy_tcag and cleancopy_novogene need to be backed up to hard drive using filezilla 4. scripts needs to be backed up using filezilla ###### tags: `tools` `Basic` `Documentation` `Genomics` `Conservation Genomics`