NCBI SRA === ###### tags: `基因體/三級分析/資料庫` ###### tags: `生物資訊`, `基因體`, `資料庫`, `NCBI SRA` <br> [TOC] <br> ## 簡介 - SRA: The Sequence Read Archive 序列讀數檔案 - 存放次世代定序序列的資料庫 ### License - ### [ChatGPT] Sequence Read Archive 的版權為何？ Sequence Read Archive（SRA）是由美國國家生物技術資訊中心（National Center for Biotechnology Information，NCBI）管理和維護的一個公共數據庫。根據NCBI的使用條款，SRA的數據可以被廣泛使用和共享，並且**沒有特定的版權限制**。使用者可以自由地訪問、下載和使用SRA中的數據，以支持科學研究和學術目的。然而，使用者應該遵守相關的使用條款和政策，並且必須引用數據的來源和相關出處。這確保了數據的正確性和責任。 ### 相關介紹 - ### [如何將高通量測序原始資料上傳到NCBI的SRA（The Sequence Read Archive）資料庫？](https://iter01.com/571226.html) ![](https://i.imgur.com/hYtLUrb.png) - ### [如何使用SRA toolkits下載 NIH 的 NGS data](https://weitinglin.com/2016/01/23/%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8sra-toolkits%E4%B8%8B%E8%BC%89nih%E7%9A%84ngs-data/) > by 我們的基因體時代 OUR "GENE"RATION - ### [[Nvidia] 下載 SRA 範例 (非 NCBI 來源)](https://docs.nvidia.com/clara/parabricks/3.7.0/How-Tos/SomaticCalling.html#download-example-fastq-files) > Download Example FASTQ Files > ```bash ## Download publically available SRA files using wget # Normal sample wget https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR7890827/SRR7890827 # should be 69.84G and will take 30 min. to download # Tumor sample wget https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR7890824/SRR7890824 # should be 64.83G and takes around 30 min. to download ## Convert SRA to FASTQ files fastq-dump -I --split-files SRR7890827 --gzip fastq-dump -I --split-files SRR7890824 --gzip ``` - [NCBI 來源 / SRA / SRR7890824](https://www.ncbi.nlm.nih.gov/sra/?term=SRR7890824) - [NCBI 來源 / SRA / SRR7890827](https://www.ncbi.nlm.nih.gov/sra/?term=SRR7890827) - ### [Fastq-dump 使用](https://www.plob.org/article/14557.html) <br> <hr> <br> ## 工具下載 ### Tool / NCBI SRA Toolkit (v2.11.2) > - SRA: Sequence Read Archive > - 資料來源：[Whole-Genome Somatic Small Variant Calling](https://docs.nvidia.com/clara/parabricks/3.7.0/How-Tos/SomaticCalling.html#) > - ### [2.11 版的下載點](https://github.com/ncbi/sra-tools/wiki/02.-Installing-SRA-Toolkit) - http://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-ubuntu64.tar.gz <br> ### Tool / NCBI SRA Toolkit (v3.0.0) > - SRA: Sequence Read Archive > - ### [下載](https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=software) - [Ubuntu Linux 64 bit architecture](https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/3.0.0/sratoolkit.3.0.0-ubuntu64.tar.gz) - 解壓縮後即可執行 - 執行工具的時候，就會要你做初始化 ``` vdb-config --interactive ``` - ### [文件說明](https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=toolkit_doc) - [Tool: fastq-dump](https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=toolkit_doc&f=fastq-dump) <br> <hr> <br> ## fastq 格式 [![](https://i.imgur.com/1FS1WaX.png)](https://www.drive5.com/usearch/manual/fastq_files.html) [![](https://i.imgur.com/hx7SQXS.png)](https://compgenomr.github.io/book/fasta-and-fastq-formats.html) - 每一段 fastq data，有 4 列資料 - 專業術語用 spot 為單位表示 (每一個 spot 有 4 列資料) :::warning :bulb: **Why spot?** 每一條序列，就是次世代定序儀上的一個奈米井(奈米孔洞)所綁定的一條序列，所以稱為 spot 也是蠻合理的 見 [[HackMD] flowcell](https://hackmd.io/9fsiG_EUQDyIN7dq6mboGg#%E6%99%B6%E7%89%87%E6%B5%81%E9%80%9A%E6%B1%A0-flowcell) ![](https://i.imgur.com/AZd0sXW.png) ::: - 第三列的 id 可有可無 <br> <hr> <br> ## fastq-dump > 部分由測試得知 ### 參數說明 - **輸出檔案：在當前所在目錄下輸出** `/root/dataset$ /root/fast-dump /root/db/SRR7890824` - 輸出的檔案，位於當前 `/root/dataset` 目錄下 - 不在「資料庫」所在 `/root/db/` 目錄下 - 不在「執行檔」所在 `/root/` 目錄下 - **參數`-X | --maxSpotId`** > Maximum spot id (最大點 ID) > 表示要抓幾段 fastq 資料 - `$ fast-dump SRR7890824 -X 3` 表示輸出 3 段 fastq - `-X 1` 或 `-X1` 表示抓取 1 段 ``` $ fast-dump SRR7890824 -X 1 $ cat SRR7890824.fastq @db24.1 1 length=300 CCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCACCCCCCACGCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCACCGCCCCCCACCCCCCACCCCCAACCCCCCCCCATCCCGCACCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCACCCCCAC +db24.1 1 length=300 <AAFFFJJFJJJF-JJJJJ<F-F-AA<--77-A<AAAAF<-<<--F7-7A<F---7<--7AF<--------7-----<<--7A--77-77AF-A<JF-A--7-7-7---7-----7-<-7A-----7A--7---------7---7-F-AF<AAAFFFJJFJJAJJJJJJJJJJJJJJJJJJJJAFFJFJJFJFFJJFFJJJJJJJJJJJFJJJJJFJJ<JAJJJJAJJJJJFAJFJ<J<JJJJAAJFJFAJJJFFJJFJJJJF-7FF<<AFAJ<FJ-A-AF)A-)7-7-)-))7<<)))7 ``` - `-X 2` 或 `-X2` 表示抓取 2 段 ``` $ fast-dump SRR7890824 -X2 $ cat SRR7890824.fastq @db24.1 1 length=300 CCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCACCCCCCACGCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCACCGCCCCCCACCCCCCACCCCCAACCCCCCCCCATCCCGCACCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCACCCCCAC +db24.1 1 length=300 <AAFFFJJFJJJF-JJJJJ<F-F-AA<--77-A<AAAAF<-<<--F7-7A<F---7<--7AF<--------7-----<<--7A--77-77AF-A<JF-A--7-7-7---7-----7-<-7A-----7A--7---------7---7-F-AF<AAAFFFJJFJJAJJJJJJJJJJJJJJJJJJJJAFFJFJJFJFFJJFFJJJJJJJJJJJFJJJJJFJJ<JAJJJJAJJJJJFAJFJ<J<JJJJAAJFJFAJJJFFJJFJJJJF-7FF<<AFAJ<FJ-A-AF)A-)7-7-)-))7<<)))7 @db24.2 2 length=300 CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCACACCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACAC +db24.2 2 length=300 AAAAF<FAJJFFAF7JJFF<JF<AAJJJJJJJJJFJJJJJJFFJJJAJFFFAJJJJJJJJJAJJ<FJJAJ-AAFJA<JFAJJFFFJJFJJJJJ<JFJFJJJJFJAJFJJJFJJJ<AJJJJJJJJJJJJF77A7F<F-AAJF--777A77-AAAFFJJJJJJJJJJFJFJJJJJJJJJJJJJFJJJJJJAJJJJJJJAFJFJFJJJJFJ7AJJAJJJFJFJJFJJJJJJJJJJJJJFJJJJJJJJJJJJJJJJJJJJJA7AJFFJFFFFFFFFJJJJ-7AJ--AA-7<F--)-<7AJ)7)7 ``` - **:warning: fastq 資料，是從「本地端」抓？還是「線上」抓？** ```$ fastq-dump SRR7890824 -X 1``` (來源：線上 NCBI > 本地端) ```$ fastq-dump ./SRR7890824 -X 1``` (來源：本地端) :::warning :bulb: **快速測試** - 如果來源是**本地端**，1.5 秒內就會立即回應 - fastq: ++**1.5**++ 秒 - fastq.gz: ++**1.5~2**++ 秒 - 如果來源是**線上 NCBI** - 但線上無此檔案，只需 2.5 秒就會有錯誤訊息 - 且線上有此檔案，需要等待 ++**21~30**++ 秒才會有回應 - fastq: ++**21~30**++ 秒 - fastq.gz: ++**23~36**++ 秒 ::: - 如果當前目錄下**有**該檔案 - 且 NCBI 上也**有**該檔案，則「**優先**」從 NCBI 上抓取 - 但 NCBI **沒有**該檔案，則直接從本地端抓取 - 如果當前目錄下**沒有**該檔案 - 則從 NCBI 上抓取 - 如果 NCBI 上**沒有**該檔案，則會顯示錯誤： > **檔案名稱**即為**登陸號(accession)** ``` $ ../fastq-dump X0123456789 -X2 2022-04-14T09:42:01 fastq-dump.3.0.0 err: name not found while resolving query within virtual file system module - failed to resolve accession 'X0123456789' - Cannot resolve accession ( 404 ) 2022-04-14T09:42:03 fastq-dump.3.0.0 err: name not found while resolving query within virtual file system module - failed to resolve accession 'X0123456789' - Cannot resolve accession ( 404 ) 2022-04-14T09:42:03 fastq-dump.3.0.0 err: item not found while constructing within virtual database module - the path 'X0123456789' cannot be opened as database or table fastq-dump quit with error code 3 ``` - **參數`--split-files`** > Write reads into separate files. Read number will be suffixed to the file name. NOTE! The `--split-3` option is recommended. In cases where not all spots have the same number of reads, this option will produce files that WILL CAUSE ERRORS in most programs which process split pair fastq files. > > - 將讀數寫入到個別檔案。 > - 讀數編號將作為檔名的後綴。 > - 注意： > - 建議使用 `--split-3` 選項。 > - 在並非所有點都有相同讀數數量的情況下，該選項(`--split-files`? `--split-3`? )所產生的檔案將使大多數的程式在處理分割後的成對 fastq 檔案時導致錯誤。 - 無添加 `--split-files` 參數 ``` $ fastq-dump ./SRR7890824 -X1 $ cat SRR7890824.fastq @SRR7890824.1 1 length=300 CCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCACCCCCCACGCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCACCGCCCCCCACCCCCCACCCCCAACCCCCCCCCATCCCGCACCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCACCCCCAC +SRR7890824.1 1 length=300 <AAFFFJJFJJJF-JJJJJ<F-F-AA<--77-A<AAAAF<-<<--F7-7A<F---7<--7AF<--------7-----<<--7A--77-77AF-A<JF-A--7-7-7---7-----7-<-7A-----7A--7---------7---7-F-AF<AAAFFFJJFJJAJJJJJJJJJJJJJJJJJJJJAFFJFJJFJFFJJFFJJJJJJJJJJJFJJJJJFJJ<JAJJJJAJJJJJFAJFJ<J<JJJJAAJFJFAJJJFFJJFJJJJF-7FF<<AFAJ<FJ-A-AF)A-)7-7-)-))7<<)))7 ``` - 添加 `--split-files` 參數 讀數被拆成兩段 ``` $ fastq-dump ./SRR7890824 -X1 --split-files $ cat SRR7890824_1.fastq @SRR7890824.1 1 length=150 CCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCACCCCCCACGCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCACCGCCCCCCACCCCCCACCCCCAACCCCCCCCCATCCCGCA +SRR7890824.1 1 length=150 <AAFFFJJFJJJF-JJJJJ<F-F-AA<--77-A<AAAAF<-<<--F7-7A<F---7<--7AF<--------7-----<<--7A--77-77AF-A<JF-A--7-7-7---7-----7-<-7A-----7A--7---------7---7-F-AF $ cat SRR7890824_2.fastq @SRR7890824.1 1 length=150 CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCACCCCCAC +SRR7890824.1 1 length=150 <AAAFFFJJFJJAJJJJJJJJJJJJJJJJJJJJAFFJFJJFJFFJJFFJJJJJJJJJJJFJJJJJFJJ<JAJJJJAJJJJJFAJFJ<J<JJJJAAJFJFAJJJFFJJFJJJJF-7FF<<AFAJ<FJ-A-AF)A-)7-7-)-))7<<)))7 ``` - 添加 `--split-3` 參數 ``` $ fastq-dump ./SRR7890824 -X1 --split-3 $ cat SRR7890824_1.fastq @SRR7890824.1 1 length=150 CCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCACCCCCCACGCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCACCGCCCCCCACCCCCCACCCCCAACCCCCCCCCATCCCGCA +SRR7890824.1 1 length=150 <AAFFFJJFJJJF-JJJJJ<F-F-AA<--77-A<AAAAF<-<<--F7-7A<F---7<--7AF<--------7-----<<--7A--77-77AF-A<JF-A--7-7-7---7-----7-<-7A-----7A--7---------7---7-F-AF $ cat SRR7890824_2.fastq @SRR7890824.1 1 length=150 CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCACCCCCAC +SRR7890824.1 1 length=150 <AAAFFFJJFJJAJJJJJJJJJJJJJJJJJJJJAFFJFJJFJFFJJFFJJJJJJJJJJJFJJJJJFJJ<JAJJJJAJJJJJFAJFJ<J<JJJJAAJFJFAJJJFFJJFJJJJF-7FF<<AFAJ<FJ-A-AF)A-)7-7-)-))7<<)))7 ``` - **參數`-I | --readids`** > Append read id after spot id as 'accession.spot.**readid**' on defline > > 在 spot ID (斑點 ID) 後附加 read ID (讀數 ID)， > 作為定義行上的 'accession.spot.**readid**' - 無添加 & 有添加`-I`的差異 [![](https://i.imgur.com/uZqiFI2.png)](https://i.imgur.com/uZqiFI2.png) - 有 `-I`參數 - `SRR7890827_1.fastq` -> `SRR7890827.${spot}.1` - `SRR7890827_2.fastq` -> `SRR7890827.${spot}.2` - 無 `-I`參數 - `SRR7890827_1.fastq` -> `SRR7890827.${spot}` - `SRR7890827_2.fastq` -> `SRR7890827.${spot}` <br> :::warning :warning: 經測試，只有在指定 `--split-files` 下，`-I` 才會有效果 ::: - **參數`--gzip`** > Compress output using gzip: deprecated, not recommended > 使用 gzip 壓縮輸出；**已被捨棄，不建議使用** - **參數與對應輸出** | `--split-files` | `--gzip` | Output | | --------------- | -------- | ------ | | - | - | `SRR7890824.fastq` | | - | v | `SRR7890824.fastq.gz` | | v | - | `SRR7890824_1.fastq`<br>`SRR7890824_2.fastq` | | v | v | `SRR7890824_1.fastq.gz`<br>`SRR7890824_2.fastq.gz` | - 參考資料 - [fastq-dump options](https://edwards.flinders.edu.au/fastq-dump-options/) <br> <hr> <br> ## 操作範例 ### 操作範例1 > [RNA-Sick@Day7 > 我相信自由自在，我詳細希望｜下載 NCBI 上的原始序列 feat. SRA Toolkit (下)](https://ithelp.ithome.com.tw/articles/10218998) > - 用 SRA Toolkit 的神秘指令下載該筆資料的 fastq 檔案 > - [SRA Run Selector / PRJNA319058 / SRR3406492](https://www.ncbi.nlm.nih.gov/Traces/study/?acc=PRJNA319058) > ![](https://i.imgur.com/3IGD6s0.png) <br> ### 操作範例2 (from Nvidia Parabricks) > 資料來源：[Download Example FASTQ Files](https://docs.nvidia.com/clara/parabricks/3.7.0/How-Tos/SomaticCalling.html#download-example-fastq-files) ```bash ## Download publically available SRA files using wget # Normal sample wget https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR7890827/SRR7890827 # should be 69.84G and will take 30 min. to download # Tumor sample wget https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR7890824/SRR7890824 # should be 64.83G and takes around 30 min. to download ## Convert SRA to FASTQ files fastq-dump -I --split-files SRR7890827 --gzip fastq-dump -I --split-files SRR7890824 --gzip ``` - 直接下載 fastq.gz ``` fastq-dump -I --split-files SRR7890827 --gzip ``` - 如果本地端有檔案 SRR7890827 到底是從本地端轉檔? 還是從線上抓取? - ESC4000 下載狀況 - **SRR7890827 (normal sample)** > Size: 69.84G, 7.03MB/s, in 5h 43m > MD5: `9e7ea85994dfebcea76f4a4f05c84a0a` :white_check_mark: ``` $ wget https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR7890827/SRR7890827 ``` - **SRR7890824 (tumor sample)** > Size: 64.83G, 6.98MB/s, in 2h 46m > MD5: `963de7d4f34748bf719070402dc48e8a` :white_check_mark: ``` $ wget https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR7890824/SRR7890824 ``` - **SRR7890827.fastq.gz** ``` $ date +%T \ /Everythings/dataset/somatic/sratoolkit.3.0.0-ubuntu64/bin/fastq-dump -I --split-files SRR7890827 --gzip \ date +%T ``` - 透過轉檔 > 花費時間: > - `15h 20m 10s` (OneAI) > - `14h 05m 32s` (ESC4000) :::warning :warning: md5 會因為**檔名不同**而不同，因為檔名會被寫入到 fastq，成為 fastq 的一部分，如下所示： [![](https://i.imgur.com/TEQb7mJ.png)](https://i.imgur.com/TEQb7mJ.png) [![](https://i.imgur.com/S0wTfyX.png)](https://i.imgur.com/S0wTfyX.png) ::: - SRR7890827_1.fastq.gz - OneAI > MD5: `3f6ba87b1c7569048b2e3087dbe7a2df` > Size: 52233107436 (49G) - ESC4000 > MD5: `b808792018e3ee0378f043e8467ae4cc` > Size: 52581190461 (49G) - SRR7890827_2.fastq.gz - OneAI > MD5: `8e1f5c7605867929d5c046d074823848` > Size: 57360709396 (54G) - ESC4000 > MD5: `9bd0526c7e4c75a89385683966cf78cc` > Size: 57732748359 (54G) - 直接抓取 > ESC4000 大小約 103G，每小時抓取約 2G，估計要 51 小時 - SRR7890827_1.fastq.gz > MD5: - SRR7890827_2.fastq.gz > MD5: - **SRR7890824.fastq.gz** ``` $ date +%T \ /Everythings/dataset/somatic/sratoolkit.3.0.0-ubuntu64/bin/fastq-dump -I --split-files SRR78908274 --gzip \ date +%T ``` - 透過轉檔 - SRR7890824_1.fastq.gz - OneAI > 檔案損毀 > ~~MD5: `cc97ad0a43c67711c29f6dcfaf58baa4`~~ > ~~Size: 29416787753 (28G)~~ - ESC4000 > MD5: `0ad14d1bc92935c90ee8ec3776403d3f` > Size: 48444535375 (46G) - SRR7890824_2.fastq.gz - OneAI > MD5: `0560489e45b2723ecc13d3c666c0deeb` > Size: 33092152300 (31G) - ESC4000 > MD5: `734e843f8ce05e05795befe2c13eb09a` > Size: 54057229470 (51G) - 直接抓取 - SRR7890824_1.fastq.gz > MD5: - SRR7890824_2.fastq.gz > MD5: - 重作 & 重新比較 | 方法 | 檔案<br>MD5 | One AI<br>(回合) | Esc4000<br>(回合) | | --- | ----------- | ----------------| ------- | | AWS | SRR7890824, 65G :white_check_mark:<br><sup>963de7d4f34748bf719070402dc48e8a</sup> | | 2h46m<br>4h28m | | AWS | SRR7890827, 70G :white_check_mark:<br><sup>9e7ea85994dfebcea76f4a4f05c84a0a</sup> | | 5h43m<br>9h59m | || | 轉檔 | SRR7890824_1.fastq, 205G :white_check_mark:<br><sup>3ba467d9f8ec4955b57cca9a17a84ab1</sup> | 1h26m24s | | | 轉檔 | SRR7890824_2.fastq, 205G :white_check_mark:<br><sup>f03846615ae3aeebb2156078f0efe46c</sup> | 1h26m24s | | | 轉檔 | SRR7890824_1.fastq<b>.gz</b> 45G :white_check_mark:<br><sup>66b4d19b071ecd6e7adcaff3972032ca</sup> | 14h25m38s | 13h13m10s | | 轉檔 | SRR7890824_2.fastq<b>.gz</b>, 50G :white_check_mark:<br><sup>0d61e81efec5f697068bf54678ecb37f</sup> | 14h25m38s | 13h13m10s | | 抓取 | SRR7890824_1.fastq<b>.gz</b>, 45G<br><sup>66b4d19b071ecd6e7adcaff3972032ca</sup> | 35h37m01s | | | 抓取 | SRR7890824_2.fastq<b>.gz</b>, 50G<br><sup>0d61e81efec5f697068bf54678ecb37f</sup> | 35h37m01s | | | 抓取<br>`+I` | SRR7890824_1.fastq<b>.gz</b>, 45G<br><sup>307249208489df3e81753502c8a05def</sup> | 36h19m49s | 46h48m09s | | 抓取<br>`+I` | SRR7890824_2.fastq<b>.gz</b>, 51G<br><sup>0067a03ac6d4b117234e369d1e1d9a1b</sup> | 36h19m49s | 46h48m09s | || | 轉檔 | SRR7890827_1.fastq, 216G :white_check_mark:<br><sup>09df88b21bf7fdfb591adba6df35d9d5</sup> | 1h22m55s || | 轉檔 | SRR7890827_2.fastq, 216G :white_check_mark:<br><sup>14f7dfe83365647524c799aca843bddf</sup> | 1h22m55s ||| | 轉檔 | SRR7890827_1.fastq<b>.gz</b>, 49G :white_check_mark: <br><sup>d71237644ffe1b9229a7ab064d4b6d70</sup> | 15h23m56s | 14h28m57s | | 轉檔 | SRR7890827_2.fastq<b>.gz</b>, 54G :white_check_mark:<br><sup>35cb18bd1f532b3a4bfb5717ca0f1c4c</sup> | 15h23m56s | 14h28m57s | | 抓取 | SRR7890827_1.fastq<b>.gz</b>, G<br>d71237644ffe1b9229a7ab064d4b6d70<sup></sup> | 38h49m18s | 50h05m50s | | 抓取 | SRR7890827_2.fastq<b>.gz</b>, G<br><sup>35cb18bd1f532b3a4bfb5717ca0f1c4c</sup> | 38h49m18s | 50h05m50s | | 抓取<br>`+I` | SRR7890827_1.fastq<b>.gz</b>, 49G :white_check_mark:<br><sup>3f6ba87b1c7569048b2e3087dbe7a2df</sup> | | 29h55m35s | | 抓取<br>`+I` | SRR7890827_2.fastq<b>.gz</b>, 54G :white_check_mark:<br><sup>8e1f5c7605867929d5c046d074823848</sup> | | 29h55m35s | - Esc4000: 使用 SSD raid 硬碟 - AWS: 從 AWS S3 抓取 (Nvidia 存放的檔案) - 抓取: 表示本地端無封存檔，直接從 NCBI 上抓 - `+I`: 表示有使用參數 `+I` - :white_check_mark:: 表示不同的來源，可經由解壓縮後，得到相同的 MD5，表示資料正確 - 例如 md5(.fastq) == md5(ungzip(.fastq.gz)) - ESC4000 第二次時間暴增的原因 - 為了抓取檔案，開兩個 process 抓取 - 一個是從線上直接抓取 - 另一個是透過轉檔 - 結果： - 那個 NV 轉檔指令根本就是錯誤 - 其實還是透過線上抓取 - 導致兩個 process 去抓取網路 - 等於頻寬被切半，時間就 double - 下載一組 fastq 要 50h (兩天) !! - ### Error - **SRR7890824_1.fastq.gz** 檔案下載不全 ![](https://i.imgur.com/0iofq7f.png) unexpected end of file - **SRR7890824_2.fastq.gz** 檔案下載不全 ![](https://i.imgur.com/ahuUwo0.png) unexpected end of file - 遭遇: - somatic pipeline 無法處理 `-I` - paired reads have different names: “SRR7890824.1.2”, “SRR7890824.1.1” - 檔案下載不全 - <br>