Try   HackMD

GPU basecalling ONT data

Author: Miles Benton (GitHub; Twitter)
Created: 2021-11-16 10:24:03
Last modified: 2022-08-17 10:37:17

tags: Nanopore GPU notes documentation benchmarks

A note on live basecalling

From ONT community forum link:

“Keep up” is defined as 80% of the theoretical flow cell output.
e.g. MinION = 4000 kHz x 512 channels x 0.8 = 1.6 M samples/s = 160 kbases/s at 400 b/s

MinION = 4000 kHz x 512 channels x 1.0 = 2,048,000 samples/s

≡ 2.048 M samples/s or 2.048e+06 samples/s

It should be noted that this is based of an ideal situation where a flowcell is sequencing at 100% it's capacity / theoretical output. This is in reality unlikely to happen, so it's probably safe to assume that a GPU that can perform a minimum of 1.6 M samples/s for a given basecalling model will be able to keep up live.

IMPORTANT: please remember these numbers and calculations are based on MinION flowcells and not indicative of PromethION flowcell performance.

Table of results (updated: 2022-08-17)

GPU\CPU FAST model+ HAC model+ SUP model+ Guppy Version
A100 (40GB) (MS) 4.42000e+07 3.40000e+07 1.17000e+07 6.2.1
A100 (40GB) 3.40604e+07 2.68319e+07 6.58227e+06 6.0.1
RTX3090 (HG) 6.09667e+07 1.90738e+07 6.24702e+06 6.0.1
RTX3080Ti (eGPU) 5.71209e+07 1.18229e+07 4.52692e+06 6.0.1
A5000 8.71738e+07 1.33596e+07 4.43743e+06 6.2.1
Tesla V100 (16GB) (MS) 2.69000e+07 1.65000e+07 4.32000e+06 6.2.1
Titan RTX (P920) 3.17412e+07 1.47765e+07 4.29710e+06 6.0.1
Telsa V100 (32GB) 2.66337e+07 1.58095e+07 3.91847e+06 5.3.4
RTX6000 (Clara AGX) 2.01672e+07 1.36405e+07 3.42290e+06 5.3.4
Titan V (DE) 4.71917e+07 1.33653e+07 3.07009e+06 6.0.1
RTX3070 (HG) 5.04924e+07 1.03841e+07 2.95291e+06 6.0.1
RTX3070 (MH) 4.59143e+07 7.32223e+06 2.40374e+06 6.0.1
RTX3060 (eGPU) 4.70238e+07 6.40374e+06 2.28163e+06 5.3.4
RTX2060 SUPER (MS) 4.12000e+07 8.28000e+06 2.24000e+06 6.2.1
Tesla T4 (16GB) (MS) 2.61000e+07 5.16000e+06 1.43000e+06 6.2.1
RTX4000 (mobile) 2.88644e+07 4.81920e+06 1.36953e+06 6.0.1
Telsa P100 (12GB) (MS) 1.41000e+07 4.00000e+06 9.37000e+05 6.2.1
Jetson Xavier AGX (16GB) 8.49277e+06 1.57560e+06 4.40821e+05 5.3.4
Jetson Xavier NX 4.36631e+06 - - 5.3.4
Jetson TX2 (modified) 2.05553e+06 - - 5.0.14
Jetson TX2 (Mk1C) 1.60000e+06 - - -
Xeon W-10885M (CPU) 6.43747e+05 DNF DNF 5.0.14

NOTE: the above table is currently sorted based on best performance in the Super Accuracy Model (SUP).

A massive thank you to all external contributors:

  • David Eccles (Titan V)
  • Martin Haagmans (RTX3070)
  • Hasindu Gamaarachchi (RTX3070, 2x RTX3090)
  • Michael Shamash (RTX2060, P100, T4, V100-SXM2, A100)

Table indicating live calling performance

The below table lists results for all the GPUs that we have currently tested. We have used the same example set of ONT fast5 files, Guppy 5.0.16, and where possible have tuned the chunks_per_runner parameter to get the most out of HAC and SUP calling based on the GPU being tested. This hopefully gives a more "real world" example of what you can expect from these types of cards in terms of basecalling rate.

The colours represent how well a given GPU and basecalling model will perform for keeping up with live basecalling during a sequencing run.

  • green - easily keeps up in real-time
  • orange - will likely keep up with 80-90% of the run in real-time
  • red - won't get anywhere close, large lag in basecalling

From ONT community forum link:

“Keep up” is defined as 80% of the theoretical flow cell output.
e.g. MinION = 4000 kHz x 512 channels x 0.8 = 1.6 M samples/s = 160 kbases/s at 400 b/s

MinION = 4000 kHz x 512 channels x 1.0 = 2,048,000 samples/s

≡ 2.048 M samples/s or 2.048e+06 samples/s

It should be noted that this is based of an ideal situation where a flowcell is sequencing at 100% it's capacity / theoretical output. This is in reality unlikely to happen, so it's probably safe to assume that a GPU that can perform a minimum of 1.6 M samples/s for a given basecalling model will be able to keep up 'live'.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

* the metric reported is samples/second - where higher is faster basecalling
DNF - did not finish (I couldn’t be bothered waiting hours/days for the CPU)

Last changed by 
 
Miles Benton
At the bottom of the world in New Zealand, I am a computational geneticist interested in all facets of biology and technology, particularly GPUs and Nanopores.
0
2
2292

Read more

GPU musings (with an eye on genomics)

DOI Author: Miles Benton (GitHub; Twitter)Created: 2021-21-01 23:15:32Last modified: 2021-10-11 08:53:33 hackmd-github-sync-badge :::warning ATTENTION: this document is an ongoing work in progress. :::

Apr 4, 2023
Tuning GPU parameters for Guppy performance

Author: Miles Benton (GitHub; Twitter; Gmail) Created: 2022-02-08 16:56:50 Last modified: 2022-02-11 20:24:06 Nvidia GPUs drastically accelerate the basecalling rate of Nanopore data. This is great, but not all GPUs are built equally, meaning that sometimes you'll need to tweak specific parameters to either increase performance, or on the other side of the coin, tune down parameters so that a lower spec'd card can work. Some examples of this: you have a card with more than 8GB of RAM (such as a RTX3080, RTX3090, A5000) and you want to get more basecalling performance from high accuracy (HAC) and super accuracy (SUP) models. you have a card with less than 8GB of RAM (such as GTX1660, RTX2060, RTX3050) and you want to be able to use Guppy for live basecalling or running the HAC/SUP models without running into CUDA memory issues. For some time I've been meaning to put together something more structured around how I go about tuning Guppy parameters to get the most out of particular GPUs for the Nanopore basecalling work that we do. This document is an attempt at that.

Feb 25, 2022
GPU price / performance comparisons for Nanopore basecalling

Author: Miles Benton (GitHub; Twitter) Created: 2021-07-16 20:15:56 Last modified: 2022-01-19 13:54:40 For some time I've been wanting to put together my thoughts around price/performance ratios of GPUs. I've been thinking that there must be a "sweet spot" for users that are wanting to run a single MinION Mk1b and have access to things such as adaptive sampling and live basecalling with FAST/HAC models. Bonus points if it has decent retrospective basecalling performance. I've been very fortunate recently in terms of being provided with a range of hardware that has allowed me to start exploring this. So I wanted to create some notes to provide the information back to the community, for any users that are interested and may be in the market for a GPU, or people that just want an idea of the type of performance you can expect from various GPU models. I'm hoping that this becomes a dynamic document and will evolve with time. For now I want to report on a comparison I was able to perform using an external GPU enclosure (eGPU) paired with two of the Nvidia Ampere cards, a RTX3060 and a RTX3080Ti. These are two cards aimed at gaming, one at the 'lower' end (RTX3060) the other very much at the higher end (RTX3080Ti). Obviously this is reflected in the price with the RTX3060 being ~$1000 NZD and the RTX3080Ti ~$3000.

Feb 23, 2022
Running (live) GPU basecalling on 21.04 [experimental]

Author: Miles Benton (GitHub; Twitter) Created: 2021-08-30 08:15:02 Last modified: 2021-08-31 20:49:50 :::warning WARNING: please ensure you are comfortable with the command line, and are willing to accept any and all responsibility for potential issues that may arise. All efforts have been made to ensure this is as safe and easy to revert if needed, but the author takes no responsibility if things go pear shaped or don't work. Below there be dragons (potentially)! ::: This is a set of notes for getting MinKNOW running with live GPU basecalling on a computer running Ubuntu 21.04 (or a base of 21.04, such as Pop!_OS).

Aug 31, 2021
Read more from Miles Benton
Published on HackMD