# Running (live) GPU basecalling on 21.04 [experimental] *Author:* [Miles Benton](https://sirselim.github.io/) ([GitHub](https://github.com/sirselim); [Twitter](https://twitter.com/miles_benton)) *Created:* 2021-08-30 08:15:02 *Last modified:* 2021-08-31 20:49:50 ###### tags: `Nanopore` `GPU` `notes` `documentation` :::warning **WARNING:** please ensure you are comfortable with the command line, and are willing to accept any and all responsibility for potential issues that may arise. All efforts have been made to ensure this is as safe and easy to revert if needed, but the author takes no responsibility if things go pear shaped or don't work. Below there be dragons (potentially)! ::: This is a set of notes for getting MinKNOW running with live GPU basecalling on a computer running Ubuntu 21.04 (or a base of 21.04, such as Pop!\_OS). I can confirm that I have this running on a Pop!\_OS 21.04 system and GPU calling works great. I also have 3 different confirmations from people that have got up and running on various 21.04 (Hirsute Hippo) based Linux systems. So it seems to be a method that is currently fairly stable. Let's crack on... ## add ONT repo We're going to use the ONT Focal (20.04) repository. It's not recommended to mix releases (i.e. packages from 21.04 and 20.04), but we'll put a few things in place to try and make it as safe as possible. We don't need to pull many packaged from the Focal repos, so we'll use pinning to make them available. First lets add the ONT Focal repository: ```shell= # update packages list sudo apt-get update # check for and install wget if needed if [ $(dpkg-query -W -f='${Status}' wget 2>/dev/null | grep -c "ok installed") -eq 0 ]; then sudo apt --yes install wget; fi # add the key wget -O- https://mirror.oxfordnanoportal.com/apt/ont-repo.pub | sudo apt-key add - # add the focal 20.04 repo echo "deb http://mirror.oxfordnanoportal.com/apt focal-stable non-free" | sudo tee /etc/apt/sources.list.d/nanoporetech.sources.list ``` Once the ONT repo has been added we can update: ```shell= sudo apt update ``` We should no have access to ONT software, let's check: ```shell= apt policy minknow-core-minion-nc minknow-core-minion-nc: Installed: 4.3.4-focal Candidate: 4.3.4-focal Version table: 4.3.4-focal 100 10 http://mirror.oxfordnanoportal.com/apt focal-stable/non-free amd64 Packages 100 /var/lib/dpkg/status ``` Now that we have that set up we'll grab the Focal (20.04) repos and get them set up with the correct privileges. ## add the Focal (20.04) repos First thing we have to do is create a list of the Focal (20.04) resources to pull. The below code steps through this. The first command uses nano to create the file `system-focal.sources` located in `/etc/apt/sources.list.d`. ```shell= sudo nano /etc/apt/sources.list.d/system-focal.sources ``` When you are in nano copy the below into the file and save it: ```shell= X-Repolib-Name: Pop_OS System Sources Enabled: yes Types: deb deb-src URIs: http://us.archive.ubuntu.com/ubuntu/ Suites: focal focal-security focal-updates focal-backports Components: main restricted universe multiverse X-Repolib-Default-Mirror: http://us.archive.ubuntu.com/ubuntu/ ``` Check that the file exists and contains the right information: ```shell= $ cat system-focal.sources X-Repolib-Name: Pop_OS System Sources Enabled: yes Types: deb deb-src URIs: http://us.archive.ubuntu.com/ubuntu/ Suites: focal focal-security focal-updates focal-backports Components: main restricted universe multiverse X-Repolib-Default-Mirror: http://us.archive.ubuntu.com/ubuntu/ ``` Right, next step is making sure we only pull packages when we want to. For this we'll use the process of package pinning. ### pin the Focal (20.04) repos Pinning is an advance package management concept that essentially allows us to set priorities on particular packages, repositories or even whole releases. To do this we need to create a file in `/etc/apt/preferences.d`. This can be done as below, we'll create a file called `focal-default-settings`: ```shell= sudo nano /etc/apt/preferences.d/focal-default-settings ``` Now that we're in nano paste the below into the file and save: ```shell= Package: * Pin: release n=focal* Pin-Priority: 10 ``` ... and check that it looks right: ```shell= $ cat focal-default-settings Package: * Pin: release n=focal* Pin-Priority: 10 ``` A pin priority of 10 is defined as: ``` 0 < P <=100 causes a version to be installed only if there is no installed version of the package ``` OK, this now means that when we update using apt we'll have access to all focal packages, but they won't be installed if there is a Hirsute (21.04) package of the same name. Focal packages will only be installed if they are the only option and a package depends on them, or if a user specifically asks for it to be installed. :::info **INFO:** if you'd like to read more about pinning there is some good information [here](https://wiki.debian.org/AptConfiguration). ::: ### update and check pinning Now we can perform a package list update and check our pinning. ```shell= sudo apt update ``` Check pinning - let's use `r-base` as an example (because I know it has different versions between Focal and Hirsute): ```shell= $ apt policy r-base r-base: Installed: (none) Candidate: 4.0.4-1build1 Version table: 4.0.4-1build1 500 500 http://us.archive.ubuntu.com/ubuntu hirsute/universe amd64 Packages 500 http://us.archive.ubuntu.com/ubuntu hirsute/universe i386 Packages 3.6.3-2 10 10 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages 10 http://us.archive.ubuntu.com/ubuntu focal/universe i386 Packages ``` In the above output you'll notice that the Hirsute version of `r-base` is 4.0.4 and has a pin-priority of 500, while the Focal version is 3.6.3 and has a pin-priority of 10. If we were to run `apt install r-base` the Hirsute version will be installed preferentially. If we check a package that I know doesn't have a Hirsute version, lets see what it looks like: ```shell= $ apt policy libhdf5-cpp-103 libhdf5-cpp-103: Installed: (none) Candidate: 1.10.4+repack-11ubuntu1 Version table: 1.10.4+repack-11ubuntu1 10 10 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages ``` *Note: we'll see the `libhdf5-cpp-103` package soon, it's going to be required by `ont-guppy`.* So this package will be 'ignored' by the package manager in terms of system updates etc, but it is able to be installed using apt. If we were to use a pin-priority lower than 0 then the package isn't able to be installed at all (unless you use a special approach). This grants us the ability to somewhat safely "mix and match" specific packages that we require to get MinKNOW correctly set up with live GPU basecalling. **Note: this isn't best practice so again only do this if you are happy with the potential risks. The way this is being implemented with pinning attempts to mitigate a lot of the risk, and the packages being installed shouldn't cause issue, but there is always a chance. You have again been warned.** ## install MinKNOW and required packages OK, we've got access to all the pieces of the puzzle. Time to install MinKNOW/MinION software and components. In the past I've spoken about the `minion-nc` metapackage that is currently causing issues, you can find my thoughts on that in the "old" notes [here](https://github.com/sirselim/jetson_nanopore_sequencing/blob/main/live_basecalling.md#install-minion-software). If/when this meta-package becomes stable and works on systems other than 16.04/18.04 I'll update these guides, but until then hopefully this approach is useful. Install the required packages as per below: ```shell= sudo apt install \ minknow-core-minion-nc \ ont-kingfisher-ui-minion \ ont-bream4-minion \ ont-configuration-customer-minion \ ont-jwt-auth \ ont-vbz-hdf-plugin ``` This should proceed just fine, there may be some extra packages that get pulled but it shouldn't be a problem. ## install GPU Guppy Herein lies the crux of the issue with getting GPU basecalling up and running, Guppy. The biggest issue is that `ont-guppy-cpu-for-minion` is the package that is bundled in the `minion-nc` meta-package. This is the CPU version and obviously not much help when you want to use the shiny GPU(s) in your computer. The `ont-guppy` package is the GPU version that is currently supported by most recent version of MinKNOW. You can also download pre-compiled binaries for GPU Guppy from the Community Software page and set up that way as well (I work through this process [here](https://github.com/sirselim/jetson_nanopore_sequencing/blob/main/live_basecalling.md) in detail). For this guide I'm trying to stick to the "official" methods as much as possible, and trying to make the process easier for people. We can check the current version of MinKNOW compatible Guppy: ```shell= $ apt policy ont-guppy ont-guppy: Installed: (none) Candidate: 5.0.11-1~focal Version table: 5.0.11-1~focal 10 10 http://mirror.oxfordnanoportal.com/apt focal-stable/non-free amd64 Packages ``` We can also explore the `ont-guppy` package (or any package for the matter): ```shell= $ apt show ont-guppy Package: ont-guppy Version: 5.0.11-1~focal Priority: optional Section: science Maintainer: unknown <unknown@unknown.com> Installed-Size: 1,376 MB Depends: libc6 (>=2.31), libzmq5, libcurl4, libhdf5-cpp-103, libssl1.1, libcuda-11.1-1 | libcuda1 (>= 455), libboost-atomic1.71.0, libboost-chrono1.71.0, libboost-date-time1.71.0, libboost-filesystem1.71.0, libboost-iostreams1.71.0, libboost-program-options1.71.0, libboost-regex1.71.0, libboost-system1.71.0, libboost-log1.71.0 Recommends: python3 Conflicts: ont-guppy-cpu Replaces: ont-guppy-cpu Download-Size: 892 MB APT-Sources: http://mirror.oxfordnanoportal.com/apt focal-stable/non-free amd64 Packages Description: Oxford Nanopore Technologies high-performance GPU basecaller Basecalls fast5 files generated by Oxford Nanopore devices, producing basecalled fastq files and a summary. Also includes standalone tools for barcoding, alignment, and running a basecall server. ``` If we just want to see the dependencies of the `ont-guppy` package: ```shell= $ apt depends ont-guppy ont-guppy Depends: libc6 (>= 2.31) Depends: libzmq5 Depends: libcurl4 Depends: libhdf5-cpp-103 Depends: libssl1.1 |Depends: <libcuda-11.1-1> libnvidia-compute-450-server libnvidia-compute-460-server libnvidia-compute-470 libnvidia-compute-470-server Depends: <libcuda1> (>= 455) libnvidia-compute-460-server libnvidia-compute-470 libnvidia-compute-470-server Depends: libboost-atomic1.71.0 Depends: libboost-chrono1.71.0 Depends: libboost-date-time1.71.0 Depends: libboost-filesystem1.71.0 Depends: libboost-iostreams1.71.0 Depends: libboost-program-options1.71.0 Depends: libboost-regex1.71.0 Depends: libboost-system1.71.0 Depends: libboost-log1.71.0 Conflicts: ont-guppy-cpu Recommends: python3 Replaces: ont-guppy-cpu ``` The things that are important here for the GPU version are the Nvidia Drivers, CUDA and BOOST libraries - all things a GPU version of most software will need. If you don't have Nvidia drivers and CUDA set up this guide isn't going to take you through that process, so please ensure these things are ready to go before proceeding. :::warning **WARNING:** what we're about to do is install a few of the BOOST libraries from the Focal branch alongside the Hirsute libraries that may already be on your system. This has potential to break things but should be OK, and is completely reversible if you do face any issues. I haven't hit any problems having the handful of libraries required on my system, but it may be different for you. ::: Time to pull the trigger and install GPU Guppy from the ONT repo: ```shell= sudo apt install ont-guppy ``` This shouldn't spit out any errors now that we have pinned the focal repos, you should get a list of packages that are required alongside `ont-guppy` - as wel saw above. I'll list the current ones again below, which provides a way to track exactly what has been installed in case it needs to be removed again. * `libaec0 libboost-atomic1.71.0` * `libboost-chrono1.71.0` * `libboost-date-time1.71.0` * `libboost-filesystem1.71.0` * `libboost-iostreams1.71.0` * `libboost-log1.71.0` * `libboost-program-options1.71.0` * `libboost-regex1.71.0` * `libboost-system1.71.0` * `libboost-thread1.71.0` * `libhdf5-103` * `libhdf5-cpp-103` * `libsz2` Once Guppy is installed you should be able to check it's paths: ```shell= $ which guppy_basecaller /usr/bin/guppy_basecaller ``` ```shell= $ guppy_basecaller --version : Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 5.0.11+2b6dbff ``` This is looking good. ## configuring MinKNOW/Guppyd :::warning **WARNING:** this is still very much in a state of rapid development. Traditionally Guppy has been launched by `minknow.service` and configuration lines in `app_conf`. This is still the case with 21.0.6, but it is moving away from this to Guppy having it's own "control" via a systemd service, `guppyd.service`. The current `ont-guppyd-for-minknow` package requires the CPU version of MinKNOW, so it's not able to be installed. However I've come up with a manual solution for now. ::: ### `minknow.service` For now we can still edit the MinKNOW `app_conf`. I usually do this manually, but there is a config_editor. We need to provide the path to the GPU versions of the Guppy binaries, the location of where they are installed. We're going to use the `ont-guppy` package, which installs these binary files to `/opt/ont/guppy/bin/`. Thus the basecall server path would be `/opt/ont/guppy/bin/guppy_basecall_server`. The below is an example of how to use the conf_editor: ```shell= sudo /opt/ont/minknow/bin/config_editor --conf application \ --filename /opt/ont/minknow/conf/app_conf \ --set guppy.server_executable="/opt/ont/guppy/bin/guppy_basecall_server" \ --set guppy.client_executable="/opt/ont/guppy/bin/guppy_basecall_client" \ --set guppy.gpu_calling=1 \ --set guppy.num_threads=16 \ --set guppy.ipc_threads=2 ``` Please note I have not tuned/optimised the above parameters, it's just the example ONT give. The above should work just fine, but you will want to make modifications to the GPU parameters based on the type of card that you have. Any time you make modifications to these systemd services make sure you restart them: ```shell= systemctl restart minknow.service ``` At any time you can check on their status: ```shell= systemctl status minknow.service ``` ### `guppyd.service` There is now a systemd service created by the package `ont-guppyd-for-minknow`. The current problem is that if we try to install that package it tries to pull and install the CPU version of Guppy (`ont-guppy-cpu-for-minion`) because they depend on each other. As this is currently in the early stages we can still use the `app_conf` method above, but in an attempt to start adopting the newer approach I have created a manual method to get `guppyd.service` running. The first thing is to create the file in the correct location: ```shell= sudo nano /lib/systemd/system/guppyd.service ``` We need to add the below content (it's the default) to the file, and then save: ```shell= [Unit] Description=Service to manage the guppy basecall server. Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018 [Service] Type=simple ExecStart=/opt/ont/guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --port 5555 -x cuda:all Restart=always User=root MemoryLimit=8G MemoryHigh=8G CPUQuota=200% [Install] Alias=guppyd.service WantedBy=multi-user.target ``` As always, I like to check the contents of the newly created file: ```shell= $ cat /lib/systemd/system/guppyd.service [Unit] Description=Service to manage the guppy basecall server. Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018 [Service] Type=simple ExecStart=/usr/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --port 5555 -x cuda:all Restart=always User=root MemoryLimit=8G MemoryHigh=8G CPUQuota=200% [Install] Alias=guppyd.service WantedBy=multi-user.target ``` :::info **NOTE:** the above content can be modified based on your preferred settings for your GPU. Any arguments you can pass to Guppy can be added to the `ExecStart=` line, i.e. you could have `--chunks_per_runner` etc. ::: We also want to enable this service to auto start upon a system restart: ```shell= systemctl enable guppyd.service ``` Once that is done restart the service: ```shell= systemctl restart guppyd.service ``` ## installation script :::danger **DANGER:** this is very experimental! At this stage I have only tested this script on a single laptop running Pop!\_OS 21.04, so please proceed with caution. Saying this, there is very little that can actually drastically harm your system here, as long as you are comfortable with some commandline and Linux functions you should be OK. ::: This is an attempt to automate all of the above into a single bash script that can be run (with sudo) to make set up easier and faster. I have created a script called `minknow_hirsute_setup.sh` which is maintained in a GitHub gist [here](https://gist.github.com/sirselim/e9ba8e54c894c760b21942f6b24a794a). It should be enough to save/copy the contents of that gist into a file called `minknow_hirsute_setup.sh`, give it execution rights (`chmod +x minknow_hirsute_setup.sh`) and then run it (`./minknow_hirsute_setup.sh`). All going well it should proceed through all the above steps and complete in a few minutes. If anyone does give this a try please do contact me with any feedback, suggestions or issues, there is only so much testing one person can do!
Last changed by  
 
Miles Benton
At the bottom of the world in New Zealand, I am a computational geneticist interested in all facets of biology and technology, particularly GPUs and Nanopores.
7326

Read more

GPU musings (with an eye on genomics)

DOI Author: Miles Benton (GitHub; Twitter)Created: 2021-21-01 23:15:32Last modified: 2021-10-11 08:53:33 hackmd-github-sync-badge :::warning ATTENTION: this document is an ongoing work in progress. :::

4/4/2023
GPU basecalling ONT data

Author: Miles Benton (GitHub; Twitter) Created: 2021-11-16 10:24:03 Last modified: 2022-08-17 10:37:17 A note on live basecalling From ONT community forum link: “Keep up” is defined as 80% of the theoretical flow cell output. e.g. MinION = 4000 kHz x 512 channels x 0.8 = 1.6 M samples/s = 160 kbases/s at 400 b/s

8/16/2022
Tuning GPU parameters for Guppy performance

Author: Miles Benton (GitHub; Twitter; Gmail) Created: 2022-02-08 16:56:50 Last modified: 2022-02-11 20:24:06 Nvidia GPUs drastically accelerate the basecalling rate of Nanopore data. This is great, but not all GPUs are built equally, meaning that sometimes you'll need to tweak specific parameters to either increase performance, or on the other side of the coin, tune down parameters so that a lower spec'd card can work. Some examples of this: you have a card with more than 8GB of RAM (such as a RTX3080, RTX3090, A5000) and you want to get more basecalling performance from high accuracy (HAC) and super accuracy (SUP) models. you have a card with less than 8GB of RAM (such as GTX1660, RTX2060, RTX3050) and you want to be able to use Guppy for live basecalling or running the HAC/SUP models without running into CUDA memory issues. For some time I've been meaning to put together something more structured around how I go about tuning Guppy parameters to get the most out of particular GPUs for the Nanopore basecalling work that we do. This document is an attempt at that.

2/25/2022
GPU price / performance comparisons for Nanopore basecalling

Author: Miles Benton (GitHub; Twitter) Created: 2021-07-16 20:15:56 Last modified: 2022-01-19 13:54:40 For some time I've been wanting to put together my thoughts around price/performance ratios of GPUs. I've been thinking that there must be a "sweet spot" for users that are wanting to run a single MinION Mk1b and have access to things such as adaptive sampling and live basecalling with FAST/HAC models. Bonus points if it has decent retrospective basecalling performance. I've been very fortunate recently in terms of being provided with a range of hardware that has allowed me to start exploring this. So I wanted to create some notes to provide the information back to the community, for any users that are interested and may be in the market for a GPU, or people that just want an idea of the type of performance you can expect from various GPU models. I'm hoping that this becomes a dynamic document and will evolve with time. For now I want to report on a comparison I was able to perform using an external GPU enclosure (eGPU) paired with two of the Nvidia Ampere cards, a RTX3060 and a RTX3080Ti. These are two cards aimed at gaming, one at the 'lower' end (RTX3060) the other very much at the higher end (RTX3080Ti). Obviously this is reflected in the price with the RTX3060 being ~$1000 NZD and the RTX3080Ti ~$3000.

2/23/2022
Read more from Miles Benton
Published on HackMD