Running (live) GPU basecalling on 21.04 [experimental]

Author: Miles Benton (GitHub; Twitter)
Created: 2021-08-30 08:15:02
Last modified: 2021-08-31 20:49:50

tags: Nanopore GPU notes documentation

WARNING: please ensure you are comfortable with the command line, and are willing to accept any and all responsibility for potential issues that may arise. All efforts have been made to ensure this is as safe and easy to revert if needed, but the author takes no responsibility if things go pear shaped or don't work. Below there be dragons (potentially)!

This is a set of notes for getting MinKNOW running with live GPU basecalling on a computer running Ubuntu 21.04 (or a base of 21.04, such as Pop!_OS).

I can confirm that I have this running on a Pop!_OS 21.04 system and GPU calling works great. I also have 3 different confirmations from people that have got up and running on various 21.04 (Hirsute Hippo) based Linux systems. So it seems to be a method that is currently fairly stable.

Let's crack on…

add ONT repo

We're going to use the ONT Focal (20.04) repository. It's not recommended to mix releases (i.e. packages from 21.04 and 20.04), but we'll put a few things in place to try and make it as safe as possible. We don't need to pull many packaged from the Focal repos, so we'll use pinning to make them available.

First lets add the ONT Focal repository:

# update packages list
sudo apt-get update
# check for and install wget if needed
if [ $(dpkg-query -W -f='${Status}' wget 2>/dev/null | grep -c "ok installed") -eq 0 ];
then
  sudo apt --yes install wget;
fi
# add the key
wget -O- https://mirror.oxfordnanoportal.com/apt/ont-repo.pub | sudo apt-key add -
# add the focal 20.04 repo
echo "deb http://mirror.oxfordnanoportal.com/apt focal-stable non-free" | sudo tee /etc/apt/sources.list.d/nanoporetech.sources.list

Once the ONT repo has been added we can update:

sudo apt update

We should no have access to ONT software, let's check:

apt policy minknow-core-minion-nc 

minknow-core-minion-nc:
  Installed: 4.3.4-focal
  Candidate: 4.3.4-focal
  Version table:
     4.3.4-focal 100
         10 http://mirror.oxfordnanoportal.com/apt focal-stable/non-free amd64 Packages
        100 /var/lib/dpkg/status

Now that we have that set up we'll grab the Focal (20.04) repos and get them set up with the correct privileges.

add the Focal (20.04) repos

First thing we have to do is create a list of the Focal (20.04) resources to pull. The below code steps through this. The first command uses nano to create the file system-focal.sources located in /etc/apt/sources.list.d.

sudo nano /etc/apt/sources.list.d/system-focal.sources

When you are in nano copy the below into the file and save it:

X-Repolib-Name: Pop_OS System Sources
Enabled: yes
Types: deb deb-src
URIs: http://us.archive.ubuntu.com/ubuntu/
Suites: focal focal-security focal-updates focal-backports
Components: main restricted universe multiverse
X-Repolib-Default-Mirror: http://us.archive.ubuntu.com/ubuntu/

Check that the file exists and contains the right information:

$ cat system-focal.sources

X-Repolib-Name: Pop_OS System Sources
Enabled: yes
Types: deb deb-src
URIs: http://us.archive.ubuntu.com/ubuntu/
Suites: focal focal-security focal-updates focal-backports
Components: main restricted universe multiverse
X-Repolib-Default-Mirror: http://us.archive.ubuntu.com/ubuntu/

Right, next step is making sure we only pull packages when we want to. For this we'll use the process of package pinning.

pin the Focal (20.04) repos

Pinning is an advance package management concept that essentially allows us to set priorities on particular packages, repositories or even whole releases.

To do this we need to create a file in /etc/apt/preferences.d. This can be done as below, we'll create a file called focal-default-settings:

sudo nano /etc/apt/preferences.d/focal-default-settings

Now that we're in nano paste the below into the file and save:

Package: *
Pin: release n=focal*
Pin-Priority: 10

… and check that it looks right:

$ cat focal-default-settings

Package: *
Pin: release n=focal*
Pin-Priority: 10

A pin priority of 10 is defined as:

0 < P <=100
    causes a version to be installed only if there is no installed version of the package

OK, this now means that when we update using apt we'll have access to all focal packages, but they won't be installed if there is a Hirsute (21.04) package of the same name. Focal packages will only be installed if they are the only option and a package depends on them, or if a user specifically asks for it to be installed.

INFO: if you'd like to read more about pinning there is some good information here.

update and check pinning

Now we can perform a package list update and check our pinning.

sudo apt update

Check pinning - let's use r-base as an example (because I know it has different versions between Focal and Hirsute):

$ apt policy r-base

r-base:
  Installed: (none)
  Candidate: 4.0.4-1build1
  Version table:
     4.0.4-1build1 500
        500 http://us.archive.ubuntu.com/ubuntu hirsute/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu hirsute/universe i386 Packages
     3.6.3-2 10
         10 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages
         10 http://us.archive.ubuntu.com/ubuntu focal/universe i386 Packages

In the above output you'll notice that the Hirsute version of r-base is 4.0.4 and has a pin-priority of 500, while the Focal version is 3.6.3 and has a pin-priority of 10. If we were to run apt install r-base the Hirsute version will be installed preferentially.

If we check a package that I know doesn't have a Hirsute version, lets see what it looks like:

$ apt policy libhdf5-cpp-103

libhdf5-cpp-103:
  Installed: (none)
  Candidate: 1.10.4+repack-11ubuntu1
  Version table:
     1.10.4+repack-11ubuntu1 10
         10 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

Note: we'll see the libhdf5-cpp-103 package soon, it's going to be required by ont-guppy.

So this package will be 'ignored' by the package manager in terms of system updates etc, but it is able to be installed using apt. If we were to use a pin-priority lower than 0 then the package isn't able to be installed at all (unless you use a special approach).

This grants us the ability to somewhat safely "mix and match" specific packages that we require to get MinKNOW correctly set up with live GPU basecalling. Note: this isn't best practice so again only do this if you are happy with the potential risks. The way this is being implemented with pinning attempts to mitigate a lot of the risk, and the packages being installed shouldn't cause issue, but there is always a chance. You have again been warned.

install MinKNOW and required packages

OK, we've got access to all the pieces of the puzzle. Time to install MinKNOW/MinION software and components. In the past I've spoken about the minion-nc metapackage that is currently causing issues, you can find my thoughts on that in the "old" notes here. If/when this meta-package becomes stable and works on systems other than 16.04/18.04 I'll update these guides, but until then hopefully this approach is useful.

Install the required packages as per below:

sudo apt install \
  minknow-core-minion-nc \
  ont-kingfisher-ui-minion \
  ont-bream4-minion \
  ont-configuration-customer-minion \
  ont-jwt-auth \
  ont-vbz-hdf-plugin

This should proceed just fine, there may be some extra packages that get pulled but it shouldn't be a problem.

install GPU Guppy

Herein lies the crux of the issue with getting GPU basecalling up and running, Guppy. The biggest issue is that ont-guppy-cpu-for-minion is the package that is bundled in the minion-nc meta-package. This is the CPU version and obviously not much help when you want to use the shiny GPU(s) in your computer.

The ont-guppy package is the GPU version that is currently supported by most recent version of MinKNOW. You can also download pre-compiled binaries for GPU Guppy from the Community Software page and set up that way as well (I work through this process here in detail). For this guide I'm trying to stick to the "official" methods as much as possible, and trying to make the process easier for people.

We can check the current version of MinKNOW compatible Guppy:

$ apt policy ont-guppy

ont-guppy:
  Installed: (none)
  Candidate: 5.0.11-1~focal
  Version table:
     5.0.11-1~focal 10
         10 http://mirror.oxfordnanoportal.com/apt focal-stable/non-free amd64 Packages

We can also explore the ont-guppy package (or any package for the matter):

$ apt show ont-guppy

Package: ont-guppy
Version: 5.0.11-1~focal
Priority: optional
Section: science
Maintainer: unknown <[e-mail]>
Installed-Size: 1,376 MB
Depends: libc6 (>=2.31), libzmq5, libcurl4, libhdf5-cpp-103, libssl1.1, 
libcuda-11.1-1 | libcuda1 (>= 455), libboost-atomic1.71.0, 
libboost-chrono1.71.0, libboost-date-time1.71.0, libboost-filesystem1.71.0, 
libboost-iostreams1.71.0, libboost-program-options1.71.0, 
libboost-regex1.71.0, libboost-system1.71.0, libboost-log1.71.0
Recommends: python3
Conflicts: ont-guppy-cpu
Replaces: ont-guppy-cpu
Download-Size: 892 MB
APT-Sources: http://mirror.oxfordnanoportal.com/apt focal-stable/non-free amd64 Packages
Description: Oxford Nanopore Technologies high-performance GPU basecaller
 Basecalls fast5 files generated by Oxford Nanopore devices,
 producing basecalled fastq files and a summary.
 Also includes standalone tools for barcoding, alignment,
 and running a basecall server.

If we just want to see the dependencies of the ont-guppy package:

$ apt depends ont-guppy

ont-guppy
  Depends: libc6 (>= 2.31)
  Depends: libzmq5
  Depends: libcurl4
  Depends: libhdf5-cpp-103
  Depends: libssl1.1
 |Depends: <libcuda-11.1-1>
    libnvidia-compute-450-server
    libnvidia-compute-460-server
    libnvidia-compute-470
    libnvidia-compute-470-server
  Depends: <libcuda1> (>= 455)
    libnvidia-compute-460-server
    libnvidia-compute-470
    libnvidia-compute-470-server
  Depends: libboost-atomic1.71.0
  Depends: libboost-chrono1.71.0
  Depends: libboost-date-time1.71.0
  Depends: libboost-filesystem1.71.0
  Depends: libboost-iostreams1.71.0
  Depends: libboost-program-options1.71.0
  Depends: libboost-regex1.71.0
  Depends: libboost-system1.71.0
  Depends: libboost-log1.71.0
  Conflicts: ont-guppy-cpu
  Recommends: python3
  Replaces: ont-guppy-cpu

The things that are important here for the GPU version are the Nvidia Drivers, CUDA and BOOST libraries - all things a GPU version of most software will need. If you don't have Nvidia drivers and CUDA set up this guide isn't going to take you through that process, so please ensure these things are ready to go before proceeding.

WARNING: what we're about to do is install a few of the BOOST libraries from the Focal branch alongside the Hirsute libraries that may already be on your system. This has potential to break things but should be OK, and is completely reversible if you do face any issues. I haven't hit any problems having the handful of libraries required on my system, but it may be different for you.

Time to pull the trigger and install GPU Guppy from the ONT repo:

sudo apt install ont-guppy

This shouldn't spit out any errors now that we have pinned the focal repos, you should get a list of packages that are required alongside ont-guppy - as wel saw above. I'll list the current ones again below, which provides a way to track exactly what has been installed in case it needs to be removed again.

• libaec0 libboost-atomic1.71.0
• libboost-chrono1.71.0
• libboost-date-time1.71.0
• libboost-filesystem1.71.0
• libboost-iostreams1.71.0
• libboost-log1.71.0
• libboost-program-options1.71.0
• libboost-regex1.71.0
• libboost-system1.71.0
• libboost-thread1.71.0
• libhdf5-103
• libhdf5-cpp-103
• libsz2

Once Guppy is installed you should be able to check it's paths:

$ which guppy_basecaller

/usr/bin/guppy_basecaller

$ guppy_basecaller --version

: Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 5.0.11+2b6dbff

This is looking good.

configuring MinKNOW/Guppyd

WARNING: this is still very much in a state of rapid development. Traditionally Guppy has been launched by minknow.service and configuration lines in app_conf. This is still the case with 21.0.6, but it is moving away from this to Guppy having it's own "control" via a systemd service, guppyd.service.

The current ont-guppyd-for-minknow package requires the CPU version of MinKNOW, so it's not able to be installed. However I've come up with a manual solution for now.

minknow.service

For now we can still edit the MinKNOW app_conf. I usually do this manually, but there is a config_editor.

We need to provide the path to the GPU versions of the Guppy binaries, the location of where they are installed. We're going to use the ont-guppy package, which installs these binary files to /opt/ont/guppy/bin/. Thus the basecall server path would be /opt/ont/guppy/bin/guppy_basecall_server.

The below is an example of how to use the conf_editor:

sudo /opt/ont/minknow/bin/config_editor --conf application \
  --filename /opt/ont/minknow/conf/app_conf \
  --set guppy.server_executable="/opt/ont/guppy/bin/guppy_basecall_server" \
  --set guppy.client_executable="/opt/ont/guppy/bin/guppy_basecall_client" \
  --set guppy.gpu_calling=1 \
  --set guppy.num_threads=16 \
  --set guppy.ipc_threads=2

Please note I have not tuned/optimised the above parameters, it's just the example ONT give. The above should work just fine, but you will want to make modifications to the GPU parameters based on the type of card that you have.

Any time you make modifications to these systemd services make sure you restart them:

systemctl restart minknow.service

At any time you can check on their status:

systemctl status minknow.service

guppyd.service

There is now a systemd service created by the package ont-guppyd-for-minknow. The current problem is that if we try to install that package it tries to pull and install the CPU version of Guppy (ont-guppy-cpu-for-minion) because they depend on each other.

As this is currently in the early stages we can still use the app_conf method above, but in an attempt to start adopting the newer approach I have created a manual method to get guppyd.service running.

The first thing is to create the file in the correct location:

sudo nano /lib/systemd/system/guppyd.service

We need to add the below content (it's the default) to the file, and then save:

[Unit]
Description=Service to manage the guppy basecall server.
Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018

[Service]
Type=simple
ExecStart=/opt/ont/guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --port 5555 -x cuda:all 
Restart=always
User=root
MemoryLimit=8G
MemoryHigh=8G
CPUQuota=200%

[Install]
Alias=guppyd.service
WantedBy=multi-user.target

As always, I like to check the contents of the newly created file:

$ cat /lib/systemd/system/guppyd.service

[Unit]
Description=Service to manage the guppy basecall server.
Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018

[Service]
Type=simple
ExecStart=/usr/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --port 5555 -x cuda:all
Restart=always
User=root
MemoryLimit=8G
MemoryHigh=8G
CPUQuota=200%

[Install]
Alias=guppyd.service
WantedBy=multi-user.target

NOTE: the above content can be modified based on your preferred settings for your GPU. Any arguments you can pass to Guppy can be added to the ExecStart= line, i.e. you could have --chunks_per_runner etc.

We also want to enable this service to auto start upon a system restart:

systemctl enable guppyd.service

Once that is done restart the service:

systemctl restart guppyd.service

installation script

DANGER: this is very experimental! At this stage I have only tested this script on a single laptop running Pop!_OS 21.04, so please proceed with caution. Saying this, there is very little that can actually drastically harm your system here, as long as you are comfortable with some commandline and Linux functions you should be OK.

This is an attempt to automate all of the above into a single bash script that can be run (with sudo) to make set up easier and faster.

I have created a script called minknow_hirsute_setup.sh which is maintained in a GitHub gist here.

It should be enough to save/copy the contents of that gist into a file called minknow_hirsute_setup.sh, give it execution rights (chmod +x minknow_hirsute_setup.sh) and then run it (./minknow_hirsute_setup.sh). All going well it should proceed through all the above steps and complete in a few minutes.

If anyone does give this a try please do contact me with any feedback, suggestions or issues, there is only so much testing one person can do!