Try   HackMD

Portable sequencing/sensor devices

'the point [ESR perspective]'

Assessing the potential of autonomous AI devices such as Nvidia’s Jetson Xavier and Nano to develop portable real-time DNA sequencers and other deployable sensors. The idea is to create a truly portable real-time sequencing device which can be easily taken into the 'field' and results are reported as the sequencing is running. This will use the Nanopore minION DNA sequencer alongside a cheap single board computer (Nvidia Jetson based) which can be powered by off-the-shelf rechargeable batteries. The Nvidia powered technology will allow real-time base calling of DNA making direct detection/identification in the field a real possibility.

Additionally, we envisage this as a totally modular device not just limited to DNA sequencing. A wide range of sensors can be attached (i.e. temperature, humidity, water flow, camera's, …) which can report back in real-time. This alongside the ability to run off portable batteries, with potential solar charging options, makes for an extremely versatile base unit.

The whole package is extremely cost effective and has potential use cases across ESRs business (Environmental, Forensics, Health), as well as in other areas such as community outreach and education. Example use cases:

  • Environmental: water ways monitoring, device could have water flow and temperature sensors attached continually sampling, as well as a minION which could be sequencing sampled contents in real-time.
  • Forensics: [long term potential] real-time sequencing at the scene, potential to develop an STR capture protocol for this purpose.
  • Health: both surveillance and diagnostics, in-house (clinic, hospital, practise) sequencing of bacteria/microbes, faster turn around, can tailor treatment sooner (i.e. target antibiotics).
  • Community: empowering and upskilling in genomics, making people aware that it exists, it's cool, and it can be useful. We would approach it from an environmental health angle, i.e. a question could be 'what is in our water?'; Where we dive for Paua what's there? Is it safe? Does it change how we engage with and/or care for ourselves, our environment, our community, … ?

Notes on Jetson Xavier setup

This started off initially as a bunch of notes, then started to morph into a blog post like structure. I now think that it's gotten so long that I might have to cut it into several different blog style posts. Something like:

  1. unboxing and setting up the hardware
  2. installing the software
  3. basecalling nanopore data

I will however leave this 'collection' available on my HackMD account.

overview

NVIDIA® JetsonTM Xavier is the latest addition to the Jetson platform. A powerful AI computer for autonomous machines, it gives you the performance of a GPU workstation in an embedded module under 30 W. With multiple operating modes at 10 W, 15 W, and 30 W, Jetson Xavier delivers more than 20X the performance and 10X the energy efficiency of its predecessor, the NVIDIA JetsonTM TX2.

unboxing

Look what got delivered to my desk… shiny new package!

specs

​​​​NVIDIA 512-Core Volta GPU with (64?) Tensor Cores
​​​​8-Core ARM v8.2 64-Bit CPU, 8 MB L2 + 4 MB L3
​​​​16 GB 256-Bit LPDDR4x Memory
​​​​32 GB eMMC 5.1 Flash Storage
​​​​(2x) NVDLA DL Accelerator Engines
​​​​7-Way VLIW Vision Processor
​​​​(2x) 4Kp60 | HEVC Video Encoder
​​​​(2x) 4Kp60 | 12-bit Video Decoder

Look at that power!

I've been quite impressed with the packaging of the Jetson products, nothing flashy but nice, clean and practical. The Xavier comes well protected.

Check out that heatsink! Big and shiny. There is a bit of heft to this device, but it's still tiny when you consider how much power is contained in such a small foot print.

Here's another angle, yep that's a full PCI express connection! The Xavier is able to take PCIe devices such as SSD drives, and Nvidia has confirmed that in the future you will be able to connect GPUs - very cool! You can also see the GPIO pins on the bottom edge. There is a pin breakout adapter included in the kit to make these more accessible.

This side of the Xavier has the bulk of the connectivity. From left to right:

  • usb-c
  • power jack
  • ethernet port
  • esata and usb port
  • HDMI port
  • micro-sd card slot?

As mentioned above there is also an additional usb-c port on the 'front' side, as well as a set of GPIO pins.

Other things included in the box (I'll update with a pic or two as well):

  • GPIO breakout pin
  • usb-c to usb-a (male) cable
  • usb-c to usb-a (female) adapter
  • power supply (USA layout, but can use adapter or replace the wall end)
  • getting started guide

installing ssd (hardware)

The Xaiver has a m.2 NVMe slot located under the large heat sink. To access this you have to remove the 4 screw located in the 'feet' of the device (2 in each foot).

Once you have removed the screw and the feet it should look like the below. Bonus image of the selected SSD.

Close up of the Western Digital Black 500GB NVMe SSD. We selected this drive as it has been tested and confirmed working previously on Jetson devices. Spoiler: it was fully supported out of the box - plug and play.

After the screws and feet are removed you are able to detach the heatsink/GPU module from the module board. You will need to apply a decent amount of pressure to pull these two components apart. WARNING: there is a fan(?) cable that connects the heatsink module to the board. Care should be taken as this is fragile. You can either detach the cable using something like a set of tweezers, or you can do as I did and place something under the heat sink to raise it up so there is no stress on the cable. This worked well for me, I figured that the cable wasn't in the way of the m.2 slot… but you have been warned.

To install the SSD first remove the screw that will hold the drive in place (see above). Next, locate the slot in the end of the SSD drive with the notch in the m.2 bay. Insert the card on a 45' angle (as below).

Close up of drive insertion below. Ensure that the drive is inserted enough so that when pushed down you are able to insert and tighten the screw to hold it in place. If the screw doesn't fit gently push the card futher into the slot untill it can be secured down.

Final installed ssd below.

Once the SSD has been installed you can reattach the heatsink module, being mindful of the tiny, fragile cable (ensure this doesn't get caught). There should be a nice 'click' as the heatsink seats back into place. Turn the Xavier over and relocate and screw down the two feet. Done.

installing wifi (hardware)

The Xaiver has an m.2e slot on the bottom of the board to install a compatible wifi module.

Example wifi module. Intel AX200NGW.

NOTE: this wifi card is only currently supported in kernel 5.1+. Currently trying to build a kernel module against the 4.9.15 that comes with Jetpack 4.2.2, not as yet successful.

Until we are able to either build a kernel module or purchase a compatible wifi module we are using a wiPi usb wifi dongle (this was plug and play).

Below is a close up of the of the m.2 socket on the bottom of the Jetson Xaiver.

The wifi module is installed in the same way as the m.2 nvme ssd. Locate the slot in the correct manner and insert the module at a 45' angle and push down. If properly inserted the screw should locate and fasten to hold the module in place.

Example of installed wifi module.

Close up of installed wifi module.

Note: remeber to attach the antenne

up and running

There are two main methods for getting the Xavier up and running with OS and software. The first one I tried was using L4T:

Jetson modules run Linux with NVIDIA® Tegra® Linux Driver Package (L4T), which provides the Linux kernel, bootloader, NVIDIA drivers, flashing utilities, sample filesystem, and more for the Jetson platform.

To do this I connected a mouse and keyboard, HDMI (to TV in this case) and then the provided power supply. Upon pressing the the power button (far left in the picture below).

NOTE: The button layout shown in the below image:

  • left: power
  • middle: recovery mode
  • right: reset

Above image borrowed from: http://linuxgizmos.com/tiny-carrier-unleashes-nvidia-xavier-power-for-robotics-and-ai/

Connected Xaiver showing Nvidia boot logo (below).

The power led is locate next to the 'front' usb-c port (below).

NOTE: this front usb-c port is used for flashing the device when using the Nvidia SDK manager. It can also be used as an additional usb device connection port. There is a usb-c to standard usb converter provided with the development kit.

Kernel modules being loaded on first boot (below).

Setup screen for 'manual' OS install (below).

Following the provided prompts you will be able to boot into an Ubuntu GUI using a deafult user and password (both are 'nvidia').

The above setup works but is lacking all the fun bells and whistles that make the Jetson family so exciting. They can probably be cobbled together manually, however there is an 'easier' approach that will provide a fully functioning environment with tools such as Cuda, TensorFlow etc etc already installed and operational.

getting sdk manager working on non-Ubuntu system

The other, generally more appropriate, method to get up and running is to use the SDK manager made available by Nvidia to flash the Xavier.

Ideally this is as simple as:

  • downloading the SDK manager
  • installing it on a host computer
  • make sure you have an Nvidia developer account
  • connect the Jetson device to the host computer (via front usb-c on Xavier)
  • running the SDK manager and selecting the appropriate options for your device
  • flashing the device
  • reboot the device and start having fun!

WARNING: this was not so easy…

…running the SDK manager…

Nvidia provide the Jetson family SDK manager made on their website. It's a Linux native piece of software (nice), which is stated to require Ubuntu 16.10 or 18.04. OK, Ubuntu is Debain right? So I downloaded the SDK manager and it installed just fine on my personal laptop running Debain Unstable (Siduction). It also ran just fine. So far so good.

DANGER: unless you know exactly what you are doing and you want dev tools etc locally, make sure to deselect the option to install 'host machine' (local computer) drivers/CUDA. If you select this option you run the risk of installing video drivers/CUDA libs that aren't compatible with your system and then experience issues.

The problem arose when selecting the Jetpack version for the selected device, see below image:

A nice helpful image saying "not supported on Linux"… sigh. After a bit of forum searching it turns out that the host OS MUST be Ubuntu for the tool to work… bigger sigh.

For those of you thinking of spinning up an Ubuntu VM, nope doesn't work. I considered it but saw the failed attempts in the forums. Turns out VMs don't handle USB device connections in a way that is suitable to flash Jetson devices with this SDK manager.

In the forums I noted a couple of people that were able to 'trick' the SDK by editing their OS release files. Really Nvidia?!

editing os-release

Running Debian Sid, have to 'trick' SDK manager

  1. create a backup of /usr/lib/os-release
sudo cp /usr/lib/os-release /usr/lib/os-release.bkup
  1. edit/create file /usr/lib/os-release:
sudo nano /usr/lib/os-release
  • replace contents with below:
NAME="Ubuntu"
VERSION="18.04 (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04"
VERSION_ID="18.04"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  1. export env variable
export LSB_OS_RELEASE=/usr/lib/os-release
  1. run nvidia jetson sdk manager from same terminal (see section below for setup if not done previously)
sdkmanager
  1. replace with original file when done
sudo cp /usr/lib/os-release.bkup /usr/lib/os-release

NOTE: could try having another file, such as /usr/lib/os-release-sdk, and export the env as such:

export LSB_OS_RELEASE=/usr/lib/os-release-sdk

it might work?

Nvidia Jetson SDK manager

[section under construction]

need online nvidia dev account and login once sdk manager is installed (and the above 'tweak' is applied if you're not on Ubuntu) you should be able to select a jetpack image in the sdk choose the most recent image change download paths if needed I chose to download and flash later confirm that all packages and OS download

connecting jetson xavier

[section under construction]

initial connection needs to be in recovery mode push and hold the middle button 'recovery' push and hold the power button release both buttons check to see if entry is present using lsusb - look for Nvidia entry

WARNING: ensure there are no other usb devices attached!! There seems to be an issue that pops up around an initial oversaturation of the USB bus.

manual flash of OS

[section under construction]

navigate to dir containing flash.sh this will be where you downloaded the OS and packages to using the SDK manager

sudo ./flash.sh jetson-xavier mmcblk0p1

NOTE: if the flash doesn't work and you get an error about probing board and not found, try moving the usb cable to a different usb port. This worked for me. Also try a different usb cable if it still doesn't work.

install packages using sdk manager

[section under construction]

You should now be able to boot out of recovery by pushing the 'reset' button. Will load into Ubunut GUI, go through standard setup (i.e. locale, user, pw). In the sdk manager you can now use the 'automatically' option to flash/install packages such as cuda and tensorflow (you shouldn't have to change any of the settings).

performance monitors

There are a few different means to monitor various aspects of the system on Jetson boards. These range from the simple through to the more complex and very detailed:

setting up nvme ssd

[section under construction]

DANGER: these are notes that I quickly jotted down for future reference. The general flow is sound but some of the drive locations etc might change across different devices. Please ensure you are comfortable working with either gparted or CLI disk tools before proceeding.

If you'd like a refresher this guide is quite good.

installed gparted:

sudo apt install gparted

Created a 'gpt' partition on the ssd (/dev/nvme0n1p1) Formatted to 'ext4'

Create a root directory for mounting:

sudo mkdir /xavier_ssd

Mount and make writable:

WARNING: the below is making an assumption that



sudo mount /dev/nvme0n1p1 /xavier_ssd/
sudo chmod 777 /xavier_ssd/

Edit /etc/fstab to allow automount first create a backup:

sudo cp /etc/fstab /etc/fstab.bkup

install nano to edit text files

sudo apt install nano

edit fstab:

Note: a drives UUID is assigned when you format and partition it. Once this is done you can find it in gparted by selecting the correct drive and right clicking on it, selected information and you'll be able to copy the UUID. This can then be pasted into fstab as per below.




sudo nano /etc/fstab
# add entry
UUID=<UUID>	/xavier_ssd	ext4	defaults	0	2

unmount:

sudo umount /xavier_ssd

test mount using fstab

sudo mount -a

try creating a test file

# touch /xavier_ssd/test.txt

If run ls you should see the created file on the newly mounted SSD:




ls /xavier_ssd

test.txt

testing CUDA

As we're wanting to run software such as Guppy from ONT we'll need to ensure Cuda and Tensorflow are installed and working as expected.

Navigate to ssd and copy cuda samples dir across:



cd /xavier_ssd
cp -r /usr/local/cuda/samples/ .

Compile the devicequery tool:



cd samples/1_Utilities/deviceQuery
make

Run this:














































../../bin/aarch64/linux/release/deviceQuery 
../../bin/aarch64/linux/release/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Xavier"
  CUDA Driver Version / Runtime Version          10.0 / 10.0
  CUDA Capability Major/Minor version number:    7.2
  Total amount of global memory:                 15697 MBytes (16459345920 bytes)
  ( 8) Multiprocessors, ( 64) CUDA Cores/MP:     512 CUDA Cores
  GPU Max Clock rate:                            1500 MHz (1.50 GHz)
  Memory Clock rate:                             1377 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 524288 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.0, CUDA Runtime Version = 10.0, NumDevs = 1
Result = PASS

Cuda looks like it's installed and running correctly.

testing tensorflow

Open a Python terminal and enter the following lines of code:







$ python3
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2019-12-03 14:45:58.715871: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0

Some further typing:













































>>> hello = tf.constant("hello TensorFlow!")
>>> sess=tf.Session() 
2019-12-03 14:47:41.860063: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-12-03 14:47:41.866189: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:41.866361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.5
pciBusID: 0000:00:00.0
2019-12-03 14:47:41.866469: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 14:47:41.897754: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 14:47:41.923847: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-03 14:47:41.959857: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-03 14:47:41.998161: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-03 14:47:42.020087: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-03 14:47:42.092069: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-03 14:47:42.092417: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:42.092711: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:42.092802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-03 14:47:42.113788: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2019-12-03 14:47:42.114982: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3b3088f0 executing computations on platform Host. Devices:
2019-12-03 14:47:42.115064: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-12-03 14:47:42.191865: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:42.192258: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3b929b10 executing computations on platform CUDA. Devices:
2019-12-03 14:47:42.192315: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Xavier, Compute Capability 7.2
2019-12-03 14:47:42.192710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:42.192811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.5
pciBusID: 0000:00:00.0
2019-12-03 14:47:42.192890: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 14:47:42.192969: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 14:47:42.193053: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-03 14:47:42.193116: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-03 14:47:42.193179: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-03 14:47:42.193235: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-03 14:47:42.193271: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-03 14:47:42.193386: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:42.193595: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:42.193658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-03 14:47:42.193762: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 14:47:44.340915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-03 14:47:44.341028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-12-03 14:47:44.341097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-12-03 14:47:44.341584: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:44.341824: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 14:47:44.341967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 501 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)

To verify your installation just type:



>>> print (sess.run(hello))
b'hello TensorFlow!'

…or, from the shell…





python3 -c 'import tensorflow as tf; print(tf.__version__)'
2019-12-03 15:24:44.136715: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
1.14.0

Again, Tensorflow installation looks good (apart from a lot of 'spammy' messages that get printed!).

Basecalling with Guppy

UPDATE (13th Dec 2019): I have been doing some more extensive benchmarking with the Xavier and guppy basecalling. This is documented in this gist: https://gist.github.com/sirselim/2ebe2807112fae93809aa18f096dbb94

In the above I have identified a series of parameters that appears to provide the optimal speed increase.

We initially had a lot of issues trying to get a version of Guppy that was compiled for an arm-based processor. ONT offer binaries in their community portal, however arm-based versions aren't on offer. Finally after several attempts we obtained a version from ONT that was able to be installed directly on the xavier, allowing us to demonstrate the ability of this device to provide a significant boost in basecalling speed.

The test data set was generated on a flongle flow cell, which arrived with very few active pores. It did still manage to generate ~500Mbps of data, which is more than enough to run some tests with.

Guppy version:



$ guppy_basecaller --version
: Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 3.4.1+213a60d0

Setting the run:

guppy_basecaller --compress_fastq -c dna_r9.4.1_450bps_fast.cfg -i flongle_fast5_pass/ -s flongle_test -x 'auto' --recursive

Parameters used:

  • --compress_fastq - to compress the output sequence data
  • -c dna_r9.4.1_450bps_fast.cfg - the fast base caller model config file
  • -x 'auto' - automatically identify the GPU device
  • --recursive - recursive search through input dir for all fast5 files

Note: we'll look at recalling with the high accuracy model at a later date. Here I wanted to compare apples with apples in terms of the work that was performed on the production server.

Running and results

Basecalling up and running on the Xavier!

Check out that 100% GPU usage:

Confirming that it is indeed Nvidia Jetson Xavier hardware:

…and the results are in:

So the Xavier finished basecalling the data from this flongle run in ~478 seconds or a hair under 8 mins. To put this into context for those that aren't excited straight off the bat:

  • this run being basecalled by CPUs took 36 hours
  • when run on our GPU server (2x Tesla V100) is took ~280 seconds

This means that the Jetson Xavier was able to basecall this data at only twice as slow a rate as 2 extremely powerful and expensive cards - that's insane!

IMPORTANT: there is a lot more testing and potential optimisation to be done on both the Xavier and our other GPU platforms. We haven't looked at modifying the default parameters etc, and I'm not convinced that both V100 GPUs were really working at full capacity. Watch this space.

fan speed

not sure if there is a direct way to control the fan speed, but I noticed in my testing that when under 100% GPU load the fan didn't seem to speed up from ~30%. the GPU temp didn't seem to rise above ~60'C but it also looked like it may have throttled a little at times.

Some searching turned up the below as a way to 'force' the fan speed by adjusting pwm between 0 (off) and 255 (100%, fully on).









# enter root
sudo su
# turn to 100%
echo 255 | tee /sys/devices/pwm-fan/target_pwm
# turn off
echo 0 | tee /sys/devices/pwm-fan/target_pwm
# about 30%
echo 77 | tee /sys/devices/pwm-fan/target_pwm

To check the current fan pwm:

sudo cat /sys/devices/pwm-fan/cur_pwm

Guppy rerun at 100% fan speed

I decided to see what would happen if we locked the fan at 100% for a Guppy run.

Here's a screen shot of jtop about a third of the way through:

You'll notice that the GPU temp is ~51'C in that shot. The highest I saw it spike was 56'C and it actually leveled off around the 47-48'C mark for the majority of the run. That's much better than the ~60'C cap that seemed to be enforced when the fan was being automatically controlled and would ramp past 30% speed. I also noted that there was no apparent thermal throttling of the GPU when running the fan at 100%.

Here are the run stats:
























ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  160
records per file:   4000
fastq compression:  ON
num basecallers:    1
gpu device:         auto
kernel path:        
runners per device: 8

Found 105 fast5 files to process.
Init time: 731 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 479127 ms, Samples called: 3970269916, samples/s: 8.28647e+06
Finishing up any open output files.
Basecalling completed successfully.

Again we come in around the 479 second mark for this data set - not bad at all!

compression of fast5 using vbz

ONT have a newish compression available for fast5 files, vbz. This is implemented in ont-fast5-api and will be supported in future versions of MinKnow (it's supported in Guppy >= 3.4).

The Jetson Xavier doesn't have pip2 installed by default and it seems this api will only work on python2. So first need to install pip2 (warning: this may cause issues, but there is no arm conda yet so…):

sudo apt install python-pip

Once installed you should be able to install the ONT api:

pip2 install ont-fast5-api

To test compression:

compress_fast5 -i INPUT_FAST%_DIR -s COMPRESSED_DIR -c vbz -t 6 --recursive

Warning: Can't currently get this working on the Xavier. Hence this is a work in progress.

[section under construction]

Last changed by 
 
Miles Benton
At the bottom of the world in New Zealand, I am a computational geneticist interested in all facets of biology and technology, particularly GPUs and Nanopores.
0
1
1428
Published on HackMD