# DPDK - Installation and Optimization
###### tags: `DPDK`

Kevin Chiu
Last update: 2021.02.05

## Introduction to DPDK
DPDK is a set of libs used for accelerating packet processing.
Applications are able to reach higher throughput and lower latency with the features DPDK provides as shown below:
**1. Kernel bypass:**
A device mapping is created in the user space by the **UIO module** so applications are able to access NICs directly and fast without the overhead of context switch and data copy between user space and kernel space.
**2. Interrupt handling:**
The specific CPU is assigned to run the **Poll Mode Driver (PMD)** that constantly poll the NIC for new packets instead of the NIC raising interrupt to the CPU.
**3. Memory management:**
* **Buffer Managers**: to optimize the allocation/deallocation of the network buffer queues.
* **NUMA awareness**: to avoid expensive memory operations across memory nodes.
* **Hugepages**: to optimise physical-to-virtual page mappings within the CPU's TLB.


## Platform
* Model: SCB-1921B-AA1
* CPU: Intel® Xeon® Gold 6242R Processor *2
* OS: Ubuntu 18.04, kernel 4.15.0-123-generic
* DPDK version: 19.11.4/16.11.1
* GCC version: 7.5.0-3
* ICC version: 19.1.2
* meson version: 0.56.0
* ninja version: 1.10.0.git.kitware.jobserver-1

## Prerequisite
**System requirements for compilation of the DPDK**
* General development tools (with GCC version 4.9 or later)

# apt install build-essential

* Python (python3.5 or later)

# apt isntall python3

* Library for handling NUMA

# apt install libnuma-dev

* Meson v0.47.1+ & ninja (only needed for meson build)

# pip3 install meson ninja

* Other tools: pkg-config

# apt install pkg-config

**System requirements for running DPDK applications:**
* Linux kernel version 3.16 or later
* glibc version 2.7 or later
* Kernel configuration
* HUGETLBFS
* PROC_PAGE_MONITOR support
* HPET and HPET_MMAP configuration options should also be enabled if HPET support is required. See the section on High Precision Event Timer (HPET) Functionality for more details.

## Install DPDK
### Download and decompress
Download DPDK: (or download from: https://core.dpdk.org/download/)

# wget http://fast.dpdk.org/rel/dpdk-19.11.4.tar.xz
# tar xvJf dpdk-19.11.4.tar.xz
# cd dpdk-stable-19.11.4

### Install DPDK by script
Run the setup script and choose the compiling options according to your target.

# cd ~/dpdk-stable-19.11.4/usertools
# ./dpdk-setup.sh

Options to choose:
1. option: [41] x86_64-native-linux-gcc
2. option: [45] Insert IGB UIO module
3. option: [62] Exit Script

The DPDK module /dpdk-gcc-19.11.4/x86_64-native-linux-gcc/kmod/igb_uio.ko will be created once you finish compiling.

> Note: if you want to build DPDK with ICC, you should choose: [42] x86_64-native-linux-icc

### Install DPDK by meson
Create a folder named "build" where you build DPDK module with meson tool.

# cd ~/dpdk-stable-19.11.4/
# meson build
# cd build
# ninja
# ninja install
# ldconfig

The DPDK module /dpdk-gcc-19.11.4/build/kernel/linux/igb_uio/igb_uio.ko will be created once you finish compiling.

## Optimization
### Optimization options are classified by the combination of:
* DPDK version
DPDK 19.11.4 or DPDK 16.11.1
* Compiler
Use GCC(GNU Compiler Collection) or ICC(Intel C++ Compiler) to build DPDK.
* Compiler optimization options
1. LTO (Link Time Optimization)
LTO is a whole program optimization option which can be enabled in Makefiles.
2. AVX2
This is an option in Makefiles which is specifically for AVX2 architecture optimization.
4. AVX512
This is an option in Makefiles which is specifically for AVX512 architecture optimization
5. Fast
A combination option of -Ofast, -ipo, -static (for static linking) and -xHost.
6. Meson
Meson is a compiling tool just similar to `make`. And it's also officially recommended from DPDK 19.11 LTS on.

### Test options
1. dpdk-16.11.1-gcc
2. dpdk-19.11.4-gcc
3. dpdk-19.11.4-gcc-lto
4. dpdk-19.11.4-gcc-meson
5. dpdk-19.11.4-icc
6. dpdk-19.11.4-icc-avx2
7. dpdk-19.11.4-icc-avx2+lto
8. dpdk-19.11.4-icc-avx512
9. dpdk-19.11.4-icc-fast
10. dpdk-19.11.4-icc-lto
11. dpdk-19.11.4-icc-meson

### Example
Here's an example we add the optimization option in the Makefile:

# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Cavium, Inc
#

include $(RTE_SDK)/mk/rte.vars.mk

#
# library name
#
LIB = librte_mempool_octeontx.a

CFLAGS += $(WERROR_FLAGS)
CFLAGS += -xCORE-AVX2 -ipo -c
CFLAGS += -I$(RTE_SDK)/drivers/common/octeontx/
CFLAGS += -DALLOW_EXPERIMENTAL_API

EXPORT_MAP := rte_mempool_octeontx_version.map

In the line `CFLAGS += -xCORE-AVX2 -ipo -c`
* `-xCORE-AVX2`: the optimization option for AVX2.
* `-ipo -c`: the optimization option for LTO.

### Makefile list
These are the Makefiles you should edit if you want to add optimization options while building DPDK.

app/test-crypto-perf/Makefile
app/test/Makefile
app/test-compress-perf/Makefile
app/test-sad/Makefile
app/test-pmd/Makefile
app/test-pipeline/Makefile
app/test-bbdev/Makefile
app/test-acl/Makefile
app/pdump/Makefile
app/test-cmdline/Makefile
app/test-eventdev/Makefile
app/proc-info/Makefile
drivers/raw/octeontx2_dma/Makefile
drivers/raw/ioat/Makefile
drivers/raw/dpaa2_qdma/Makefile
drivers/raw/ntb/Makefile
drivers/raw/skeleton/Makefile
drivers/raw/dpaa2_cmdif/Makefile
drivers/mempool/octeontx/Makefile
drivers/mempool/bucket/Makefile
drivers/mempool/stack/Makefile
drivers/mempool/octeontx2/Makefile
drivers/mempool/dpaa2/Makefile
drivers/mempool/ring/Makefile
drivers/mempool/dpaa/Makefile
drivers/net/liquidio/Makefile
drivers/net/bnxt/Makefile
drivers/net/tap/Makefile
drivers/net/af_packet/Makefile
drivers/net/i40e/Makefile
drivers/net/virtio/Makefile
drivers/net/vhost/Makefile
drivers/net/vmxnet3/Makefile
drivers/net/axgbe/Makefile
drivers/net/cxgbe/Makefile
drivers/net/iavf/Makefile
drivers/net/octeontx/Makefile
drivers/net/octeontx2/Makefile
drivers/net/ice/Makefile
drivers/net/enic/Makefile
drivers/net/atlantic/Makefile
drivers/net/e1000/Makefile
drivers/net/bonding/Makefile
drivers/net/null/Makefile
drivers/net/fm10k/Makefile
drivers/net/netvsc/Makefile
drivers/net/nfp/Makefile
drivers/net/thunderx/Makefile
drivers/net/vdev_netvsc/Makefile
drivers/net/memif/Makefile
drivers/net/ark/Makefile
drivers/net/qede/Makefile
drivers/net/ifc/Makefile
drivers/net/softnic/Makefile
drivers/net/dpaa2/Makefile
drivers/net/ixgbe/Makefile
drivers/net/ring/Makefile
drivers/net/kni/Makefile
drivers/net/avp/Makefile
drivers/net/failsafe/Makefile
drivers/net/ena/Makefile
drivers/net/enetc/Makefile
drivers/net/dpaa/Makefile
drivers/bus/fslmc/Makefile
drivers/bus/pci/Makefile
drivers/bus/vdev/Makefile
drivers/bus/ifpga/Makefile
drivers/bus/vmbus/Makefile
drivers/bus/dpaa/Makefile
drivers/event/octeontx/Makefile
drivers/event/octeontx2/Makefile
drivers/event/dsw/Makefile
drivers/event/sw/Makefile
drivers/event/dpaa2/Makefile
drivers/event/opdl/Makefile
drivers/event/skeleton/Makefile
drivers/event/dpaa/Makefile
drivers/crypto/virtio/Makefile
drivers/crypto/nitrox/Makefile
drivers/crypto/octeontx/Makefile
drivers/crypto/octeontx2/Makefile
drivers/crypto/null/Makefile
drivers/crypto/caam_jr/Makefile
drivers/crypto/dpaa_sec/Makefile
drivers/crypto/dpaa2_sec/Makefile
drivers/crypto/scheduler/Makefile
drivers/baseband/turbo_sw/Makefile
drivers/baseband/fpga_lte_fec/Makefile
drivers/baseband/null/Makefile
drivers/common/octeontx2/Makefile
drivers/common/qat/Makefile
drivers/common/dpaax/Makefile
drivers/common/cpt/Makefile
drivers/compress/octeontx/Makefile
lib/librte_eal/linux/eal/Makefile
lib/librte_cryptodev/Makefile
lib/librte_ipsec/Makefile
lib/librte_gso/Makefile
lib/librte_distributor/Makefile
lib/librte_bpf/Makefile
lib/librte_ethdev/Makefile
lib/librte_metrics/Makefile
lib/librte_pipeline/Makefile
lib/librte_rib/Makefile
lib/librte_reorder/Makefile
lib/librte_bbdev/Makefile
lib/librte_eventdev/Makefile
lib/librte_timer/Makefile
lib/librte_ip_frag/Makefile
lib/librte_pci/Makefile
lib/librte_port/Makefile
lib/librte_pdump/Makefile
lib/librte_fib/Makefile
lib/librte_table/Makefile
lib/librte_rawdev/Makefile
lib/librte_net/Makefile
lib/librte_stack/Makefile
lib/librte_cfgfile/Makefile
lib/librte_kvargs/Makefile
lib/librte_mempool/Makefile
lib/librte_ring/Makefile
lib/librte_lpm/Makefile
lib/librte_bitratestats/Makefile
lib/librte_vhost/Makefile
lib/librte_cmdline/Makefile
lib/librte_efd/Makefile
lib/librte_rcu/Makefile
lib/librte_sched/Makefile
lib/librte_acl/Makefile
lib/librte_jobstats/Makefile
lib/librte_hash/Makefile
lib/librte_compressdev/Makefile
lib/librte_kni/Makefile
lib/librte_meter/Makefile
lib/librte_gro/Makefile
lib/librte_latencystats/Makefile
lib/librte_member/Makefile
lib/librte_mbuf/Makefile
lib/librte_flow_classify/Makefile
lib/librte_security/Makefile
lib/librte_power/Makefile

### Testing environment
Make sure to setup the environment and insert module by the following commands before testing.

mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
echo 1 > /proc/sys/net/ipv4/ip_forward
insmod ./dpdk-gcc-19.11.4/x86_64-native-linux-gcc/kmod/igb_uio.ko

After that you can start testing.

## Reference
[1] https://doc.dpdk.org/guides-19.11/linux_gsg/quick_start.html
[2] https://www.packetflow.co.uk/what-is-dpdk/