# DPDK introduction and building note (later than DPDK ver.20) ###### tags: `DPDK` ## Brief introduction of DPDK Point of DPDK: kernel-bypass networking. Most of network latency is caused by application, so there is a radical way to prevent latency caused by application: bypass kernel. With DPDK, application can send package to userspace TCP/IP stack directory. NIC send and receive packages by userspace TCP/IP stack (usermode?) directory. (I am not sure) ## Platform Mother Board: CB-1932 OS: RHEL 8.2, kernel 4.18.0 DPDK: 20.11.3 LTS ## Install DPDK ### Required tools ```bash= # install 'build_essential_tool of RHEL' yum groupinstall 'Development Tools' # install python3 yum install python3 # install numactl yum install numactl # install pip yum install python3-pip # install meson and ninja pip3 install meson ninja # install pkg-config yum install pkg-config # change grub vim /etc/sysconfig/grub ``` ### Download DPDK source: https://core.dpdk.org/download/ ### Unpacking the file ```bash= tar Jxvf dpdk-20.11.3.tar.xz cd dpdk-stable-20.11.3 ``` In this example, I put DPDK source file at /home/user. ### Compile DPDK ```bash= meson -Dexamples=all build cd build ninja ninja install ldconfig ``` ### Download igb_uio.ko from git provided by DPDK ```bash= cd /root yum install git git clone git://dpdk.org/dpdk-kmods dpdk_igb_uio_driver cd ./dpdk_igb_uio_driver/linux/igb_uio make clean && make ``` ### Setting hugepages and isolcpus ```javascript= $ sudo vim /etc/default/grub ``` Add these content into the end of GRUB_CMDLINE_LINUX_DEFAULT. ```javascript= default_hugepagesz=1G hugepagesz=1G hugepages=8 isolcpus=16,17,18,19 //isolcpus means OS would not run process on those chosen cores ``` After that... ```javascript= # update-grub # reboot ``` After reboot, check hugepages has been set. ```javascript= cat /proc/meminfo ``` Use hugepages after hugepages set. ```javascript= $ sudo mkdir /mnt/huge $ mount -t hugetlbfs nodev /mnt/huge ``` Disable firewall. ```javascript= service iptables stop ``` Enable IP forward. ```javascript= $ sudo sysctl net.ipv4.ip_forward=1 ``` Install kernel module. ```javascript= $ sudo modprobe uio $ sudo insmod /home/user/dpdk-stable-19.11.4/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko ``` Check lan card information and bind lan card with igb_uio module. ```javascript= # cd /home/user/dpdk-stable-19.11.4/usertools $ sudo ./dpdk-devbind.py --status //to know lan card status $ sudo ./dpdk-devbind.py -b igb_uio 18:00.0 //to bind lan card in dpdk ``` ## Using l3fwd to test. ```javascript= # cd /home/user/dpdk-stable-19.11.4/examples/l3fwd/build $ sudo ./l3fwd -c f0001 -n 4 -- -P -p 0xf --config="(0,0,16),(1,0,17),(2,0,18),(3,0,19)" ``` ※-c f0001 => 1111 0000 0000 0000 0001, 1 means binded, 0 means unbinded, 0th core must open(000"1"), 16~19th core binded for l3fwd(1111), others closed("0000 0000 0000 000"1) ※-n 4 => 4 RAM channels ※-P => Enable promiscuous mode? ※-p port amount to configure, 0xf means 4 ports ※config=(0,0,16)... => (no. of lan port, queue, CPU core no.) ## Using testpmd to test (l2) 1. Install kernel module. ```javascript= $ sudo modprobe uio $ sudo insmod /home/user/dpdk-stable-19.11.4/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko ``` 2. Mount hugepages ```javascript= $ sudo mkdir /mnt/huge $ mount -t hugetlbfs nodev /mnt/huge ``` 3. Check lan card information and bind lan card with igb_uio module. ```javascript= # cd /home/user/dpdk-stable-19.11.4/usertools $ sudo ./dpdk-devbind.py --status //to know lan card status $ sudo ./dpdk-devbind.py -b igb_uio 18:00.0 //to bind lan card in dpdk ``` 4. Run testpmd ```javascript= # cd /home/user/dpdk-stable-19.11.4/x86_64-native-linux-gcc/app $ sudo ./testpmd -c 1f -n 4 -- -i --portmask=0xf --nb-cores=4 ``` ※-c 1f => 0001 1111, 1 means binded, 0 means unbinded, 0th core must open(000"1"), 2~5th core binded, so result: 1f ※-n 4 => 4 RAM channels ※-portmask => port amount to configure, 0xf means 4 ports ※--nb-cores => how many CPU cores binded for testpmd ## Notice Every single version of DPDK, path of tool, kernel module such as l3fwd, igb_uio.ko dpdk-setup.sh testpmd and others are different Use find command in linux to find path of the tool you need ## Source https://hackmd.io/@sysprog/linux-dev-review https://www.slideshare.net/garyachy/dpdk-44585840 https://www.net.in.tum.de/fileadmin/bibtex/publications/theses/2014-gallenmueller-high-speed-packet-processing.pdf https://core.dpdk.org/download/ ## Special thanks Rick Xiao