# 2020q1 Homework8 (khttpd)
contributed by < `IepIweidieng` >
## 完成清單
- [ ] [自我檢查清單](https://hackmd.io/@sysprog/linux2020-khttpd#-%E8%87%AA%E6%88%91%E6%AA%A2%E6%9F%A5%E6%B8%85%E5%96%AE)
- [x] 參照 [fibdrv 作業說明](https://hackmd.io/@sysprog/linux2020-fibdrv) 裡頭的「Linux 核心模組掛載機制」一節，解釋 `$ sudo insmod khttpd.ko port=1999` 這命令是如何讓 `port=1999` 傳遞到核心，作為核心模組初始化的參數呢？
- `insmod` 會讓 kernel 執行函式 `finit_module`，而該函式有個參數 `uargs`：([kernel/module.c](<https://elixir.bootlin.com/linux/v4.18/source/kernel/module.c>))
```c=3855
SYSCALL_DEFINE3(finit_module, int, fd, const char __user *, uargs, int, flags)
{
[...]
pr_debug("finit_module: fd=%d, uargs=%p, flags=%i\n", fd, uargs, flags);
[...]
return load_module(&info, uargs, flags);
}
```
- - - `load_module`:
```c=3654
- /* Allocate and load the module: note that size of section 0 is always
zero, and we rely on this for optional sections. */
static int load_module(struct load_info *info, const char __user *uargs,
int flags)
{
```
- - - 而 `uargs` 只出現一次：
```c=3735
/* Now copy in args */
mod->args = strndup_user(uargs, ~0UL >> 1);
if (IS_ERR(mod->args)) {
err = PTR_ERR(mod->args);
goto free_arch_cleanup;
}
```
- - - 接下來則使用到 `mod->args`：
```c=3756
/* Module is ready to execute: parsing args may do that. */
after_dashes = parse_args(mod->name, mod->args, mod->kp, mod->num_kp,
-32768, 32767, mod,
unknown_module_param_cb);
if (IS_ERR(after_dashes)) {
err = PTR_ERR(after_dashes);
goto coming_cleanup;
} else if (after_dashes) {
pr_warn("%s: parameters '%s' after `--' ignored\n",
mod->name, after_dashes);
}
```
- - - `parse_args`: ([kernel/params.c](<https://elixir.bootlin.com/linux/v4.18/source/kernel/params.c>))
```c=163
/* Args looks like "foo=bar,bar2 baz=fuz wiz". */
char *parse_args(const char *doing,
char *args,
const struct kernel_param *params,
unsigned num,
s16 min_level,
s16 max_level,
void *arg,
int (*unknown)(char *param, char *val,
const char *doing, void *arg))
{
```
- - - `params` 的指向型別上的 `const` 不是說不能改變 `params` 所有直接或間接指向的內容，因為裡面指向非 `const` 型別的指標成員可以透過 dereference 修改其指向內容，只是不能更改指標指向的位置而已。
`struct kernel_param`: ([include/linux/moduleparam.h](<https://elixir.bootlin.com/linux/v4.18/source/include/linux/moduleparam.h>))
```c=71
struct kernel_param {
const char *name;
struct module *mod;
const struct kernel_param_ops *ops;
const u16 perm;
s8 level;
u8 flags;
union {
void *arg;
const struct kparam_string *str;
const struct kparam_array *arr;
};
};
```
- - [ ] 參照 [CS:APP 第 11 章](https://hackmd.io/s/ByPlLNaTG)，給定的 kHTTPd 和書中的 web 伺服器有哪些流程是一致？又有什麼是你認為 kHTTPd 可改進的部分？
- [ ] `htstress.c` 用到 [epoll](http://man7.org/linux/man-pages/man7/epoll.7.html) 系統呼叫，其作用為何？這樣的 HTTP 效能分析工具原理為何？
- [ ] 在 GitHub 上 fork [khttpd](https://github.com/sysprog21/khttpd)，目標是將 fibdrv 作業的成果整合進 kHTTPd。過程中應一併完成以下:
1. [ ] 當來自 web 客戶端 (即網頁瀏覽器或類似的軟體) 請求的網址為 `/fib/N` 時 ($N$ 是自然數，最大可到 $2^{31} - 1$)，例如 `$ wget http://localhost:8081/fib/10` 應該要讓 web 客戶端得到 $Fibonacci(10)$ 即 `55` 這個字串，這個實作需要考慮到大數運算 $\to$ 不要小看這個需求，由於大數運算無可避免會用到更大的記憶體空間和更長的執行時間，你要考慮手動安插 `reschedule()`, `preempt_enable()`, `preempt_disable()`
2. [ ] 自 [Facebook 討論串](https://www.facebook.com/groups/system.software2020/permalink/345124729755066/) 找出合適的 Fibonacci 運算檢驗程式，用以比對上述 (1) 的運算結果，並且要能詳實記錄和分析處理時間 (從 HTTP 請求到 kHTTPd 回應)
3. [ ] 指出 kHTTPd 實作的缺失 (特別是安全疑慮) 並予以改正
4. [ ] 引入 [Concurrency Managed Workqueue](https://www.kernel.org/doc/html/v4.15/core-api/workqueue.html) (cmwq)，改寫 kHTTPd，分析效能表現和提出改進方案，可參考 [kecho](https://github.com/sysprog21/kecho)
5. [ ] 參照 [in-kernel HTTP server](https://hackmd.io/@sysprog/kernel-web-server)，學習其中分析和實作手法，從而改進 kHTTPd
- [ ] 彙整同學們的共筆材料，並依據作業要求予以補強
## 環境準備
先準備支援 Linux kernel module 的環境，而 WSL 不符條件。
但 WSL 2 支援。
```cmd
Microsoft Windows [版本 10.0.19041.330]
(c) 2020 Microsoft Corporation。著作權所有，並保留一切權利。
C:\Users\user>wsl --list --verbose
NAME STATE VERSION
* Ubuntu Running 1
C:\Users\user>wsl --set-version Utuntu 2
沒有所提供名稱的發佈。
C:\Users\user>wsl --set-version Ubuntu 2
正在進行轉換，這可能需要幾分鐘的時間...
有關 WSL 2 的主要差異詳細資訊，請瀏覽 https://aka.ms/wsl2
轉換完成。
C:\Users\user>
```
### 系統資訊
```shell
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
$ uname -a
Linux IID-PC-acer-N17Q3 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release
No LSB modules are available.
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
Vendor ID: AuthenticAMD
CPU family: 21
Model: 112
Model name: AMD A9-9420 RADEON R5, 5 COMPUTE CORES 2C+3G
Stepping: 0
CPU MHz: 2994.380
BogoMIPS: 5988.76
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 32K
L1i cache: 96K
L2 cache: 1024K
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw xop fma4 topoext ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 xsaveopt virt_ssbd arat
$
```
### 準備 kernel module 所需環境
#### 檢查核心版本
```shell
$ uname -r
4.19.104-microsoft-standard
```
```shell
$ sudo aptitude install linux-headers-$(uname -r)
[sudo] password for iid:
Couldn't find any package whose name or description matched "linux-headers-4.19.104-microsoft-standard"
Unable to apply some actions, aborting
```
沒有現成套件。
#### 手動設定 kernel module 環境
Clone WSL 2 所使用的 kernel 的 repository。只擷取需要的 history。
```shell
$ git clone --depth=1 --branch=4.19.104-microsoft-standard https://github.com/microsoft/WSL2-Linux-Kernel.git
Cloning into 'WSL2-Linux-Kernel'...
remote: Enumerating objects: 65600, done.
remote: Counting objects: 100% (65600/65600), done.
remote: Compressing objects: 100% (60527/60527), done.
remote: Total 65600 (delta 5197), reused 31211 (delta 4157), pack-reused 0
Receiving objects: 100% (65600/65600), 177.72 MiB | 1.31 MiB/s, done.
Resolving deltas: 100% (5197/5197), done.
Note: checking out '506f7fa21ff5fe49bc99e173f59c965bbc7ed0c7'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
Checking out files: 100% (61781/61781), done.
$ cd WSL2-Linux-Kernel/
```
安裝編譯 kernel 所需要的套件。
```shell
WSL2-Linux-Kernel$ sudo aptitude install build-essential flex bison libssl-dev libelf-dev
[...]
```
複製系統的 kernel config 過來。
```shell
WSL2-Linux-Kernel$ zcat /proc/config.gz > .config
```
然後只編譯及安裝 `modules` 相關程式。
```shell
WSL2-Linux-Kernel$ make modules
Makefile:590: include/config/auto.conf: No such file or directory
Makefile:621: include/config/auto.conf.cmd: No such file or directory
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
YACC scripts/kconfig/zconf.tab.c
LEX scripts/kconfig/zconf.lex.c
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --syncconfig Kconfig
SYSTBL arch/x86/include/generated/asm/syscalls_32.h
SYSHDR arch/x86/include/generated/asm/unistd_32_ia32.h
SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h
SYSTBL arch/x86/include/generated/asm/syscalls_64.h
SYSHDR arch/x86/include/generated/uapi/asm/unistd_32.h
SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h
SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h
HOSTCC arch/x86/tools/relocs_32.o
HOSTCC arch/x86/tools/relocs_64.o
HOSTCC arch/x86/tools/relocs_common.o
HOSTLD arch/x86/tools/relocs
UPD include/config/kernel.release
WRAP arch/x86/include/generated/uapi/asm/bpf_perf_event.h
WRAP arch/x86/include/generated/uapi/asm/poll.h
WRAP arch/x86/include/generated/asm/dma-contiguous.h
WRAP arch/x86/include/generated/asm/early_ioremap.h
WRAP arch/x86/include/generated/asm/export.h
WRAP arch/x86/include/generated/asm/mcs_spinlock.h
WRAP arch/x86/include/generated/asm/mm-arch-hooks.h
UPD include/generated/uapi/linux/version.h
UPD include/generated/utsrelease.h
CC kernel/bounds.s
UPD include/generated/bounds.h
UPD include/generated/timeconst.h
CC arch/x86/kernel/asm-offsets.s
UPD include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
DESCEND objtool
HOSTCC /home/bbs/WSL2-Linux-Kernel/tools/objtool/fixdep.o
HOSTLD /home/bbs/WSL2-Linux-Kernel/tools/objtool/fixdep-in.o
LINK /home/bbs/WSL2-Linux-Kernel/tools/objtool/fixdep
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/exec-cmd.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/help.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/pager.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/parse-options.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/run-command.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/sigchain.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/subcmd-config.o
LD /home/bbs/WSL2-Linux-Kernel/tools/objtool/libsubcmd-in.o
AR /home/bbs/WSL2-Linux-Kernel/tools/objtool/libsubcmd.a
GEN /home/bbs/WSL2-Linux-Kernel/tools/objtool/arch/x86/lib/inat-tables.c
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/arch/x86/decode.o
LD /home/bbs/WSL2-Linux-Kernel/tools/objtool/arch/x86/objtool-in.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/builtin-check.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/builtin-orc.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/check.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/orc_gen.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/orc_dump.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/elf.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/special.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/objtool.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/libstring.o
CC /home/bbs/WSL2-Linux-Kernel/tools/objtool/str_error_r.o
LD /home/bbs/WSL2-Linux-Kernel/tools/objtool/objtool-in.o
LINK /home/bbs/WSL2-Linux-Kernel/tools/objtool/objtool
HOSTCC scripts/genksyms/genksyms.o
YACC scripts/genksyms/parse.tab.c
HOSTCC scripts/genksyms/parse.tab.o
LEX scripts/genksyms/lex.lex.c
YACC scripts/genksyms/parse.tab.h
HOSTCC scripts/genksyms/lex.lex.o
HOSTLD scripts/genksyms/genksyms
CC scripts/mod/empty.o
HOSTCC scripts/mod/mk_elfconfig
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/modpost.o
CC scripts/mod/devicetable-offsets.s
UPD scripts/mod/devicetable-offsets.h
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
HOSTCC scripts/bin2c
HOSTCC scripts/kallsyms
HOSTCC scripts/conmakehash
HOSTCC scripts/recordmcount
HOSTCC scripts/sortextable
HOSTCC scripts/asn1_compiler
HOSTCC scripts/extract-cert
Building modules, stage 2.
MODPOST 0 modules
WSL2-Linux-Kernel$ sudo make modules_install
DEPMOD 4.19.104-microsoft-standard+
Warning: modules_install: missing 'System.map' file. Skipping depmod.
```
最後將剛才得到的目錄設定為 kernel modules 的路徑 (WSL 2 下原無此目錄)。
```shell
WSL2-Linux-Kernel$ sudo ln -s /lib/modules/4.19.104-microsoft-standard+ /lib/modules/$(uname -r)
```
### 測試 `khttpd`
```shell
khttpd$ make
cc -std=gnu11 -Wall -Wextra -Werror -o htstress htstress.c -lpthread
make -C /lib/modules/4.19.104-microsoft-standard/build M=/home/iid/Linux-Kernel-Design-2020/khttpd modules
make[1]: Entering directory '/home/bbs/WSL2-Linux-Kernel'
CC [M] /home/iid/Linux-Kernel-Design-2020/khttpd/http_parser.o
CC [M] /home/iid/Linux-Kernel-Design-2020/khttpd/http_server.o
CC [M] /home/iid/Linux-Kernel-Design-2020/khttpd/main.o
LD [M] /home/iid/Linux-Kernel-Design-2020/khttpd/khttpd.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/iid/Linux-Kernel-Design-2020/khttpd/khttpd.mod.o
LD [M] /home/iid/Linux-Kernel-Design-2020/khttpd/khttpd.ko
make[1]: Leaving directory '/home/bbs/WSL2-Linux-Kernel'
khttpd$ make check
make -C /lib/modules/4.19.104-microsoft-standard/build M=/home/iid/Linux-Kernel-Design-2020/khttpd modules
make[1]: Entering directory '/home/bbs/WSL2-Linux-Kernel'
Building modules, stage 2.
MODPOST 1 modules
make[1]: Leaving directory '/home/bbs/WSL2-Linux-Kernel'
0 requests
10000 requests
20000 requests
30000 requests
40000 requests
50000 requests
60000 requests
70000 requests
80000 requests
90000 requests
requests: 100000
good requests: 100000 [100%]
bad requests: 0 [0%]
socker errors: 0 [0%]
seconds: 20.734
requests/sec: 4822.889
Complete
khttpd$
```
把 Google Chrome 關掉試試……
```shell
khttpd$ make check
[...]
requests: 100000
good requests: 100000 [100%]
bad requests: 0 [0%]
socker errors: 0 [0%]
seconds: 19.614
requests/sec: 5098.430
Complete
khttpd$
```
效能差異不大。
檢視 `khttpd.ko` 的資訊：
```shell
khttpd$ modinfo khttpd.ko
filename: /home/iid/Linux-Kernel-Design-2020/khttpd/khttpd.ko
version: 0.1
description: in-kernel HTTP daemon
author: National Cheng Kung University, Taiwan
license: Dual MIT/GPL
srcversion: 557851B109241B26D04B2D2
depends:
retpoline: Y
name: khttpd
vermagic: 4.19.104-microsoft-standard+ SMP mod_unload modversions
parm: port:ushort
parm: backlog:ushort
khttpd$ lsmod
Module Size Used by
khttpd 40960 0
khttpd$
```
Benchmark
```shell
khttpd$ ./htstress -n 1000 -c 1 -t 4 http://www.google.com/
0 requests
100 requests
200 requests
300 requests
400 requests
500 requests
600 requests
700 requests
800 requests
900 requests
requests: 1000
good requests: 1000 [100%]
bad requests: 0 [0%]
socker errors: 0 [0%]
seconds: 42.700
requests/sec: 23.419
```
```shell
khttpd$ sudo insmod khttpd.ko port=8081
khttpd$ wget localhost:8081
--2020-07-01 11:59:42-- http://localhost:8081/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8081... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12 [text/plain]
Saving to: ‘index.html’
index.html 100%[===============================>] 12 --.-KB/s in 0s
2020-07-01 11:59:42 (168 KB/s) - ‘index.html’ saved [12/12]
khttpd$ cat index.html
Hello World!khttpd$
```
## 處理 `/fib/N`
這個字串是在哪裡產生的？
```shell
khttpd$ rg "Hello World!"
http_server.c
16: "Connection: Close" CRLF CRLF "Hello World!" CRLF
21: "Connection: Keep-Alive" CRLF CRLF "Hello World!" CRLF
```
```c=12
#define HTTP_RESPONSE_200_DUMMY \
"" \
"HTTP/1.1 200 OK" CRLF "Server: " KBUILD_MODNAME CRLF \
"Content-Type: text/plain" CRLF "Content-Length: 12" CRLF \
"Connection: Close" CRLF CRLF "Hello World!" CRLF
```
```c=76
static int http_server_response(struct http_request *request, int keep_alive)
{
char *response;
pr_info("requested_url = %s\n", request->request_url);
if (request->method != HTTP_GET)
response = keep_alive ? HTTP_RESPONSE_501_KEEPALIVE : HTTP_RESPONSE_501;
else
response = keep_alive ? HTTP_RESPONSE_200_KEEPALIVE_DUMMY
: HTTP_RESPONSE_200_DUMMY;
http_server_send(request->socket, response, strlen(response));
return 0;
}
```
`request->request_url` 應是所要的 url。驗證如下：
```shell
khttpd$ wget localhost:8081/fib/42
--2020-07-01 13:22:45-- http://localhost:8081/fib/42
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8081... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12 [text/plain]
Saving to: ‘42.1’
42.1 100%[===============================>] 12 --.-KB/s in 0s
2020-07-01 13:22:45 (945 KB/s) - ‘42.1’ saved [12/12]
khttpd$ dmesg -k | tail
[703899.551653] khttpd: requested_url = /
[703899.554815] khttpd: recv error: -104
[703900.925947] khttpd: requested_url = /
[703900.927536] khttpd: recv error: -104
[703901.353702] khttpd: requested_url = /
[703901.355572] khttpd: recv error: -104
[703901.726378] khttpd: requested_url = /
[703901.727973] khttpd: recv error: -104
[703904.739368] khttpd: requested_url = /fib/42
[703904.740990] khttpd: recv error: -104
khttpd$
```
Sign in with Wallet
Connect another wallet