Linux 核心專題: 高度並行的核心模式網頁伺服器

執行人: Jordymalone

TODO: 進行 ktcp

依據 ktcp 作業規範，提交 CMWQ 以外的 pull request
完成所有的作業要求

自我檢查清單

研讀〈Linux 核心設計: RCU 同步機制〉並測試相關 Linux 核心模組以理解其用法
- 筆記
如何測試網頁伺服器的效能，針對多核處理器場景調整
如何利用 Ftrace 找出 khttpd 核心模組的效能瓶頸，該如何設計相關實驗學習。搭配閱讀《Demystifying the Linux CPU Scheduler》第 6 章
解釋 drop-tcp-socket 核心模組運作原理。TIME-WAIT sockets 又是什麼？
- 筆記
參照〈測試 Linux 核心的虛擬化環境〉和〈建構 User-Mode Linux 的實驗環境〉，在原生的 Linux 系統中，利用 UML 或 virtme 建構虛擬化執行環境，搭配 GDB 追蹤 khttpd 核心模組

Cppcheck 問題

$ cppcheck --version
Cppcheck 2.13.0
$ git commit -a
http_server.h:-1:0: information: Unmatched suppression: unusedStructMember [unmatchedSuppression]

^
nofile:0:0: information: Active checkers: 132/592 (use --checkers-report=<filename> to see details) [checkersReport]

Fail to pass static analysis.

執行 pre-commit.hook 會出現以上錯誤，說明 cppcheck 回報這條 suppression 是多餘的，因此觸發了 unmatchedSuppression 警告。

參考 lab0-c 的 Commit 0180045，@komark06，Cppcheck manual

目前是先修改 pre-commit.hook 如下

+CPPCHECK_unmatched=
+for f in *.c *.h; do
+    CPPCHECK_unmatched="$CPPCHECK_unmatched --suppress=unmatchedSuppression:$f"
+done
 # http-parser was taken from Node.js, and we don't tend to validate it.
 CPPCHECK_suppresses="--suppress=missingIncludeSystem \
                      --suppress=unusedFunction:http_parser.c \
@@ -8,7 +12,7 @@ CPPCHECK_suppresses="--suppress=missingIncludeSystem \
                     --suppress=unknownMacro:http_server.c \
                      --suppress=unusedStructMember:http_parser.h \
                      --suppress=unusedStructMember:http_server.h"
-CPPCHECK_OPTS="-I. --enable=all --error-exitcode=1 --force $CPPCHECK_suppresses"
+CPPCHECK_OPTS="-I. --enable=all --error-exitcode=1 --force $CPPCHECK_suppresses $CPPCHECK_unmatched"

一次性地抑制所有可能出現的 unmatchedSuppression 警告。

目前不清楚 cppcheck 從哪個版本有做相關的更動
是否要以這改動提交 PR?

實驗環境

$ gcc --version
gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   8
  On-line CPU(s) list:    0-7
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz
    CPU family:           6
    Model:                94
    Thread(s) per core:   2
    Core(s) per socket:   4
    Socket(s):            1
    Stepping:             3
    CPU(s) scaling MHz:   92%
    CPU max MHz:          3900.0000
    CPU min MHz:          800.0000
    BogoMIPS:             6999.82

測試原始專案

編譯 khttpd 專案

$ make

出現以下 warning

  CC [M]  /home/jordan/linux2025/khttpd/http_parser.o
/home/jordan/linux2025/khttpd/http_parser.c: In function ‘http_parser_execute’:
/home/jordan/linux2025/khttpd/http_parser.c:1105:16: warning: this statement may fall through [-Wimplicit-fallthrough=]
 1105 |             if (parser->method == HTTP_SOURCE) {
      |                ^
/home/jordan/linux2025/khttpd/http_parser.c:1110:11: note: here
 1110 |           default:
      |           ^~~~~~~
/home/jordan/linux2025/khttpd/http_parser.c:1539:23: warning: this statement may fall through [-Wimplicit-fallthrough=]
 1539 |               h_state = h_content_length_num;
      |               ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
/home/jordan/linux2025/khttpd/http_parser.c:1542:13: note: here
 1542 |             case h_content_length_num:
      |             ^~~~
/home/jordan/linux2025/khttpd/http_parser.c:1854:31: warning: this statement may fall through [-Wimplicit-fallthrough=]
 1854 |               parser->upgrade = 1;
      |               ~~~~~~~~~~~~~~~~^~~
/home/jordan/linux2025/khttpd/http_parser.c:1857:13: note: here
 1857 |             case 1:
      |             ^~~~
/home/jordan/linux2025/khttpd/http_parser.c:1401:12: warning: this statement may fall through [-Wimplicit-fallthrough=]
 1401 |         if (ch == LF) {
      |            ^
/home/jordan/linux2025/khttpd/http_parser.c:1408:7: note: here
 1408 |       case s_header_value_start:
      |       ^~~~
/home/jordan/linux2025/khttpd/http_parser.c: In function ‘parse_url_char’:
/home/jordan/linux2025/khttpd/http_parser.c:548:10: warning: this statement may fall through [-Wimplicit-fallthrough=]
  548 |       if (ch == '@') {
      |          ^
/home/jordan/linux2025/khttpd/http_parser.c:553:5: note: here
  553 |     case s_req_server_start:
      |     ^~~~
/home/jordan/linux2025/khttpd/http_parser.c: In function ‘http_parser_parse_url’:
/home/jordan/linux2025/khttpd/http_parser.c:2460:18: warning: this statement may fall through [-Wimplicit-fallthrough=]
 2460 |         found_at = 1;
      |         ~~~~~~~~~^~~
/home/jordan/linux2025/khttpd/http_parser.c:2463:7: note: here
 2463 |       case s_req_server:
      |       ^~~~
/home/jordan/linux2025/khttpd/http_parser.c: In function ‘http_parse_host_char’:
/home/jordan/linux2025/khttpd/http_parser.c:436:59: warning: this statement may fall through [-Wimplicit-fallthrough=]
  436 | #define IS_HOST_CHAR(c)     (IS_ALPHANUM(c) || (c) == '.' || (c) == '-')
      |                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
/home/jordan/linux2025/khttpd/http_parser.c:2279:11: note: in expansion of macro ‘IS_HOST_CHAR’
 2279 |       if (IS_HOST_CHAR(ch)) {
      |           ^~~~~~~~~~~~
/home/jordan/linux2025/khttpd/http_parser.c:2284:5: note: here
 2284 |     case s_http_host_v6_end:
      |     ^~~~
/home/jordan/linux2025/khttpd/http_parser.c:2292:10: warning: this statement may fall through [-Wimplicit-fallthrough=]
 2292 |       if (ch == ']') {
      |          ^
/home/jordan/linux2025/khttpd/http_parser.c:2297:5: note: here
 2297 |     case s_http_host_v6_start:
      |     ^~~~
/home/jordan/linux2025/khttpd/http_parser.c:2308:10: warning: this statement may fall through [-Wimplicit-fallthrough=]
 2308 |       if (ch == ']') {
      |          ^
/home/jordan/linux2025/khttpd/http_parser.c:2313:5: note: here
 2313 |     case s_http_host_v6_zone_start:
      |     ^~~~

~~TODO: 待釐清~~
已提交 pull request

掛載 khttpd 模組並指定 Port，若不指定預設就是 8081

$ sudo insmod khttpd.ko port=<Port number>

執行 wget localhost:8081

$ wget localhost:8081
--2025-05-24 16:20:58--  http://localhost:8081/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8081... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12 [text/plain]
Saving to: ‘index.html’

index.html                         100%[==================================================================================================================================>]      12  --.-KB/s    in 0s      

2025-05-24 16:20:58 (7.74 MB/s) - ‘index.html’ saved [12/12]

可正常下載並獲得 index.html 的檔案，裡面有 Hello World!!! 表示成功

但透過 dmesg 觀察會發現有以下 error

[ 7326.973496] khttpd: http server worker Begin
[ 7326.973519] khttpd: requested_url = /
[ 7326.974314] khttpd: recv error: -104
[ 7326.974327] khttpd: http server worker Stop

TODO: 待釐清

導入 CMWQ 改寫 khttpd

參考作業描述
 Commit e70dd99

在導入 CMWQ 之前先使用 htstress 這個 Benchmark Tool 來測試效果。

導入前:

$ ./htstress http://localhost:8081 -t 3 -c 20 -n 200000

requests:      200000
good requests: 200000 [100%]
bad requests:  0 [0%]
socket errors: 0 [0%]
seconds:       13.001
requests/sec:  15383.569

khttpd 最初的設計思路是為每個連線都生成獨立的一條 kthread。
在模組初始化時，

static int __init khttpd_init(void)
{
    int err = open_listen_socket(port, backlog, &listen_socket);
    ...
    http_server = kthread_run(http_server_daemon, &param, KBUILD_MODNAME);
    ...
    return 0;
}

可以看到我們會使用 kthread_run 啟動一條 daemon thread 來扮演 server 的角色去負責接收連線。同時他會去執行 http_server_daemon 這個函式，可以看到以下:

kthread_run 是個巨集，詳情可參閱 /include/linux/kthread.h

int http_server_daemon(void *arg)
{
    ...
    while (!kthread_should_stop()) {
        int err = kernel_accept(param->listen_socket, &socket, 0);
        ...
        worker = kthread_run(http_server_worker, socket, KBUILD_MODNAME);
        ...
    return 0;
}

當有 client 被 accept 進來時，一樣透過 kthread_run 生成一條獨立的 thread 執行 http_server_worker 去處理這個 client 的要求。

參考 kecho 的實作，我們先在 http_server.h 中加入:

+#define MODULE_NAME "khttpd"
 #include <net/sock.h>
+#include <linux/workqueue.h>
+#include <linux/list.h>
 
 struct http_server_param {
     struct socket *listen_socket;
 };
 
+struct httpd_service {
+    bool is_stopped;
+    struct list_head head;
+};

新增 httpd_service 結構的用意是，我們可以使用鏈結串列來去追蹤 work item 的狀況。

is_stopped 用來記錄是否停止
head 用來記錄 work item

接著，修改 main.c 中模組載入時初始化的部分，

static int __init khttpd_init(void)
{
    ...
    param.listen_socket = listen_socket;

+   khttpd_wq = alloc_workqueue(MODULE_NAME, 0, 0);
+   if (!khttpd_wq) {
+       pr_err("can't create workqueue\n");
+       close_listen_socket(listen_socket);
+       return -ENOMEM;
+   }
    http_server = kthread_run(http_server_daemon, &param, KBUILD_MODNAME);
    ...
    return 0;
}

這邊的 alloc_workqueue 若設定 WQ_UNBOUND 不知道會不會比較好。

透過 alloc_workqueue 建立 workqueue，將後續所有 HTTP 請求封裝為 work item，並交由 kworker 於背景處理。

而主要改動都在 http_server.c 中，新增了 create_work 和 free_work 函式，前者的用途是每當有新連線 accept 成功時，通過 kmalloc 分配一塊 http_request 結構，然後用 INIT_WORK(work->khttpd_work, http_server_worker); 將他的成員 khttpd_work 初始化為一個 work item，綁定 http_server_worker 函式，一旦 kworker 拿到這個 work item 就會自動呼叫 http_server_worker 這個函式。

後者 free_work 則是在模組卸載時走訪鏈結串列上還沒被釋放的 http_request。

同時我們針對 http_server_daemon 做以下修改:

 int http_server_daemon(void *arg)
 {
     struct socket *socket;
-    struct task_struct *worker;
     struct http_server_param *param = (struct http_server_param *) arg;
+    struct work_struct *work;
     ...
+    INIT_LIST_HEAD(&daemon_list.head);

     while (!kthread_should_stop()) {
         ...
-        worker = kthread_run(http_server_worker, socket, KBUILD_MODNAME);
-        if (IS_ERR(worker)) {
-            pr_err("can't create more worker process\n");
+        work = create_work(socket);
+        if (!work) {
+            pr_err("can't create work for socket\n");
+            kernel_sock_shutdown(socket, SHUT_RDWR);
+            sock_release(socket);
             continue;
         }
+        queue_work(khttpd_wq, work);
     }
+    daemon_list.is_stopped = true;
+    free_work();
+    pr_info("http server daemon stopped\n");
+
     return 0;
 }

初始化一個全域的 daemon_list，紀錄尚未釋放的 http_request，在迴圈當中，每次有新連線 accept 進來，將接收到的 socket 透過 create_work 封裝成 struct http_request，再使用 queue_work 交由系統的 CMWQ 排程。

同時也對 http_server_worker 做適度的修改，詳情請見對應的 Commit。

導入後:

$ ./htstress http://localhost:8081 -t 3 -c 20 -n 200000

requests:      200000
good requests: 200000 [100%]
bad requests:  0 [0%]
socket errors: 0 [0%]
seconds:       7.907
requests/sec:  25293.244

從上述兩組數據可以看出，在相同的測試條件下，導入 CMWQ 之前和之後:

指標	導入前	導入後
總耗時（秒）	13.001	7.907
吞吐量（req/sec）	15383.6	25293.244

總耗時下降
- 由 13.001s 降到 7.907 s，降低了約 39.2 %。
吞吐量提升
- 從約 15.4 k req/s 提升到約 25.3 k req/s，增幅約 64.3 %。

執行 rmmod 會出現以下訊息，但不影響移除模組

[ 2206.623504] BUG: kernel NULL pointer dereference, address: 0000000000000068
[ 2206.623511] #PF: supervisor read access in kernel mode
[ 2206.623514] #PF: error_code(0x0000) - not-present page
[ 2206.623516] PGD 0 P4D 0 
[ 2206.623520] Oops: Oops: 0000 [#5] PREEMPT SMP PTI
[ 2206.623525] CPU: 4 UID: 0 PID: 319077 Comm: khttpd Tainted: G S    D    OE      6.11.0-26-generic #26~24.04.1-Ubuntu
[ 2206.623531] Tainted: [S]=CPU_OUT_OF_SPEC, [D]=DIE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 2206.623533] Hardware name: ASUSTeK COMPUTER INC. ESC500 G4/P10S WS, BIOS 0801 09/02/2016
[ 2206.623535] RIP: 0010:kernel_sock_shutdown+0xa/0x20
...
[ 2206.623832] note: khttpd[319077] exited with irqs disabled
[ 2206.623873] khttpd: module unloaded

待處理

基本 directory listing 功能

Commit 5dac8d7

TODO: 利用核心的 TLS 機制來實作 HTTPS

善用 Kernel TLS offload

Eric Chou

2025/06/14 10:30:08

如果 fall through 的設計是故意為之，加上 " // fall through " 避免 warning ，參考 https://stackoverflow.com/questions/45129741/gcc-7-wimplicit-fallthrough-warnings-and-portable-way-to-clear-them。 (但我沒辦法 re-create 這個 warning ) 反之，使用 break 避免 fall through。不論哪種，都是提交 PR 的機會! (Edited)

Jim Huang

2025/06/22 05:23:55

是否要以這改動提交 PR?

來！