# 2024q1 final
contributed by < `SimonLee0316` >
> github : [SimonLee0316/c-web-server](https://github.com/SimonLee0316/c-web-server): a blocking web server
> [專題解說錄影](https://www.youtube.com/watch?v=bxRA_5DZr50)

[專題發表連結](https://hackmd.io/zOnqpAnBRs68WZJ3XhP_CA?both)
## 實驗環境

$ gcc --version
gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
    CPU family:          6
    Model:               165
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            5
    CPU max MHz:         4800.0000
    CPU min MHz:         800.0000
    BogoMIPS:            5799.77
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    2 MiB (8 instances)
  L3:                    16 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15

## Quiz8 並擴充為 web server
[Quiz8筆記](https://hackmd.io/NQdK92x4TeetRz5GUBsdwA)
[cserv筆記](https://hackmd.io/PY04VBDJTN-yGvEvHpTD4Q)

高效的網頁伺服器，採用阻塞式 I/O 的事件驅動架構。單執行緒、支援多核並善用 CPU affinity。

透過 fork (**cpu 可用數量**)個監聽行程 ，並綁定在特定 cpu ，這樣可以達到同時處理多個連線請求。

監聽特定 port

#define PORT 9999

透過 `start , stop`，開啟跟關閉伺服器

./tcp_server start   # Start the web server.
./tcp_server stop    # Stop the web server.

### 啟動伺服器
先創建 socket 並將網路協定設定為 IPV4，類型為 TCP。
為了防止 TCP/IP 在關閉 socket 後會處於 `TIME-WAIT` 狀態，使用 `setsockopt` 設定 `SO_REUSEADDR` 選項，好讓我們可以重新綁定 socket。
設定 [TCP_CORK](https://linux.die.net/man/7/tcp) ，提高傳輸效率。
使用 `bind and listen` 將 socket 綁定到特定 port 和 address ，使其成為一個監聽 socket ，準備接受連線請求。

/* Eliminate 'Address already in use' error during the binding process */
    if (setsockopt(listenfd, SOL_SOCKET, SO_REUSEADDR, (const void *)&optval, sizeof(int)) < 0) {
        perror("Set socket option SO_REUSEADDR failed");
        return -1;
    }

    /* Setting TCP_CORK option */
    if (setsockopt(listenfd, IPPROTO_TCP, TCP_CORK, (const void *)&optval, sizeof(int)) < 0) {
        perror("Set socket option TCP_CORK failed");
        return -1;
    }

建立 cpu 可用數量個行程並綁定在特定 cpu 上面，先取得目前可以使用的 cpu 數量。
[sysconf](https://www.man7.org/linux/man-pages/man3/sysconf.3.html) 取得目前可用 cpu 數量。

int num_workers = sysconf(_SC_NPROCESSORS_ONLN);

[cpu_set_t](https://man7.org/linux/man-pages/man3/CPU_OR.3.html)
初始化一個 CPU 集合。
將0-15 號 CPU 添加到集合中。
使用 [sched_setaffinity](https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html) 函数将行程的 CPU 親和力設定為指定的 CPU 集合。
建立 [coroutine](https://hackmd.io/NQdK92x4TeetRz5GUBsdwA#2024q1-quiz8) 去循環接受請求。

void create_workers(int listenfd, int num_workers) {

    for (int i = 0; i < num_workers; i++) {
        pid_t pid = fork();
        if (pid == 0) {  // Child
            cpu_set_t cpuset;
            CPU_ZERO(&cpuset);
            CPU_SET(i % num_workers, &cpuset);
            if (sched_setaffinity(0, sizeof(cpu_set_t), &cpuset) < 0) {
                perror("Could not set CPU affinity");
                exit(1);
            }
            quick_start(worker_process_cycle, co_cleanup, &(int){listenfd});
            
        } else if (pid > 0) {  // Parent
            pids[i] = pid;
            printf("Child process %d created and assigned to CPU %d\n", pid, i % num_workers);
        } else {
            perror("Fork failed");
        }
    }
    create_pidfile(PID_FILE, pids, num_workers);
    // Parent process waits for children to exit
    while (wait(NULL) > 0);
}

確認行程是否有綁定在特定 cpu 。

$ ps -ef | grep tcp_serv
simon      76329   47602  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76330   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76331   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76332   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76333   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76334   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76335   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76336   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76337   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76338   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76339   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76340   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76341   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76342   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76343   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76344   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
simon      76345   76329  0 00:12 pts/0    00:00:00 ./tcp_server start
$ ./test.sh 
pid 76330's current affinity list: 0
pid 76331's current affinity list: 1
pid 76332's current affinity list: 2
pid 76333's current affinity list: 3
pid 76334's current affinity list: 4
pid 76335's current affinity list: 5
pid 76336's current affinity list: 6
pid 76337's current affinity list: 7
pid 76338's current affinity list: 8
pid 76339's current affinity list: 9
pid 76340's current affinity list: 10
pid 76341's current affinity list: 11
pid 76342's current affinity list: 12
pid 76343's current affinity list: 13
pid 76344's current affinity list: 14
pid 76345's current affinity list: 15

`worker_process_cycle` 負責接受請求，`handle_client` 處理請求並給予回應。
accept 是一個 [Blocking I/O](https://hackmd.io/@sysprog/linux-io-model/https%3A%2F%2Fhackmd.io%2F%40sysprog%2Fevent-driven-server#Blocking-IO)，它會一直在這裡阻塞直到收到請求。
收到請求之後會建立 [coroutine](https://hackmd.io/NQdK92x4TeetRz5GUBsdwA#2024q1-quiz8) 去處理請求並給予回應。

void worker_process_cycle(void *udata) {
    int listenfd = *(int *) udata;
    int connfd;
    struct sockaddr_in client_addr;
    socklen_t client_addr_len = sizeof(client_addr);
    
    // accept cilent request
    while (1) {
        connfd = accept(listenfd, (struct sockaddr *)&client_addr, &client_addr_len);
        if (connfd < 0) {
            perror("Accept failed");
            continue;
        }
        quick_start(handle_client, co_cleanup, &(int){connfd});
        task_yield(); // Re-enqueue
    }
}
void handle_client(int connfd) {
    char buffer[BUFFER_SIZE];
    int bytes_read;

    // read client request
    bytes_read = read(connfd, buffer, BUFFER_SIZE - 1);
    if (bytes_read < 0) {
        perror("Failed to read from socket");
        close(connfd);
        return;
    }
    /*....*/

    // build http response
    char *response =
        "HTTP/1.1 200 OK\r\n"
        "Content-Type: text/html\r\n"
        "Content-Length: 13\r\n"
        "\r\n"
        "Hello, world!";

    // send http response to client
    write(connfd, response, strlen(response));

    // close client connect
    close(connfd);
}

### 關閉伺服器

在開啟伺服器之後，會先將 fork 的子行程，紀錄在 `tmp/tcp_server.pid` ，這樣當要停止的時候才可以透過 `read_pidfile` 將所有子行程停止。

在 `create_worker` 中如果是父母行程 (pid > 0)，會將子行程的 pid 紀錄在 `pids[]` ，在使用 `create_pidfile` 將值存入目錄裡面。

int main(int argc, char** argv) {
    /*...*/
    if (!strcmp(argv[1], "stop")) {
        send_signal(SIGTERM);
        printf("cserv: stopped.\n");
        return 0;
    }
    /*...*/
	return 0;
}
static void send_signal(int signo)
{
    int num_pids = 0;
    read_pidfile(PID_FILE, pids, &num_pids);
    for (int i = 0; i < num_pids; i++) {
        if (kill(pids[i], signo) == 0) {
            printf("Stopped process %d successfully.\n", pids[i]);
        } else {
            perror("Failed to stop the process");
        }
    }
    unlink(PID_FILE);
}

而父母行程會在 `create_workers` 裡面等到所有子行程都停止才會關閉 **listenfd** 並停止。

void create_workers(int listenfd, int num_workers) {

    /*...*/
    create_pidfile(PID_FILE, pids, num_workers);
    // Parent process waits for children to exit
    while (wait(NULL) > 0);
}

./htstress -n 100000 -c 500 http://localhost:9999/

| | request/sec | seconds |
| ---------- | ----------- | ------- |
| tcp_server | 44181.476 | 2.263 |

### 使用 [coroutine](https://hackmd.io/NQdK92x4TeetRz5GUBsdwA#2024q1-quiz8) 去接收請求還有處理請求

fork 完子行程之後，會去產生一個 coroutine 去接收請求，而在接受到請求之後會在產生一個 coroutine 去處理請求。

void create_workers(int listenfd, int num_workers) {
    for (int i = 0; i < num_workers; i++) {
        pid_t pid = fork();
        if (pid == 0) {  // Child
           /*...*/
            quick_start(worker_process_cycle, co_cleanup, &(int){listenfd});
        }
        /*...*/
    }
    /*...*/
}

void worker_process_cycle(void *udata) {
    /*...*/
    // accept cilent request
    while (1) {
        connfd = accept(listenfd, (struct sockaddr *)&client_addr, &client_addr_len);
        /*...*/
        quick_start(handle_client, co_cleanup, &(int){connfd}); // handle request
    }
}

### 壓力測試與比較
**工具**
* [htstress](https://github.com/sysprog21/khttpd/blob/master/htstress.c)

**比較對象**
* [xampp apache server](https://www.apachefriends.org/zh_tw/index.html)
* [cserv](https://github.com/sysprog21/cserv)
* [nginx](https://ubuntu.com/tutorials/install-and-configure-nginx#1-overview)

htstress
100000 requests with 500 requests at a time(concurrency)

 ./htstress -n 100000 -c 500 http://localhost:9999/
0 requests
10000 requests
20000 requests
30000 requests
40000 requests
50000 requests
60000 requests
70000 requests
80000 requests
90000 requests

requests:      100000
good requests: 100000 [100%]
bad requests:  0 [0%]
socket errors: 0 [0%]
seconds:       2.263
requests/sec:  44194.207

| | request/sec | seconds |
| -------------- | ----------- | ------- |
| apache | 37318.529 | 2.680 |
| nginx | 45460.600 | 2.200 |
| cserv | 46730.129 | 2.140 |
| **tcp_server** | 44194.207 | 2.263 |