# [Linux 核心設計](https://beta.hackfoldr.org/linux/): 賦予應用程式生命的系統呼叫 Copyright (**慣C**) 2018 [宅色夫](http://wiki.csie.ncku.edu.tw/User/jserv) ## Linux 系統呼叫作為真正的「九二共識」 ==[直播錄影](https://youtu.be/rPWt6KgL8uQ)== 儘管 1991 年 Linux 核心的原始程式碼已公開釋出,但[到了 1992 年才開始以 GNU GPLv2 釋出原始程式碼](https://en.wikipedia.org/wiki/History_of_Linux#Linux_under_the_GNU_GPL),自此透過大量開發者的投入,創造真正的「九二共識」。 Linus Torvalds 在 2001 年紀錄片《[Revolution OS](https://hackmd.io/s/SyuRJIPI-)》說過: > 「作業系統就是你永遠不會看到的東西,因為沒有人直接使用作業系統,人們使用的是程式。在他們的電腦上,作業系統唯一的使命就是,幫助其它程式執行,所以作業系統從未獨立運行,而僅是默默等待程式,來向它要求現有資源、某個存在硬碟上的檔案或要求其它程式將這個程式連接到外面去,然後作業系統再一步步地,試著讓人們寫程式容易一些」 這席話背後的機制,恰好就是系統呼叫。 Linus Torvalds 在 2001 年 10 月重申「九二共識」: > From a technical standpoint, I believe the kernel will be "more of the same" and that all the _really_interesting staff will be going out in user space. ![](https://i.imgur.com/eoX2FDC.png) 推薦閱讀:論文 [Analyzing a Decade of Linux System Calls](http://research.cs.queensu.ca/~cordy/Papers/BKBHDC_ESE_Linux.pdf) ![](https://i.imgur.com/OWdx5EL.png) ## 從縮減 Hello World 程式談起 《[C 語言編程透視](https://github.com/tinyclub/open-c-book)》是中國友人 [falcon](https://github.com/lzufalcon) 著眼於透視 C 的前世今生,所撰寫的電子書,在 [打造史上最小可執行 ELF 檔案](https://github.com/tinyclub/open-c-book/blob/master/zh/chapters/02-chapter8.markdown) 談及系統呼叫使用的機制。 給定 `hello.c`: ```cpp #include <stdio.h> int main(void) { printf("hello, world!\n"); return 0; } ``` 以 gcc/clang 編譯 `hello.c`,隨後用 [ltrace](http://man7.org/linux/man-pages/man1/ltrace.1.html) 追蹤: ```cpp ltrace hello __libc_start_main(0x400526, 1, 0x7fffb57761d8, 0x400540 <unfinished ...> puts("hello, world!"hello, world! ) = 14 +++ exited (status 0) +++ ``` 幾個觀察: * `__libc_start_main` 是程式真正的進入點,詳見 [你所不知道的 C 語言: 執行階段程式庫 (CRT)](https://hackmd.io/@sysprog/c-runtime) * 字串作為 `puts` 函式的參數 * `puts` 函式返回值為 `14`,表示字串長度 * 最後的 `status 0` 為 status code,也就是 main 函式的`return` 值 對 gcc 輸出的組合語言進行縮減,得到以下 `hello.s`: ```cpp .LC0: .string "Hello world" .text .globl main .type main, @function main: leal 4(%esp), %ecx andl $-16, %esp pushl -4(%ecx) pushl %ebp movl %esp, %ebp pushl %ecx subl $4, %esp movl $.LC0, (%esp) call puts movl $0, (%esp) call _exit ``` 組譯和連結: ```shell $ as --32 -o hello.o hello.s $ ld -melf_i386 -o hello hello.o ``` 在 [你所不知道的 C 語言: 編譯器和最佳化原理篇](https://hackmd.io/@sysprog/c-compiler-optimization) 提及 gcc 會將 `printf("hello, world!\n")` 最佳化為 `puts("hello, world!\n")`,以降低解析 format string 和對應處理的成本。[qrintf - sprintf accelerator](https://github.com/h2o/qrintf) 展示了字串處理的執行時期開銷。對應到組合語言就是 `call puts`,以及之前參數傳遞的指令。 可避免動態連結函式庫中的 `printf` 或 `puts`,也不用直接呼叫 _exit,而在組合語言裡頭使用系統呼叫,即可可以去掉和動態連結關聯的內容。重寫後得到如下 x86 程式碼: ```cpp .LC0: .string "Hello world!\xa\x0" .text .global _start _start: xorl %eax, %eax movb $4, %al #eax = 4, sys_write(fd, addr, len) xorl %ebx, %ebx incl %ebx #ebx = 1, standard output movl $.LC0, %ecx #ecx = $.LC0, the address of string xorl %edx, %edx movb $14, %dl #edx = 14, the length of .string int $0x80 xorl %eax, %eax movl %eax, %ebx #ebx = 0 incl %eax #eax = 1, sys_exit int $0x80 ``` 對應的 x86_64 版本: ```clike .data msg: .ascii "Hello, world!\n" len = . - msg .text .global _start _start: movq $1, %rax movq $1, %rdi movq $msg, %rsi movq $len, %rdx syscall movq $60, %rax xorq %rdi, %rdi syscall ``` 組譯和連結: ```shell $ gcc -c hello.s $ ld -o hello hello.o ``` 原來,無論是 x86 的 `int $0x80` 抑或 x86_64 的 `syscall` 都是系統呼叫的 [call gate](https://en.wikipedia.org/wiki/Call_gate_(Intel)),後期 Intel 引入快速系統呼叫 (fast system call)。 ![](https://i.imgur.com/eI0WONT.png) 搭配閱讀: [System Calls Make the World Go Round](https://manybutfinite.com/post/system-calls/) 使用 [syscall(2)](http://man7.org/linux/man-pages/man2/syscall.2.html) 改寫為以下: ```cpp #define _GNU_SOURCE #include <unistd.h> #include <sys/syscall.h> int main() { return syscall(__NR_write, 1, "Hello, world!\n", 14); } ``` 透過 [strace(1)](http://man7.org/linux/man-pages/man1/strace.1.html) 追蹤系統呼叫: ```cpp write(1, "hello, world!\n", 14hello, world! ) = 14 ``` 查閱 syscall(2): | arch/ABI | instruction | syscall # | retval | | -----------|-----------------:| ---------:|--------| | arm/OABI | `swi` NR | - | a1 | | arm/EABI | `swi 0x0` | r7 | r0 | | arm64 | `svc #0` | x8 | x0 | | i386 | `int $0x80` | eax | eax | | x86_64 | `syscall` | rax | rax | | x32 | `syscall` | rax | rax | > [x32 ABI](https://en.wikipedia.org/wiki/X32_ABI) allows programs to take advantage of the benefits of x86-64 instruction set (larger number of CPU registers, better floating-point performance, faster position-independent code, shared libraries, function parameters passed via registers, faster syscall instruction) while using 32-bit pointers and thus avoiding the overhead of 64-bit pointers. > [_syscall(2)](http://man7.org/linux/man-pages/man2/_syscall.2.html) 已棄置 搭配閱讀: [Computer Science from the Bottom Up: System Calls](https://www.bottomupcs.com/system_calls.xhtml) ## 透過 kprobes + eBPF 來追蹤系統呼叫 [Kernel Probes (Kprobes)](https://www.kernel.org/doc/Documentation/kprobes.txt) 僅能讀取系統呼叫的參數和返回值,無法變更暫存器內含值。 複習 [透過 eBPF 觀察作業系統行為](https://hackmd.io/@sysprog/linux-ebpf)。 準備 `call.py` ```python= from bcc import BPF bpf_text = """ #include <net/inet_sock.h> #include <bcc/proto.h> int kprobe__inet_listen(struct pt_regs *ctx, struct socket *sock, int backlog) { bpf_trace_printk("Hello World!\\n"); return 0; }; """ b = BPF(text=bpf_text) while True: print b.trace_readline() ``` 先在一個終端機執行: ```shell $ sudo python call.py ``` 等待五秒,再開啟另一個終端機並執行以下命令: ```shell $ nc -l 0 4242 ``` 依據 [nc(1)](https://linux.die.net/man/1/nc): > arbitrary TCP and UDP connections and listens > -l' Used to specify that nc should ==listen for an incoming connection== rather than initiate a connection to a remote host. `0` 是 hostname (也就是本地端),`42` 是 port 預期將看到以下訊息: ``` nc-10348 [027] .... 12355941.551058: 0x00000001: Hello World! ``` 修改上述程式的第 8 行為: ```cpp bpf_trace_printk("Listening with %d pending connections!\\n", backlog); ``` 重作實驗,預期得到以下訊息: ``` nc-10842 [025] .... 12356439.738348: 0x00000001: Listening with 1 pending connections! ``` 修改上述 kprobe__inet_listen 函式,取代為以下: ```cpp struct inet_sock *inet = inet_sk(sock->sk); u32 laddr = 0; u16 lport = 0; bpf_probe_read(&laddr, sizeof(laddr), &(inet->inet_rcv_saddr)); bpf_probe_read(&lport, sizeof(lport), &(inet->inet_sport)); bpf_trace_printk("Listening on %x %d\\n", ntohl(laddr), ntohs(lport)); return 0; ``` 重作實驗,預期得到以下訊息: ``` nc-11250 [020] .... 12356849.241049: 0x00000001: Listening on 0 4242 ``` 延伸閱讀: * [How to turn any syscall into an event: Introducing eBPF Kernel probes](https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/) ## Linux 核心對系統呼叫的實作機制 檔案 [linux/arch/x86/entry/syscall_64.c](https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscall_64.c): ```cpp /* this is a lie, but it does not hurt as sys_ni_syscall just returns -EINVAL */ extern asmlinkage long sys_ni_syscall(const struct pt_regs *); #define __SYSCALL_64(nr, sym, qual) extern asmlinkage long sym(const struct pt_regs *); #include <asm/syscalls_64.h> #undef __SYSCALL_64 #define __SYSCALL_64(nr, sym, qual) [nr] = sym, asmlinkage const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = { /* * Smells like a compiler bug -- it doesn't work * when the & below is removed. */ [0 ... __NR_syscall_max] = &sys_ni_syscall, #include <asm/syscalls_64.h> }; ``` 搭配閱讀 [How does the Linux kernel handle a system call](https://0xax.gitbooks.io/linux-insides/SysCall/linux-syscall-2.html) 和 [Anatomy of a system call, part 1 ](https://lwn.net/Articles/604287/) ## vsyscall 和 vDSO 問問 git (有飯桶」和「笨蛋」的意思) 需要什麼? ```shell $ ldd /usr/bin/git linux-vdso.so.1 => (0x00007ffd83ff1000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f7035583000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7035369000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f703514e000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7034f31000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7034d29000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f703495f000) /lib64/ld-linux-x86-64.so.2 (0x00007f70357f3000) ``` 等等,檔案 `linux-vdso.so.1` 在哪? 參見 [什麼是 Linux vDSO 與 vsyscall?——發展過程](https://alittleresearcher.blogspot.com/2017/04/linux-vdso-and-vsyscall-history.html) > benefits to gettimeofday() is implemented with a userspace-only vsyscall/vdso, which avoids the syscall overhead. > [gettimeofday、clockgettime 以及不同時鐘源的影響](https://www.cnblogs.com/raymondshiquan/articles/gettimeofday_vs_clock_gettime.html) 在 32-bit 和 64-bit 環境還有更多考量,參見 [vDSO on arm64](https://blog.linuxplumbersconf.org/2016/ocw/system/presentations/3711/original/LPC_vDSO.pdf) 搭配閱讀: * [Anatomy of a system call, part 2](https://lwn.net/Articles/604515/) * [vDSO: 快速的 Linux 系統呼叫機制](https://hackmd.io/@sysprog/linux-vdso)