# 2022/08/23~24 OSSLA Records ###### tags: `ocf` `ossla` ## How Open Source Helped My Career: A History from an Average Joe - Fabricio Buzeto, bxblue [Presented in English] > https://osslatam22.sched.com/event/15Bqd/how-open-source-helped-my-career-a-history-from-an-average-joe-fabricio-buzeto-bxblue-presented-in-english ### Intro. 講者為巴西人,從 2002 開始 full-time coding 一開始跟一般人一樣,不知道 how does computer work, 不知道什麼是 open source 有天他的朋友問他,你知道什麼是 Linux 嘛? 他說不知道 朋友說就是作業系統呀,他還是不知道 最後朋友說,就像是 windows,但是是不同作業系統 他說,不是只有 windows, dos 嘛? 在大學是講者第一次接觸到 open source,就是從 Linux 系統開始。 因為身邊很多人參與、投入,所以也就自然而然加入。 Open source 不是一個 project,更像是一種社會運動。有時候難以說服人們投入,因為要怎麼保證他的 quality, 獲得報酬, 他如何運作? 等等的都是議題。 Windows 是許多人第一次接觸的作業系統,如果直接得跟他們說 "不,你應該擺脫 windows,投入 open source,投入 Linux 的懷抱",很多人不會接受,也會因此造成摩擦,這也不是一個良好的推廣方式。 很多人第一次接觸到的 open source 就是作業系統,而許多與 development 無關的 open source 人員其實並不知道什麼是 operating system。 當你看到其他 developer 使用 windows 而不是 Linux,如果你直接跟他說 "嘿,你應該使用 Linux 而不是 winodws",這樣只會造成更多的仇恨與摩擦,一眛的霸凌是沒有意義的,重點在於不管是哪個作業系統,這些公司與工程師都有貢獻 open source 就好,更多的討論才能使大家進步。 ### To contribute 作者提到 The Apache Software Foundation,有非常多的 open source 計畫 https://www.apache.org/index.html#projects-list Some open source programming languages: * ruby * python * javascript Jakarta Server Faces (JSF) 是講者第一次貢獻的 open source,也是 Apache 的計畫之一 講者改善了原本 unstable, unworked 的 code,試圖用他的方法 correct it,貢獻這個專案。 現在有非常非常多的 open source projects,鼓勵所有人都可以動手貢獻,與 open source 接觸。 ### Today's situation - 27M developers - 15M github accounts - 50M OSS projects - 3B working population 即便有這麼多的工程師跟 githuber,但實際在貢獻 open source 的人相對來講卻不多,unbalanced 相較於勞動力人口,developer 的比例很低 Open source 的領域很廣,一流的公司也都在努力推廣 - docker - HashiCorp - CONFLUENT - AUTOMATTIC TO build good open source -> good projects -> good company - React - ELECTRON - TensorFlow - git - oh my zsh ### The path/ways to open source The path: - Use - Learn - Share - Contribute - Create 起初可以先找個專案來用,學習怎麼使用,分享使用經驗與遇到的問題 learn 與 share 是可以大量接觸社群的好機會 等到更熟悉後,就可以貢獻回專案,甚至是創造專案 當然,流程是自由的,no limit Impact people in the world! The ways: - Community - Teach - Write - Talk about it 參與社群,教導別人,寫寫文章,或跟朋友討論 當更多人使用並貢獻,你的 software becomes more valuable 希望大家能從 Tech people -> Occasional OSS people ### 截圖證明 ![](https://i.imgur.com/jeHujGt.png) --- ## Tools and Techniques to Debug an Embedded Linux System - Sergio Prado, Embedded Labworks [Presented in English] > https://osslatam22.sched.com/event/15Br1/tools-and-techniques-to-debug-an-embedded-linux-system-sergio-prado-embedded-labworks-presented-in-english > 這個議程很多案例的講解,建議看投影片比較清楚 這些技巧除了 Embedded system, 在 desktop 上也可以用 講者有 25+ years 的經驗 on Embedded Linux, Embedded Android, RTOS ### THE SIX STAGES OF DEBUGGING - Denial: That can't happen. - Frustration: That doesn't happen on my machine. - Disbelief: That shouldn't happen. - Testing: Why does that happen? - Confirmation: Oh, I see. - Relief: How did that ever work? ---- ### Debugging a software problem might involve the following steps: - Understanding the problem. - Reproducing the problem. - Identifying the root cause. - Fixing the problem. - Fixed? If so, celebrate! If not, go back to step 1 了解問題 -> 重現問題 -> 確定問題根本原因 -> 解決問題 -> 確認是否解決 ---- ### THE 5 TYPES OF PROBLEMS - Crash. - Lockup/Hang. - Logic/implementation. worked, 但跟你想的不一樣 - Resource leakage. - (Lack of) performance ---- ### TOOLS AND TECHNIQUES - Our brain (aka knowledge). - Post mortem analysis (logging analysis, memory dump analysis, etc). 分析 log, not in runtime - Tracing/profiling (specialized logging). e.g. ftrace, strace - Interactive debugging (eg: GDB). - Debugging frameworks (eg: Valgrind). e.g. find memory leaks ---- ### Post mortem analysis 透過分析 logs, memory dump 找出問題 logs: any information stored by system Memory dump: 當有 application crash 時,kernel 會紀錄當時 memory 的 snapshot ---- ### Example: kernel crash There are program counter, backtrace in report use vmlinux: recompile with debug symbol ``` $ arm-linux-addr2line -f -p -e vmlinux 0xc06a21cc storage_probe at /opt/labs/ex/linux/drivers/usb/storage/usb.c:1118 $ arm-linux-gdb vmlinux (gdb) list *(storage_probe+0x60) 0xc06a21cc is in storage_probe (drivers/usb/storage/usb.c:1118). 1113 */ 1114 if (usb_usual_ignore_device(intf)) 1115 return -ENXIO; 1116 1117 /* Print vendor and product name */ 1118 v = (char *)unusual_dev->vendorName; 1119 p = (char *)unusual_dev->productName; 1120 if (v && p) 1121 dev_dbg(&intf->dev, "vendor=%s product=%s\n" , v, p); ``` ---- ### Example: user space crash - when `fping` segmentation fault - use `ulimit -c unlimited` cannot solve it - check file status `ls -la core` `file core` - `$ arm-linux-gdb src/fping -c core` open it by gdb ``` Core was generated by `fping -c 3 192.168.0.1'. Program terminated with signal SIGSEGV, Segmentation fault. #0 optparse_long (options=0xbe8e8914, longopts=0xbe8e89f8, longindex=0x0) at optparse.c:217 217 char *option = options->argv[options->optind]; ``` (in gdb) you can print variables to find problems (in gdb) `up`/`down` to check stack status ---- ### Tracing Runtime analysis tools collect & store some specialized form of logs. It's implemented via static and dynamic tracepoints (probes) in the code to instrument the software at runtime. Can be used for: - debugging - latency profiling - performance profiling ---- ### Example: Kernel Tracing 放了一個 bug 在 LED 燈號 利用 ftrace 來做 tracing > 講者用 `zcat` 來設定 tracing config 有些方法可以拿到 trace information - `mount -t tracefs tracefs /sys/kernel/tracing/` - `trace-cmd` 可以用 kernel-shark 來做視覺化 (btw, 我自己用過 perfetto 來做視覺化) ---- ### Example: User space Tracing #### strace - Run `netcat -l -p 1234` fail - Use `strace netcat -l -p 1234` #### uprobe - Use `zcat /proc/config.gz | grep CONFIG_UPROBE` - CONFIG_UPROBES=y - CONFIG_UPROBE_EVENTS=y - combine with `perf` ---- ### INTERACTIVE DEBUGGING - Allows us to interact with the application at runtime - On Linux systems, the most used interactive debugging tool is GDB. https://www.sourceware.org/gdb/ > can dprintf in gdb ---- ### EXAMPLE: KERNEL DEBUGGING WITH GDB 同樣用 LED 當例子 - use `zcat /proc/config.gz | grep ^CONFIG_KGDB` ``` CONFIG_KGDB=y CONFIG_KGDB_HONOUR_BLOCKLIST=y CONFIG_KGDB_SERIAL_CONSOLE=y ``` 再來要 boot linux kernel 進入 debuggy mode,可以在 boot time 或 runtime 做到,這裡是在 runtime - `echo g > /proc/sysrq-trigger` put kernel into debuggy mode ---- ### Summary ![](https://i.imgur.com/JF5r9av.png) print 並不總是有用,所以還是建議使用 debug tool 這些 tool 都很建議學習,總有一天會用上的 :) ### 截圖證明 ![](https://i.imgur.com/4VJr0uc.png) --- ## Keynote: Trust and the Linux Kernel - Greg Kroah-Hartman, Kernel Maintainer & Fellow, The Linux Foundation [Presented in English] > https://osslatam22.sched.com/event/15Bpc/keynote-trust-and-the-linux-kernel-greg-kroah-hartman-kernel-maintainer-fellow-the-linux-foundation-presented-in-english 講者是 Linux kernel maintainer Open souce is more trustworthy than closed source, because it can be audited by anyone Trust: 像 Linux kernel,open source 並不代表著 trust,我們想要讓使用者有一個 trust kernel 可以用,那我們就必須 verify all developers 但人數太多,這是 impossible task ![](https://i.imgur.com/V0ZGd00.png =600x) 79662 commits in 2021 - 17% is fix tag - 在 -final 前被 fixes 的只有 26% - 12% 在處理 old bugs 講者以自己為例: 說明貢獻越多的 developer,同時也會產出越多 bug XD "Make it easy to find and fix those bugs." ---- ### Lifecycle of a kernel change - Submit through email - Change -> reviewed / rejected - Resubmit - Average change takes 3 attempts #### Example: ![](https://i.imgur.com/hFNVjSo.png) #### 0-Day CI Linux* Kernel Performance Intel 提供的 Linux kernel 全面性測試服務 對不同 subsystem, developer tree 提供測試與 benchmark > 0-Day CI is an automated Linux* kernel test service that provides comprehensive test coverage of the Linux kernel. It covers kernel build, static analysis, boot, functional, performance and power tests. - [Linux* Kernel Performance](https://www.intel.com/content/www/us/en/developer/topic-technology/open/linux-kernel-performance/overview.html) - [0-Day CI Linux* Kernel Performance Report (V5.16)](https://www.intel.com/content/www/us/en/developer/articles/technical/0-day-ci-linux-kernel-performance-report-v5-16.html) #### KernelCI > [KernelCI](https://foundation.kernelci.org/) > [KernelCI Dashboard](https://linux.kernelci.org/) > KernelCI is a community-based open source distributed test automation system focused on upstream kernel development. The primary goal of KernelCI is to use an open testing philosophy to ensure the quality, stability and long-term maintenance of the Linux kernel. 由多個大公司共同合作開發 #### LKFT > [Linaro's Linux Kernel Functional Test framework | LKFT](https://lkft.linaro.org/) > [LKFT reports](https://qa-reports.linaro.org/lkft/) Linaro 開發的測試系統 - Testing for stable and Linus/s -rc realease - Testing for linux-next #### Guenter Roeck's test system ![](https://i.imgur.com/D3MtjXq.png) 利用以上工具來協助測試與開發 ---- ### Test every release Android, Nvidia, Huawei, Fedora, etc. 很多人與公司在做測試 可能是私人的,可能是公開的 誰做了測試? 測試了什麼? 使用 linux kernel 就是這些整合起來的成果。 你不應該只測試某一個獨立的部分,這樣只會造成問題,測試應該要涵蓋全部。 ---- ### Trust Linux kernel, but verify ant test 我們 trust 的不是 "永遠都是正確的",而是 "當發現錯誤時可以修正" 每個人都可以做這件事,信任、創造、修改 不論是新增一個 subsystem, driver, core 等 --- ## Revisit Multi-Reader Synchronizations for Scalable Applications - Jim Huang, BiiLabs & Shao-Tse Hung, National Yang Ming Chiao Tung University [Presented in English] > 我自己的 talk > https://osslatam22.sched.com/event/15BsE/revisit-multi-reader-synchronizations-for-scalable-applications-jim-huang-biilabs-shao-tse-hung-national-yang-ming-chiao-tung-university-presented-in-english