Yocto Project (6)：Testing and Debugging Openbmc service

## Yocto Project (6)：Testing and Debugging Openbmc service 前篇：[Yocto Project (5)：在樹梅派 4 上跑 Openbmc](https://hackmd.io/@zhonwei/BJqZEhSTbe) 目標：測試與確認 Openbmc service 是否可以正常運作 --- 確認 OpenBMC 版本 ```bash $ cat /etc/os-release ``` 確認基本服務是否正常運作 ```bash $ systemctl list-units --state=failed ``` 出現三個 fail ``` bmcweb.service failed obmc-phosphor-sysd.service failed start-ipkvm.service failed ```` 查看那三個服務的狀態 ```bash $ systemctl status bmcweb.service $ systemctl status obmc-phosphor-sysd.service $ systemctl status start-ipkvm.service ``` --- ## bmcweb.service ::: info bmcweb 是 OpenBMC 的前端網頁伺服器與 API 通道。它負責處理所有進入 BMC 的 HTTP/HTTPS 請求 ::: 研讀：[[OpenBMC] 快速上手OpenBMC的Redfish](https://iris123321.blogspot.com/2022/02/openbmc-openbmcredfish.html) 查看日記，裡面會記載錯誤原因 ```b $ journalctl -u bmcweb.service -n 50 Process: 215 ExecStart=/usr/bin/bmcweb (code=exited, status=255/EXCEPTION) ``` 參考 https://github.com/openbmc/bmcweb/issues/126 ::: danger 錯誤的原因是缺少了 `/var/log/redfish`，因為 bmcweb 啟動時會嘗試初始化 Redfish 的事件日誌系統，它的程式碼裡有一段邏輯會去開啟或寫入 `/var/log/redfish` 這個資料夾。所以我們就手動去建立資料夾，就可以解決這個問題了。 ::: 手動建立資料夾，重新啟動服務 ```bash $ mkdir -p /var/log/redfish $ systemctl daemon-reload $ systemctl restart bmcweb.service $ systemctl status bmcweb.service ``` 在重新查看一下 bmcweb.service 就會是 active（running）了但之後重開機的話又會顯示 bmcweb.service failed，這是因為 ```b $ df -h /var/log tmpfs 3.9G 8.0K 3.9G 0% /var/volatile ``` ```b $ ls -la /var/log /var/log -> volatile/log ``` `/var/log` 是指向 `/var/volatile/log` 的符號連結 ::: danger 又發生錯誤的原因是 OpenBMC 把 /var/volatile 掛載為 tmpfs（記憶體檔案系統），/var/log 只是指向 /var/volatile/log 的符號連結。 tmpfs 的特性是： * 資料存在記憶體裡，不寫入儲存裝置 * 每次重開機記憶體清空，所有資料消失 * 速度快，適合暫存資料所以手動建立的 `/var/log/redfish` 實際上是建立在記憶體裡，重開機後會自然消失。 ::: 解決方法是寫一個開機自動建立 /var/log/redfish 目錄的 service ```b cat > /etc/systemd/system/create-redfish-log-dir.service << EOF [Unit] Description=Create /var/log/redfish directory Before=bmcweb.service [Service] Type=oneshot ExecStart=/bin/mkdir -p /var/volatile/log/redfish RemainAfterExit=yes [Install] WantedBy=multi-user.target EOF ``` 啟用服務 ``` $ systemctl enable create-redfish-log-dir.service $ systemctl start create-redfish-log-dir.service ``` 重新開機確認 ``` $ reboot $ systemctl status bmcweb.service ``` --- ## obmc-phosphor-sysd.service ::: info obmc-phosphor-sysd 是 OpenBMC 架構中負責「系統狀態監控與管理」的背景服務，監控 BMC（基板管理控制器）內部的各種系統事件，並確保系統依照預定的狀態執行。 ::: > 參考：https://github.com/openbmc/phosphor-state-manager 查看 obmc-phosphor-sysd 的狀態 ```b $ systemctl status obmc-phosphor-sysd.service Active: failed (Result: exit-code) since Thu 1970-01-01 00:00:06 UTC Process: 328 ExecStart=/usr/bin/env obmc-phosphor-sysd (code=exited, status=1/FAILURE) Main PID: 328 (code=exited, status=1/FAILURE) Scheduled restart job, restart counter is at 2. Failed to start Phosphor System Manager. ``` 查看日記 ```bash $ journalctl -u obmc-phosphor-sysd.service -n 50 Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Python runtime state: core initialized LookupError: no codec search functions registered: can't find encoding ``` 確認 Python encodings 資料夾內容 ```b $ ls /usr/lib/python3.8/encodings/ __pycache__ ``` 確認只有 __pycache__，沒有任何 .py 檔案。測試 Python 是否能正常執行 ```b $ python3 -c "print('hello')" Could not find platform independent libraries <prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] Python configuration: PYTHONHOME = (not set) PYTHONPATH = (not set) program name = 'python3' sys.path = [ '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', ] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding LookupError: no codec search functions registered: can't find encoding ``` ::: danger 錯誤原因是 OpenBMC 為了縮小 image 大小，在編譯時透過 python3_%.bbappend 這個檔案，把所有 Python .py 檔案刪除，只保留 __pycache__。導致 obmc-phosphor-sysd 啟動時呼叫 Python，Python 找不到 encodings 模組，直接崩潰。 ::: > 參考：https://github.com/openbmc/openbmc/issues/3661 查看 python3_%.bbappend 內容（在 Ubuntu 虛擬機上） ```b $ cat ~/openbmc/meta-phosphor/recipes-devtools/python/python3_%.bbappend inherit update-alternatives ALTERNATIVE_${PN}-core += "python" ALTERNATIVE_LINK_NAME[python] = "${bindir}/python" ALTERNATIVE_TARGET[python] = "${bindir}/python3" # python3 takes up a lot of space that most embedded systems # do not have, so remove some un-needed files from the rootfs do_install_append_class-target() { # Even though python3 is built with --without-ensurepip, it still installs # a large, compressed version of pip. Remove it to free up the space. rm -rf ${D}${libdir}/python${PYTHON_MAJMIN}/ensurepip # Remove all python .py files from python recipe. Only the .pyc # files are required. # The _sysconfigdata*.py files are system configuration files generated # during build time. It's used in the yocto packaging process so # it is required to remain in the image. find ${D}${libdir}/python${PYTHON_MAJMIN} -name \*.py ! -name _sysconfigdata*.py -exec rm {} \; } ``` 把這行整段刪除 ```bash $ find ${D}${libdir}/python${PYTHON_MAJMIN} -name \*.py ! -name _sysconfigdata*.py -exec rm {} \; ``` 之後再重新編譯 ```b $ cd ~/openbmc $ source openbmc-env $ bitbake obmc-phosphor-image ``` 編譯好之後，image 檔會在 `/home/zhonwei/openbmc/build/tmp/deploy/images/` 複製 image 檔到共享資料夾 ```b $ sudo cp /home/zhonwei/openbmc/build/tmp/deploy/images/raspberrypi4/obmc-phosphor-image-raspberrypi4-20260427072528.rootfs.rpi-sdimg /media/sf_shared/ ``` 重新燒錄到 SD 卡中開機，會出現前面有出現的錯誤，再按照前面的方式解決一次就可以了確認 Python 正常 ```b $ python3 -c "print('hello')" hello ``` 確認 obmc-phosphor-sysd 服務正不正常 ```b $ systemctl status obmc-phosphor-sysd. Active: failed ``` 還是失敗，再次查看詳細錯誤原因 ```b $ journalctl -u obmc-phosphor-sysd.service -n 30 SyntaxError: Missing parentheses in call to 'print'. Did you mean print("obmc-phosphor-watchdog starting...")? File "/usr/sbin/obmc-phosphor-sysd", line 26 ``` ::: danger 錯誤原因：obmc-phosphor-sysd 是用 Python 2 語法寫的，但系統上的 python 已經指向 Python 3.8.2，Python 3 不支援舊的 print 語法。 ::: 確認 python 版本 ```b $ python --version $ python3 --version $ ls -la /usr/bin/python Python 3.8.2 Python 3.8.2 /usr/bin/python -> /usr/bin/python3 ``` 修正 /usr/sbin/obmc-phosphor-sysd 裡的 print 語法 ```bash $ sed -i 's/print "obmc-phosphor-watchdogd starting..."/print("obmc-phosphor-watchdogd starting...")/' /usr/sbin/obmc-phosphor-sysd ``` 重新啟動服務 ```bash $ systemctl restart obmc-phosphor-sysd.service $ systemctl status obmc-phosphor-sysd.service ``` obmc-phosphor-sysd.service 成功啟動！ ::: info 補充：可以查看 service 的 Unit file 來了解這個 service 的內容 ::: ```b $ cat /lib/systemd/system/obmc-phosphor-sysd.service [Unit] Description=Phosphor System Manager [Service] Restart=always ExecStart=/usr/bin/env obmc-phosphor-sysd SyslogIdentifier=obmc-phosphor-sysd [Install] WantedBy=multi-user.target ``` --- ## start-ipkvm.service ``` $ journalctl -u start-ipkvm.service -n 50 Process: 343 ExecStart=/usr/bin/obmc-ikvm -v /dev/video0 -k /dev/hidg0 -p /dev/hidg1 (code=dumped, signal=ABRT) ``` KVM 服務直接崩潰，這個在樹莓派上是正常的，因為樹莓派沒有 KVM 硬體，可以忽略。

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.