[TOC]
總體來說，因為我要opengl 所以我寧可是整個虛擬機而不單是cross compiler。
再來 docker command 太近shell script，所以
* opengl 必要, glibc 必要
1. 虛擬機
* 我更傾向虛擬機，因為更還原整體。build tool 指定路徑很繁瑣，很多語言的build tool 甚至是沒有考慮。
* 我不是單純要有 cross compiler 的虛擬機。而是要有包含opengl的整個環境
* 關鍵字是要有 QEMU
* [ Why should we use vm for cross compilation ? ](https://www.reddit.com/r/linuxquestions/comments/123aiji/comment/jdvehqb/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
3. rsync + cross compiler
1. 直接拿 SDK 而非 rsync 是更正規的作法。[Getting started with NXP MPUs: i.MX 7 - OpenGL - fbdev](https://doc.embedded-wizard.de/getting-started-imx7-opengl-fbdev)
2. glibc 一併copy 過來。
# 別人的部份範例
* [ CI/CD for embedded software development ](https://www.reddit.com/r/embedded/comments/mo8ha5/cicd_for_embedded_software_development/)
* 整套流程都有，但沒說compiler是？
* > Our whole toolchain is put in a docker container so jenkins uses the same setup as we developers. Additionaly we use spinnacker to create a sandboxed backend for all conncetion tests. (Creates e.g. s3 buckets spin ups all needed pods in the kubernetics and setups public access point to test the system). The hardware is connected with a special testadapter to a rasperry pi, and automated tests for Hardware in the loop are exectued on the pis.
* [ Must-Haves in CI/CD Software Development for Embedded | Parasoft ](https://www.youtube.com/watch?v=0FspWijeAOs)
* [How to Build a Continuous Integration and Delivery Process for Embedded Software](https://medium.com/jumperiot/how-to-build-a-continuous-integration-and-delivery-flow-for-embedded-software-b0b5bf220a2)
* [CI/CD For Embedded Systems](https://www.linkedin.com/pulse/cicd-embedded-systems-martin-fogarty/)
* 這是cross compile
* [ What you need to know for CI/CD for the IoT.](https://beetlebox.org/what-you-need-to-know-for-ci-cd-for-the-iot/)
* > Simulation and emulation. The first challenge we can overcome is dealing with the need for hundreds of hardware boards in our testing system. We also need to ensure that the team’s code is tested within the same clean environment. This can be done by simulating or emulating our hardware to form virtual platforms where hundreds of tests can be run. These virtual platforms form a single source of truth for developers to test from as well. To do this, code is taken from the version control system and cross compiled for our chosen IoT device. Our server then acts as a host machine for up to hundreds of virtual platforms, which can run the builds as if running on actual hardware. We can then encourage developers to keep building and testing without added expenditure.
* cross compile 是必要的。
* [Cross-Compiling Tauri Applications for ARM-based Devices](https://tauri.app/v1/guides/building/linux/#experimental-automatic-cross-compilation-using-a-github-action)
* tauri 的方法是直接在QEMU+ raspberry 官方image 裡compile。
* 一定要online 執行 action 很不好，但 local 可能有command 不支援 e.g. `upload-artifact`
* [ Support artifacts #169 ](https://github.com/nektos/act/issues/169)
* 現在可以了，但功能有限。[ Asset server implementation #677 ](https://github.com/nektos/act/pull/677) 非正式記載
* 要我說，tauri 做的不好，光強制 `upload-artifact` 就out了，自己shell script manual compile。
## 結論
1. docker with cross compiler compile code
2. docker with QEMU run testing
嚴格來說兩者都必要，無法偏廢。只是說，當target system 在某些case ，可以以虛擬機執行起來compile(not all OS like linux have compiler)，可以省略掉cross compiler。 情況是活的。
在虛擬機compile，理論上可以，但還是無法保證動態連接相同 e.g. glibc, opengl。所以是有不確定性的。
但是現在先基本功，維持完整。
# 虛擬化
* [ QEMU vs KVM: A Quick Comparison ](https://www.linkedin.com/pulse/qemu-vs-kvm-quick-comparison-raja-nagori-/)
* [如何用QEMU在x86電腦模擬ARM版樹莓派系統，虛擬機跑Raspberry Pi OS](https://ivonblog.com/posts/emulate-raspberry-pi-os-on-x86-linux/)
* qemu 性能1/5，太低了。他是軟體runtime 翻譯。
* 我猜要實用了話，也是先開KVM 虛擬機，裡面在跑docker。
* [ TCG-Continuous-Benchmarking Public](https://github.com/ahmedkrmn/TCG-Continuous-Benchmarking)
* 看來qemu 比較適合 執行，compile 還是要靠 cross compiler。
* [KVM: Linux 虛擬化基礎建設](https://hackmd.io/@sysprog/linux-kvm#QEMU)
* QEMU-KVM 把 QEMU 的底層換成KVM 加速。
可以預料的是，如果有用到KVM，相容性的問題會比較大，但我們頂多用arm而已，應該沒差。
看來qemu 比較適合 執行，compile 還是要靠 cross compiler。
# 軟體相容性
* cross compile 有分 build 和 host(target) machine
* [relationship of c compiler and c standard library](https://stackoverflow.com/questions/50459459/relationship-of-c-compiler-and-c-standard-library)
* glibc 綁的是distro 而非kernel。distro compile 時的lib 都是要被dynamic link的。所以有版本問題。
* compiler 包含glibc，可以解決 host(target) machine glibc問題。
* 我如果host 的glibc 和 build machine 不同，能不能透過切換 cross-compiler 解決？:**能！！！**
* 可以將distro 比喻為 鎖，compiler 比喻為 鑰匙，package manager 比喻為代理人。軟體大到一個程度，必須要 package manager 代為compile (開鎖)，package manager 能否達成目標是一大關鍵。
* compiler 是否是glibc的最小版本。或是說 我可以要求任意cross compiler link 任一版 glibc 以完成compile 嗎？
* [How can I link to a specific glibc version?](https://stackoverflow.com/a/52550158)
* [Verify the glibc version corresponding to a specific gcc compiler](https://copyprogramming.com/howto/check-glibc-version-for-a-particular-gcc-compiler)
* **不行**，已經是最小單位了。
## stdc++
then C++ also suffer from similar problem but much easier, since it's not coupling to OS, distro.
[How do you find what version of libstdc++ library is installed on your linux machine?](https://stackoverflow.com/a/10355215)
`/sbin/ldconfig -p | grep stdc++`
`strings /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 | grep LIBCXX`
to update
`apt list libstdc++6`
[ How do I run a C++20 program on my Pi 1B? ](https://www.reddit.com/r/raspberry_pi/comments/13aktpk/how_do_i_run_a_c20_program_on_my_pi_1b/)
直接 copy libstdc++ 過去 or static link
先static link (失敗，conan cxx flag 怪怪)
copy lib, then LD_LIBRARRY_PATH 成功。但 create window failed
## static link
static link 是一個可能的接決方式，但需要os level support。
[Musl libc：为什么我们会需要另一个 libc？](https://linuxstory.org/musl-libc-yet-another-libc/)
[ Alpine Linux does not make the news ](https://news.ycombinator.com/item?id=36867992)
## opengl
opengl 只能 dynamic link
遇到glfw init 成功， open window fail
[glfwCreateWindow returns null](https://github.com/glfw/glfw/issues/2019)
find opengl version
`glxinfo -B"`
`glxgears -info`
會從中間到下 ，分別顯示opengl 和 opengl es 所有的驅動，e.g. mesa, amd 最後是es的。
注意 並非都support，建議先用demo example build 試試，可以用再選該backend。opengl 常常是 compile 都能過，static link 才知道有問題。
# basic
## docker
* [docker run](https://docs.docker.com/engine/reference/commandline/run/)
* -it interactive+pseudo tty
* --rm Automatically remove the container when it exits
* --volume -v Bind mount a volume
* --platform API 1.32+ Set platform if server is multi-platform capable
* [Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide](https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/)
* --privileged Give extended privileges to this container
* --publish -p Publish a container's port(s) to the host
* -d daemon
* copy
* `docker cp 464ab:/root/c_difeqtrol /run/user/1000/gvfs/sftp:host=192.168.0.101/home/pi/conan_demo/`
* docker manifest
* [ 【Day 3】 - Docker 基本指令操作 ](https://ithelp.ithome.com.tw/articles/10186431)
* [ Day6：把 Docker Image Push 到 Docker Hub ](https://ithelp.ithome.com.tw/articles/10191139)
* `sudo docker container prune` delete all stopped container.
* [[小抄] Docker 基本命令](https://yingclin.github.io/2018/docker-basic.html)
*
* [Install Docker Desktop Dashboard on Ubuntu 22.04 LTS Linux](https://linux.how2shout.com/how-to-install-docker-desktop-gui-on-ubuntu-22-04-lts-linux/)
* 怪怪的
* 是root 和 user ，所以沒顯示？
* [Portainer：一款圖形化容器管理工具 ](https://vocus.cc/article/643e9f74fd89780001b51739)
* [How can I use Docker without sudo?](https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo)
* [針對已啟用 Docker Container 修改 Port](https://roychou121.github.io/2020/07/15/docker-container-change-port/)
* [How to SSH into a Running Docker Container and Run Commands](https://phoenixnap.com/kb/how-to-ssh-into-docker-container#ftoc-heading-5)
* I don't like to modify port(port forwarding).
* 直接用zerotier更快?
## rpi
* [Check Whether Raspberry Pi OS is 32-bit or 64-bit](https://lindevs.com/check-whether-raspberry-pi-os-is-32-bit-or-64-bit)
* `log-bit`
* `uname -a`
* `getconf LONG_BIT`
* `cat /etc/os-release`
* usr:`pi`,pwd:`raspberry`
# QEMU itself
恐怕要先熟悉qemu正常運行的方法。
[How To Use QEMU From the Linux Command-Line](https://www.baeldung.com/linux/qemu-from-terminal)
# KVM itself
先裝arm vm, 上面再docker
[ Linux QEMU-KVM 教學 ](https://www.youtube.com/watch?v=702H2gkJzCE&t=1s)
如果說沒connection
[virt-manager can't connect to libvirt](https://askubuntu.com/questions/345218/virt-manager-cant-connect-to-libvirt)
tpm emulate 在 armhf 上沒有，所以我直接uninstall 它。
進uefi 界面，失敗。
* [How to run the Raspberry Pi Os in a virtual machine with Qemu and Kvm](https://linuxconfig.org/how-to-run-the-raspberry-pi-os-in-a-virtual-machine-with-qemu-and-kvm)
* 可惜是raspberry
* 看起來kvm 要針對 distro, hardware 修改
* [qemu-rpi-kernel](https://github.com/dhruvvyas90/qemu-rpi-kernel)
* 不太知道這裡說的kernel 是什麼
* arm kvm on x86 好像是沒有加速的
* [How to instantiate an ARM-based VM through Linux KVM API on x86?](https://stackoverflow.com/a/69523523)
* [QEMU with KVM with differing host/guest architectures](https://unix.stackexchange.com/a/341068)
* 那沒救，除非買一台arm server
* 要回去arm qemu了。
# docker + QEMU
* [Running and Building ARM Docker Containers on x86](https://devopstales.github.io/linux/running_and_building_multi_arch_containers/)
* [qemu-user-static](https://github.com/multiarch/qemu-user-static)
* 關鍵，但我還看不太懂。
* 不只單一arch，好像支援多種
* [docker-qemu-rpi-os](https://github.com/carlosperate/docker-qemu-rpi-os)
* [dockerpi](https://github.com/lukechilds/dockerpi)
* 圖解
* 但不要直接用這個，是rpi2的
* 一般docker 關掉下次不要 create new container 而是start container，資料還在？
* [Emulating a Raspberry Pi in QEMU](https://interrupt.memfault.com/blog/emulating-raspberry-pi-in-qemu)
* detail teach to create rpi image from source. run it with QEMU. set up dockerfile to build image with rpi image and QEMU.
* [ Using Docker to Cross-Compile ](https://www.ev3dev.org/docs/tutorials/using-docker-to-cross-compile/)
* 這個比較沒激進，先run 起container 在 volume 裡compile。
* `docker run --rm -it -v C:\Users\myname\example\:/src -w /src ev3cc`
## docker image
* [How to cross compile using Docker](https://www.bytescale.com/blog/how-to-cross-compile-using-docker/)
* [[教學] 用 Docker 的 buildx 輕鬆多架構編譯 (multi-architecture build)](https://blog.jks.coffee/docker-multi-architecture-build/)
* [Multi-platform](https://docs.docker.com/build/building/multi-platform/)
* [docker debian](https://hub.docker.com/_/debian/tags?page=1&name=buster)
* docker 官方提供的 ubuntu 沒有ver 18的(是debian buster)。debian 就有。
* buster 原版只有gcc8，選backport(向後移植)也沒有
* gcc
* [ Upgrading the GCC, GNU compiler in Buster ](https://www.reddit.com/r/debian/comments/mmure2/upgrading_the_gcc_gnu_compiler_in_buster/)
* [gcc-10-debian-buster.sh](https://gist.github.com/s3rvac/76ac07f21d5635209accc11fa56bd3fb)
* [install gcc-9 on Ubuntu 18.04?](https://askubuntu.com/a/1149383)
* gpg keyserver error ，加不上
* 而且這是ubuntu。
* 還是直接compile了？不會永遠都有人做好的。
`docker run --platform linux/arm/v7 debian:buster-backports`
can try some command
`docker run --platform linux/arm/v7 debian:buster-backports uname -a`
我們要的是compile，要 volume
`docker run -it -v ./foc_simu:/src -w /src --platform linux/arm/v7 debian:buster-backports`
> 下載和安裝速度明顯比之前rpi qemu快，不知原因
```bash
apt update
apt-get -y install curl build-essential python3 python3-pip software-properties-common wget git nano pkg-config
pip3 install -U pip setuptools wheel
add-apt-repository --yes ppa:ubuntu-toolchain-r/test
apt update
~~apt install -y gcc-9 g++-9~~
apt install -y gcc-11 g++-11
add-apt-repository ppa:deadsnakes/ppa
apt-get update
apt-get install -y python3.8
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
apt-get install python3.8-venv
python3 -m venv conan_build
source conan_build/bin/activate
pip3 install -U pip setuptools wheel
yes | pip3 install conan
apt install -y libgl1-mesa-dev libegl1-mesa-dev
sudo apt-get install libglfw3-dev libgles2-mesa-dev
```
[update python version](https://www.itsupportwale.com/blog/how-to-upgrade-to-python-3-10-on-ubuntu-18-04-and-20-04-lts/)
[How do I get EGL and OpenGLES libraries for Ubuntu running on VirtualBox?](https://askubuntu.com/questions/244133/how-do-i-get-egl-and-opengles-libraries-for-ubuntu-running-on-virtualbox)
for x86 cross compiler
```bash
gcc-arm-linux-gnueabihf
```
---
### rust part
建議新的container
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
#tauri
sudo apt update
sudo apt install libwebkit2gtk-4.0-dev \
build-essential \
curl \
wget \
file \
libssl-dev \
libgtk-3-dev \
libayatana-appindicator3-dev \
librsvg2-dev
rustup update
cargo install create-tauri-app --locked
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash #install nvm
nvm install lts/gallium #install npm and nodejs, prevent encounter glibc problem
npm install
cargo tauri build
```
`/tmp` 有問題 `chmod 1777 /tmp`
好像是cargo 自己的
[cargo install fails with tmpdir noexec](https://github.com/rust-lang/cargo/issues/4350)
總體來說，在tmp裡有exec 都是不對的。
但因為我不知為何，user create 給權限沒生效(docker 限制，[ Running a Docker Container with a Custom Non-Root User: Syncing Host and Container Permissions ](https://dev.to/izackv/running-a-docker-container-with-a-custom-non-root-user-syncing-host-and-container-permissions-26mb))，remount也沒用, 就先用bypass的了。
[cargo install: specify a /tmp substitute?](https://stackoverflow.com/questions/64572901/cargo-install-specify-a-tmp-substitute)
最後成功
# docker + cross compiler
單純speed up CICD
* [How to Cross Compile for Raspberry Pi Using Docker](https://desertbot.io/blog/how-to-cross-compile-for-raspberry-pi)
* 看 docker file 的 `FROM` 可以清楚判斷。
* 用 docker volume pass file in
*
* [steeve/cross-compiler](https://github.com/dockcross/dockcross)
* 還是x86架構下的cross compiler，但還會附emulator
* 有內建 conan,vcpkg,cmkae,meason 之類的 C++ build tools
* 如果不需要distro 對應 glibc, opengl，是很好的solution
* [Continuous Integration for C/C++ on Raspberry Pi: Part1 – Cross Compile Using Docker](http://funputer.com/en/2022/continuous-integration-c-on-raspberry-pi-part2-jenkins/)
* 問題是它強制其他人把docker container 當時cli。 然後conan 要先detect profile。永遠用不了。
* [dockcross/linux-armv7/Dockerfile.in](https://github.com/dockcross/dockcross/blob/master/linux-armv7/Dockerfile.in)
* [CI/CD for Amiberry using Github Actions (pt 3)](https://blitterstudio.com/ci-cd-for-amiberry-using-github-actions-pt-3/)
* [Building the Cross-Compilation Toolchain](https://tttapa.github.io/Pages/Raspberry-Pi/C++-Development/Building-The-Toolchain.html)
* 這個用法的docker 是拿來產生cross compile toolchain的。拿到output 的toolcahin 在local使用。
* 有gcc13
* [docker-arm-cross-toolchain](https://github.com/tttapa/docker-arm-cross-toolchain)
* default 沒有 opengl，要自己重compile toolchaine 加入。
* [Cross-Compile OpenCV 3.4.0 to Raspberry Pi 3](https://vbob.medium.com/cross-compile-opencv-3-4-0-to-raspberry-pi-3-1f62a455989b)
* 一樣是沒opengl沒用。
# copy(e.g.rsync)+toolchain
* 這的前提是拿到的croos compiler 版本glibc 和host 一樣，否則沒用。
* [Raspberry Pi: How to cross-compile against third-party libraries](https://deardevices.com/2019/12/25/raspberry-pi-sysroot/)
* code 一定是裡面原版的
* 很多langugae 不一定能設 toolchain path or 設定繁瑣。
* copy 表示要該硬體，可得性差。cobine with docker?
## conan
this command can check if sysroot take effect
`conan graph info . -pr rpi3_gcc13 -c tools.build:sysroot=/run/user/1000/gvfs/sftp:host=10.147.17.77/`
* conan install 可能下載binary, conan create 強制compile from source
* [conan config](https://docs.conan.io/2/reference/commands/config.html?highlight=sysroot)
* conan 有很多config e.g. sysroot 算方便很多。 但有take effect？
* 雖然graph info裡沒寫 sysroot，但我如果用x86的 compiler+對應 rpi sysroot 編譯，會變成編不了，所以應該是有
* 問題可能在opengl的 check condition 是用 apt?
* 沒辦法關掉dependency，只能覆蓋。[Dependencies conflicts](https://docs.conan.io/2/tutorial/versioning/conflicts.html)
* [ [question] How to make system packages from CCI work with an alternative sysroot, i.e. Yocto #14990 ](https://github.com/conan-io/conan/issues/14990)
* 已經有人發現任何system package 的recipe 都會造成這個問題
* 必須要手動更改recipe 並且安裝。
*
* 總體來說，短期內，沒有任何正確長遠的方法讓conan with opengl work。這對c++ 開發是一大傷害，因為不可能再退回以前那種開發方式。
* 好在backend 這邊沒有opengl 只要搞定glibc 還能沿用conan。
* boost 不知道什麼原因，就算 cross compile也有問題，可能是裡面
* 照著改，[ [question] How to cross compile boost to armhf in a Docker context #15480 ](https://github.com/conan-io/conan-center-index/issues/15480#issuecomment-1488941694)
* [ What is CHOST? ](https://www.linuxquestions.org/questions/linux-general-1/what-is-chost-701439/)
* 再把compiler 原本 `arch=armv8` --> `arch=armv7` boost 有過，換openssl。有compile過。
*
* [ [question] What is Conan 2's story for deploying cross compiled binaries to a remote machine? #13565 ](https://github.com/conan-io/conan/issues/13565)
* [Deployer](https://docs.conan.io/2.0/reference/extensions/deployers.html) 是 conan 處理cross compile 的方法，否則share lib 無法一起到remote。
* `conan build . --deployer=full_deploy`
* [Creating a Conan-agnostic deploy of dependencies for developer use](https://docs.conan.io/2.0/examples/extensions/deployers/dev/development_deploy.html#examples-extensions-builtin-deployers-development)
* `sed -i 's|/home/morgana/c_difeqtrol/build/Release/generators|/home/pi/conan_demo/c_difeqtrol/build/Release/generators|g' *`
* > The binary variability is limited to Release/Debug. The generated files are exclusively for the current configuration, changing any other setting (os, compiler, architecture) will require a different deploy
* deploy是完全copy 操作，比如croos compile arch 是 x86，deploy 出的 cmake 還是 x86 (非arm)。
* `export PATH=$PATH:/home/pi/conan_demo/c_difeqtrol/full_deploy/host/`
* `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/pi/conan_demo/c_difeqtrol/full_deploy/host/`
* `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/pi/conan_demo/c_difeqtrol/full_deploy/host/cmocka/1.1.5/Release/armv7/lib/`
* this work
* but encouter glibc unmatch problem
* 更換完compiler 就能跑了，所以compiler 包含glibc，可以解決 host(target) machine glibc問題。
* 注意，我所使用的compiler是完整toolchain (該dir下有sysroot, `libc.so`...)
* p.s.每次改完profile 要 delete build folder 重來。不然會按原本的cmake。
*
# npm
* [從零開始，在Raspberry Pi上執行及打包Electron](https://medium.com/@wrre/%E5%BE%9E%E9%9B%B6%E9%96%8B%E5%A7%8B-%E5%9C%A8raspberry-pi%E4%B8%8A%E5%9F%B7%E8%A1%8Celectron-9f1e4ae5b62)
* [Build Instructions (Linux)](https://www.electronjs.org/docs/latest/development/build-instructions-linux)
* 可以看到肯定還是會遇到glibc問題。
*
## cargo
* rust compiler called Rustc
* [如何在Linux作業系統上交叉編譯Rust程式？](https://magiclen.org/rust-cross-compile/)
* rust 有用到 C/C++ 程式--> 繼承 glibc 問題。
* 純rust 程式，只要有對應的 rustc，就能搞定。
* [Building targets for cross-compilation](https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html#building-targets-for-cross-compilation)
* 可以自己compile，但明顯是個工程。
* [cross-compilation](https://rust-lang.github.io/rustup/cross-compilation.html#cross-compilation)
* `rustup target list`
* [Compile Rust for Raspberry Pi ARM](https://medium.com/swlh/compiling-rust-for-raspberry-pi-arm-922b55dbb050)
* 可以看到rust 需要的 glibc, linker ... 都是
* [Toolchains](https://rust-lang.github.io/rustup/concepts/toolchains.html#toolchains)
* 看起來可以support custom toolchain?
* [cc-rs](https://github.com/rust-lang/cc-rs)
* rust 核心功能的一部分
* A library to compile C/C++/assembly into a Rust library/application.
* `rustup target add` ...
* [ cross](https://github.com/cross-rs/cross)
* 用docker 來處理 compile C++的部份
* glibc required fix at `2.31`
* [How can I specify the GLIBC version in cargo build for Rust?](https://stackoverflow.com/a/75264059)
* 最後也選 docker 來得到舊的glibc
# summary
其實rust, cargo 就是所有非C/C++語言的縮影，這些語言或多或少都以C/C++ 技術為底。所以如果在舊的glibc平台下 glibc的問題要優先解決，才有可能輪到後面。這個過程就是先 glibc checking --> gcc compiler build --> compile language specific runtime, compiler (nodejs, chromium, python...) --> compile c/c++ lib dependency --> your application code。
虛擬機在這幾個 proccess 都有機會縮減，理想上能大幅縮減，具體能減少多少不一定。
理論上，可以把任何program compile 到 任何target machine 上，實際上，肯定是要一個平衡點，而非無限支援的。
要我說，這個平衡點（停損點）就是glibc。更確切的說，當你想要使用的某個 語言（版本）（或是很大的framework，大到需要自己的基礎設旋 e.g. GN, qmake），在各種docker image都找不到linux distro 提供該glibc版的基礎語言設施(compiler, runtime)了話，就是升級target/host 的時候。
ubuntu/debian --> centos8/opensuse。這4個以外就別搞了，連pkg server 的速度可能都有問題。
因為，一個語言的底層非常龐大，上面更有各式各樣的lib，會一直卡在 lib <--> compile tool 的循環。如果你需要先搞定gcc 後從0開始搭建，肯定不用做事了。這也是C/C++的優勢，因為是其它語言的起源，所以沒有這個問題。
至於 arch是否相同無所謂，有利有弊，build machine 只要能夠compile to target 就好。
> 非x86 e.g. arm 的distro support 會比較差，有些 package, update 沒有。cross compile速度更快。但是 apt預設可以抓到 opengl 之類的system lib，且可以直接執行target code。
> 總之，重點是repo 群，誰有package誰優先。
>
## experiment
e.g. rpi (target) 是 glibc 2.28，目標是有c++17的compiler，最低 gcc9
ubuntu18 是 glibc2.28 但沒有 official docker 。debian 有buster，但沒有透過apt upgrade gcc 的方法。debian 直接out。試試3rd ubuntu18?
> openSUSE Leap 15.2 是另一符合版
[lj020326/ubuntu1804-systemd-python](https://hub.docker.com/r/lj020326/ubuntu1804-systemd-python)，這個有x86 & arm
建議不要用default run command，鎖太多東西
`docker run --platform linux/arm/v7 -d -t --name systemd-ubuntu -v ./content:/content -w /content lj020326/ubuntu1804-systemd-python`
先用x86 ver，因為需要apt different server to update gcc。
[Package gcc-9-arm-linux-gnueabihf](https://packages.ubuntu.com/search?keywords=gcc-9-arm-linux-gnueabihf)
沒有 cross compiler, x86 ver 直接下去。
換arm 版，過了！！就用這個。
馬上遇到x86下沒有的問題 [No module named Markupsafe Error in jetson nano](https://stackoverflow.com/questions/70316777/no-module-named-markupsafe-error-in-jetson-nano) ~~照著改沒用，先update python version to 10
無法，卡postgresql10，解決不了。
改裝python8，不知為何過了。
還是遇到markupsafe，改用 `python3.8 -m pip install conan==2.0.6` ~~
[ "No module named 'markupsafe'" when installing MarkupSafe #285 ](https://github.com/pallets/markupsafe/issues/285)
after update pip, now I can install conan
不知是何原因，cmake版本小就一直拿到conan1 的recipe。
版本大，就只install binary 說只支援armv8
包括virtual env 都用了。
受不了，用add apt server, install. then disable tool require
最終compile foc_simu 成功
copy conan recipe
`conan create 3.x.x/ --version 3.25.3 --build=missing` 才成功。應該是 recipe 搞混到binary 那裡去了。
我compile 有加 -static flag，避免copy dynamic lib.
編譯c_difetrol 有過，可以跑。
但是foc_simu 用到opengl，基本上沒有任何一般方法可得到opengl 的static lib，所以就報錯了
profile 轉回 `-static-libgcc` 雖然這flag 好像沒用。
發現rpi 是用 opengl 2.1/glsl 120, and opengl es 2.0/glsl es 1.0.16。same as official document [VC4](https://docs.mesa3d.org/drivers/vc4.html)。 我原code 是用 opegl 3的。
原code 就有，只是我把它刪了
[example_glfw_opengl3](https://github.com/ocornut/imgui/blob/master/examples/example_glfw_opengl3/main.cpp)
gl es 和gl差很大。
[OpenGL ES 2.0 vs OpenGL 3 - Similarities and Differences](https://stackoverflow.com/questions/9629897/opengl-es-2-0-vs-opengl-3-similarities-and-differences)
又遇到 Failed to initialize OpenGL loader，應該是 [ImGui_ImplOpenGL3_Init](https://github.com/ocornut/imgui/blob/3733b5064e1820dc8517fedba2b34bfbdef569ab/backends/imgui_impl_opengl3.cpp#L269)
[ Vsync and run problems with example_sdl_opengl3 on RPi4b Bullseye #5065 ](https://github.com/ocornut/imgui/issues/5065)
最後懷疑是opengl2, opengl3 的版本問題，先下載imgui的demo。發現opengl3 example可以compile不能執行，opengl2 example可以正常執行。
做新的opgl2的foc_simu 版，就可以了。
奇怪的地方是opengl3 backend 的有opgl es 2，不知道為何不行。
---
rust_init file can't at /tmp/
dns error --> `/etc/resolv.conf` add
```
nameserver 8.8.8.8
nameserver 8.8.4.4
```
# build cross compiler(compiler build)
* 不用想，就用 [crosstoll-ng](https://github.com/crosstool-ng/crosstool-ng) 就對了
* [How to Build a GCC Cross-Compiler](https://www.cntofu.com/book/46/raspberry_pi/how_to_build_a_gcc_cross-compiler.md)
* [Building GCC from source](https://medium.com/@darrenjs/building-gcc-from-source-dcc368a3bb70)
* 連不上，先ping 一下。連不到就加name server。[How to Resolve the "Temporary failure in name resolution" Error](https://phoenixnap.com/kb/temporary-failure-in-name-resolution)
* `arch_flags` 要先改掉。`-march=armv7`, modify `build_target=armv7-unknown-linux-gnu`
* ~~`with-tune` 改 `--with-tune=cpu`~~，直接註銷
* 可以compile了，發現沒zlib, `apt install zlib1g-dev`
* ["fatal error: bits/libc-header-start.h: No such file or directory" while compiling HTK](https://stackoverflow.com/questions/54082459/fatal-error-bits-libc-header-start-h-no-such-file-or-directory-while-compili)
* [ Hi, does anyone know why I can't install the package gcc-multilib on Ubuntu 20.04? It is absolutly necessary for starting a Yocto project. Does anyone know how I can fix this? (see DockerFile in description) Thanks for the support! ](https://www.reddit.com/r/docker/comments/10uya5l/hi_does_anyone_know_why_i_cant_install_the/)
* can't install `gcc-multilib`
* [Cross-Compiling for Raspberry Pi](https://github.com/crosstool-ng/crosstool-ng)
# compile
* [Linux gcc鏈接動態庫出錯：LIBRARY_PATH和LD_LIBRARY_PATH的區別](https://www.cntofu.com/book/46/linked_and_loader/linux_gcclian_jie_dong_tai_ku_chu_cuo_ff1a_library.md)
* before compile: LIBRARY_PATH, after get executable:LD_LIBRARY_PATH
*
# log
* 2023/11/14
* docker 裡 image run conan 遇到 `ERROR: Can't write version file in '/home/pi/.conan2/version.txt': [Errno 28] No space left on device: '/home/pi/.conan2'` ，但宿主機明顯還有container space。
* 我下 `sudo conan --version` 就可以。就好像 pi default user 無權用space。
* `dockcross/linux-armv7-lts` not useful
* 2023/11/20
* docker arm qemu image compile tauri success. tauri is slow on rpi3b+.
*
Sign in with Wallet
Connect another wallet