# SUSE Edge Image Builder 實作
## 實作目標
OS 在第一次開機後自動安裝和設定好 SL Micro 6.0 + RKE2 v1.31.3 (ALL in One) + Rancher Prime v2.10.1。
## 環境準備
1. 一台 x86_64 實體主機或虛擬主機，作業系統是 SLES 15 SP6, openSUSE Leap 15.6 或 openSUSE Tumbleweed
2. 在建立 EIB image 之前，請確定您的系統上已安裝 `gpgme`、`device-mapper` 和 `libbtrfs` 的開發標頭和函式庫：
```
# 在 SLES 15 SP6 上安裝
$ sudo zypper install -y gpgme-devel device-mapper-devel libbtrfs-devel
```
3. 安裝 podman container runtime。
4. SLE Micro 6.0 SelfInstall ISO 的下載點，可在[此](https://www.suse.com/download/sle-micro/)找到
## 實作
### 1. 下載 EIB Container Image
```
$ sudo podman pull registry.suse.com/edge/3.2/edge-image-builder:1.1.0
```
### 2. 建立 CRB Disk Image 設定目錄
由於 EIB 在 Container 內運行，我們需要從主機掛載一個設定目錄，以便可以指定所需的設定，並且在構建過程中 EIB 可以訪問所需的設定文件和 supporting artifacts。此目錄必須遵循特定的結構，我們假設這個目錄位於您的主目錄中，並命名為 `eib`，讓我們來創建它：
```
$ export CONFIG_DIR=$HOME/eib && \
mkdir -p $CONFIG_DIR/base-images
```
在前一步中，我們創建了一個 `base-images` 目錄，用來存放 SL Micro 6.0 的 input ISO，現在我們要確保已下載的 ISO 被複製到設定目錄中：
```!
$ cp /path/to/downloads/SL-Micro.x86_64-6.0-Default-SelfInstall-GM2.install.iso $CONFIG_DIR/base-images/slemicro.iso
```
在 EIB 執行期間，原始的基本映像不會被修改；新的自訂版本會在 EIB config 目錄的根目錄中，以所需的組態建立。
此時的設定目錄應該如下所示：
```
$ tree $CONFIG_DIR
/root/eib
└── base-images
└── slemicro.iso
```
## 3. 建立描述 CRB Disk Image 的定義檔
#### 3.1 設定 Operating System
完整欄位說明，請參考此[連結](https://github.com/suse-edge/edge-image-builder/blob/main/docs/building-images.md#operating-system)
#### 3.2 設定 OS Users
使用 OpenSSL 建立單向加密密碼：
```
$ openssl passwd -6 rancher
```
執行結果應與以下類似 :
```
$6$C7vQ3j959WZ68/N3$torcva.LV2/O9LiwNGHHDNG0Ia0Wsd8UktcePEUAvqKV.tgtaNYYoOp5vnCFaCdaptzMtsGnxI4tKiKA035Bg/
```
#### 3.3 編輯定義檔
* 這裡產生了 `root`、`rancher` 帳號，並且密碼都是 `rancher`
```
$ cat << 'EOF' > $CONFIG_DIR/iso-definition.yaml
apiVersion: 1.1
image:
imageType: iso
arch: x86_64
baseImage: slemicro.iso
outputImageName: eib-image.iso
operatingSystem:
isoConfiguration:
installDevice: /dev/sda
time:
timezone: Asia/Taipei
ntp:
forceWait: true
servers:
- time.google.com
keymap: tw
users:
- username: root
encryptedPassword: $6$HrcBepMLjMsAzdTY$FY87XYphK4cL0NaFoLktOJZv2FK1xR2s8.QmMgBSrehPO3zx3cz/easlPVBHOal7TigH8x7x6ULIGCEDF8mE2/
- username: rancher
uid: 1000
encryptedPassword: $6$HrcBepMLjMsAzdTY$FY87XYphK4cL0NaFoLktOJZv2FK1xR2s8.QmMgBSrehPO3zx3cz/easlPVBHOal7TigH8x7x6ULIGCEDF8mE2/
createHomeDir: true
primaryGroup: users
secondaryGroups:
- rancher
groups:
- name: users
gid: 1000
- name: rancher
gid: 1001
systemd:
disable:
- transactional-update.timer
- rebootmgr
enable:
- cockpit.socket
EOF
```
### 4. 設定 Kubernetes cluster 和 Rancher
#### 4.1 編輯 RKE2 叢集設定檔
```
$ mkdir -p $CONFIG_DIR/kubernetes/config && \
cat << EOF > $CONFIG_DIR/kubernetes/config/server.yaml
node-name:
- "rms"
token: my-shared-secret
kubelet-arg:
- "container-log-max-files=3"
- "container-log-max-size=10Mi"
etcd-extra-env: TZ=Asia/Taipei
kube-apiserver-extra-env: TZ=Asia/Taipei
kube-controller-manager-extra-env: TZ=Asia/Taipei
kube-proxy-extra-env: TZ=Asia/Taipei
kube-scheduler-extra-env: TZ=Asia/Taipei
cloud-controller-manager-extra-env: TZ=Asia/Taipei
selinux: true
cni: "canal"
EOF
```
#### 4.2 編輯定義檔
* `apiVIP` : 是給 apiserver 一個對外的 ip，這裡需要注意設定為自己的可用 ip。
* `urls` : 是在 build iso 時會用到的地址來去下載需要的 yaml。`192.168.11.101` 這個地址是待會會自建一個 nginx web server，提供自定義的 yaml 下載。
* `embeddedArtifactRegistry` : 在 build iso 時會先下載的 image，使我們可以離線安裝 rms。
```
$ cat << 'EOF' >> $CONFIG_DIR/iso-definition.yaml
kubernetes:
version: v1.31.3+rke2r1
network:
apiVIP: 192.168.11.102
manifests:
urls:
- https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml
- http://192.168.11.101:8080/config/rancher-namespace.yaml
helm:
charts:
- name: rancher
version: 2.10.1
repositoryName: rancher-prime
valuesFile: rancher-values.yaml
targetNamespace: cattle-system
createNamespace: true
installationNamespace: cattle-system
repositories:
- name: rancher-prime
url: https://charts.rancher.com/server-charts/prime
embeddedArtifactRegistry:
images:
- name: registry.rancher.com/rancher/backup-restore-operator:v6.0.0
- name: registry.rancher.com/rancher/calico-cni:v3.29.0-rancher1
- name: registry.rancher.com/rancher/cis-operator:v1.3.4
- name: registry.rancher.com/rancher/flannel-cni:v1.4.1-rancher1
- name: registry.rancher.com/rancher/fleet-agent:v0.11.2
- name: registry.rancher.com/rancher/fleet:v0.11.2
- name: registry.rancher.com/rancher/hardened-addon-resizer:1.8.20-build20241001
- name: registry.rancher.com/rancher/hardened-calico:v3.29.0-build20241104
- name: registry.rancher.com/rancher/hardened-cluster-autoscaler:v1.8.11-build20241014
- name: registry.rancher.com/rancher/hardened-cni-plugins:v1.6.0-build20241022
- name: registry.rancher.com/rancher/hardened-coredns:v1.11.3-build20241018
- name: registry.rancher.com/rancher/hardened-dns-node-cache:1.23.1-build20241008
- name: registry.rancher.com/rancher/hardened-etcd:v3.5.16-k3s1-build20241106
- name: registry.rancher.com/rancher/hardened-flannel:v0.26.1-build20241107
- name: registry.rancher.com/rancher/hardened-k8s-metrics-server:v0.7.1-build20241008
- name: registry.rancher.com/rancher/hardened-kubernetes:v1.31.3-rke2r1-build20241121
- name: registry.rancher.com/rancher/hardened-multus-cni:v4.1.3-build20241028
- name: registry.rancher.com/rancher/hardened-node-feature-discovery:v0.15.6-build20240822
- name: registry.rancher.com/rancher/hardened-whereabouts:v0.8.0-build20241011
- name: registry.rancher.com/rancher/helm-project-operator:v0.2.1
- name: registry.rancher.com/rancher/k3s-upgrade:v1.31.3-k3s1
- name: registry.rancher.com/rancher/klipper-helm:v0.9.3-build20241008
- name: registry.rancher.com/rancher/klipper-lb:v0.4.9
- name: registry.rancher.com/rancher/kube-api-auth:v0.2.3
- name: registry.rancher.com/rancher/kubectl:v1.31.1
- name: registry.rancher.com/rancher/local-path-provisioner:v0.0.30
- name: registry.rancher.com/rancher/machine:v0.15.0-rancher124
- name: registry.rancher.com/rancher/mirrored-cluster-api-controller:v1.8.3
- name: registry.rancher.com/rancher/nginx-ingress-controller:v1.10.5-hardened4
- name: registry.rancher.com/rancher/prometheus-federator:v0.4.3
- name: registry.rancher.com/rancher/pushprox-client:v0.1.4-rancher2-client
- name: registry.rancher.com/rancher/pushprox-proxy:v0.1.4-rancher2-proxy
- name: registry.rancher.com/rancher/rancher-agent:v2.10.1
- name: registry.rancher.com/rancher/rancher-csp-adapter:v5.0.1
- name: registry.rancher.com/rancher/rancher-webhook:v0.6.2
- name: registry.rancher.com/rancher/rancher:v2.10.1
- name: registry.rancher.com/rancher/rke-tools:v0.1.105
- name: registry.rancher.com/rancher/rke2-cloud-provider:v1.31.0-build20240910
- name: registry.rancher.com/rancher/rke2-runtime:v1.31.3-rke2r1
- name: registry.rancher.com/rancher/rke2-upgrade:v1.31.3-rke2r1
- name: registry.rancher.com/rancher/security-scan:v0.5.2
- name: registry.rancher.com/rancher/shell:v0.3.0
- name: registry.rancher.com/rancher/system-agent-installer-k3s:v1.31.3-k3s1
- name: registry.rancher.com/rancher/system-agent-installer-rke2:v1.31.3-rke2r1
- name: registry.rancher.com/rancher/system-agent:v0.3.11-suc
- name: registry.rancher.com/rancher/system-upgrade-controller:v0.14.2
- name: registry.rancher.com/rancher/ui-plugin-catalog:3.2.0
EOF
```
#### 4.3 建立 Rancher Helm values 檔:
* 設定 rancher 的 hostname，以及預設登入密碼等
```
$ mkdir -p $CONFIG_DIR/kubernetes/helm/values/ && \
cat << EOF > $CONFIG_DIR/kubernetes/helm/values/rancher-values.yaml
hostname: andy-rancher.example.com
replicas: 1
bootstrapPassword: "rancheradmin"
systemDefaultRegistry: registry.rancher.com
useBundledSystemChart: true
EOF
```
#### 4.4 設定 rancher namespace:
```
$ mkdir -p ~/config/ && \
cat << EOF > ~/config/rancher-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cattle-system
EOF
```
#### 4.5 建立 Nginx Web Server
* 透過我們自己建立的 nginx，可以把自定義的 yaml 提供給 eib 在 build iso 時使用
```
$ sudo podman run -d -p 8080:80 \
--name web \
-v $HOME/config:/usr/share/nginx/html/config \
-v $HOME/eib:/usr/share/nginx/html/eib/ \
docker.io/library/nginx:stable
```
```
$ ls -l $HOME/config
total 4
-rw-r--r-- 1 root root 63 May 7 09:18 rancher-namespace.yaml
```
* 測試是否可以獲取 yaml
```
$ curl http://192.168.11.101:8080/config/rancher-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cattle-system
```
### 5. 設定 sle micro 網路
* 這裡 VM 的 MAC Address 要先確定，因此如果是虛擬環境要先創建 VM 然後確認這台 VM 的 MAC Address 是多少。
* 這裡使用 PVE 建立虛擬機。
* 設定自己環境所需要的 ip、dns、default gateway
```
$ mkdir -p $CONFIG_DIR/network/ && \
cat << EOF > $CONFIG_DIR/network/admin-rancher.yaml
interfaces:
- name: eth0
type: ethernet
state: up
mac-address: BC:24:11:C4:AA:AA
ipv4:
address:
- ip: 192.168.11.103
prefix-length: 24
enabled: true
ipv6:
dhcp: true
enabled: true
routes:
config:
- destination: 0.0.0.0/0
next-hop-interface: eth0
next-hop-address: 192.168.11.254
dns-resolver:
config:
server:
- 8.8.8.8
EOF
```
最後的目錄結構 :
```
$ tree $CONFIG_DIR ~/config
/root/eib
├── base-images
│ └── slemicro.iso
├── iso-definition.yaml
├── kubernetes
│ ├── config
│ │ └── server.yaml
│ └── helm
│ └── values
│ └── rancher-values.yaml
└── network
└── admin-rancher.yaml
/root/config
└── rancher-namespace.yaml
```
### 6. Building the CRB Disk Image
現在我們已經有了 base iso image 和 image definition 供 EIB 使用，讓我們繼續建立 ISO。 為此，我們只需使用 podman 以「建立」指令呼叫 EIB container，並指定定義檔案：
```
$ sudo podman run --rm -it --privileged \
-v $CONFIG_DIR:/eib \
registry.suse.com/edge/3.2/edge-image-builder:1.1.0 \
build --definition-file iso-definition.yaml
```
執行結果 :
```
SELinux is enabled in the Kubernetes configuration. The necessary RPM packages will be downloaded.
Downloading file: rancher-public.key 100% |████████████████████████████████████████████| (2.4/2.4 kB, 2.7 MB/s)
Setting up Podman API listener...
Downloading file: dl-manifest-1.yaml 100% |████████████████████████████████████████████| (987/987 kB, 1.4 MB/s)
Downloading file: dl-manifest-2.yaml 100% |███████████████████████████████████████████████| (63/63 B, 1.9 MB/s)
Pulling selected Helm charts... 100% |███████████████████████████████████████████████████████| (3/3, 28 it/min)
Generating image customization components...
Identifier ................... [SUCCESS]
Custom Files ................. [SKIPPED]
Time ......................... [SUCCESS]
Network ...................... [SUCCESS]
Groups ....................... [SUCCESS]
Users ........................ [SUCCESS]
Proxy ........................ [SKIPPED]
Resolving package dependencies...
Rpm .......................... [SUCCESS]
Os Files ..................... [SKIPPED]
Systemd ...................... [SUCCESS]
Fips ......................... [SKIPPED]
Elemental .................... [SKIPPED]
Suma ......................... [SKIPPED]
Populating Embedded Artifact Registry... 100% |███████████████████████████████████████████████████| (53/53, 5 it/min)
Embedded Artifact Registry ... [SUCCESS]
Keymap ....................... [SUCCESS]
Configuring Kubernetes component...
The Kubernetes CNI is not explicitly set, defaulting to 'cilium'.
Downloading file: rke2_installer.sh
Downloading file: rke2-images-core.linux-amd64.tar.zst 100% |███████████████████████████████████| (644/644 MB, 17 MB/s)
Downloading file: rke2-images-cilium.linux-amd64.tar.zst 100% |█████████████████████████████████| (400/400 MB, 17 MB/s)
Downloading file: rke2.linux-amd64.tar.gz 100% |███████████████████████████████████████████████████| (36/36 MB, 18 MB/s)
Downloading file: sha256sum-amd64.txt 100% |█████████████████████████████████████████████████████| (4.3/4.3 kB, 35 MB/s)
Kubernetes ................... [SUCCESS]
Certificates ................. [SKIPPED]
Cleanup ...................... [SKIPPED]
Building ISO image...
Kernel Params ................ [SKIPPED]
Build complete, the image can be found at: eib-image.iso
```
檢視以客製化後的 ISO，大小大約 7G
```
$ ls -lh eib/eib-image.iso
-rw-r--r-- 1 root root 7.0G May 7 10:02 eib/eib-image.iso
```
### 7. 設定 VM 透過 CRB Disk Image 開機
以下使用 PVE 環境示範
1. 確認 VM 透過 ISO 開機
2. 將 VM 開機
3. 開機成功後，登入 VM ，需要等一下讓他自動部屬 rke2、rancher
```
$ ssh rancher@192.168.11.103
```
4. 設定 kubeconfig
```
$ sudo cp /var/lib/rancher/rke2/bin/* /usr/local/bin/ && \
sudo cp /opt/rke2/bin/* /usr/local/bin/ && \
mkdir -p $HOME/.kube && \
sudo cp /etc/rancher/rke2/rke2.yaml $HOME/.kube/config && \
sudo chown $(id -u):$(id -g) $HOME/.kube/config && \
sudo /usr/local/bin/crictl config \
--set runtime-endpoint=unix:///run/k3s/containerd/containerd.sock
```
5. 確認 rke2、rancher 已自動部屬完成
> Endpoint Copier Operator : 目的是創建 Kubernetes 服務和 Endpoint 的副本並使它們保持同步。
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
rms Ready control-plane,etcd,master 3m10s v1.31.3+rke2r1
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-fleet-system fleet-controller-5ff79b7698-gdkn2 3/3 Running 0 10s
cattle-fleet-system gitjob-55fc987d94-2qzc6 1/1 Running 0 10s
cattle-system helm-install-rancher-xzjqb 0/1 Completed 3 2m55s
cattle-system helm-operation-njstl 0/2 Completed 0 44s
cattle-system helm-operation-pgp97 0/2 Completed 0 17s
cattle-system helm-operation-rqskx 1/2 NotReady 0 12s
cattle-system rancher-99d599967-n27kc 1/1 Running 0 97s
cattle-system rancher-webhook-79798674c5-44bpd 1/1 Running 0 42s
cert-manager cert-manager-74b56b6655-9bdfk 1/1 Running 0 2m54s
cert-manager cert-manager-cainjector-55d94dc4cc-sc8cd 1/1 Running 0 2m54s
cert-manager cert-manager-webhook-564f647c66-9hbgv 1/1 Running 0 2m54s
endpoint-copier-operator endpoint-copier-operator-7bf97b9d45-2qnbq 1/1 Running 0 2m36s
endpoint-copier-operator endpoint-copier-operator-7bf97b9d45-jkt44 1/1 Running 0 2m36s
kube-system cloud-controller-manager-rms 1/1 Running 1 (2m58s ago) 2m59s
kube-system etcd-rms 1/1 Running 0 2m22s
kube-system helm-install-endpoint-copier-operator-rp6xn 0/1 Completed 0 2m55s
kube-system helm-install-metallb-cqpzl 0/1 Completed 0 2m55s
kube-system helm-install-rke2-canal-wk2jl 0/1 Completed 0 2m55s
kube-system helm-install-rke2-coredns-jkmv9 0/1 Completed 0 2m55s
kube-system helm-install-rke2-ingress-nginx-5t7p9 0/1 Completed 0 2m54s
kube-system helm-install-rke2-metrics-server-sxpjz 0/1 Completed 0 2m53s
kube-system helm-install-rke2-snapshot-controller-66rgh 0/1 Completed 0 2m52s
kube-system helm-install-rke2-snapshot-controller-crd-4q5sf 0/1 Completed 0 2m52s
kube-system helm-install-rke2-snapshot-validation-webhook-xnbxm 0/1 Completed 0 2m51s
kube-system kube-apiserver-rms 1/1 Running 0 2m58s
kube-system kube-controller-manager-rms 1/1 Running 0 2m50s
kube-system kube-proxy-rms 1/1 Running 0 2m41s
kube-system kube-scheduler-rms 1/1 Running 0 2m50s
kube-system rke2-canal-gltxf 2/2 Running 0 2m41s
kube-system rke2-coredns-rke2-coredns-9579797d8-b45k9 1/1 Running 0 2m40s
kube-system rke2-coredns-rke2-coredns-autoscaler-78db5d674-tqvr5 1/1 Running 0 2m40s
kube-system rke2-ingress-nginx-controller-x9fsj 1/1 Running 0 2m26s
kube-system rke2-metrics-server-7c85d458bd-zbhnf 1/1 Running 0 2m36s
kube-system rke2-snapshot-controller-65bc6fbd57-mtfkx 1/1 Running 0 2m34s
kube-system rke2-snapshot-validation-webhook-859c7896df-p7jgm 1/1 Running 0 2m34s
metallb-system metallb-controller-5756c8898-vw7q6 1/1 Running 0 2m33s
metallb-system metallb-speaker-b6mnz 1/1 Running 0 2m33s
```
* `kubernetes-vip` 是我們前面設定提供的 vip 地址
```
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 4m16s
kubernetes-vip LoadBalancer 10.43.229.38 192.168.11.102 9345:30766/TCP,6443:31548/TCP 4m11s
```
* 登入 rancher，預設密碼 `rancheradmin`
## 故障排除
### 在 OS 安裝階段出現問題在開機引導畫面加入參數
若是 GRUB 畫面（常見於 EFI 系統）：
1. 用鍵盤選擇 Install SL Micro 開機選項，不要馬上按 Enter。
2. 按下 `e` 進入編輯模式。
3. 找到 `linux` 開頭的那一行。在行尾加上：
```
systemd.log_level=debug systemd.debug_shell=1 rd.debug
```
4. 按下 `Ctrl + x` 或 `F10` 開始以 debug 模式開機。
### 參數說明：
1. `systemd.log_level=debug` : 讓 systemd 顯示更多 debug 訊息
2. `systemd.debug_shell=1` : 在 TTY 9 開啟 root shell（可用 Ctrl+Alt+F9 切過去）
3. `rd.debug` : 開啟 Dracut（initramfs）開機階段的詳細除錯輸出
## 參考
https://hackmd.io/@QI-AN/eib
https://documentation.suse.com/suse-edge/3.0/html/edge/id-air-gapped-deployments-with-edge-image-builder.html#rancher-install
https://documentation.suse.com/suse-edge/3.2/html/edge/components-eco.html
https://github.com/suse-edge/edge-image-builder/blob/main/docs/building-images.md
Sign in with Wallet
Connect another wallet