---
# System prepended metadata

title: SUSE Edge Image Builder 實作
tags: [work]

---

# SUSE Edge Image Builder 實作

## 實作目標

OS 在第一次開機後自動安裝和設定好 SL Micro 6.0 + RKE2 v1.31.3 (ALL in One) + Rancher Prime v2.10.1。

## 環境準備

1. 一台 x86_64 實體主機或虛擬主機，作業系統是 SLES 15 SP6, openSUSE Leap 15.6 或 openSUSE Tumbleweed
2. 在建立 EIB image 之前，請確定您的系統上已安裝 `gpgme`、`device-mapper` 和 `libbtrfs` 的開發標頭和函式庫：

```
# 在 SLES 15 SP6 上安裝
$ sudo zypper install -y gpgme-devel device-mapper-devel libbtrfs-devel
```
3. 安裝 podman container runtime。
4. SLE Micro 6.0 SelfInstall ISO 的下載點，可在[此](https://www.suse.com/download/sle-micro/)找到

## 實作

### 1. 下載 EIB Container Image
```
$ sudo podman pull registry.suse.com/edge/3.2/edge-image-builder:1.1.0
```
### 2. 建立 CRB Disk Image 設定目錄

由於 EIB 在 Container 內運行，我們需要從主機掛載一個設定目錄，以便可以指定所需的設定，並且在構建過程中 EIB 可以訪問所需的設定文件和 supporting artifacts。此目錄必須遵循特定的結構，我們假設這個目錄位於您的主目錄中，並命名為 `eib`，讓我們來創建它：
```
$ export CONFIG_DIR=$HOME/eib && \
mkdir -p $CONFIG_DIR/base-images
```
在前一步中，我們創建了一個 `base-images` 目錄，用來存放 SL Micro 6.0 的 input ISO，現在我們要確保已下載的 ISO 被複製到設定目錄中：
```!
$ cp /path/to/downloads/SL-Micro.x86_64-6.0-Default-SelfInstall-GM2.install.iso $CONFIG_DIR/base-images/slemicro.iso
```
在 EIB 執行期間，原始的基本映像不會被修改；新的自訂版本會在 EIB config 目錄的根目錄中，以所需的組態建立。

此時的設定目錄應該如下所示：
```
$ tree $CONFIG_DIR
/root/eib
└── base-images
    └── slemicro.iso
```

## 3. 建立描述 CRB Disk Image 的定義檔

#### 3.1 設定 Operating System

完整欄位說明，請參考此[連結](https://github.com/suse-edge/edge-image-builder/blob/main/docs/building-images.md#operating-system)

#### 3.2 設定 OS Users

使用 OpenSSL 建立單向加密密碼：

```
$ openssl passwd -6 rancher
```
執行結果應與以下類似 :
```
$6$C7vQ3j959WZ68/N3$torcva.LV2/O9LiwNGHHDNG0Ia0Wsd8UktcePEUAvqKV.tgtaNYYoOp5vnCFaCdaptzMtsGnxI4tKiKA035Bg/
```
#### 3.3 編輯定義檔
* 這裡產生了 `root`、`rancher` 帳號，並且密碼都是 `rancher`
```
$ cat << 'EOF' > $CONFIG_DIR/iso-definition.yaml
apiVersion: 1.1
image:
  imageType: iso
  arch: x86_64
  baseImage: slemicro.iso
  outputImageName: eib-image.iso
operatingSystem:
  isoConfiguration:
    installDevice: /dev/sda
  time:
    timezone: Asia/Taipei
    ntp:
      forceWait: true
      servers:
        - time.google.com
  keymap: tw
  users:
    - username: root
      encryptedPassword: $6$HrcBepMLjMsAzdTY$FY87XYphK4cL0NaFoLktOJZv2FK1xR2s8.QmMgBSrehPO3zx3cz/easlPVBHOal7TigH8x7x6ULIGCEDF8mE2/
    - username: rancher
      uid: 1000
      encryptedPassword: $6$HrcBepMLjMsAzdTY$FY87XYphK4cL0NaFoLktOJZv2FK1xR2s8.QmMgBSrehPO3zx3cz/easlPVBHOal7TigH8x7x6ULIGCEDF8mE2/
      createHomeDir: true
      primaryGroup: users
      secondaryGroups:
        - rancher
  groups:
    - name: users
      gid: 1000
    - name: rancher
      gid: 1001
  systemd:
    disable:
      - transactional-update.timer
      - rebootmgr
    enable:
      - cockpit.socket
EOF
```

### 4. 設定 Kubernetes cluster 和 Rancher

#### 4.1 編輯 RKE2 叢集設定檔

```
$ mkdir -p $CONFIG_DIR/kubernetes/config && \
cat << EOF > $CONFIG_DIR/kubernetes/config/server.yaml
node-name:
  - "rms"
token: my-shared-secret
kubelet-arg:
  - "container-log-max-files=3"
  - "container-log-max-size=10Mi"
etcd-extra-env: TZ=Asia/Taipei
kube-apiserver-extra-env: TZ=Asia/Taipei
kube-controller-manager-extra-env: TZ=Asia/Taipei
kube-proxy-extra-env: TZ=Asia/Taipei
kube-scheduler-extra-env: TZ=Asia/Taipei
cloud-controller-manager-extra-env: TZ=Asia/Taipei
selinux: true
cni: "canal"
EOF
```

#### 4.2 編輯定義檔
* `apiVIP` : 是給 apiserver 一個對外的 ip，這裡需要注意設定為自己的可用 ip。
* `urls` : 是在 build iso 時會用到的地址來去下載需要的 yaml。`192.168.11.101` 這個地址是待會會自建一個 nginx web server，提供自定義的 yaml 下載。
* `embeddedArtifactRegistry` : 在 build iso 時會先下載的 image，使我們可以離線安裝 rms。

```
$ cat << 'EOF' >> $CONFIG_DIR/iso-definition.yaml
kubernetes:
  version: v1.31.3+rke2r1
  network:
    apiVIP: 192.168.11.102
  manifests:
    urls:
    - https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml
    - http://192.168.11.101:8080/config/rancher-namespace.yaml
  helm:
    charts:
      - name: rancher
        version: 2.10.1
        repositoryName: rancher-prime
        valuesFile: rancher-values.yaml
        targetNamespace: cattle-system
        createNamespace: true
        installationNamespace: cattle-system
    repositories:
      - name: rancher-prime
        url:  https://charts.rancher.com/server-charts/prime
embeddedArtifactRegistry:
  images:
    - name: registry.rancher.com/rancher/backup-restore-operator:v6.0.0
    - name: registry.rancher.com/rancher/calico-cni:v3.29.0-rancher1
    - name: registry.rancher.com/rancher/cis-operator:v1.3.4
    - name: registry.rancher.com/rancher/flannel-cni:v1.4.1-rancher1
    - name: registry.rancher.com/rancher/fleet-agent:v0.11.2
    - name: registry.rancher.com/rancher/fleet:v0.11.2
    - name: registry.rancher.com/rancher/hardened-addon-resizer:1.8.20-build20241001
    - name: registry.rancher.com/rancher/hardened-calico:v3.29.0-build20241104
    - name: registry.rancher.com/rancher/hardened-cluster-autoscaler:v1.8.11-build20241014
    - name: registry.rancher.com/rancher/hardened-cni-plugins:v1.6.0-build20241022
    - name: registry.rancher.com/rancher/hardened-coredns:v1.11.3-build20241018
    - name: registry.rancher.com/rancher/hardened-dns-node-cache:1.23.1-build20241008
    - name: registry.rancher.com/rancher/hardened-etcd:v3.5.16-k3s1-build20241106
    - name: registry.rancher.com/rancher/hardened-flannel:v0.26.1-build20241107
    - name: registry.rancher.com/rancher/hardened-k8s-metrics-server:v0.7.1-build20241008
    - name: registry.rancher.com/rancher/hardened-kubernetes:v1.31.3-rke2r1-build20241121
    - name: registry.rancher.com/rancher/hardened-multus-cni:v4.1.3-build20241028
    - name: registry.rancher.com/rancher/hardened-node-feature-discovery:v0.15.6-build20240822
    - name: registry.rancher.com/rancher/hardened-whereabouts:v0.8.0-build20241011
    - name: registry.rancher.com/rancher/helm-project-operator:v0.2.1
    - name: registry.rancher.com/rancher/k3s-upgrade:v1.31.3-k3s1
    - name: registry.rancher.com/rancher/klipper-helm:v0.9.3-build20241008
    - name: registry.rancher.com/rancher/klipper-lb:v0.4.9
    - name: registry.rancher.com/rancher/kube-api-auth:v0.2.3
    - name: registry.rancher.com/rancher/kubectl:v1.31.1
    - name: registry.rancher.com/rancher/local-path-provisioner:v0.0.30
    - name: registry.rancher.com/rancher/machine:v0.15.0-rancher124
    - name: registry.rancher.com/rancher/mirrored-cluster-api-controller:v1.8.3
    - name: registry.rancher.com/rancher/nginx-ingress-controller:v1.10.5-hardened4
    - name: registry.rancher.com/rancher/prometheus-federator:v0.4.3
    - name: registry.rancher.com/rancher/pushprox-client:v0.1.4-rancher2-client
    - name: registry.rancher.com/rancher/pushprox-proxy:v0.1.4-rancher2-proxy
    - name: registry.rancher.com/rancher/rancher-agent:v2.10.1
    - name: registry.rancher.com/rancher/rancher-csp-adapter:v5.0.1
    - name: registry.rancher.com/rancher/rancher-webhook:v0.6.2
    - name: registry.rancher.com/rancher/rancher:v2.10.1
    - name: registry.rancher.com/rancher/rke-tools:v0.1.105
    - name: registry.rancher.com/rancher/rke2-cloud-provider:v1.31.0-build20240910
    - name: registry.rancher.com/rancher/rke2-runtime:v1.31.3-rke2r1
    - name: registry.rancher.com/rancher/rke2-upgrade:v1.31.3-rke2r1
    - name: registry.rancher.com/rancher/security-scan:v0.5.2
    - name: registry.rancher.com/rancher/shell:v0.3.0
    - name: registry.rancher.com/rancher/system-agent-installer-k3s:v1.31.3-k3s1
    - name: registry.rancher.com/rancher/system-agent-installer-rke2:v1.31.3-rke2r1
    - name: registry.rancher.com/rancher/system-agent:v0.3.11-suc
    - name: registry.rancher.com/rancher/system-upgrade-controller:v0.14.2
    - name: registry.rancher.com/rancher/ui-plugin-catalog:3.2.0
EOF
```
#### 4.3 建立 Rancher Helm values 檔:
* 設定 rancher 的 hostname，以及預設登入密碼等
```
$ mkdir -p $CONFIG_DIR/kubernetes/helm/values/ && \
cat << EOF > $CONFIG_DIR/kubernetes/helm/values/rancher-values.yaml
hostname: andy-rancher.example.com
replicas: 1
bootstrapPassword: "rancheradmin"
systemDefaultRegistry: registry.rancher.com
useBundledSystemChart: true
EOF
```

#### 4.4 設定 rancher namespace:
```
$ mkdir -p ~/config/ && \
cat << EOF > ~/config/rancher-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cattle-system
EOF
```
#### 4.5 建立 Nginx Web Server
* 透過我們自己建立的 nginx，可以把自定義的 yaml 提供給 eib 在 build iso 時使用
```
$ sudo podman run -d -p 8080:80 \
  --name web \
  -v $HOME/config:/usr/share/nginx/html/config \
  -v $HOME/eib:/usr/share/nginx/html/eib/ \
  docker.io/library/nginx:stable
```
```
$ ls -l $HOME/config
total 4
-rw-r--r-- 1 root root 63 May  7 09:18 rancher-namespace.yaml
```
* 測試是否可以獲取 yaml
```
$ curl http://192.168.11.101:8080/config/rancher-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cattle-system
```
### 5. 設定 sle micro 網路
* 這裡 VM 的 MAC Address 要先確定，因此如果是虛擬環境要先創建 VM 然後確認這台 VM 的 MAC Address 是多少。
* 這裡使用 PVE 建立虛擬機。


![image](https://hackmd.io/_uploads/SJqsIN_xge.png)

* 設定自己環境所需要的 ip、dns、default gateway
```
$ mkdir -p $CONFIG_DIR/network/ && \
cat << EOF > $CONFIG_DIR/network/admin-rancher.yaml
interfaces:
- name: eth0
  type: ethernet
  state: up
  mac-address: BC:24:11:C4:AA:AA
  ipv4:
    address:
    - ip: 192.168.11.103
      prefix-length: 24
    enabled: true
  ipv6:
    dhcp: true
    enabled: true
routes:
  config:
    - destination: 0.0.0.0/0
      next-hop-interface: eth0
      next-hop-address: 192.168.11.254
dns-resolver:
  config:
    server:
    - 8.8.8.8
EOF
```
最後的目錄結構 :
```
$ tree $CONFIG_DIR ~/config
/root/eib
├── base-images
│   └── slemicro.iso
├── iso-definition.yaml
├── kubernetes
│   ├── config
│   │   └── server.yaml
│   └── helm
│       └── values
│           └── rancher-values.yaml
└── network
    └── admin-rancher.yaml
/root/config
└── rancher-namespace.yaml
```
### 6. Building the CRB Disk Image
現在我們已經有了 base iso image 和 image definition 供 EIB 使用，讓我們繼續建立 ISO。 為此，我們只需使用 podman 以「建立」指令呼叫 EIB container，並指定定義檔案：

```
$ sudo podman run --rm -it --privileged \
  -v $CONFIG_DIR:/eib \
  registry.suse.com/edge/3.2/edge-image-builder:1.1.0 \
  build --definition-file iso-definition.yaml
```
執行結果 :
```
SELinux is enabled in the Kubernetes configuration. The necessary RPM packages will be downloaded.
Downloading file: rancher-public.key 100% |████████████████████████████████████████████| (2.4/2.4 kB, 2.7 MB/s)
Setting up Podman API listener...
Downloading file: dl-manifest-1.yaml 100% |████████████████████████████████████████████| (987/987 kB, 1.4 MB/s)
Downloading file: dl-manifest-2.yaml 100% |███████████████████████████████████████████████| (63/63 B, 1.9 MB/s)
Pulling selected Helm charts... 100% |███████████████████████████████████████████████████████| (3/3, 28 it/min)
Generating image customization components...
Identifier ................... [SUCCESS]
Custom Files ................. [SKIPPED]
Time ......................... [SUCCESS]
Network ...................... [SUCCESS]
Groups ....................... [SUCCESS]
Users ........................ [SUCCESS]
Proxy ........................ [SKIPPED]
Resolving package dependencies...
Rpm .......................... [SUCCESS]
Os Files ..................... [SKIPPED]
Systemd ...................... [SUCCESS]
Fips ......................... [SKIPPED]
Elemental .................... [SKIPPED]
Suma ......................... [SKIPPED]
Populating Embedded Artifact Registry... 100% |███████████████████████████████████████████████████| (53/53, 5 it/min)
Embedded Artifact Registry ... [SUCCESS]
Keymap ....................... [SUCCESS]
Configuring Kubernetes component...
The Kubernetes CNI is not explicitly set, defaulting to 'cilium'.
Downloading file: rke2_installer.sh
Downloading file: rke2-images-core.linux-amd64.tar.zst 100% |███████████████████████████████████| (644/644 MB, 17 MB/s)
Downloading file: rke2-images-cilium.linux-amd64.tar.zst 100% |█████████████████████████████████| (400/400 MB, 17 MB/s)
Downloading file: rke2.linux-amd64.tar.gz 100% |███████████████████████████████████████████████████| (36/36 MB, 18 MB/s)
Downloading file: sha256sum-amd64.txt 100% |█████████████████████████████████████████████████████| (4.3/4.3 kB, 35 MB/s)
Kubernetes ................... [SUCCESS]
Certificates ................. [SKIPPED]
Cleanup ...................... [SKIPPED]
Building ISO image...
Kernel Params ................ [SKIPPED]
Build complete, the image can be found at: eib-image.iso
```
檢視以客製化後的 ISO，大小大約 7G
```
$ ls -lh eib/eib-image.iso
-rw-r--r-- 1 root root 7.0G May  7 10:02 eib/eib-image.iso
```

### 7. 設定 VM 透過 CRB Disk Image 開機
以下使用 PVE 環境示範

1. 確認 VM 透過 ISO 開機

![image](https://hackmd.io/_uploads/rJHeVHuexl.png)
2. 將 VM 開機

![image](https://hackmd.io/_uploads/H1HfVrOgle.png)

3. 開機成功後，登入 VM ，需要等一下讓他自動部屬 rke2、rancher
```
$ ssh rancher@192.168.11.103
```
4. 設定 kubeconfig
```
$ sudo cp /var/lib/rancher/rke2/bin/* /usr/local/bin/ && \
  sudo cp /opt/rke2/bin/* /usr/local/bin/ && \
  mkdir -p $HOME/.kube && \
  sudo cp /etc/rancher/rke2/rke2.yaml $HOME/.kube/config && \
  sudo chown $(id -u):$(id -g) $HOME/.kube/config && \
  sudo /usr/local/bin/crictl config \
    --set runtime-endpoint=unix:///run/k3s/containerd/containerd.sock
```
5. 確認 rke2、rancher 已自動部屬完成
> Endpoint Copier Operator : 目的是創建 Kubernetes 服務和 Endpoint 的副本並使它們保持同步。
```
$ kubectl get no
NAME   STATUS   ROLES                       AGE     VERSION
rms    Ready    control-plane,etcd,master   3m10s   v1.31.3+rke2r1

$ kubectl get po -A
NAMESPACE                  NAME                                                   READY   STATUS      RESTARTS        AGE
cattle-fleet-system        fleet-controller-5ff79b7698-gdkn2                      3/3     Running     0               10s
cattle-fleet-system        gitjob-55fc987d94-2qzc6                                1/1     Running     0               10s
cattle-system              helm-install-rancher-xzjqb                             0/1     Completed   3               2m55s
cattle-system              helm-operation-njstl                                   0/2     Completed   0               44s
cattle-system              helm-operation-pgp97                                   0/2     Completed   0               17s
cattle-system              helm-operation-rqskx                                   1/2     NotReady    0               12s
cattle-system              rancher-99d599967-n27kc                                1/1     Running     0               97s
cattle-system              rancher-webhook-79798674c5-44bpd                       1/1     Running     0               42s
cert-manager               cert-manager-74b56b6655-9bdfk                          1/1     Running     0               2m54s
cert-manager               cert-manager-cainjector-55d94dc4cc-sc8cd               1/1     Running     0               2m54s
cert-manager               cert-manager-webhook-564f647c66-9hbgv                  1/1     Running     0               2m54s
endpoint-copier-operator   endpoint-copier-operator-7bf97b9d45-2qnbq              1/1     Running     0               2m36s
endpoint-copier-operator   endpoint-copier-operator-7bf97b9d45-jkt44              1/1     Running     0               2m36s
kube-system                cloud-controller-manager-rms                           1/1     Running     1 (2m58s ago)   2m59s
kube-system                etcd-rms                                               1/1     Running     0               2m22s
kube-system                helm-install-endpoint-copier-operator-rp6xn            0/1     Completed   0               2m55s
kube-system                helm-install-metallb-cqpzl                             0/1     Completed   0               2m55s
kube-system                helm-install-rke2-canal-wk2jl                          0/1     Completed   0               2m55s
kube-system                helm-install-rke2-coredns-jkmv9                        0/1     Completed   0               2m55s
kube-system                helm-install-rke2-ingress-nginx-5t7p9                  0/1     Completed   0               2m54s
kube-system                helm-install-rke2-metrics-server-sxpjz                 0/1     Completed   0               2m53s
kube-system                helm-install-rke2-snapshot-controller-66rgh            0/1     Completed   0               2m52s
kube-system                helm-install-rke2-snapshot-controller-crd-4q5sf        0/1     Completed   0               2m52s
kube-system                helm-install-rke2-snapshot-validation-webhook-xnbxm    0/1     Completed   0               2m51s
kube-system                kube-apiserver-rms                                     1/1     Running     0               2m58s
kube-system                kube-controller-manager-rms                            1/1     Running     0               2m50s
kube-system                kube-proxy-rms                                         1/1     Running     0               2m41s
kube-system                kube-scheduler-rms                                     1/1     Running     0               2m50s
kube-system                rke2-canal-gltxf                                       2/2     Running     0               2m41s
kube-system                rke2-coredns-rke2-coredns-9579797d8-b45k9              1/1     Running     0               2m40s
kube-system                rke2-coredns-rke2-coredns-autoscaler-78db5d674-tqvr5   1/1     Running     0               2m40s
kube-system                rke2-ingress-nginx-controller-x9fsj                    1/1     Running     0               2m26s
kube-system                rke2-metrics-server-7c85d458bd-zbhnf                   1/1     Running     0               2m36s
kube-system                rke2-snapshot-controller-65bc6fbd57-mtfkx              1/1     Running     0               2m34s
kube-system                rke2-snapshot-validation-webhook-859c7896df-p7jgm      1/1     Running     0               2m34s
metallb-system             metallb-controller-5756c8898-vw7q6                     1/1     Running     0               2m33s
metallb-system             metallb-speaker-b6mnz                                  1/1     Running     0               2m33s
```
* `kubernetes-vip` 是我們前面設定提供的 vip 地址
```
$ kubectl get svc
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                         AGE
kubernetes       ClusterIP      10.43.0.1      <none>           443/TCP                         4m16s
kubernetes-vip   LoadBalancer   10.43.229.38   192.168.11.102   9345:30766/TCP,6443:31548/TCP   4m11s
```
* 登入 rancher，預設密碼 `rancheradmin`

![image](https://hackmd.io/_uploads/HyDk8Hullg.png)

## 故障排除

### 在 OS 安裝階段出現問題在開機引導畫面加入參數

若是 GRUB 畫面（常見於 EFI 系統）：

1. 用鍵盤選擇 Install SL Micro 開機選項，不要馬上按 Enter。

![image](https://hackmd.io/_uploads/S1wu8ncbeg.png)

2. 按下 `e` 進入編輯模式。
3. 找到 `linux` 開頭的那一行。在行尾加上：

```
systemd.log_level=debug systemd.debug_shell=1 rd.debug
```

![image](https://hackmd.io/_uploads/HyAc8n5Wlg.png)

4. 按下 `Ctrl + x` 或 `F10` 開始以 debug 模式開機。

### 參數說明：

1. `systemd.log_level=debug` : 讓 systemd 顯示更多 debug 訊息
2. `systemd.debug_shell=1` : 在 TTY 9 開啟 root shell（可用 Ctrl+Alt+F9 切過去）
3. `rd.debug` : 開啟 Dracut（initramfs）開機階段的詳細除錯輸出


## 參考
https://hackmd.io/@QI-AN/eib

https://documentation.suse.com/suse-edge/3.0/html/edge/id-air-gapped-deployments-with-edge-image-builder.html#rancher-install

https://documentation.suse.com/suse-edge/3.2/html/edge/components-eco.html

https://github.com/suse-edge/edge-image-builder/blob/main/docs/building-images.md