# Install Longhorn on Talos Kubernetes
## PreRequest
### System Extensions
要額外在 Talos K8S 的 Worker Node 上安裝 `iscsi-tools` 和 `util-linux-tools` 這兩個套件
```
customization:
systemExtensions:
officialExtensions:
- siderolabs/iscsi-tools
- siderolabs/util-linux-tools
```
> `siderolabs/iscsi-tools`: this extension enables iscsid daemon and iscsiadm to be available to all nodes for the Kubernetes persistent volumes operations.
> `siderolabs/util-linux-tools`: this extension enables linux tool to be available to all nodes. For example, the fstrim binary is used for Longhorn volume trimming.
### Pod Security
Longhorn 要求在 `longhorn-system` Namespace 要有 `enforce: "privileged"` 的權限
### Data Path Mounts
You need provide additional data path mounts to be accessible to the Kubernetes Kubelet container.
These mount is necessary to provide access to the host directories and attaching volumes required by the Longhorn components.
```
machine:
kubelet:
extraMounts:
- destination: /var/lib/longhorn
type: bind
source: /var/lib/longhorn
options:
- bind
- rshared
- rw
```
## 環境準備
### 獲得 Talos System Extensions 的 image ID
```bash!
## create the schematic file
$ cat <<EOF > longhorn.yaml
customization:
systemExtensions:
officialExtensions:
- siderolabs/iscsi-tools
- siderolabs/util-linux-tools
- siderolabs/gvisor
- siderolabs/qemu-guest-agent
EOF
## 到 Talos 官網的 Image Factory 查詢包含指定套件的 Image ID:
$ curl -X POST --data-binary @longhorn.yaml https://factory.talos.dev/schematics
{"id":"c527b6b20fb22847304656677e9bd4c4055dfcce95f3385da5db80e35f5fa1dc"}
```
> Image Factory is a service that generates Talos boot assets on-demand. Image Factory allows to generate boot assets for the official Talos Linux releases and official Talos Linux system extensions.
### Controlplane node patch yaml
以下是 Controlplane node 使用 machine config 檔需要的 patch yaml
```yaml!
machine:
install:
image: factory.talos.dev/installer/c527b6b20fb22847304656677e9bd4c4055dfcce95f3385da5db80e35f5fa1dc:v1.6.5
cluster:
inlineManifests:
- name: namespace-longhorn-system
contents: |-
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system
```
### Worker node patch yaml
以下是 Worker node 使用 machine config 檔需要的 patch yaml
```yaml!
machine:
kubelet:
extraMounts:
- destination: /var/lib/longhorn
type: bind
source: /var/lib/longhorn
options:
- bind
- rshared
- rw
install:
image: factory.talos.dev/installer/c527b6b20fb22847304656677e9bd4c4055dfcce95f3385da5db80e35f5fa1dc:v1.6.5
```
### 透過 patch yaml 修改 machineconfig
```bash!
## 修改 k1m1
$ talosctl machineconfig patch /home/bigred/k1/v1.6.5/controlplane.yaml --patch @/home/bigred/k1/v1.6.5/k1m1.patch --output /home/bigred/k1/v1.6.5/k1m1.yaml
## 修改 k1w1
$ talosctl machineconfig patch /home/bigred/k1/v1.6.5/worker.yaml --patch @/home/bigred/k1/v1.6.5/k1w1.patch --output /home/bigred/k1/v1.6.5/k1w1.yaml
## 修改 k1w2
$ talosctl machineconfig patch /home/bigred/k1/v1.6.5/worker.yaml --patch @/home/bigred/k1/v1.6.5/k1w2.patch --output /home/bigred/k1/v1.6.5/k1w2.yaml
```
### 建立 Talos K8S
```
$ 1m2w.sh
Talos Control Plane (172.22.1.11) config
waiting 120
k1m1(172.22.1.11) config
waiting 300
k1w1(172.22.1.15) config
k1w2(172.22.1.16) config
waiting 120
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k1m1 Ready control-plane 5m56s v1.29.2 172.22.1.11 <none> Talos (v1.6.5) 6.1.78-talos containerd://1.7.13
k1w1 Ready worker 63s v1.29.2 172.22.1.15 <none> Talos (v1.6.5) 6.1.78-talos containerd://1.7.13
k1w2 Ready worker 56s v1.29.2 172.22.1.16 <none> Talos (v1.6.5) 6.1.78-talos containerd://1.7.13
```
### 確認 System Extension 是否安裝成功
```bash!
$ talosctl -n 172.22.1.11,172.22.1.15,172.22.1.16 --talosconfig=/home/bigred/k1/v1.6.5/talosconfig get extensions
NODE NAMESPACE TYPE ID VERSION NAME VERSION
172.22.1.11 runtime ExtensionStatus 0 1 iscsi-tools v0.1.4
172.22.1.11 runtime ExtensionStatus 1 1 util-linux-tools $VERSION
172.22.1.11 runtime ExtensionStatus 2 1 schematic 613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
172.22.1.15 runtime ExtensionStatus 0 1 iscsi-tools v0.1.4
172.22.1.15 runtime ExtensionStatus 1 1 util-linux-tools $VERSION
172.22.1.15 runtime ExtensionStatus 2 1 schematic 613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
172.22.1.16 runtime ExtensionStatus 0 1 iscsi-tools v0.1.4
172.22.1.16 runtime ExtensionStatus 1 1 util-linux-tools $VERSION
172.22.1.16 runtime ExtensionStatus 2 1 schematic 613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
```
## 安裝 Longhorn
```bash!
$ curl -s https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/longhorn.yaml | sed 's/numberOfReplicas: "3"/numberOfReplicas: "2"/' | kubectl apply -f -
```
### 檢查是否部屬成功
```bash!
$ kubectl get pods --namespace longhorn-system
NAME READY STATUS RESTARTS AGE
csi-attacher-57689cc84b-btwsj 1/1 Running 0 12m
csi-attacher-57689cc84b-h7zw9 1/1 Running 0 12m
csi-attacher-57689cc84b-zsb4f 1/1 Running 0 12m
csi-provisioner-6c78dcb664-pc2f5 1/1 Running 0 12m
csi-provisioner-6c78dcb664-vjw8z 1/1 Running 0 12m
csi-provisioner-6c78dcb664-xfvc6 1/1 Running 0 12m
csi-resizer-7466f7b45f-b8qq9 1/1 Running 0 12m
csi-resizer-7466f7b45f-pbhwp 1/1 Running 0 12m
csi-resizer-7466f7b45f-pdqr5 1/1 Running 0 12m
csi-snapshotter-58bf69fbd5-54w76 1/1 Running 0 12m
csi-snapshotter-58bf69fbd5-8vznv 1/1 Running 0 12m
csi-snapshotter-58bf69fbd5-ps7nj 1/1 Running 0 12m
engine-image-ei-acb7590c-6jq2h 1/1 Running 0 12m
engine-image-ei-acb7590c-bhjs5 1/1 Running 0 12m
instance-manager-45ef10875bdd08b79ebfe014faeb1722 1/1 Running 0 12m
instance-manager-55c27a0ae5e3ee437940cd838ea3ccef 1/1 Running 0 12m
longhorn-csi-plugin-f8zqf 3/3 Running 0 12m
longhorn-csi-plugin-g22hx 3/3 Running 0 12m
longhorn-driver-deployer-576d574c8-j2lzq 1/1 Running 0 12m
longhorn-manager-2jf5d 1/1 Running 1 (12m ago) 12m
longhorn-manager-tfh54 1/1 Running 0 12m
longhorn-ui-556f7bb76c-fwf5d 1/1 Running 0 12m
longhorn-ui-556f7bb76c-ljflg 1/1 Running 0 12m
```
### 測試 mysql database 飄移在不同節點上不會遺失資料
使用以下 yaml 測試
```yaml!
$ cat <<EOF > test-mysql.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
labels:
app: mysql
spec:
selector:
matchLabels:
app: mysql # has to match .spec.template.metadata.labels
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
restartPolicy: Always
containers:
- image: mysql:5.6
name: mysql
livenessProbe:
exec:
command:
- ls
- /var/lib/mysql/lost+found
initialDelaySeconds: 5
periodSeconds: 5
env:
- name: MYSQL_ROOT_PASSWORD
value: changeme
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-volume
mountPath: /var/lib/mysql
volumes:
- name: mysql-volume
persistentVolumeClaim:
claimName: mysql-pvc
EOF
```
### 建立 mysql
```
$ kubectl apply -f test-mysql.yaml
```
### 檢查部屬狀態
```!
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mysql-7b79bcfb7c-2zrxh 1/1 Running 0 33s 10.244.2.18 k1w2 <none> <none>
```
### 在 mysql 產生資料
```!
$ mysql -u root -p -h 10.244.2.18
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.51 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> create database haha;
Query OK, 1 row affected (0.001 sec)
MySQL [(none)]> use haha;
Database changed
MySQL [haha]> CREATE TABLE customer (
-> ID int NOT NULL,
-> NAME varchar(100) NOT NULL,
-> AGE int,
-> ADDRESS varchar(255),
-> SALARY DECIMAL(10,2),
-> PRIMARY KEY (ID)
-> );
Query OK, 0 rows affected (0.028 sec)
MySQL [haha]> INSERT INTO customer
-> VALUES (1,'Ramesh',32,'Ahmedabad','2000.00'),
-> (2,'Khilan',25,'Delhi','1500.00'),
-> (3,'kaushik',23,'kota','2000.00'),
-> (4,'Chaitali',25,'Mumbai','6500.00'),
-> (5,'Hardik',27,'Bhopal','8500.00'),
-> (6,'Komal',22,'MP','4500.00'),
-> (7,'Muffy',24,'Indore','10000.00');
Query OK, 7 rows affected (0.005 sec)
Records: 7 Duplicates: 0 Warnings: 0
MySQL [haha]> select * from customer;
+----+----------+------+-----------+----------+
| ID | NAME | AGE | ADDRESS | SALARY |
+----+----------+------+-----------+----------+
| 1 | Ramesh | 32 | Ahmedabad | 2000.00 |
| 2 | Khilan | 25 | Delhi | 1500.00 |
| 3 | kaushik | 23 | kota | 2000.00 |
| 4 | Chaitali | 25 | Mumbai | 6500.00 |
| 5 | Hardik | 27 | Bhopal | 8500.00 |
| 6 | Komal | 22 | MP | 4500.00 |
| 7 | Muffy | 24 | Indore | 10000.00 |
+----+----------+------+-----------+----------+
7 rows in set (0.001 sec)
```
### 將 Pod 轉移到其他節點上，並觀察資料是否還在
```
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mysql-7b79bcfb7c-2zrxh 1/1 Running 0 5m 10.244.2.18 k1w2 <none> <none>
bigred@kube-kadm:~$ kubectl cordon k1w2
node/k1w2 cordoned
bigred@kube-kadm:~$ kubectl scale deploy mysql --replicas=0
deployment.apps/mysql scaled
bigred@kube-kadm:~$ kubectl cordon k1w2
node/k1w2 already cordoned
bigred@kube-kadm:~$ kubectl get pods -o wide
No resources found in default namespace.
bigred@kube-kadm:~$ kubectl scale deploy mysql --replicas=1
deployment.apps/mysql scaled
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mysql-7b79bcfb7c-fz7pg 1/1 Running 0 30s 10.244.1.24 k1w1 <none> <none>
bigred@kube-kadm:~$ mysql -u root -p -h 10.244.1.24
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.51 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> select * from haha.customer;
+----+----------+------+-----------+----------+
| ID | NAME | AGE | ADDRESS | SALARY |
+----+----------+------+-----------+----------+
| 1 | Ramesh | 32 | Ahmedabad | 2000.00 |
| 2 | Khilan | 25 | Delhi | 1500.00 |
| 3 | kaushik | 23 | kota | 2000.00 |
| 4 | Chaitali | 25 | Mumbai | 6500.00 |
| 5 | Hardik | 27 | Bhopal | 8500.00 |
| 6 | Komal | 22 | MP | 4500.00 |
| 7 | Muffy | 24 | Indore | 10000.00 |
+----+----------+------+-----------+----------+
7 rows in set (0.005 sec)
```
---
### Block Volume 測試
* 建立 pvc
```
$ echo 'apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: longhorn-block-vol
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
storageClassName: longhorn
resources:
requests:
storage: 3Gi' | kubectl apply -f -
```
* 建立 pod 並且掛載 pvc
```
$ echo 'apiVersion: apps/v1
kind: Deployment
metadata:
name: pod-with-raw-block-volume
labels:
os: alpine
spec:
replicas: 1
selector:
matchLabels:
os: alpine
template:
metadata:
labels:
os: alpine
spec:
containers:
- name: alpine
image: taiwanese/alpine:stable
imagePullPolicy: IfNotPresent
command: ["/bin/sleep", "infinity"]
volumeDevices:
- name: data
devicePath: /dev/sdd
securityContext:
privileged: true
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- |
set -e
mkdir /longhorn
checkformat=$(blkid | grep -w /dev/sdd | cut -d ':' -f1)
[[ "$checkformat" != /dev/sdd ]] && (mkfs.ext4 /dev/sdd && mount /dev/sdd /longhorn) || mount /dev/sdd /longhorn
preStop:
exec:
command:
- /bin/sh
- -c
- |
umount /dev/sdd
volumes:
- name: data
persistentVolumeClaim:
claimName: longhorn-block-vol' | kubectl apply -f -
```
```
$ kubectl get po
NAME READY STATUS RESTARTS AGE
pod-with-raw-block-volume-57b7bf4d5d-nmrzn 1/1 Running 0 20s
```
```
$ kubectl exec -it pod-with-raw-block-volume-57b7bf4d5d-5wl9l -- bash
bash-5.1# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 4K 1 loop
loop1 7:1 0 28K 1 loop
loop2 7:2 0 2.3M 1 loop
loop3 7:3 0 53.8M 1 loop
loop4 7:4 0 16M 0 loop
loop5 7:5 0 3G 0 loop
sda 8:0 0 800G 0 disk
├─sda1 8:1 0 100M 0 part
├─sda2 8:2 0 1M 0 part
├─sda3 8:3 0 1000M 0 part
├─sda4 8:4 0 1M 0 part
├─sda5 8:5 0 100M 0 part
└─sda6 8:6 0 798.8G 0 part /etc/resolv.conf
/etc/hostname
/dev/termination-log
/etc/hosts
sdb 8:16 0 120G 0 disk
sdc 8:32 0 120G 0 disk
sdd 8:48 0 3G 0 disk /longhorn
sr0 11:0 1 81.4M 0 rom
bash-5.1# echo 123 > /longhorn/test
bash-5.1# cat /longhorn/test
123
```
* 測試 pod 移轉
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-with-raw-block-volume-57b7bf4d5d-5wl9l 1/1 Running 0 59s 10.244.2.24 k1w1 <none> <none>
$ kubectl scale deploy pod-with-raw-block-volume --replicas=0
$ kubectl cordon k1w1
$ kubectl scale deploy pod-with-raw-block-volume --replicas=1
$ kubectl get no
NAME STATUS ROLES AGE VERSION
k1m1 Ready control-plane 47m v1.29.2
k1w1 Ready,SchedulingDisabled worker 42m v1.29.2
k1w2 Ready worker 42m v1.29.2
$ kubectl uncordon k1w1
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-with-raw-block-volume-6cfcd9889-t6pzt 1/1 Running 0 25s 10.244.1.21 k1w2 <none> <none>
$ kubectl exec -it pod-with-raw-block-volume-6cfcd9889-t6pzt -- cat /longhorn/test
123
```
## 參考資料
- [System Extensions - Talos Docs](https://www.talos.dev/v1.6/talos-guides/configuration/system-extensions/)
- [Boot Assets - Talos Docs](https://www.talos.dev/v1.6/talos-guides/install/boot-assets/#image-factory)
- [Talos Linux Image Factory](https://factory.talos.dev/)
Sign in with Wallet
Connect another wallet