Learning and Practicing Harvester
Preface
本篇文章會主要會介紹以下主題 :
- Harvester 的架構
- 如何安裝 Harvester,並將額外的節點加入 Harveter Cluster
- 將 Harvester 與 Rancher 整合,做出 Nested Kubernetes Clusters 以及 Node Pool
可以透過點擊以下目錄,選擇想看的內容,跳轉至特定章節
Harvester Architecture
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
Harvester 架構由以下開源技術組成:
- Linux 作業系統 : Elemental for SLE-Micro 5.3 是 Harvester 的核心,它是一個不可變的 Linux 發行版,旨在盡可能消除 Kubernetes 叢集中的作業系統維護工作。
- 建構於 Kubernetes(RKE2) 之上 : Kubernetes 已成為所有形式因素中最主要的基礎架構語言,而 Harvester 就是一個在引擎蓋下裝有 Kubernetes 的 HCI 解決方案。
- 利用 Kubevirt 進行虛擬化管理 : Kubevirt 在 Kubernetes 的基礎上使用 KVM 提供虛擬化管理。
- 利用 Longhorn 進行儲存管理 : Longhorn 提供分散式區塊儲存。
Harvester Features
Harvester 是一個 enterprise-ready、易於使用的基礎架構平台,它利用本地直接連接的 Storage,而不是複雜的外部 SAN。 它利用 Kubernetes API 作為跨 Container 和 虛擬機器 workloads 的統一自動化語言。 Harvester 的一些主要功能包括 :
- 易於上手 : 由於 Harvester 是以可啟動裝置鏡像的形式提供的,因此您可以使用 ISO 映像直接將其安裝到裸機伺服器上,也可以使用 iPXE 腳本自動安裝。
- 虛擬機器生命週期管理 : 輕鬆建立、編輯、複製和刪除虛擬機,包括 SSH 金鑰注入、Cloud Init 以及圖形化界面 (VNC) 和 serial port (22 port) 控制台。
- 虛擬機器即時遷移 : 零停機時間將虛擬機器遷移到不同的主機或節點。
- 虛擬機器備份、快照和還原 : 從 NFS、S3 伺服器或 NAS 設備備份虛擬機器。 使用備份好的虛擬硬碟檔還原故障的虛擬機,或在不同的叢集上建立新的虛擬機。
- 儲存管理 : Harvester 支援分散式區塊的儲存和 tiering (分層)。 Volumes 代表 storage;你可以輕鬆建立、編輯、複製或匯出磁碟區。
- 網路管理 : 支援使用虛擬 IP(VIP)和多個網路介面卡(NIC)。 如果虛擬機器需要連接外部網絡,可建立 VLAN 或 untagged (無標記) 網路。
- 與 Rancher 整合 : 透過 Rancher 的虛擬化管理頁面直接存取 Rancher 中的 Harvester,並與 Kubernetes 叢集一起管理 VM workloads (虛擬機器工作負載),實現 Node Pool 的功能。
PreRequest
Hardware and Network Requirements
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
詳細資訊,請以下參考 Harvester 官網 Docs :
Hardware and Network Requirements
Install Single-Node Harvester Clusters with USB
1. 下載 ISO 並透過 Rufus 將 USB 做成可開機裝置
1.1. 下載 ISO
可以使用 ISO 鏡像直接將 Harvester 安裝到裸機伺服器上,若要取得 ISO 映像,請從 Harvester releases 頁面下載 💿 harvester-v1.x.x-amd64.iso
。
螢幕輸出 :
1.2. 開啟 Rufus,將乾淨的 USB 碟插入電腦,Rufus 會自動偵測 USB。 從 "裝置 "下拉式選單中選擇要使用的 USB 裝置,在 "啟動選擇 "中選擇 "選擇",然後找到要燒錄到 USB 上的 Harvester 安裝 ISO 映像。
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
2. 安裝步驟
2.1. 使用 USB 開機,並透過選擇 Harvester Installer 選項啟動伺服器。
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
使用預設選項,直接按 Enter 按鍵。
2.2. 選擇建立一個新的 Harvester Cluster
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
使用預設選項,直接按 Enter 按鍵。
2.3. 選擇要安裝的 Harvester 系統碟和要儲存虛擬機器資料的資料碟。
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
按 Tab 鍵可展開下拉式選單,按 Enter 鍵儲存設定值。
2.4. 設定主機的 Hostname
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
輸入 h1
2.5. 設定管理網路
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
網路卡的部分,按 Tab 展開下拉式選單,按空白鍵選擇網路卡,確定之後按 Enter 鍵
IPv4 Address 欄位,輸入 : 172.20.0.34/24
Gateway 欄位,輸入 : 172.20.0.254
2.6. 設定 DNS Server
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
DNS Server 欄位 : 172.20.0.254
2.7. 設定 VIP
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
按 Tab 展開下拉式選單,按向下方向鍵選擇 Static
,確認後按 Enter 鍵。
VIP 欄位 : 172.20.0.36
2.8. 設定 Cluster Token
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
Cluster token 欄位 : lab
Cluster token 用於將其他節點新增至 Harvester 叢集。
2.9. 設定使用者密碼
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
預設的使用者是 rancher
。
2.10. 設定 NTP Server
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
NTP Server 欄位 : time.google.com
。
2.11. 設定 HTTP Proxy Server ( 此步驟不一定要設定,取決於當下的環境 )
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
如果不設定就空白,直接按 Enter 鍵。
2.12. 匯入 SSH Keys ( 此步驟不一定要設定 )

如果不設定就空白,直接按 Enter 鍵。
2.13. 設定 Harvester 設定檔自訂主機 ( 此步驟不一定要設定 )

如果不設定就空白,直接按 Enter 鍵。
2.14. 查看並確認安裝選項

確認安裝選項沒問題後,按 Enter 鍵,Harvester 將會安裝到主機上。
2.15. 安裝完成後,節點將重新啟動,以下是啟動完成畫面

重新啟動後,Harvester 控制台會顯示管理 URL 和狀態。網路介面的預設 URL 是 https://your-virtual-ip
。 可以使用 F12 按鍵從 Harvester 控制台切換到 Shell,然後鍵入 exit 返回 Harvester 控制台。
2.16. 第一次登入的時候,系統會提示您設定預設管理員使用者的密碼。

使用者預設是 admin
Update Harvester Configuration After Installation
DNS servers
- SSH 連線至 Harvester Node
- 備份基本的 cloud-init 檔案
/oem/90_custom.yaml
- 修改
/oem/90_custom.yaml
檔案,並且更新以下 Yaml 檔中路徑的值 stages.initramfs[0].commands
修改範例內容如下 :
name: Harvester Configuration
stages:
initramfs:
- commands:
- modprobe kvm
- modprobe vhost_net
- sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="172.20.0.37"/'
/etc/sysconfig/network/config
將原本的 172.20.0.254
改成 172.20.0.37
- 重新開機
- 確認 DNS Server 修改正確
設定 Storage Class
- 預設的 StorageClass 他的 Replicas 會被設為 3 ,但因為只有單一台 Node,所以要額外新增一個 StorageClass 並將 Replicas 設為 1,否則 VM 會無法建立。
- 點選左側選單 Advanced -> Storage Classes -> Create
- 輸入以下資訊 :
- Name:
new-sc
- Number of Replicas 設為
1
確認資訊輸入正確後,點選 Create

- 設成 Default
透過 URL 下載 Cloud Images
- 左側選單選擇 Images -> Create
- 透過 URL 下載 OpenSUSE Leap 15.5 的 Cloud Images

URL 如下 :
https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.5/images/openSUSE-Leap-15.5.x86_64-NoCloud.qcow2
- 確認下載完成
新增 Cloud Config Templates
User Data
- 點選左側選單 Cloud Config Templates -> Create
- 輸入以下資訊 :
package_update
,會先更新系統的套件
packages
,代表要安裝的套件
write_files
,針對特定檔案寫入特定內容,這邊的範例是關 ipv6
runcmd
,系統開機以後要執行的命令,這邊有設定啟動 qemu-guest-agent 服務,並設為開機自動啟動,還有設定 DNS Server 的位置,還有設定時區,特別設定 DNS Server 的原因是因為要可以跟 Rancher 溝通
ssh_authorized_keys
,讓跳板機可以免密碼登入使用這個 Cloud init config 做出來的 VM
users
,新增使用者,並設定使用者的群組、sudo 權限和密碼。
確認資訊輸入正確後,點選 Create

Network Data
Static
DHCP
建立 VM Network
- 點選左側選單 Networks -> VM Networks -> Create
- 編輯以下設定資訊
- Name:
vlan
- Vlan ID:
100
- Cluster Netwrok:
mgmt
確認沒問題後,點選 Create

- 確認是否建立成功
State 和 Route Connectivity 的狀態都要顯示 Active

將其他節點加入 Harvester 叢集
1. 將燒有 Harvester ISO 的 USB 可開機裝置插入伺服器,並使用 USB 開機。
2. 安裝步驟
2.1. 使用 USB 開機,並透過選擇 Harvester Installer 選項啟動伺服器。
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
使用預設選項,直接按 Enter 按鍵。
2.2. 選擇加入一個已經存在的 Harvester Cluster

選擇 Join an existing Harvester cluster
後,按 Enter 按鍵。
2.3. 選擇要安裝的 Harvester 系統碟和要儲存虛擬機器資料的資料碟。

按 Tab 鍵可展開下拉式選單,按 Enter 鍵儲存設定值。
2.4. 設定主機的 Hostname

h2
2.5. 設定網路

網路卡的部分,按 Tab 展開下拉式選單,按空白鍵選擇網路卡,確定之後按 Enter 鍵
IPv4 Address 欄位,輸入 : 172.20.0.35/24
Gateway 欄位,輸入 : 172.20.0.254
2.6. 設定 DNS Server

DNS Server 欄位 : 172.20.0.37
2.7. 設定 管理 IP Address

Management address 欄位 : 172.20.0.36
2.8. 設定 Cluster Token
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
Cluster token 欄位 : lab
Cluster token 用於將其他節點新增至 Harvester 叢集。
2.9. 設定使用者密碼
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
預設的使用者是 rancher
。
2.10. 設定 NTP Server

NTP Server 欄位 : time.google.com
。
2.11. 設定 HTTP Proxy Server ( 此步驟不一定要設定,取決於當下的環境 )
Image Not Showing
Possible Reasons
- The image was uploaded to a note which you don't have access to
- The note which the image was originally uploaded to has been deleted
Learn More →
如果不設定就空白,直接按 Enter 鍵。
2.12. 匯入 SSH Keys ( 此步驟不一定要設定 )

如果不設定就空白,直接按 Enter 鍵。
2.13. 設定 Harvester 設定檔自訂主機 ( 此步驟不一定要設定 )

如果不設定就空白,直接按 Enter 鍵。
2.14. 查看並確認安裝選項

確認安裝選項沒問題後,按 Enter 鍵,Harvester 將會安裝到主機上。
2.15. 安裝完成後,節點將重新啟動,以下是啟動完成畫面
2.16. 打開 Harvester 網站,檢查 Node 是否變 2 台
點選左側選單 Hosts

使用者預設是 admin
與 Rancher 整合
PreRequest
Rancher 啟動 Virtualization management
- 打開虛擬化管理介面
點選左側選單 Virtualization Management

- Import Harvester Cluster
點選 Import Existing

- 填入叢集資訊
- 填入叢集資訊
- 複製讓 Harvester 與 Rancher 註冊的 URL
- 回到 Harvester 的網站,與 Rancher 註冊 (1/2)
點選左側選單 Settings -> 找到 cluster-registration-url -> 點選 Edit Setting

- 回到 Harvester 的網站,與 Rancher 註冊 (2/2)
將註冊 URL 貼上 -> 點選 Save

- 到 Rancher UI 確認
Creating an K3s Kubernetes Cluster
- 在 Home Page 點選 Create
- Node Driver 選擇 Harvester
- 建立 Cloud Credential
- Credential Name:
harvester
- Imported Harvester Cluster 選擇 harvester
- 點選 Continue

- 編輯 Machine Pools
VM 必須要能透過 DNS Server 解析 Rancher 的 FQDN,否則 K3S 將無法建立。
- Cluster Name:
bobo
- CPU:
4
- Memory:
8
- Namespace:
default
- SSH User:
Opensuse
- Volumes:
- Image Volume:
openSUSE-Leap-15.5.x86_64-NoCloud.qcow2
- Disk:
100
- Networks:
- Network Name:
default/vlan (vlanId=100)
- 點選
Show Advanced
按鈕
- User Data:
- User Data Template:
opensuse
- 編輯 Cluster Configuration
- 選擇 Kubernetes 版本 : k3s v1.26.13+k3s2
- 點選 Create
- 檢查 K3S 是否建立成功
點選 Provisioning Log

要 Troubleshooting 的話,可以 ssh 連線到 VM 中,看以下兩個檔案 :
/var/log/cloud-init.log
。
/var/log/cloud-init-output.log
,這檔案裡面紀錄 cloud init 命令執行的結果,包含透過 Rancher 安裝 K3S 的過程都會記在這個檔案中。
K3s node Scale up to 2
- Scale up node
點選 Machine Pools -> +

- 檢查是否符合預期
K3s node Scale down to 1
- Scale down node
點選 Machine Pools -> -

- 檢查是否符合預期
Add Additional Disks
- 再新增一顆硬碟後,要先 Format 成 ext4 檔案系統再重新開機, Harvester 才認的到
- Go to the Hosts page.
- On the node you want to modify, click ⋮ > Edit Config.
- Select the Storage tab and click Add Disk.
- Select an additional raw block device to add as an additional data volume.
- The
Force Formatted
option is required if the block device has never been force-formatted.
- Last, you can click ⋮ > Edit Config again to check the newly added disk.
Resize Disk (虛擬化環境)
- 先在虛擬化那層擴硬碟空間
- SSH 連線進那台 node,並切換成 root 使用者
- Resize Partitaion 大小
- 執行以下命令 Resize Longhorn 的 Filesystem
- 再進 Longhorn UI 看就會發現空間擴出來了
Stop VM with cli
參考文章