Helm3 - 入門
===
###### tags: `K8s / Helm`
###### tags: `Kubernetes`, `k8s`, `Helm Chart`, `HelmChart`
<br>
[TOC]
<br>
## Helm 官網
- 英文 - https://helm.sh/docs/intro/using_helm/
- 中文 - https://helm.sh/zh/docs/topics/charts/
<br>
<hr>
<br>
## 前言
### Helm
- 管理 K8s Chart 套件的工具
### Chart
- 預先配置好的 K8s 資源物件集合
### [輔助] K8s 的 debug 資源 [HackMD筆記]
- [K8s / Hello World](/WTFM-2OsRXK-tW-YwKWzzg)
- [K8s / 指令集](/kD5ynMNsRH-TWq9KUwXfCQ)
<br>
<hr>
<br>
## 安裝 Helm
### 安裝 Helm3 (Helm v3)
- [[NVIDIA] Install NVIDIA GPU Operator / Install Helm](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-helm)
```bash
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
&& chmod 700 get_helm.sh \
&& ./get_helm.sh
```
- ### 參數說明：
| 參數 | 說明 |
|-----|------|
| `-f` | **Fail silently (on HTTP errors)**：如果 HTTP 回應狀態碼是 400 以上的錯誤，不會輸出錯誤訊息，也不會將錯誤 HTML 顯示出來。適合在腳本中使用。 |
| `-s` | **Silent mode**：不顯示任何進度條或錯誤訊息（除非搭配 `-f` 時有特殊行為）。這樣可以讓輸出更乾淨。|
| `-S` | **Show error**：與 `-s` 搭配使用時，會在失敗時顯示錯誤訊息。 |
| `-L` | **Location**：如果 URL 回傳的是 3xx 的重新導向，會自動跟隨新的位置。 |
| `-o get_helm.sh` | 將下載的內容輸出到名為 `get_helm.sh` 的檔案中（output to file）。|
- 沒有 `-s` 會有底下 verbose log
```
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11913 100 11913 0 0 59703 0 --:--:-- --:--:-- --:--:-- 59864
```
- ### 綜合解釋：
這條指令的意思是：
> 使用 `curl` 靜默且安全地下載 Helm 安裝腳本，如果有重新導向也會跟過去，並將檔案儲存成 `get_helm.sh`，接著給它執行權限，然後執行這個腳本。
這樣的寫法常用在自動化腳本中，避免不必要的輸出干擾使用者或其他工具的判斷。
<br>
## Hello World
> 預設的 image 樣板範例為 nginx (port:80)
- 建立 Chart 參考樣板
```bash=
$ helm create tj-helm-for-nginx
Creating tj-helm-for-nginx
```
- 查看檔案結構
```bash=
$ tree
.
└── tj-helm-for-nginx
├── Chart.yaml <--- 紀錄 Chart 的版本、名稱、描述
├── charts <--- 放置 sub charts
├── templates <--- K8s 資源物件定義
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ ├── serviceaccount.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml <--- 定義 K8s 資源物件的參數，要帶入到 templates 中
4 directories, 10 files
```
- 切到 剛建立出來的 Chart 目錄
```bash=
$ cd tj-helm-for-nginx/
```
- 查看要佈署到 K8s 的樣板： deployment.yaml
```bash=
$ cat templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "tj-helm-for-nginx.fullname" . }}
labels:
{{- include "tj-helm-for-nginx.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
...
```
- 可以看到：很多的「**參數代碼 {{ ... }}**」
- 這些參數，將由 values.yaml 餵入
- 其他檔案，以此類推
<br>
- 查看 K8s 資源物件的參數：```values.yaml```
```bash=
$ cat values.yaml
# Default values for tj-helm-for-nginx.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: nginx
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
...
```
- 可以看到：很多的「參數」
- 預設是使用 nginx 這個 web app，port 使用 80
<br>
- 將當前目錄下的 Chart 安裝到 K8s
```bash=
# 查詢已安裝的 Chart (helm list 或 helm ls)
$ helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
# 將當前目錄下的 Chart，安裝到 K8s
## helm 2
$ helm install .
# 或是
$ helm install . -n tj-helm-demo1
$ helm install -n tj-helm-demo1 .
## helm 3
$ helm install tj-helm-demo1 .
NAME: tj-helm-demo1
LAST DEPLOYED: Tue Dec 1 17:30:24 2020
NAMESPACE: helm-lab
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace helm-lab -l "app.kubernetes.io/name=tj-helm-for-nginx,app.kubernetes.io/instance=tj-helm-demo1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helm-lab port-forward $POD_NAME 8080:80
# 查詢已安裝的 Chart (helm list 或 helm ls)
$ helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
tj-helm-demo1 helm-lab 1 2020-12-01 17:30:24.483291444 +0800 CST deployed tj-helm-for-nginx-0.1.01.16.0
```
- 測試佈署到 K8s 的結果
```bash=
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
tj-helm-demo1-tj-helm-for-nginx-9f5b9949d-nmbzf 1/1 Running 0 7m33s
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tj-helm-demo1-tj-helm-for-nginx-9f5b9949d-nmbzf 1/1 Running 0 8m45s 10.244.1.3 tj-cluster-worker2 <none> <none>
```
- 測試 pod 的連線
```bash=
# 安裝完 Chart 後，最後三行會顯示連線資訊
export POD_NAME=$(kubectl get pods --namespace helm-lab -l "app.kubernetes.io/name=tj-helm-for-nginx,app.kubernetes.io/instance=tj-helm-demo1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helm-lab port-forward $POD_NAME 8080:80
```
- 執行「上述三行指令」的過程：
[](https://i.imgur.com/lgbng5c.png)
<br>
- 開啟另一個 terminal 測試連線
```
$ curl -X GET 127.0.0.1:8080
```
[](https://i.imgur.com/of8clcy.png)
<br>
- 刪除佈署到 K8s 的 Chart
```bash=
$ helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
tj-helm-demo1 helm-lab 1 2020-12-01 17:30:24.483291444 +0800 CST deployed tj-helm-for-nginx-0.1.0 1.16.0
$ helm delete --purge tj-helm-demo1
release "tj-helm-demo1" uninstalled
$ helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
$ kubectl get pods
No resources found in helm-lab namespace.
```
- 測試沒問題，就將當前的 Chart 目錄，打包成壓縮檔 (tgz 檔)
```bash=
# This command packages a chart into a versioned chart archive file.
# If a path is given, this will look at that path for a chart
# (which must contain a Chart.yaml file) and then package that directory.
$ helm package .
Successfully packaged chart and saved it to: .../tj-helm-for-nginx/tj-helm-for-nginx-0.1.0.tgz
$ ll
total 32
drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 1 17:54 ./
drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 1 17:15 ../
-rw-r--r-- 1 ubuntu ubuntu 349 Dec 1 17:15 .helmignore
-rw-r--r-- 1 ubuntu ubuntu 1108 Dec 1 17:15 Chart.yaml
drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 1 17:15 charts/
drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 1 17:15 templates/
-rw-rw-r-- 1 ubuntu ubuntu 3589 Dec 1 17:54 tj-helm-for-nginx-0.1.0.tgz <---
-rw-r--r-- 1 ubuntu ubuntu 1810 Dec 1 17:15 values.yaml
```
- 可將打包好的 tgz 檔，進行佈署到其他節點
```
# helm 3
$ helm install tj-helm-demo2 tj-helm-for-nginx-0.1.0.tgz
$ helm install tj-helm-demo3 tj-helm-for-nginx-0.1.0.tgz
$ helm install tj-helm-demo4 tj-helm-for-nginx-0.1.0.tgz
$ helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
tj-helm-demo2 helm-lab 1 2020-12-01 17:59:34.291777489 +0800 CST deployed tj-helm-for-nginx-0.1.0 1.16.0
tj-helm-demo3 helm-lab 1 2020-12-01 18:01:37.284712894 +0800 CST deployed tj-helm-for-nginx-0.1.0 1.16.0
tj-helm-demo4 helm-lab 1 2020-12-01 18:01:42.366486885 +0800 CST deployed tj-helm-for-nginx-0.1.0 1.16.0
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
tj-helm-demo2-tj-helm-for-nginx-777d9c67d7-4cnwh 1/1 Running 0 2m15s
tj-helm-demo3-tj-helm-for-nginx-86654488-7bd78 1/1 Running 0 12s
tj-helm-demo4-tj-helm-for-nginx-549df6d677-tkzzg 1/1 Running 0 7s
```
- 每一次的佈署，都會產生新的 instance (實例)
- K8s 與 Java 有異曲同工之妙
| Java | K8s |
| ------------- | --------------- |
| 類別(class) / 多個物件(object) | 資源物件(resource object) / 實體(instance) |
| my-utils-**v1**.jar | apiVersion: **v1** |
| package **a.b.c** | metadata:<br> namespace: **a.b.c** |
| class **Abc** extends **SomeClass** | kind: **SomeClass**<br>metadata:<br> name: **Abc** |
| 欄位(fields) | 後設資料(metadata) |
| 方法(method) | 技術規格(spec) |
<br>
- 參考資料
- [Kubernetes 基礎教學（三）Helm 介紹與建立 Chart]()
<br>
<hr>
<hr>
<br>
## Hello World 2 (v2 & v3) - with NodePort
- 指令
```bash
# 建立 Chart 參考樣板
$ helm create tj-helm-for-nginx2
```
```bash
# 進入 Chart 根目錄
$ cd tj-helm-for-nginx2
```
```bash
# 到 service.yaml 中，新增 nodePort 樣板
# nodePort: {{ .Values.service.nodePort }}
$ nano templates/service.yaml
```
- service.yaml 變更前
```yaml
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
```
- service.yaml 變更後
```yaml
ports:
- port: {{ .Values.service.port }}
nodePort: {{ .Values.service.nodePort }} # <---
targetPort: http
protocol: TCP
name: http
```
```bash
# 到 values.yaml 中，新增 nodePort 參數
$ nano values.yaml
```
- values.yaml 變更前
```yaml
service:
type: ClusterIP
port: 80
```
- values.yaml 變更後
```yaml
service:
type: NodePort # <---
nodePort: 30080 # <---
port: 80
```
```bash
# 將當前目錄下的 Chart 安裝到 K8s
$ helm install tj-helm-for-nginx-demo2 .
```
- 連線測試
http://192.168.34.8:30080
http://203.145.218.3:30080/ (若有公開 IP 的話)
- 若沒問題，就可以打包 Chart，並測試 chart 檔(壓縮檔)
```bash
# 刪除已經佈署的 Chart
$ helm ls
$ helm delete tj-helm-for-nginx-demo2
$ helm ls
# 打包成 Chart
$ helm package .
# 安裝 Chart 檔
$ helm install tj-helm-for-nginx-demo2 tj-helm-for-nginx2-0.1.0.tgz
# 可用 tar -xvcf xxx.tgz 解壓縮
# 查看佈署結果
$ helm ls
$ kubectl get pods -o wide
$ kubectl get services -o wide
```
<br>
<hr>
<hr>
<br>
## Hello World 3 (v2 & v3) - with Ingress
- 指令
```bash
# 建立 Chart 參考樣板
$ helm create tj-helm-for-ingress3
```
```bash
# 進入 Chart 根目錄
$ cd tj-helm-for-ingress3
```
---
```bash
# 到 values.yaml 中，新增 nodePort 參數
$ nano values.yaml
```
- 差異性 (變更前 v.s. 變更後)
```diff
image:
- repository: nginx
+ repository: hcwxd/blue-whale
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
- tag: ""
+ tag: "latest"
+containerPort: 3000
ingress:
- enabled: false
+ enabled: true
- annotations: {}
+ annotations:
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
+ nginx.ingress.kubernetes.io/rewrite-target: /
hosts:
- - host: chart-example.local
- paths: []
+ - paths: ["/blue"]
```
- 注意：host 要拿掉
不然綁訂了 host name，就無法用 IP 存取，
會出現 502 Bad Gateway，例如
```
$ kubectl get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tjdemo3-tj-helm-for-ingress3 <none> chart-example.local 10.98.112.86 80 74s
$ curl 10.98.112.86/blue
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.17.8</center>
</body>
</html>
```
---
```bash
# 到 depoyment.yaml 中，新增 containerPort 參數
$ nano templates/deployment.yaml
```
- 差異性 (變更前 v.s. 變更後)
```diff
ports:
- name: http
- containerPort: 80
+ containerPort: {{ .Values.containerPort }}
protocol: TCP
```
---
```bash
# 將當前目錄下的 Chart 安裝到 K8s
$ helm install tjdemo3 .
```
- 連線測試1: 測試內部的 ingress
```
$ kubectl get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tjdemo3-tj-helm-for-ingress3 <none> * 10.98.112.86 80 16m
$ curl 10.98.112.86:80/blue
```
- 連線測試2: 測試對外的 ingress
```
$ kubectl get svc -o wide --all-namespaces
$ kubectl get svc -o wide -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
ingress-nginx NodePort 10.98.112.86 <none> 80:30511/TCP,443:30152/TCP 4d3h app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
$ curl 127.0.0.1:30511/blue
```
http://203.145.218.3:30511/blue (若有公開 IP 的話)
- 若沒問題，就可以打包 Chart，並測試 chart 檔(壓縮檔)
```bash
# 刪除已經佈署的 Chart
$ helm ls
$ helm delete tj-helm-for-ingress3
$ helm ls
# 打包成 Chart
$ helm package .
# 安裝 Chart 檔
$ helm install tj-helm-for-ingress3 tj-helm-for-ingress3 -0.1.0.tgz
# 可用 tar -xvcf xxx.tgz 解壓縮
# 查看佈署結果
$ helm ls
$ kubectl get pods -o wide
$ kubectl get services -o wide
```
<br>
<hr>
<hr>
<br>
- [[Helm] Helm v3 使用簡介 -- 小信豬的原始部落](https://godleon.github.io/blog/DevOps/Helm-v3-Chart-Template-Guide/)
## [[helm 2][小信豬的原始部落] 什麼是 Kubernetes Helm?](https://godleon.github.io/blog/Kubernetes/k8s-Helm-Introduction/)
- **Helm** is a tool for managing Kubernetes charts.
**Helm** 是管理 K8s Chart 的工具
- **Charts** are packages of pre-configured Kubernetes resources.
**Charts** 是預先配置好 K8s 資源的軟體套件
<br>
- Helm 就有點像是 Ubuntu 中的 APT 系統
- Chart 是多個 k8s resource object configuration 的集合
### 安裝 Helm client
```
$ wget https://storage.googleapis.com/kubernetes-helm/helm-$(curl -s https://api.github.com/repos/helm/helm/releases/latest | jq --raw-output '.tag_name')-linux-amd64.tar.gz -O helm-linux-amd64.tar.gz
$ tar -zxvf helm-linux-amd64.tar.gz
$ sudo mv linux-amd64/helm /usr/local/bin/helm
```
### 將 Helm 佈署至 Cluster Level
- [helm init is not recognized #6996](https://github.com/helm/helm/issues/6996)
### 更新 Helm Repo (v3)
```bash
$ helm repo update
# Usage: helm search [command]
# Available Commands:
# - hub search for charts in the Helm Hub or an instance of Monocular
# - repo search repositories for a keyword in charts
$ helm search repo
$ helm search repo | grep nginx
ingress-nginx/ingress-nginx 3.12.0 0.41.2 Ingress controller for Kubernetes using NGINX a...
stable/nginx-ingress 1.41.3 v0.34.1 DEPRECATED! An nginx Ingress controller that us...
stable/nginx-ldapauth-proxy 0.1.6 1.13.5 DEPRECATED - nginx proxy with ldapauth
stable/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
```
### helm command
```bash=
# 取得最新的 chart 資訊
$ helm repo update
# 搜尋 helm charts (會出現很多 charts info)
$ helm search -l
# 安裝 mysql helm chart (必要時，好像要加上 sudo ?)
$ sudo helm install stable/mysql
# 刪除佈署
$ helm delete wayfaring-butterfly --purge
# 列出所有的佈署
$ helm list -a # helm ls -a
```
### 小節複習
- Helm 是什麼？
- Helm 是一項工具，是什麼工具？
- 用來管理 K8s Charts 的工具
- 簡單來講，Helm 就如同 Ubuntu 的 apt 系統
- Helm 可以將一個或多個應用打包成一個服務，方便佈署到 K8s 上
- Charts 是什麼？
- Charts 是一種軟體套件，是什麼樣的軟體套件？
- 預先配置好 K8s 資源的的軟體套件
<br>
## Helm v.s. Helm-based Operator
- Helm
- K8s 的 Chart 套件管理員
- 透過 Helm 工具，將 Chart 佈署到 K8s
```helm install -n release_name path_to_chart```
- Helm-based Operator
- 利用 K8s 的 CRD 機制
- operator-sdk 工具提供 "helm-based operator" 功能
可將 Helm Chart 資源轉成 operator
- 透過 kubectl apply -f xxx.yaml 方式部署
<br>
## 參考資料
### Nginx
- [Nginx: Installation with Helm](https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/)
### Prometheus
- [Installing the Prometheus-Operator on Kubernetes with Helm 3](http://www.dcasati.net/posts/installing-prometheus-on-kubernetes-v1.16.9/)
- [Install prometheus-operator doesn't work on AWS EC2 keeps produce “Error: failed to download ”stable/prometheus-operator"](https://stackoverflow.com/questions/64226913/install-prometheus-operator-doesnt-work-on-aws-ec2-keeps-produce-error-failed)
```
$ helm install prometheus-community/prometheus-operator --generate-name --namespace monitor
WARNING: This chart is deprecated
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: prometheus-operator-1607335590
LAST DEPLOYED: Mon Dec 7 18:06:34 2020
NAMESPACE: monitor
STATUS: deployed
REVISION: 1
NOTES:
DEPRECATED - This chart will be renamed. See https://github.com/prometheus-community/community/issues/28#issuecomment-670406329
---
The Prometheus Operator has been installed. Check its status by running:
kubectl --namespace monitor get pods -l "release=prometheus-operator-1607335590"
Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.
$ kubectl --namespace monitor get pods -l "release=prometheus-operator-1607335590"
NAME READY STATUS RESTARTS AGE
prometheus-operator-160733-operator-589476996d-5nlxb 2/2 Running 0 32s
prometheus-operator-1607335590-prometheus-node-exporter-fbxnb 1/1 Running 0 32s
prometheus-operator-1607335590-prometheus-node-exporter-ngdhk 1/1 Running 0 32s
```
- [[GitHub] prometheus-community / helm-charts](https://github.com/prometheus-community/helm-charts/)
- helm repo
```
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
```
- charts 資源列表
https://github.com/prometheus-community/helm-charts/tree/main/charts
- Install Charts
```
$ helm install [RELEASE_NAME] prometheus-community/prometheus
```
- 參考資料
- [prometheus 12.0.2 · prometheus/prometheus-community](https://artifacthub.io/packages/helm/prometheus-community/prometheus)
### Others
- [How To Install Software on Kubernetes Clusters with the Helm 3 Package Manager](https://www.digitalocean.com/community/tutorials/how-to-install-software-on-kubernetes-clusters-with-the-helm-3-package-manager)
Sign in with Wallet
Connect another wallet