# 實戰 Gitlab CI/CD

## 事前準備

* k8s 叢集需要先準備好
- Ingress controller(這裡使用 nginx)
- cert-manager
- Metrics server
- CSI(這裡使用 nfs-csi)

## 透過 cert-manager 產生自簽憑證

$ kubectl create namespace gitlab-system

$ nano cert.yaml
# cert.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gitlab-root-ca
  namespace: gitlab-system
spec:
  isCA: true
  commonName: "GitLab Internal Root CA"
  secretName: gitlab-root-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: gitlab-ca-issuer
  namespace: gitlab-system
spec:
  ca:
    secretName: gitlab-root-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gitlab-wildcard-tls
  namespace: gitlab-system
spec:
  secretName: gitlab-wildcard-tls  # 這張 Secret 會被所有人共用
  duration: 2160h
  renewBefore: 360h
  # 設定通用名稱與萬用字元
  commonName: "*.example.com"
  dnsNames:
  - "*.example.com"
  - "example.com"
  issuerRef:
    name: gitlab-ca-issuer
    kind: Issuer

$ kubectl apply -f cert.yaml

## 部署 Gitlab Operator

* Gitlab Operator 可以[參考](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/releases)版本清單

* 下載 Gitlab Operator yaml

$ wget https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/2.7.1/gitlab-operator-kubernetes-2.7.1.yaml

* 部署 Gitlab Operator

$ kubectl apply -f gitlab-operator-kubernetes-2.7.1.yaml

* 確認部署狀態

$ kubectl -n gitlab-system get all
NAME                                             READY   STATUS    RESTARTS   AGE
pod/gitlab-controller-manager-5d987b79bd-bvmrx   1/1     Running   0          72s

NAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/gitlab-controller-manager-metrics-service   ClusterIP   10.96.149.252   <none>        8443/TCP   72s
service/gitlab-webhook-service                      ClusterIP   10.96.131.105   <none>        443/TCP    72s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gitlab-controller-manager   1/1     1            1           72s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/gitlab-controller-manager-5d987b79bd   1         1         1       72s

## Installing GitLab
* 檢視環境 ingressclass

$ kubectl get ingressclass nginx
NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       9d

$ nano gitlab.yaml
apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
  namespace: gitlab-system
spec:
  chart:
    version: "9.7.1" # https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/<OPERATOR_VERSION>/CHART_VERSIONS
    values:
      global:
        hosts:
          domain: example.com # use a real domain here
        ingress:
          configureCertmanager: false
          class: nginx
          tls:
            enabled: true
            secretName: gitlab-wildcard-tls
      nginx-ingress:
        enabled: false
      prometheus:
        install: false
      gitlab:
        gitaly:
          persistence:
            storageClass: "nfs-csi"
            enabled: true
            size: 50Gi
            accessMode: ReadWriteOnce
      postgresql:
        primary:
          persistence:
            enabled: true
            size: 10Gi
            storageClass: "nfs-csi"
            accessMode: ReadWriteOnce
      minio:
        persistence:
          storageClass: "nfs-csi"
          enabled: true
          size: 50Gi
          accessMode: ReadWriteOnce
      redis:
        master:
          persistence:
            storageClass: "nfs-csi"
            enabled: true
            size: 5Gi

$ kubectl apply -f gitlab.yaml

* 環境檢查

$ kubectl -n gitlab-system get pod
NAME                                          READY   STATUS      RESTARTS      AGE
gitlab-controller-manager-5d987b79bd-m4fbl    1/1     Running     0             3h56m
gitlab-gitaly-0                               1/1     Running     0             12m
gitlab-gitlab-exporter-7cfd458df5-rrvhq       1/1     Running     0             12m
gitlab-gitlab-shell-64777bdfb6-jrss2          1/1     Running     0             5m31s
gitlab-gitlab-shell-64777bdfb6-rd8jj          1/1     Running     0             12m
gitlab-kas-f7598578f-5lzsg                    1/1     Running     0             5m31s
gitlab-kas-f7598578f-d8wnd                    1/1     Running     3 (12m ago)   12m
gitlab-migrations-a4161db-56b-1-k2gph         0/1     Completed   2             12m
gitlab-minio-create-buckets-60bc2fa-mjt2p     0/1     Completed   0             12m
gitlab-minio-dcfc68b7d-c8bvf                  1/1     Running     0             12m
gitlab-postgresql-0                           2/2     Running     0             12m
gitlab-redis-master-0                         2/2     Running     0             12m
gitlab-registry-848c6f5567-8d9v6              1/1     Running     0             5m31s
gitlab-registry-848c6f5567-dmkkr              1/1     Running     0             12m
gitlab-shared-secrets-4ecb0da-sgxb9           0/1     Completed   0             13m
gitlab-sidekiq-all-in-1-v2-779d4f8d9c-9qcgd   1/1     Running     0             3m15s
gitlab-sidekiq-all-in-1-v2-779d4f8d9c-sshk8   1/1     Running     0             5m46s
gitlab-sidekiq-all-in-1-v2-779d4f8d9c-wmvq7   0/1     Pending     0             3m15s
gitlab-toolbox-56f65c88d7-lnwff               1/1     Running     0             12m
gitlab-webservice-default-69b6f7574c-grcnb    2/2     Running     0             5m31s
gitlab-webservice-default-69b6f7574c-sd2gq    2/2     Running     0             5m47s

$ kubectl -n gitlab-system get ing
NAME                        CLASS   HOSTS                  ADDRESS      PORTS     AGE
gitlab-kas                  nginx   kas.example.com        10.10.7.83   80, 443   12m
gitlab-minio                nginx   minio.example.com      10.10.7.83   80, 443   12m
gitlab-registry             nginx   registry.example.com   10.10.7.83   80, 443   12m
gitlab-webservice-default   nginx   gitlab.example.com     10.10.7.83   80, 443   12m

$ kubectl -n gitlab-system get pvc
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-gitlab-postgresql-0           Bound    pvc-354e53e5-afe7-40e4-acd2-1af1fa93c80b   10Gi       RWO            nfs-csi        <unset>                 2m30s
gitlab-minio                       Bound    pvc-4f5d6e76-9eea-4177-987d-6675f67914c7   50Gi       RWO            nfs-csi        <unset>                 2m29s
redis-data-gitlab-redis-master-0   Bound    pvc-b7f96f8a-ac5a-4993-8d03-b8a28bb09a52   5Gi        RWO            nfs-csi        <unset>                 2m30s
repo-data-gitlab-gitaly-0          Bound    pvc-e315fb67-3a92-4393-bc4f-4baa08d6198b   50Gi       RWO            nfs-csi        <unset>                 2m29s

## 匯出 ca 到瀏覽器

$ kubectl -n gitlab-system get secret gitlab-root-ca-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

* 測試使用證書訪問

$ curl -I --cacert ca.crt https://gitlab.example.com
HTTP/2 302

## 訪問 gitlab 網站

* 獲取預設 root 密碼

$ kubectl -n gitlab-system get secret gitlab-gitlab-initial-root-password -o jsonpath='{.data.password}' | base64 --decode ; echo
pr68BgQkuqFaKc8NwtvQ4Bdtgj9u2GZIZzoeEAONgGFY1ARDVqBC7d710KSYGpwH


### 建立一個專案


* 選擇 Create blank project



## 部署 gitlab runner

$ helm repo add gitlab https://charts.gitlab.io

$ helm search repo -l gitlab/gitlab-runner


* 勾選 Run untagged jobs


* 獲取 token


$ nano runner-values.yaml
# 1. GitLab Server 的網址 (確保 Runner 解析得到這個網域)
gitlabUrl: "https://gitlab.example.com"

# 2. 您的 Runner Authentication Token
# 請去 GitLab Admin > CI/CD > Runners > New Instance Runner 取得 (glrt-開頭)
runnerToken: "glrt-WfFb6mdPsUM5p61SJBNBXG86MQpwOjEKdDozCnU6MQ8.01.170wik78c"

# 3. 併發數設定 (同時可以跑幾個 Job)
concurrent: 10

preEntrypointScript: |
  echo "--- Start setup CA ---"
  cp /home/gitlab-runner/.gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/gitlab-custom-ca.crt
  update-ca-certificates --verbose
  echo "--- End setup CA ---"

securityContext:
  runAsNonRoot: false
podSecurityContext:
  runAsUser: 0
rbac:
  create: true
  rules:
    - resources: ["pods", "pods/exec", "pods/attach", "pods/log", "secrets", "services", "configmaps"]
      verbs: ["get", "list", "watch", "create", "patch", "delete", "update"]
envVars:
  - name: CI_SERVER_TLS_CA_FILE
    value: "/home/gitlab-runner/.gitlab-runner/certs/ca.crt"
certsSecretName: gitlab-root-ca-secret
runners:
  runUntagged: true
  config: |
    [[runners]]
      environment = [
        "SSL_CERT_FILE=/home/gitlab-runner/.gitlab-runner/certs/tls.crt",
        "GIT_SSL_CAINFO=/home/gitlab-runner/.gitlab-runner/certs/tls.crt"
      ]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:latest"
        privileged = true
      [[runners.kubernetes.volumes.secret]]
            name = "gitlab-root-ca-secret"
            mount_path = "/home/gitlab-runner/.gitlab-runner/certs/"
            read_only = true

$ helm upgrade --install gitlab-runner gitlab/gitlab-runner \
  -n gitlab-system \
  -f runner-values.yaml

$ kubectl -n gitlab-system get po -l app=gitlab-runner -owide
NAME                            READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
gitlab-runner-688d97df4-lsjxm   1/1     Running   0          2m26s   10.244.190.110   w1     <none>           <none>

* 確認 gitlab runner 已成功連線


## 部署 Gitlab agent

* 進入剛建立的專案，點選 Operate > Kubernetes clusters


* 在 UI 註冊並取得 Token


* 按照步驟在 k8s 叢集安裝 gitlab agent


* 匯出 gitlab-kas-tls 使用的憑證

$ kubectl -n gitlab-system get secret gitlab-kas-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt

$ helm repo add gitlab https://charts.gitlab.io
$ helm repo update
$ helm upgrade --install my-k8s-agent gitlab/gitlab-agent \
    --namespace gitlab-agent-my-k8s-agent \
    --create-namespace \
    --set image.tag=v18.7.0 \
    --set config.token=glagent-F-5ScIF0ohJaOjg1mgfsNm86MQpwOjEH.01.0w0mh1w6v \
    --set config.kasAddress=wss://kas.example.com \
    --set-file config.kasCaCert=ca.crt   # 新增此行

$ kubectl -n gitlab-agent-my-k8s-agent get pod
NAME                                            READY   STATUS    RESTARTS   AGE
my-k8s-agent-gitlab-agent-v2-5b59d8d685-8ngnh   1/1     Running   0          23s
my-k8s-agent-gitlab-agent-v2-5b59d8d685-kt57b   1/1     Running   0          23s

* 部署後確認已連線


### 建立 Agent 設定檔


* 輸入 `.gitlab/agents/my-k8s-agent/config.yaml` 名稱，這邊要根據 agent 名稱變動


* 這個設定檔主要是允許 `root/mytest` 這個專案的 Pipeline，用他的身份來操作 K8s。
* 如果沒有設定 `ci_access` 這一段，在 `.gitlab-ci.yml` 裡面打 `kubectl config use-context ...` 是連不通的。

ci_access:      # 開啟 CI/CD 通道 (Push-based)
  projects:
    - id: "root/mytest"   # 只有這個 GitLab 專案底下的 Pipeline 有資格連線
      access_as:
        agent: {}         # 使用 Agent 自己的 ServiceAccount 權限，就是 my-k8s-agent-gitlab-agent-v2-xxx 這個 pod

* 建立 `.gitlab-ci.yml`，這個設定檔主要是在定義整個 CI/CD 的 pipelines 流程。
* 這邊測試一下是否可以連到 k8s 叢集。

# .gitlab-ci.yml

stages:
  - deploy

deploy-k8s:
  stage: deploy
  # 使用包含 kubectl 的 image
  image: bitnami/kubectl:latest
  
  script:
    # 1. 切換 Context
    # 格式：<您的專案路徑>:<您的 agent 名稱>
    # 範例：root/mytest:my-k8s-agent
    - kubectl config use-context root/mytest:my-k8s-agent
    
    # 2. 測試連線 (如果這行能列出 Node，就代表通了！)
    - kubectl get nodes

* commit 後他會自動部署 job 到 k8s 裡，然後可以查看 pod 的 log，名稱是 `runner-xxxx-xxxx` 開頭。

$ kubectl -n gitlab-system logs runner-wffb6mdps-project-1-concurrent-0-g9w0p2co
Defaulted container "build" out of: build, helper, init-permissions (init)
{"script": "/scripts-1-3/step_script"}
$ kubectl config use-context root/mytest:my-k8s-agent
Switched to context "root/mytest:my-k8s-agent".
$ kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
crio-m1   Ready    control-plane   80d   v1.32.9
m2        Ready    control-plane   80d   v1.32.9
m3        Ready    control-plane   80d   v1.32.9
w1        Ready    <none>          80d   v1.32.9

{"command_exit_code": 0, "script": "/scripts-1-3/step_script"}

* 在 pipelines 可以看到部署狀況


## 透過 gitlab 建立應用

* 建立 `manifests` 目錄


* 在 `manifests` 目錄下產生 yaml


# manifests/nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

* 編輯 `.gitlab-ci.yml`

# .gitlab-ci.yml

stages:
  - deploy
deploy-k8s:
  stage: deploy
  image: bitnami/kubectl:latest
  
  script:
    - kubectl config use-context root/mytest:my-k8s-agent
      # 部署 manifests 目錄下的 yaml
    - kubectl apply -f manifests/nginx.yaml

* 在 k8s 檢查應用已部署

$ kubectl get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/nginx-demo-96b9d695-2bf6d   1/1     Running   0          13s
pod/nginx-demo-96b9d695-6rjv2   1/1     Running   0          13s

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP   80d
service/nginx-service   ClusterIP   10.96.26.182   <none>        80/TCP    13s

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-demo   2/2     2            2           13s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-demo-96b9d695   2         2         2       13s

* 測試修改 yaml 後，commit 後就會自動部署到 k8s


$ kubectl get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/nginx-demo-96b9d695-kg7h9   1/1     Running   0          117s

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP   80d
service/nginx-service   ClusterIP   10.96.26.182   <none>        80/TCP    27m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-demo   1/1     1            1           117s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-demo-96b9d695   1         1         1       117s

## 參考
https://docs.gitlab.com/operator/installation/

https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/main/values.yaml