# 在 k8s 安裝 harbor，並透過 cert-manager 自簽憑證透過 istio ingressgateway 存取

## 環境準備

1. 已安裝好 istio 和 cert-manager

$ kubectl -n istio-system get pod
NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-5kmfh                    1/1     Running   0          19m
istio-cni-node-975fn                    1/1     Running   0          19m
istio-cni-node-9jqsq                    1/1     Running   0          19m
istio-cni-node-9n8s6                    1/1     Running   0          19m
istio-cni-node-d5d4z                    1/1     Running   0          19m
istio-ingressgateway-5b6548c6d6-n6x64   1/1     Running   0          11m
istiod-d498fb64c-d9w5x                  1/1     Running   0          19m
jaeger-76444674f-s2nqj                  1/1     Running   0          10m
kiali-6d774d8bb8-6n6zj                  1/1     Running   0          16m
ztunnel-dc56s                           1/1     Running   0          18m
ztunnel-f4t5s                           1/1     Running   0          18m
ztunnel-mfxxn                           1/1     Running   0          18m
ztunnel-vrbs2                           1/1     Running   0          18m
ztunnel-w4xkp                           1/1     Running   0          18m

$ kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-58dd99f969-w2tz2              1/1     Running   0          20m
cert-manager-cainjector-55cd9f77b5-5lkh8   1/1     Running   0          20m
cert-manager-webhook-7987476d56-j8r8z      1/1     Running   0          20m

2. 已安裝好 csi driver

$ kubectl get sc
NAME      PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi   nfs.csi.k8s.io   Retain          Immediate           false                  4s

## 建立 harbor namespace

* 在 `harbor` namespace 設定 `istio.io/dataplane-mode=ambient` label

$ kubectl create ns harbor

$ kubectl label namespace harbor istio.io/dataplane-mode=ambient

## 使用 cert-manager 自簽憑證
### 1. 先建立 SelfSigned Issuer → 再用它簽一個 CA → 用 CA 建一個 ClusterIssuer

$ nano cert.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: root-ca
  namespace: cert-manager
spec:
  isCA: true
  duration: 87600h            # 10 years
  renewBefore: 720h           # 30d
  commonName: root-ca
  secretName: root-ca-secret
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: istio-ca-clusterissuer
spec:
  ca:
    secretName: root-ca-secret

* 部署以及檢查資源

$ kubectl apply -f cert.yaml

$ kubectl -n cert-manager get Issuer
NAME                READY   AGE
selfsigned-issuer   True    32s

$ kubectl -n cert-manager get Certificate
NAME      READY   SECRET           AGE
root-ca   True    root-ca-secret   8s

$ kubectl get ClusterIssuer
NAME                     READY   AGE
istio-ca-clusterissuer   True    13s

$ kubectl -n cert-manager get secret root-ca-secret
NAME             TYPE                DATA   AGE
root-ca-secret   kubernetes.io/tls   3      32s

### 2. 建立 Harbor 憑證
* 使用 `istio-ca-clusterissuer` 這個 ClusterIssuer 來簽發憑證，憑證簽發後會把 公鑰和私鑰放在 `nginx-cert-tls` secret。
* `dnsNames:` 就是 憑證的 主體名稱（SANs），也就是 Harbor 服務對外的 domain，也需要將名稱解析到 istio ingressgateway 的 svc ip。

$ nano harbor-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-cert
  namespace: istio-system
spec:
  duration: 8760h             # 1 year
  renewBefore: 720h           # 30d
  secretName: harbor-cert-tls
  dnsNames:
    - "*.myk8s.com"
  issuerRef:
    name: istio-ca-clusterissuer
    kind: ClusterIssuer

$ kubectl apply -f harbor-cert.yaml

$ kubectl -n istio-system get Certificate harbor-cert
NAME          READY   SECRET            AGE
harbor-cert   True    harbor-cert-tls   24s

$ kubectl -n istio-system get secret harbor-cert-tls
NAME              TYPE                DATA   AGE
harbor-cert-tls   kubernetes.io/tls   3      32

## 安裝 harbor

$ helm repo add harbor https://helm.goharbor.io

$ helm search repo harbor/harbor --versions

# 指定 helm chart 版本
$ helm fetch harbor/harbor --version 1.17.1 --untar

* 編輯 harbor chart 的 values.yaml
- `expose.type` 修改為 clusterIP。
- `expose.type.tls.enabled` 修改為 false。
- `externalURL` 設定 harbor 對外的 domain。
- 在 `persistence` 以下都需設定自己的 storageclass。

$ nano harbor/values.yaml
expose:
  # Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
  # and fill the information in the corresponding section
  type: clusterIP
......
  tls:
    enabled: false
......
externalURL: https://harbor.myk8s.com
......
persistence:
  enabled: true
  # Setting it to "keep" to avoid removing PVCs during a helm delete
  # operation. Leaving it empty will delete PVCs after the chart deleted
  # (this does not apply for PVCs that are created for internal database
  # and redis components, i.e. they are never deleted automatically)
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound,
      # and specify the "subPath" if the PVC is shared with other components
      existingClaim: ""
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used(the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: "nfs-csi"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 300Gi
    chartmuseum:
      existingClaim: ""
      storageClass: "nfs-csi"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    jobservice:
      existingClaim: ""
      storageClass: "nfs-csi"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    # If external database is used, the following settings for database will
    # be ignored
    database:
      existingClaim: ""
      storageClass: "nfs-csi"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    # If external Redis is used, the following settings for Redis will
    # be ignored
    redis:
      existingClaim: ""
      storageClass: "nfs-csi"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    trivy:
      existingClaim: ""
      storageClass: "nfs-csi"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi

$ helm install harbor harbor/ -n harbor

$ kubectl -n harbor get pod
NAME                                 READY   STATUS    RESTARTS      AGE
harbor-core-848b4d75dd-9xskw         1/1     Running   3 (97s ago)   2m4s
harbor-database-0                    1/1     Running   0             2m4s
harbor-jobservice-778d59bd67-lpdjq   1/1     Running   0             28s
harbor-nginx-8549449f48-9fgsz        1/1     Running   0             2m4s
harbor-portal-5b6b5f7494-9n7pd       1/1     Running   0             2m4s
harbor-redis-0                       1/1     Running   0             2m4s
harbor-registry-584776c6dd-kslvc     2/2     Running   0             2m4s
harbor-trivy-0                       1/1     Running   0             2m4s

## 設定 istio
### 建立 Istio Gateway 與 VirtualService
* 設定 Istio 的流量路由，目標是讓外部流量透過 HTTPS 進入 Istio Ingress Gateway，然後轉發到內部 harbor 的 `harbor-portal` Service。

$ nano harbor-istio.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: harbor-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: harbor-cert-tls  # 對應憑證 Secret
    hosts:
    - harbor.myk8s.com
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: harbor-vs
  namespace: istio-system
spec:
  hosts:
  - harbor.myk8s.com
  gateways:
  - harbor-gateway
  http:
  - match:
    - uri:
        prefix: /api/
    route:
    - destination:
        host: harbor-core.harbor.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /service/
    route:
    - destination:
        host: harbor-core.harbor.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /v2/
    route:
    - destination:
        host: harbor-core.harbor.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /c/
    route:
    - destination:
        host: harbor-core.harbor.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: harbor-portal.harbor.svc.cluster.local
        port:
          number: 80


$ kubectl apply -f harbor-istio.yaml

$ kubectl -n istio-system get gateway.networking
NAME             AGE
harbor-gateway   14s

$ kubectl -n istio-system get virtualservice
NAME        GATEWAYS             HOSTS                  AGE
harbor-vs   ["harbor-gateway"]   ["harbor.myk8s.com"]   30s

## 將 ca.pem 匯入到每一個 client 上

* 將 `ca.pem` 匯入到瀏覽器

$ kubectl -n cert-manager get secret root-ca-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.pem

* ssh 進入到 client 並且把 ca.pem 複製到 OS

$ sudo cp ca.pem /usr/local/share/ca-certificates/harbor.crt

$ sudo update-ca-certificates

# 把憑證塞入 docker 就可以使用 ca login
$ sudo mkdir /etc/docker/certs.d

$ sudo cp ca.pem /etc/docker/certs.d/

$ sudo systemctl daemon-reload

$ sudo systemctl restart docker