## Helm Install
[官網教學](https://goharbor.io/docs/1.10/install-config/harbor-ha-helm/)
### Environment
1. Ingress Controller
[NGINX Ingress Controller](https://platform9.com/learn/v1.0/tutorials/nginix-controller-via-yaml)
2. PostgreSQL
[PostgreSQL](https://adamtheautomator.com/postgres-to-kubernetes/)
[PV Setting](https://stackoverflow.com/questions/63283477/helm-postgres-cannot-create-directory)

    helm install postgresql -f values.yaml bitnami/postgresql --set volumePermissions.enabled=true

3. Redis
4. K8s
5. Helm

    $ sudo wget https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz
    $ sudo tar -zxvf helm-v3.12.0-linux-amd64.tar.gz
    $ sudo mv linux-amd64/helm /usr/local/bin/helm
    $ helm version

### Install

1. 取得部屬Harbor的values.yaml

$ helm repo add harbor  https://helm.goharbor.io
$ helm show values harbor/harbor --version 1.11.1 >> values.yaml

2. 建立Namespace

kubectl create ns harbor

3. 修改values.yaml的設定
- expose
- type: nodePort
- tls.auto.commonName: "" 改成自己的domain
- nodePort.ports.https.modePort: 30003 使用port號
- externalURL: http://10.0.2.15:30003 外部連接的portURL
- harborAdminPassword: Harbor12345 預設Harbor密碼
- nginx.nodeSelector: {kubernetes.io/hostname: name} 指定使用的node

4. 設定完成後helm install

helm install harbor -f value.yaml -n harbor harbor/harbor --version 1.11.1

`externalURL`：Harbor的URL地址，用于访问Harbor。
`expose.type：用于配置Harbor的服务类型，如ClusterIP、NodePort等。
`tls.enabled`：是否启用TLS加密通信。
`persistence.enabled`：是否启用持久化存储。
`database.type`：Harbor使用的数据库类型，支持的选项有mysql和postgresql。
`redis.enabled`：是否启用Redis，用于缓存和存储Harbor的元数据。
`jobservice.maxJobWorkers`：Job Service的最大工作线程数。

## Docker
### Install
#### 先備條件
- Docker Engine 23.0.4_用以部屬docker image

$ curl -fsSL "https://get.docker.com/" | sh
$ sudo usermod -aG docker $USER

- Docker Compose 1.24.1_透過指令關閉重啟docker container

$ sudo curl -L https://github.com/docker/compose/releases/download/1.24.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

- Openssl 3.0.2_用來生成憑證

#### 下載安裝包版本
下載v2.7.2，解壓縮
https://goharbor.io/docs/2.7.0/

$ wget https://github.com/goharbor/harbor/releases/download/v2.7.2/harbor-online-installer-v2.7.2.tgz
$ tar xzvf harbor-online-installer-v2.7.2.tgz

#### 設定HTTPS Access
> 因為Harbor需要足夠的安全性，所以需要HTTPS的協定來規範，這邊會透過OpenSSL以自簽名的憑證來申請。除了創建CA機構證書之外，還需要server端與client端的配置以滿足HTTPS的規範。

- Generate a Certificate Authority Certificate
1. Generate a CA certificate private key.

    $ openssl genrsa -out ca.key 4096

2. Generate the CA certificate.

    $ openssl req -x509 -new -nodes -sha512 -days 3650 \
     -subj "/C=TW/ST=Taipei/L=Taipei/O=AA/OU=Personal/CN=hub.harbor.com" \
     -key ca.key \
     -out ca.crt

- Generate a Server Certificate
1. Generate a private key.

    $ openssl genrsa -out hub.harbor.com.key 4096

2. Generate a certificate signing request (CSR).

    $ openssl req -sha512 -new \
        -subj "/C=TW/ST=Taipei/L=Taipei/O=AA/OU=Personal/CN=hub.harbor.com" \
        -key hub.harbor.com.key \
        -out hub.harbor.com.csr

3. Generate an x509 v3 extension file.

    $ cat > v3.ext <<-EOF
    authorityKeyIdentifier=keyid,issuer
    basicConstraints=CA:FALSE
    keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names

    [alt_names]
    DNS.1=hub.harbor.com
    DNS.2=hub.harbor.com
    EOF

4. Use the v3.ext file to generate a certificate for your Harbor host.

    $ openssl x509 -req -sha512 -days 3650 \
        -extfile v3.ext \
        -CA ca.crt -CAkey ca.key -CAcreateserial \
        -in hub.harbor.com.csr \
        -out hub.harbor.com.crt

- Provide the Certificates to Harbor and Docker
1. Copy the server certificate and key into the certficates folder on your Harbor host.

    $ sudo mkdir -p /data/cert/
    $ sudo cp hub.harbor.com.crt /data/cert/
    $ sudo cp hub.harbor.com.key /data/cert/

2. Convert hub.harbor.com.crt to hub.harbor.com.cert, for use by Docker.

    $ openssl x509 -inform PEM -in hub.harbor.com.crt -out hub.harbor.com.cert

3. Copy the server certificate, key and CA files into the Docker certificates folder on the Harbor host. You must create the appropriate folders first.

    $ mkdir -p /etc/docker/certs.d/hub.harbor.com/
    $ cp hub.harbor.com.cert /etc/docker/certs.d/hub.harbor.com/
    $ cp hub.harbor.com.key /etc/docker/certs.d/hub.harbor.com/
    $ cp ca.crt /etc/docker/certs.d/hub.harbor.com/

4. Restart Docker Engine.

    $ systemctl restart docker

### 設定部屬資料
- 將harbor.yml.tem存成harbor.yml

- 更改設定

- hostman: hub.harbor.com
- // http
- https
- certificate: /data/cert/hub.harbor.com.crt
- private_key: /data/cert/hub.harbor.com.key
- harbor_admin_password: Harbor12345
- Windows
- 透過工具mmc將ca.crt放進受信任列表
- 在管理單位的local端新增
- 選擇受信任的證書機構新增
- 匯入Harbor端透過OpenSSL生成的ca.crt
- 重啟會使用到證書的程式
- Linux
- 在信任列表中新增Harbor端生成的證書
- 在Client目錄下/usr/local/share/ca-certificates/去新增Harbor端透過OpenSSL生成的ca.crt
- 透過以下指令更新信任列表

            $ sudo update-ca-certificates

### Installation

$ sudo ./install.sh

### Operation
- Stop Harbor

$ sudo docker-compose down -v

- Restart Harbor

$ sudo docker-compose up -d

- Check Status

$ sudo docker-compose ps

### Operate Harbor via Termial
#### Login

# 設定接受不安全連線 /etc/docker/daemon.json
{
    "insecure-registries": ["hub.harbor.com"]
}

$ docker login hub.harbor.com

-------------
Username: admin
Password:
Login Succeeded
-------------

#### Images
- Push Images

# Example
$ docker pull nginx:1.24.0

# 新增tag到image上
$ docker tag <source_image>:<version> hub.harbor.com/<project>/<source_image>:<version>
$ docker tag nginx:1.24.0 hub.harbor.com/test/nginx:1.24.0

# push to harbor
$ docker push hub.harbor.com/test/nginx:1.24.0

- Pull Images

# pull image from harbor

# version 顯示 none
$ docker pull hub.harbor.com/test/xxx@sha256:f3c37d8a26f7a7d8a547470c58733f270bcccb7e785da17af81ec41576170da8
# version 顯示 1.08
$ docker pull hub.qsaharborn.com/test/xxx:1.08

### 參考
[IT 1](https://ithelp.ithome.com.tw/articles/10218046)
[IT 2](https://ithelp.ithome.com.tw/articles/10223099)
[Download the Harbor Installer官網](https://goharbor.io/docs/2.0.0/install-config/download-installer/)
[在Linux信任自訂CA憑證-放在extra外面](https://blog.darkthread.net/blog/install-caroot-cert-in-linux/)
[Test an insecure registry-允許不安全](https://docs.docker.com/registry/insecure/)

### 訪問Port方式
1. 防火牆開啟指定port
- 在防火牆進階設定加入輸入規則

2. port forward local to LAN

$ netsh interface portproxy add v4tov4 listenport=8888 connectaddress=192.168.56.1 connectport=8089



架設harbor
https://ithelp.ithome.com.tw/articles/10223099
https://goharbor.io/docs/1.10/install-config/download-installer/
接受不安全連線設定
https://ithelp.ithome.com.tw/articles/10218046

問題還解決不了
https://github.com/goharbor/harbor/issues/9160

## Harbor Image Error
### QA沒辦法執行Harbor's Image as Pod
遇到問題應該要怎麼按照順序做檢查
EX: K8s Pull Image Error

Failed to pull image "hub.harbor.com/test/qqq:1.2.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://hub.harbor.com/v2/": dial tcp: lookup hub.harbor.com: no such host

檢查流程
- 從我上傳在Harbor的image先試
- ping hub.harbor.com
- docker pull

# kubectl get ns
The connection to the server localhost:8080 was refused - did you specify the right host or port?
- swapoff -a
- .kube/config

最後問題是worker node放在別的主機上，那台並沒有設定DNS

ERROR
1. k8s taint error

error
default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

...
  tolerations:
  - key: node-role.kubernetes.io/master
    operator: Exists
    effect: NoSchedule
...

2. k8s net error

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fbbc062efac5259e52a3d1abb188ac3c2174acda0907f1f7834eea2c5119a786" network for pod "test": networkPlugin cni failed to set up pod "test_default" network: loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

封包檢查工具
tcpdump
wireshark(tshark)

下載憑證

# get intermediate
wget https://certs.starfieldtech.com/repository/sfig2.crt.pem
# rename
mv sfig2.crt.pem sfig2.crt
# copy to certificate file
sudo cp sfig2.crt /usr/local/share/ca-certificates/
# refresh
sudo update-ca-certificates