---
# System prepended metadata

title: Harbor
tags: [container]

---


## Helm Install
[官網教學](https://goharbor.io/docs/1.10/install-config/harbor-ha-helm/)
### Environment
1. Ingress Controller 
    [NGINX Ingress Controller](https://platform9.com/learn/v1.0/tutorials/nginix-controller-via-yaml)
2. PostgreSQL
    [PostgreSQL](https://adamtheautomator.com/postgres-to-kubernetes/)
    [PV Setting](https://stackoverflow.com/questions/63283477/helm-postgres-cannot-create-directory)
    ```
    helm install postgresql -f values.yaml bitnami/postgresql --set volumePermissions.enabled=true
    ```
3. Redis
4. K8s
5. Helm
    ```
    $ sudo wget https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz
    $ sudo tar -zxvf helm-v3.12.0-linux-amd64.tar.gz
    $ sudo mv linux-amd64/helm /usr/local/bin/helm
    $ helm version
    ```

### Install

1. 取得部屬Harbor的values.yaml
```
$ helm repo add harbor  https://helm.goharbor.io
$ helm show values harbor/harbor --version 1.11.1 >> values.yaml
```
2. 建立Namespace
```
kubectl create ns harbor
```
3. 修改values.yaml的設定
- expose
    - type: nodePort
    - tls.auto.commonName: "" 改成自己的domain
    - nodePort.ports.https.modePort: 30003 使用port號
- externalURL: http://10.0.2.15:30003 外部連接的portURL
- harborAdminPassword: Harbor12345 預設Harbor密碼
- nginx.nodeSelector: {kubernetes.io/hostname: name} 指定使用的node

4. 設定完成後helm install
```
helm install harbor -f value.yaml -n harbor harbor/harbor --version 1.11.1
```
```
`externalURL`：Harbor的URL地址，用于访问Harbor。
`expose.type：用于配置Harbor的服务类型，如ClusterIP、NodePort等。
`tls.enabled`：是否启用TLS加密通信。
`persistence.enabled`：是否启用持久化存储。
`database.type`：Harbor使用的数据库类型，支持的选项有mysql和postgresql。
`redis.enabled`：是否启用Redis，用于缓存和存储Harbor的元数据。
`jobservice.maxJobWorkers`：Job Service的最大工作线程数。
```

## Docker
### Install
#### 先備條件
- Docker Engine 23.0.4_用以部屬docker image
```
$ curl -fsSL "https://get.docker.com/" | sh
$ sudo usermod -aG docker $USER
```
- Docker Compose 1.24.1_透過指令關閉重啟docker container
```
$ sudo curl -L https://github.com/docker/compose/releases/download/1.24.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
```
- Openssl 3.0.2_用來生成憑證

#### 下載安裝包版本
下載v2.7.2，解壓縮
https://goharbor.io/docs/2.7.0/
```
$ wget https://github.com/goharbor/harbor/releases/download/v2.7.2/harbor-online-installer-v2.7.2.tgz
$ tar xzvf harbor-online-installer-v2.7.2.tgz
```

#### 設定HTTPS Access
> 因為Harbor需要足夠的安全性，所以需要HTTPS的協定來規範，這邊會透過OpenSSL以自簽名的憑證來申請。除了創建CA機構證書之外，還需要server端與client端的配置以滿足HTTPS的規範。

- Generate a Certificate Authority Certificate
    1. Generate a CA certificate private key.
    ```
    $ openssl genrsa -out ca.key 4096
    ```
    2. Generate the CA certificate.
    ```
    $ openssl req -x509 -new -nodes -sha512 -days 3650 \
     -subj "/C=TW/ST=Taipei/L=Taipei/O=AA/OU=Personal/CN=hub.harbor.com" \
     -key ca.key \
     -out ca.crt
    ```

- Generate a Server Certificate
    1. Generate a private key.
    ```
    $ openssl genrsa -out hub.harbor.com.key 4096
    ```
    2. Generate a certificate signing request (CSR).
    ```
    $ openssl req -sha512 -new \
        -subj "/C=TW/ST=Taipei/L=Taipei/O=AA/OU=Personal/CN=hub.harbor.com" \
        -key hub.harbor.com.key \
        -out hub.harbor.com.csr
    ```
    3. Generate an x509 v3 extension file.
    ```
    $ cat > v3.ext <<-EOF
    authorityKeyIdentifier=keyid,issuer
    basicConstraints=CA:FALSE
    keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names

    [alt_names]
    DNS.1=hub.harbor.com
    DNS.2=hub.harbor.com
    EOF
    ```

    4. Use the v3.ext file to generate a certificate for your Harbor host.
    ```
    $ openssl x509 -req -sha512 -days 3650 \
        -extfile v3.ext \
        -CA ca.crt -CAkey ca.key -CAcreateserial \
        -in hub.harbor.com.csr \
        -out hub.harbor.com.crt
    ```
- Provide the Certificates to Harbor and Docker
    1. Copy the server certificate and key into the certficates folder on your Harbor host.
    ```
    $ sudo mkdir -p /data/cert/
    $ sudo cp hub.harbor.com.crt /data/cert/
    $ sudo cp hub.harbor.com.key /data/cert/
    ```
    2. Convert hub.harbor.com.crt to hub.harbor.com.cert, for use by Docker.
    ```
    $ openssl x509 -inform PEM -in hub.harbor.com.crt -out hub.harbor.com.cert
    ```
    3. Copy the server certificate, key and CA files into the Docker certificates folder on the Harbor host. You must create the appropriate folders first.
    ```
    $ mkdir -p /etc/docker/certs.d/hub.harbor.com/
    $ cp hub.harbor.com.cert /etc/docker/certs.d/hub.harbor.com/
    $ cp hub.harbor.com.key /etc/docker/certs.d/hub.harbor.com/
    $ cp ca.crt /etc/docker/certs.d/hub.harbor.com/
    ```
    4. Restart Docker Engine.
    ```
    $ systemctl restart docker
    ```
### 設定部屬資料
- 將harbor.yml.tem存成harbor.yml

- 更改設定

    - hostman: hub.harbor.com
    - // http
    - https
        - certificate: /data/cert/hub.harbor.com.crt
        - private_key: /data/cert/hub.harbor.com.key
    - harbor_admin_password: Harbor12345
    - Windows
        - 透過工具mmc將ca.crt放進受信任列表
        - 在管理單位的local端新增
        - 選擇受信任的證書機構新增
        - 匯入Harbor端透過OpenSSL生成的ca.crt
        - 重啟會使用到證書的程式
    - Linux
        - 在信任列表中新增Harbor端生成的證書
        - 在Client目錄下/usr/local/share/ca-certificates/去新增Harbor端透過OpenSSL生成的ca.crt
        - 透過以下指令更新信任列表
            ```
            $ sudo update-ca-certificates
            ```
### Installation
```
$ sudo ./install.sh
```
### Operation
- Stop Harbor
```
$ sudo docker-compose down -v
```
- Restart Harbor
```
$ sudo docker-compose up -d
```
- Check Status
```
$ sudo docker-compose ps
```
### Operate Harbor via Termial
#### Login
```
# 設定接受不安全連線 /etc/docker/daemon.json
{
    "insecure-registries": ["hub.harbor.com"]
}

$ docker login hub.harbor.com

-------------
Username: admin
Password:
Login Succeeded
-------------
```
#### Images
- Push Images
```
# Example
$ docker pull nginx:1.24.0

# 新增tag到image上
$ docker tag <source_image>:<version> hub.harbor.com/<project>/<source_image>:<version>
$ docker tag nginx:1.24.0 hub.harbor.com/test/nginx:1.24.0

# push to harbor
$ docker push hub.harbor.com/test/nginx:1.24.0
```
- Pull Images
```
# pull image from harbor

# version 顯示 none
$ docker pull hub.harbor.com/test/xxx@sha256:f3c37d8a26f7a7d8a547470c58733f270bcccb7e785da17af81ec41576170da8
# version 顯示 1.08
$ docker pull hub.qsaharborn.com/test/xxx:1.08
```
### 參考
[IT 1](https://ithelp.ithome.com.tw/articles/10218046)
[IT 2](https://ithelp.ithome.com.tw/articles/10223099)
[Download the Harbor Installer官網](https://goharbor.io/docs/2.0.0/install-config/download-installer/)
[在Linux信任自訂CA憑證-放在extra外面](https://blog.darkthread.net/blog/install-caroot-cert-in-linux/)
[Test an insecure registry-允許不安全](https://docs.docker.com/registry/insecure/)


### 訪問Port方式
1. 防火牆開啟指定port
- 在防火牆進階設定加入輸入規則

2. port forward local to LAN
```
$ netsh interface portproxy add v4tov4 listenport=8888 connectaddress=192.168.56.1 connectport=8089
```
![](https://i.imgur.com/xMdxkog.png)


![](https://i.imgur.com/RdAB4Ix.png)

架設harbor
https://ithelp.ithome.com.tw/articles/10223099
https://goharbor.io/docs/1.10/install-config/download-installer/
接受不安全連線設定
https://ithelp.ithome.com.tw/articles/10218046

問題還解決不了
https://github.com/goharbor/harbor/issues/9160

## Harbor Image Error
### QA沒辦法執行Harbor's Image as Pod
遇到問題應該要怎麼按照順序做檢查
EX: K8s Pull Image Error
```
Failed to pull image "hub.harbor.com/test/qqq:1.2.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://hub.harbor.com/v2/": dial tcp: lookup hub.harbor.com: no such host
```

檢查流程
- 從我上傳在Harbor的image先試
- ping hub.harbor.com
- docker pull

```
# kubectl get ns
The connection to the server localhost:8080 was refused - did you specify the right host or port?
- swapoff -a
- .kube/config
```
最後問題是worker node放在別的主機上，那台並沒有設定DNS

ERROR
1. k8s taint error
```
error
default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
```
```
...
  tolerations:
  - key: node-role.kubernetes.io/master
    operator: Exists
    effect: NoSchedule
...
```
2. k8s net error
```
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fbbc062efac5259e52a3d1abb188ac3c2174acda0907f1f7834eea2c5119a786" network for pod "test": networkPlugin cni failed to set up pod "test_default" network: loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
```
```
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```

封包檢查工具
tcpdump
wireshark(tshark)

下載憑證
```
# get intermediate
wget https://certs.starfieldtech.com/repository/sfig2.crt.pem
# rename
mv sfig2.crt.pem sfig2.crt
# copy to certificate file
sudo cp sfig2.crt /usr/local/share/ca-certificates/
# refresh
sudo update-ca-certificates
```
![](https://hackmd.io/_uploads/rklMFB7Hh.png)