# Install Harbor using Podman
## 先決條件
- [x] 已安裝 `podman`,版本 4.9.x 或以上
- [x] 已安裝 `curl`, `wget`, `jq`, `apache2-utils` 工具
## Setp1: 建立 podman network
```
sudo podman network create --subnet 10.8.0.0/16 harbor
```
> 若不想建立網路,可以使用預設 podman 網路,但需要將 dns 解析打開
> 預設網路目前版本沒有開啟 dns 解析,要不然容器之間沒法透過主機名稱進行互訪。
> 修改設定檔後需要將網路介面刪除或重新啟動主機才能生效。
## Step2: 下載 Harbor 離線安裝所需的檔案
1. 下載官方發布的離線版本
    ```
    HARBOR_VERSION=$(curl -s https://api.github.com/repos/goharbor/harbor/releases/latest | jq -r .tag_name)
    wget https://github.com/goharbor/harbor/releases/download/${HARBOR_VERSION}/harbor-offline-installer-${HARBOR_VERSION}.tgz
    ```
    > 第一行命令會直接到 github 抓 Harbor 官方最新 release 的版本,如果要安裝其他版本,請自行定義
2. 解壓縮檔案,並切換工作目錄
    ```
    tar zxf harbor-offline-installer-${HARBOR_VERSION}.tgz; cd harbor
    ```
3. 匯入 image
    ```
    sudo podman load -i harbor.${HARBOR_VERSION}.tar.gz
    ```
4. 檢查是否成功
    ```
    sudo podman images
    ```
    
    執行結果如下 : 
    
    ```
    REPOSITORY                               TAG         IMAGE ID      CREATED       SIZE
    localhost/goharbor/harbor-exporter       v2.13.1     8323e56fa034  2 months ago  130 MB
    localhost/goharbor/redis-photon          v2.13.1     27079bef6812  2 months ago  168 MB
    localhost/goharbor/trivy-adapter-photon  v2.13.1     9d2de710e1bc  2 months ago  388 MB
    localhost/goharbor/harbor-registryctl    v2.13.1     1ca7f7dffcb8  2 months ago  163 MB
    localhost/goharbor/registry-photon       v2.13.1     251eb949b8fc  2 months ago  86.8 MB
    localhost/goharbor/nginx-photon          v2.13.1     3a0ac2771512  2 months ago  153 MB
    localhost/goharbor/harbor-log            v2.13.1     49f7cdb104f3  2 months ago  166 MB
    localhost/goharbor/harbor-jobservice     v2.13.1     b964386ce624  2 months ago  176 MB
    localhost/goharbor/harbor-core           v2.13.1     701038c9f9cf  2 months ago  200 MB
    localhost/goharbor/harbor-portal         v2.13.1     254c145df624  2 months ago  162 MB
    localhost/goharbor/harbor-db             v2.13.1     8645cd204f13  2 months ago  278 MB
    localhost/goharbor/prepare               v2.13.1     eeb5b545352d  2 months ago  214 MB
    ```
## Step3: 部署 Harbor 各元件
注意,Harbor 各別的元件相互依賴,需要依照順序部署,否則元件無法啟動
### 3.1. 部署 redis
1. 建立 redis container
    ```
    # 定義 Redis 資料永存目錄區
    REDIS_PV="/data/redis"
    [ ! -d ${REDIS_PV} ] && sudo mkdir -p ${REDIS_PV}
    sudo chown 999:999 ${REDIS_PV} 
    
    sudo podman run --name redis --detach \
        --network=harbor \
        --cap-add=chown --cap-add=setuid \
        --cap-add=setgid --cap-drop=all \
        --volume=${REDIS_PV}:/var/lib/redis:z \
        goharbor/redis-photon:${HARBOR_VERSION}
    ```
### 3.2. 部署 postgresql
1. 建立 secrt:資料庫 root 使用者的密碼,這裡使用隨機密碼
    ```
    tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 16 | sudo podman secret create db-secret -
    ```
2. 建立 pg container
    ```
    # 定義 Redis 資料永存目錄區
    PG_PV="/data/database"
    [ ! -d ${PG_PV} ] && sudo mkdir -p ${PG_PV}
    sudo chown -R 999:999 ${PG_PV}
    
    sudo podman run --name postgresql \
        --detach --network=harbor \
        --cap-add=chown --cap-add=setuid \
        --cap-add=setgid --cap-add=dac_override \
        --cap-drop=all --shm-size=1gb \
        --secret=db-secret,type=env,target=POSTGRES_PASSWORD \
        --volume=${PG_PV}:/var/lib/postgresql/data:z \
        goharbor/harbor-db:${HARBOR_VERSION}
    ```
### 3.3. 部署 Registry
1. 編輯 registry 設定檔
    ```
    sudo mkdir -p /etc/harbor
    sudo nano /etc/harbor/registry.yml
    ```
    檔案內容如下 : 
    ```
    version: 0.1
    log:
      level: info
      fields:
        service: registry
    storage:
      cache:
        layerinfo: redis
      filesystem:
        rootdirectory: /storage
      maintenance:
        uploadpurging:
          enabled: true
          age: 168h
          interval: 24h
          dryrun: false
      delete:
        enabled: true
    redis:
      addr: redis:6379
      readtimeout: 10s
      writetimeout: 10s
      dialtimeout: 10s
      password:
      db: 1
      enableTLS: false
      pool:
        maxidle: 100
        maxactive: 500
        idletimeout: 60s
    http:
      addr: :5000
      secret: placeholder
      debug:
        addr: :9090
        prometheus:
          enabled: true
          path: /metrics
    auth:
      htpasswd:
        realm: harbor-registry-basic-realm
        path: /etc/registry/passwd
    validation:
      disabled: true
    compatibility:
      schema1:
        enabled: true
    ```
2. 建立密碼
    ```
    USERNAME="harbor_registry_user"
    PASSWORD="password"
    HASH=$(htpasswd -nbB -C 5 "$USERNAME" "$PASSWORD" | cut -d: -f2)
    echo "$USERNAME:$HASH" | sudo podman secret create registry-auth -
    ```
3. 啟動 registry container
    ```
    # 定義 REGISTRY 資料永存目錄區
    REGISTRY_PV="/data/registry"
    [ ! -d ${REGISTRY_PV} ] && sudo mkdir -p ${REGISTRY_PV}
    sudo chown 10000:10000 ${REGISTRY_PV}
    
    sudo podman run --name=registry \
        --detach --network=harbor \
        --cap-add=chown --cap-add=setuid \
        --cap-add=setgid --cap-drop=all \
        --secret=registry-auth,target=/etc/registry/passwd \
        --volume=${REGISTRY_PV}:/storage:z \
        --volume=/etc/harbor/registry.yml:/etc/registry/config.yml:ro \
        --volume=/etc/ssl/certs/ca-certificates.crt:/etc/registry/root.crt:ro \
        goharbor/registry-photon:${HARBOR_VERSION}
    ```
### 3.4. 部署 portal
1.  建立設定檔
    ```
    sudo nano /etc/harbor/portal.conf
    ```
    檔案內容如下 : 
    ```
    worker_processes auto;
    pid /tmp/nginx.pid;
    events {
        worker_connections  1024;
    }
    http {
        client_body_temp_path /tmp/client_body_temp;
        proxy_temp_path /tmp/proxy_temp;
        fastcgi_temp_path /tmp/fastcgi_temp;
        uwsgi_temp_path /tmp/uwsgi_temp;
        scgi_temp_path /tmp/scgi_temp;
        server {
            listen 8080;
            server_name  localhost;
            root   /usr/share/nginx/html;
            index  index.html index.htm;
            include /etc/nginx/mime.types;
            gzip on;
            gzip_min_length 1000;
            gzip_proxied expired no-cache no-store private auth;
            gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
            location /devcenter-api-2.0 {
                try_files $uri $uri/ /swagger-ui-index.html;
            }
            location / {
                try_files $uri $uri/ /index.html;
            }
            location = /index.html {
                add_header Cache-Control "no-store, no-cache, must-revalidate";
            }
        }
    }
    ```
2. 建立 portal container
    ```
    sudo podman run --name=portal \
        --detach --network=harbor \
        --cap-add=setuid --cap-add=setgid \
        --cap-add=net_bind_service --cap-drop=all \
        --volume=/etc/harbor/portal.conf:/etc/nginx/nginx.conf:ro \
        goharbor/harbor-portal:${HARBOR_VERSION}
    ```
### 3.5. 部署 registryctl
1. 編輯設定檔
    ```
    sudo nano /etc/harbor/registryctl.yml
    ```
    檔案內容如下 : 
    ```
    ---
    protocol: "http"
    port: 8080
    log_level: info
    registry_config: "/etc/registry/config.yml"
    ```
2. 建立密碼
    ```
    # 產生 harbor 私鑰: 用於建立 token
    openssl genrsa 4096 | openssl pkey -traditional | sudo podman secret create harbor-key -
    # 產生 harbor secretkey
    openssl rand -base64 12 | head -c 16| sudo podman secret create harbor-secretkey -
    # harbor 密碼:用於其他元件與 harbor 通訊認證
    openssl rand -base64 12 | head -c 16 | sudo podman secret create harbor-secret -
    # jobservice 密碼: 用於與 jobservice 通訊認證
    openssl rand -base64 12 | head -c 16 | sudo podman secret create jobservice-secret -
    
    # registry 密碼: 主要給 harbor 和 jobservice 使用
    printf "password" | sudo podman secret create registry-passwd -
    ```
    
3. 建立 registryctl container
    ```
    sudo podman run --name=registryctl \
        --detach --network=harbor \
        --cap-add=setuid --cap-add=setgid --cap-drop=all \
        --secret=harbor-secret,type=env,target=CORE_SECRET \
        --secret=jobservice-secret,type=env,target=JOBSERVICE_SECRET \
        --volume=${REGISTRY_PV}:/storage:z \
        --volume=/etc/harbor/registry.yml:/etc/registry/config.yml:ro \
        --volume=/etc/harbor/registryctl.yml:/etc/registryctl/config.yml:ro \
        --volume=/etc/ssl/certs/ca-certificates.crt:/etc/registry/root.crt:ro \
        goharbor/harbor-registryctl:${HARBOR_VERSION}
    ```
### 3.6. 部署 harbor
1. 編輯設定檔
    ```
    sudo nano /etc/harbor/harbor.conf
    ```
    檔案內容如下 : 
    ```
    appname = Harbor
    runmode = prod
    enablegzip = true
    [prod]
    httpport = 8080
    ```
2. 編輯 Harbor 運作所需之環境變數
    ```
    sudo nano /etc/harbor/coreenv
    ```
    檔案內容如下 : 
    ```
    CONFIG_PATH=/etc/core/app.conf
    UAA_CA_ROOT=/etc/core/certificates/uaa_ca.pem
    _REDIS_URL_CORE=redis://redis:6379?idle_timeout_seconds=30
    SYNC_QUOTA=true
    _REDIS_URL_REG=redis://redis:6379/1?idle_timeout_seconds=30
    LOG_LEVEL=info
    DATABASE_TYPE=postgresql
    POSTGRESQL_HOST=postgresql
    POSTGRESQL_PORT=5432
    POSTGRESQL_USERNAME=postgres
    POSTGRESQL_DATABASE=registry
    POSTGRESQL_SSLMODE=disable
    POSTGRESQL_MAX_IDLE_CONNS=100
    POSTGRESQL_MAX_OPEN_CONNS=900
    POSTGRESQL_CONN_MAX_LIFETIME=5m
    POSTGRESQL_CONN_MAX_IDLE_TIME=0
    REGISTRY_URL=http://registry:5000
    PORTAL_URL=http://portal:8080
    TOKEN_SERVICE_URL=http://core:8080/service/token
    HARBOR_ADMIN_PASSWORD=Harbor12345
    MAX_JOB_WORKERS=10
    WITH_TRIVY=True
    CORE_URL=http://core:8080
    CORE_LOCAL_URL=http://127.0.0.1:8080
    JOBSERVICE_URL=http://jobservice:8080
    TRIVY_ADAPTER_URL=http://trivy-adapter:8080
    REGISTRY_STORAGE_PROVIDER_NAME=filesystem
    READ_ONLY=false
    RELOAD_KEY=
    REGISTRY_CONTROLLER_URL=http://registryctl:8080
    REGISTRY_CREDENTIAL_USERNAME=harbor_registry_user
    CSRF_KEY=jPcpPYw2HU4V0r2LG2kyHWWSIwRE4Ln8
    ROBOT_SCANNER_NAME_PREFIX=2GA78XLs
    PERMITTED_REGISTRY_TYPES_FOR_PROXY_CACHE=docker-hub,harbor,azure-acr,ali-acr,aws-ecr,google-gcr,quay,docker-registry,github-ghcr,jfrog-artifactory
    HTTP_PROXY=
    HTTPS_PROXY=
    NO_PROXY=localhost,postgresql,portal,trivy-adapter,jobservice,127.0.0.1,.local,.internal,core,registryctl,redis,registry,nginx,db,log,exporter
    PORT=8080
    METRIC_ENABLE=true
    METRIC_PATH=/metrics
    METRIC_PORT=9090
    METRIC_NAMESPACE=harbor
    METRIC_SUBSYSTEM=core
    QUOTA_UPDATE_PROVIDER=db
    ```
4. 建立 Harbor container
    ```
    # 定義 Harbor 對外的 url
    HARBOR_FQDN="pharbor.example.com"
    
    # 定義 Harbor 資料永存目錄區
    HARBOR_PV="/data"
    [ ! -d ${HARBOR_PV} ] && sudo mkdir -p ${HARBOR_PV}
    sudo chown 10000:10000 ${HARBOR_PV}
    
    # Run Harbor Container 
    sudo podman run --name=core --detach \
        --network=harbor \
        --cap-add=setuid \
        --cap-add=setgid \
        --cap-drop=all \
        --env-file=/etc/harbor/coreenv \
        --env=EXT_ENDPOINT=https://${HARBOR_FQDN} \
        --secret=harbor-secret,type=env,target=CORE_SECRET \
        --secret=db-secret,type=env,target=POSTGRESQL_PASSWORD \
        --secret=jobservice-secret,type=env,target=JOBSERVICE_SECRET \
        --secret=registry-passwd,type=env,target=REGISTRY_CREDENTIAL_PASSWORD \
        --secret=harbor-secretkey,type=mount,target=/etc/core/key \
        --secret=harbor-key,type=mount,target=/etc/core/private_key.pem \
        --volume=${HARBOR_PV}:/data:Z \
        --volume=/etc/harbor/harbor.conf:/etc/core/app.conf:ro \
        --volume=/etc/ssl/certs/ca-certificates.crt:/etc/core/certificates/root.crt:ro \
        --volume=/usr/share/zoneinfo/Asia/Taipei:/etc/localtime:ro \
        goharbor/harbor-core:${HARBOR_VERSION}
    ```
### 3.7. 部署 jobservice
1. 編輯設定檔
    ```
    sudo nano /etc/harbor/jobservice.yml
    ```
    檔案內容如下 : 
    ```
    ---
    #Protocol used to serve
    protocol: "http"
    #Server listening port
    port: 8080
    #Worker pool
    worker_pool:
      #Worker concurrency
      workers: 10
      backend: "redis"
      #Additional config if use 'redis' backend
      redis_pool:
        #redis://[arbitrary_username:password@]ipaddress:port/database_index
        redis_url: redis://redis:6379/2?idle_timeout_seconds=30
        namespace: "harbor_job_service_namespace"
        idle_timeout_second: 3600
    #Loggers for the running job
    job_loggers:
      # The jobLoggers backend name, only support "STD_OUTPUT", "FILE" and/or "DB"
      - name: "STD_OUTPUT"
        level: "INFO" # INFO/DEBUG/WARNING/ERROR/FATAL
      - name: "FILE"
        level: "INFO"
        settings: # Customized settings of logger
          base_dir: "/var/log/jobs"
        sweeper:
          duration: 1 #days
          settings: # Customized settings of sweeper
            work_dir: "/var/log/jobs"
    #Loggers for the job service
    loggers:
      - name: "STD_OUTPUT" # Same with above
        level: "INFO"
    metric:
      enabled: true
      path: /metrics
      port: 9090
    reaper:
      # the max time to wait for a task to finish, if unfinished after max_update_hours, the task will be mark as error, but the task will continue to run, default value is 24,
      max_update_hours: 24
      # the max time for execution in running state without new task created
      max_dangling_hours: 168
    # the max size of job log returned by API, default is 10M
    max_retrieve_size_mb: 10
    ```
2. 編輯環境變數
    ```
    sudo nano /etc/harbor/jobservice.env
    ```
    檔案內容 : 
    ```
    REGISTRY_URL=http://registry:5000
    CORE_URL=http://core:8080
    REGISTRY_CONTROLLER_URL=http://registryctl:8080
    JOBSERVICE_WEBHOOK_JOB_MAX_RETRY=3
    JOBSERVICE_WEBHOOK_JOB_HTTP_CLIENT_TIMEOUT=3
    LOG_LEVEL=info
    HTTP_PROXY=
    HTTPS_PROXY=
    NO_PROXY=localhost,postgresql,portal,trivy-adapter,jobservice,127.0.0.1,.local,.internal,core,registryctl,redis,registry,nginx,db,log,exporter
    REGISTRY_CREDENTIAL_USERNAME=harbor_registry_user
    MAX_JOB_DURATION_SECONDS=86400
    METRIC_NAMESPACE=harbor
    METRIC_SUBSYSTEM=jobservice
    ```
3. 建立 jobservice container
    ```
    # 定義 Jobservice 資料永存目錄區
    JOBSVC_PV="/data/job_logs"
    [ ! -d ${JOBSVC_PV} ] && sudo mkdir -p ${JOBSVC_PV}
    sudo chown 10000:10000 ${JOBSVC_PV}
    
    sudo podman run --name=jobservice --detach \
        --network=harbor \
        --cap-add=setuid \
        --cap-add=setgid \
        --cap-add=chown \
        --cap-drop=all \
        --env-file=/etc/harbor/jobservice.env \
        --secret=harbor-secret,type=env,target=CORE_SECRET \
        --secret=jobservice-secret,type=env,target=JOBSERVICE_SECRET \
        --secret=registry-passwd,type=env,target=REGISTRY_CREDENTIAL_PASSWORD \
        --volume=/etc/harbor/jobservice.yml:/etc/jobservice/config.yml \
        --volume=${JOBSVC_PV}:/var/log/jobs:z \
        --volume=/etc/ssl/certs/ca-certificates.crt:/harbor_cust_cert/root.crt:ro \
        goharbor/harbor-jobservice:${HARBOR_VERSION}
    ```
### 3.8. 部署 exporter
1. 編輯環境變數檔
    ```
    sudo nano /etc/harbor/exporter.env
    ```
    檔案內容如下 : 
    ```
    LOG_LEVEL=info
    HARBOR_EXPORTER_PORT=8080
    HARBOR_EXPORTER_METRICS_PATH=/metrics
    HARBOR_EXPORTER_METRICS_ENABLED=true
    HARBOR_EXPORTER_MAX_REQUESTS=30
    HARBOR_EXPORTER_CACHE_TIME=23
    HARBOR_EXPORTER_CACHE_CLEAN_INTERVAL=14400
    HARBOR_METRIC_NAMESPACE=harbor
    HARBOR_METRIC_SUBSYSTEM=exporter
    HARBOR_SERVICE_HOST=core
    HARBOR_REDIS_URL=redis://redis:6379/2?idle_timeout_seconds=30
    HARBOR_REDIS_NAMESPACE=harbor_job_service_namespace
    HARBOR_REDIS_TIMEOUT=3600
    HARBOR_SERVICE_PORT=8080
    HARBOR_SERVICE_SCHEME=http
    HARBOR_DATABASE_HOST=postgresql
    HARBOR_DATABASE_PORT=5432
    HARBOR_DATABASE_USERNAME=postgres
    HARBOR_DATABASE_DBNAME=registry
    HARBOR_DATABASE_SSLMODE=disable
    HARBOR_DATABASE_MAX_IDLE_CONNS=100
    HARBOR_DATABASE_MAX_OPEN_CONNS=900
    HARBOR_DATABASE_CONN_MAX_LIFETIME=5m
    HARBOR_DATABASE_CONN_MAX_IDLE_TIME=0
    ```
2. 建立 exporter container
    ```
    sudo podman run -d \
      --name exporter --network harbor \
      --env-file /etc/harbor/exporter.env \
      --secret=db-secret,type=env,target=HARBOR_DATABASE_PASSWORD \
      --volume=/etc/ssl/certs/ca-certificates.crt:/harbor_cust_cert/root.crt:ro \
      goharbor/harbor-exporter:${HARBOR_VERSION}
    ```
### 3.9. 部署 Nginx
1. 產生自簽憑證
    ```
    git clone https://github.com/braveantony/mkcert.git
    ./mkcert/certctl
    sudo mkdir -p /data/secret/cert
    sudo chown -R 10000:10000 /data/secret/
    
    sudo cp mkcert/example.com/example.com.crt /data/secret/cert/server.crt
    sudo cp mkcert/example.com/example.com.key /data/secret/cert/server.key
    ```
2. 建立設定檔
    ```
    sudo nano /etc/harbor/nginx.conf
    ```
    檔案內容 :
    ```
    worker_processes auto;
    pid /tmp/nginx.pid;
    events {
      worker_connections 3096;
      use epoll;
      multi_accept on;
    }
    http {
      client_body_temp_path /tmp/client_body_temp;
      proxy_temp_path /tmp/proxy_temp;
      fastcgi_temp_path /tmp/fastcgi_temp;
      uwsgi_temp_path /tmp/uwsgi_temp;
      scgi_temp_path /tmp/scgi_temp;
      tcp_nodelay on;
      include /etc/nginx/conf.d/*.upstream.conf;
      # this is necessary for us to be able to disable request buffering in all cases
      proxy_http_version 1.1;
      upstream core {
        server core:8080;
      }
      upstream portal {
        server portal:8080;
      }
      log_format timed_combined '$remote_addr - '
        '"$request" $status $body_bytes_sent '
        '"$http_referer" "$http_user_agent" '
        '$request_time $upstream_response_time $pipe';
      access_log /dev/stdout timed_combined;
      map $http_x_forwarded_proto $x_forwarded_proto {
        default $http_x_forwarded_proto;
        ""      $scheme;
      }
      include /etc/nginx/conf.d/*.server.conf;
      server {
        listen 8443 ssl;
    #    server_name harbordomain.com;
        server_tokens off;
        # SSL
        ssl_certificate /etc/cert/server.crt;
        ssl_certificate_key /etc/cert/server.key;
        # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers '!aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES:';
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        # disable any limits to avoid HTTP 413 for large image uploads
        client_max_body_size 0;
        # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
        chunked_transfer_encoding on;
        # Add extra headers
        add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
        add_header X-Frame-Options DENY;
        add_header Content-Security-Policy "frame-ancestors 'none'";
        # customized location config file can place to /etc/nginx dir with prefix harbor.https. and suffix .conf
        include /etc/nginx/conf.d/harbor.https.*.conf;
        location / {
          proxy_pass http://portal/;
          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $x_forwarded_proto;
          proxy_cookie_path / "/; HttpOnly; Secure";
          proxy_buffering off;
          proxy_request_buffering off;
        }
        location /c/ {
          proxy_pass http://core/c/;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $x_forwarded_proto;
          proxy_cookie_path / "/; Secure";
          proxy_buffering off;
          proxy_request_buffering off;
        }
        location /api/ {
          proxy_pass http://core/api/;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $x_forwarded_proto;
          proxy_cookie_path / "/; Secure";
          proxy_buffering off;
          proxy_request_buffering off;
        }
        location /v1/ {
          return 404;
        }
        location /v2/ {
          proxy_pass http://core/v2/;
          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $x_forwarded_proto;
          proxy_buffering off;
          proxy_request_buffering off;
          proxy_send_timeout 900;
          proxy_read_timeout 900;
        }
        location /service/ {
          proxy_pass http://core/service/;
          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $x_forwarded_proto;
          proxy_cookie_path / "/; Secure";
          proxy_buffering off;
          proxy_request_buffering off;
        }
        location /service/notifications {
          return 404;
        }
      }
      server {
          listen 8080;
          #server_name harbordomain.com;
          return 308 https://$host:443$request_uri;
      }
      upstream core_metrics {
        server core:9090;
      }
      upstream js_metrics {
        server jobservice:9090;
      }
      upstream registry_metrics {
        server registry:9090;
      }
      upstream harbor_exporter {
        server exporter:8080;
      }
      server {
        listen 9090;
        location = /metrics {
          if ($arg_comp = core) { proxy_pass http://core_metrics; }
          if ($arg_comp = jobservice) { proxy_pass http://js_metrics; }
          if ($arg_comp = registry) { proxy_pass http://registry_metrics; }
          proxy_pass http://harbor_exporter;
        }
      }
    }
    ```
3. 建立 nginx container
    ```
    sudo podman run --name=proxy \
        --detach --network=harbor \
        --cap-add=setuid --cap-add=setgid --cap-add=chown \
        --cap-add=net_bind_service --cap-drop=all \
        --volume=/etc/harbor/jobservice.yml:/etc/jobservice/config.yml \
        --volume=/etc/harbor/nginx.conf:/etc/nginx/nginx.conf:ro \
        --volume=/data/secret/cert:/etc/cert:ro \
        --publish=80:8080/tcp \
        --publish=443:8443/tcp \
        goharbor/nginx-photon:${HARBOR_VERSION}
    ```
### 3.10. 部署 trivy
1. 編輯環境變數檔
    ```
    sudo nano /etc/harbor/trivy.env
    ```
    檔案內容如下 : 
    ```
    REGISTRY_URL=http://registry:5000
    SCANNER_LOG_LEVEL=info
    SCANNER_REDIS_URL=redis://redis:6379/5?idle_timeout_seconds=30
    SCANNER_STORE_REDIS_URL=redis://redis:6379/5?idle_timeout_seconds=30
    SCANNER_STORE_REDIS_NAMESPACE=harbor.scanner.trivy:store
    SCANNER_STORE_REDIS_NAMESPACE=harbor.scanner.trivy:store
    SCANNER_JOB_QUEUE_REDIS_URL=redis://redis:6379/5?idle_timeout_seconds=30
    SCANNER_JOB_QUEUE_REDIS_NAMESPACE=harbor.scanner.trivy:job-queue
    SCANNER_TRIVY_CACHE_DIR=/home/scanner/.cache/trivy
    SCANNER_TRIVY_REPORTS_DIR=/home/scanner/.cache/reports
    SCANNER_TRIVY_VULN_TYPE=os,library
    SCANNER_TRIVY_SEVERITY=UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
    SCANNER_TRIVY_IGNORE_UNFIXED=False
    SCANNER_TRIVY_SKIP_UPDATE=False
    SCANNER_TRIVY_SKIP_JAVA_DB_UPDATE=False
    SCANNER_TRIVY_OFFLINE_SCAN=True
    SCANNER_TRIVY_SECURITY_CHECKS=vuln
    SCANNER_TRIVY_GITHUB_TOKEN=
    SCANNER_TRIVY_INSECURE=False
    SCANNER_TRIVY_TIMEOUT=5m0s
    ```
2. 建立 trivy container
    ```
    sudo podman run --name=trivy --detach \
        --network=harbor \
        --cap-drop=all \
        --env-file=/etc/harbor/trivy.env \
        goharbor/trivy-adapter-photon:${HARBOR_VERSION}
    ```
### 3.11. 檢查所有服務是否異常
1. 執行以下命令檢查 container 運作狀態
    ```
    sudo podman ps -a
    ```
    正確執行結果 : 
    ```
    CONTAINER ID  IMAGE                                            COMMAND               CREATED         STATUS                   PORTS                                        NAMES
    2fbbebdfa7b7  localhost/goharbor/redis-photon:v2.13.1          redis-server /etc...  2 hours ago     Up 37 minutes (healthy)                                               redis
    1ee927703c43  localhost/goharbor/harbor-db:v2.13.1                                   2 hours ago     Up 2 hours (healthy)                                                  postgresql
    6609abf6abd3  localhost/goharbor/registry-photon:v2.13.1                             2 hours ago     Up 2 hours (healthy)                                                  registry
    0ac760f6bd97  localhost/goharbor/harbor-portal:v2.13.1         nginx -g daemon o...  2 hours ago     Up 2 hours (healthy)                                                  portal
    d5b932627e1c  localhost/goharbor/harbor-registryctl:v2.13.1                          2 hours ago     Up 2 hours (healthy)                                                  registryctl
    4ae9fe3acffd  localhost/goharbor/harbor-core:v2.13.1                                 50 minutes ago  Up 36 minutes (healthy)                                               core
    efaa7d97283a  localhost/goharbor/harbor-jobservice:v2.13.1                           20 minutes ago  Up 20 minutes (healthy)                                               jobservice
    f5d6b65fbcf8  localhost/goharbor/harbor-exporter:v2.13.1                             9 minutes ago   Up 9 minutes                                                          exporter
    f2d2ce9bb84e  localhost/goharbor/nginx-photon:v2.13.1          nginx -g daemon o...  6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  proxy
    5c4f83c677a2  localhost/goharbor/trivy-adapter-photon:v2.13.1                        5 seconds ago   Up 6 seconds (healthy)                                                trivy
    ```
### 3.12. 連線 Harbor UI
打開瀏覽器,連線至
```
https://$HARBOR_FQDN
```

輸入身分資訊 : 
- 帳號 : `admin`
- 密碼 : `Harbor12345`

### 3.13. 設定 Harbor 開機自動啟動
:::info
目前只適用 systemd 的 OS,如果是 OpenRC 就不適用
:::
1. 產生 Systemd Unit 並控制順序
    ```
    sudo podman generate systemd --name redis --restart-policy=always --after network-online.target --files
    sudo podman generate systemd --name postgresql --restart-policy=always --after container-redis.service --files
    sudo podman generate systemd --name registry --restart-policy=always --after container-postgresql.service --files
    sudo podman generate systemd --name portal --restart-policy=always --after container-registry.service --files
    sudo podman generate systemd --name registryctl --restart-policy=always --after container-portal.service --files
    sudo podman generate systemd --name core --restart-policy=always --after container-registryctl.service --files
    sudo podman generate systemd --name jobservice --restart-policy=always --after container-core.service --files
    sudo podman generate systemd --name exporter --restart-policy=always --after container-jobservice.service --files
    sudo podman generate systemd --name proxy --restart-policy=always --after container-exporter.service --files
    sudo podman generate systemd --name trivy --restart-policy=always --after container-proxy.service --files
    ```
2. 將這些 `.service` 檔案複製到 systemd 系統目錄
    ```
    sudo cp container-*.service /etc/systemd/system/
    ```
3. 重新載入 daemon
    ```
    sudo systemctl daemon-reload
    ```
4. 設定服務開機自動啟動
    ```
    sudo systemctl enable container-redis.service container-postgresql.service container-registry.service   container-portal.service container-registryctl.service container-core.service   container-jobservice.service container-exporter.service container-proxy.service container-trivy.service
    ```
5. 重開機測試
    ```
    sudo reboot
    ```
6. ssh 連線進 podman host 主機
7. 確認 harbor 各元件的 containers 運作狀態
    ```
    sudo podman ps -a
    ```
    執行結果 : 
    ```
    CONTAINER ID  IMAGE                                            COMMAND               CREATED            STATUS                  PORTS                                        NAMES
    2fbbebdfa7b7  localhost/goharbor/redis-photon:v2.13.1          redis-server /etc...  3 hours ago        Up 3 minutes (healthy)                                               redis
    1ee927703c43  localhost/goharbor/harbor-db:v2.13.1                                   3 hours ago        Up 3 minutes (healthy)                                               postgresql
    6609abf6abd3  localhost/goharbor/registry-photon:v2.13.1                             3 hours ago        Up 3 minutes (healthy)                                               registry
    0ac760f6bd97  localhost/goharbor/harbor-portal:v2.13.1         nginx -g daemon o...  3 hours ago        Up 3 minutes (healthy)                                               portal
    d5b932627e1c  localhost/goharbor/harbor-registryctl:v2.13.1                          3 hours ago        Up 3 minutes (healthy)                                               registryctl
    4ae9fe3acffd  localhost/goharbor/harbor-core:v2.13.1                                 2 hours ago        Up 3 minutes (healthy)                                               core
    efaa7d97283a  localhost/goharbor/harbor-jobservice:v2.13.1                           About an hour ago  Up 3 minutes (healthy)                                               jobservice
    f5d6b65fbcf8  localhost/goharbor/harbor-exporter:v2.13.1                             About an hour ago  Up 3 minutes                                                         exporter
    f2d2ce9bb84e  localhost/goharbor/nginx-photon:v2.13.1          nginx -g daemon o...  About an hour ago  Up 3 minutes (healthy)  0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  proxy
    5c4f83c677a2  localhost/goharbor/trivy-adapter-photon:v2.13.1                        About an hour ago  Up 3 minutes (healthy)                                               trivy
    ```
---
## 參考資料
- [Podman 安裝 harbor - zggzcgy blog](https://www.cnblogs.com/zggzcgy/p/18639254)