Install Harbor using Podman

# Install Harbor using Podman ## 先決條件 - [x] 已安裝 `podman`，版本 `4.9.x` 或以上 - [x] 已安裝 `curl`, `wget`, `jq`, `git` 和 `httpd-tools` 工具 ``` sudo dnf install -y podman curl wget jq httpd-tools git ``` - 本篇文章環境資訊 - OS 發行版: `Red Hat Enterprise Linux 9.7 (Plow)` - Kernel 版本: `5.14.0-611.24.1.el9_7.x86_64` - podman 版本: `5.6.0` - SELinux: `Enforcing` - Firewalld systemd service: `active and enabled` ### 硬體資源需求 | 資源項目 | 最低配置 | 建議配置 | | --- | --- | --- | | **CPU** | 2 核 (CPU) | 4 核 (CPU) | | **記憶體 (Mem)** | 4 GB | 8 GB | | **磁碟空間 (Disk)** | 40 GB | 160 GB | ## Setp1: 建立 podman network ``` sudo podman network create --subnet 10.8.0.0/16 harbor ``` > 若不想建立網路，可以使用預設 podman 網路，但需要將 dns 解析打開 > 預設網路目前版本沒有開啟 dns 解析，要不然 containers 之間沒法透過主機名稱進行互訪。 > 修改設定檔後需要將網路介面刪除或重新啟動主機才能生效。 ## Step2: 下載 Harbor 離線安裝所需的檔案 1. 下載官方發布的離線版本 ``` HARBOR_VERSION=$(curl -s https://api.github.com/repos/goharbor/harbor/releases/latest | jq -r .tag_name) wget https://github.com/goharbor/harbor/releases/download/${HARBOR_VERSION}/harbor-offline-installer-${HARBOR_VERSION}.tgz ``` > 第一行命令會直接到 github 抓 Harbor 官方最新 release 的版本，如果要安裝其他版本，請自行定義 2. 解壓縮檔案，並切換工作目錄 ``` tar zxf harbor-offline-installer-${HARBOR_VERSION}.tgz; cd harbor ``` 3. 匯入 image ``` sudo podman load -i harbor.${HARBOR_VERSION}.tar.gz ``` 4. 檢查是否成功 ``` sudo podman images ``` 執行結果如下 : ``` REPOSITORY TAG IMAGE ID CREATED SIZE localhost/goharbor/harbor-exporter v2.14.2 26fc5ce17425 13 days ago 133 MB localhost/goharbor/redis-photon v2.14.2 d62cec11627f 13 days ago 190 MB localhost/goharbor/trivy-adapter-photon v2.14.2 6165b96861b7 13 days ago 403 MB localhost/goharbor/harbor-registryctl v2.14.2 1364c6d80c42 13 days ago 167 MB localhost/goharbor/registry-photon v2.14.2 4161f69794cf 13 days ago 88.4 MB localhost/goharbor/nginx-photon v2.14.2 9383173f860e 13 days ago 175 MB localhost/goharbor/harbor-log v2.14.2 77a92995972a 13 days ago 188 MB localhost/goharbor/harbor-jobservice v2.14.2 5b11a15a5cb0 13 days ago 180 MB localhost/goharbor/harbor-core v2.14.2 604ea0c2c17f 13 days ago 205 MB localhost/goharbor/harbor-portal v2.14.2 b936a8ce0efb 13 days ago 183 MB localhost/goharbor/harbor-db v2.14.2 d80918ffd062 13 days ago 293 MB localhost/goharbor/prepare v2.14.2 6337f8dedb39 13 days ago 203 MB ``` ## Step3: 部署 Harbor 各元件注意，Harbor 各別的元件相互依賴，需要依照順序部署，否則元件無法啟動 ### 3.1. 部署 redis 1. 建立 redis container ``` # 定義 Redis 資料永存目錄區 REDIS_PV="/data/redis" [ ! -d ${REDIS_PV} ] && sudo mkdir -p ${REDIS_PV} sudo chown 999:999 ${REDIS_PV} sudo podman run --name redis --detach \ --network=harbor \ --cap-add=chown --cap-add=setuid \ --cap-add=setgid --cap-drop=all \ --volume=${REDIS_PV}:/var/lib/redis:z \ goharbor/redis-photon:${HARBOR_VERSION} ``` ### 3.2. 部署 postgresql 1. 建立 secret：資料庫 root 使用者的密碼，這裡使用隨機密碼 ``` tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 16 | sudo podman secret create db-secret - ``` 2. 建立 pg container ``` # 定義 Redis 資料永存目錄區 PG_PV="/data/database" [ ! -d ${PG_PV} ] && sudo mkdir -p ${PG_PV} sudo chown -R 999:999 ${PG_PV} sudo podman run --name postgresql \ --detach --network=harbor \ --cap-add=chown --cap-add=setuid \ --cap-add=setgid --cap-add=dac_override \ --cap-drop=all --shm-size=1gb \ --secret=db-secret,type=env,target=POSTGRES_PASSWORD \ --volume=${PG_PV}:/var/lib/postgresql/data:z \ goharbor/harbor-db:${HARBOR_VERSION} ``` ### 3.3. 部署 Registry 1. 編輯 registry 設定檔 ``` sudo mkdir -p /etc/harbor sudo nano /etc/harbor/registry.yml ``` 檔案內容如下 : ``` version: 0.1 log: level: info fields: service: registry storage: cache: layerinfo: redis filesystem: rootdirectory: /storage maintenance: uploadpurging: enabled: true age: 168h interval: 24h dryrun: false delete: enabled: true redis: addr: redis:6379 readtimeout: 10s writetimeout: 10s dialtimeout: 10s password: db: 1 enableTLS: false pool: maxidle: 100 maxactive: 500 idletimeout: 60s http: addr: :5000 secret: placeholder debug: addr: :9090 prometheus: enabled: true path: /metrics auth: htpasswd: realm: harbor-registry-basic-realm path: /etc/registry/passwd validation: disabled: true compatibility: schema1: enabled: true ``` 2. 建立密碼 ``` USERNAME="harbor_registry_user" PASSWORD="password" HASH=$(htpasswd -nbB -C 5 "$USERNAME" "$PASSWORD" | cut -d: -f2) echo "$USERNAME:$HASH" | sudo podman secret create registry-auth - ``` 3. 啟動 registry container ``` # 定義 REGISTRY 資料永存目錄區 REGISTRY_PV="/data/registry" [ ! -d ${REGISTRY_PV} ] && sudo mkdir -p ${REGISTRY_PV} sudo chown 10000:10000 ${REGISTRY_PV} sudo podman run --name=registry \ --detach --network=harbor \ --cap-add=chown --cap-add=setuid \ --cap-add=setgid --cap-drop=all \ --secret=registry-auth,target=/etc/registry/passwd \ --volume=${REGISTRY_PV}:/storage:z \ --volume=/etc/harbor/registry.yml:/etc/registry/config.yml:ro \ --volume=$(find /etc/ssl/certs/ -name ca-*.crt ! -name "*trust*"):/etc/registry/root.crt:ro \ goharbor/registry-photon:${HARBOR_VERSION} ``` ### 3.4. 部署 portal 1. 建立設定檔 ``` sudo nano /etc/harbor/portal.conf ``` 檔案內容如下 : ``` worker_processes auto; pid /tmp/nginx.pid; events { worker_connections 1024; } http { client_body_temp_path /tmp/client_body_temp; proxy_temp_path /tmp/proxy_temp; fastcgi_temp_path /tmp/fastcgi_temp; uwsgi_temp_path /tmp/uwsgi_temp; scgi_temp_path /tmp/scgi_temp; server { listen 8080; server_name localhost; root /usr/share/nginx/html; index index.html index.htm; include /etc/nginx/mime.types; gzip on; gzip_min_length 1000; gzip_proxied expired no-cache no-store private auth; gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; location /devcenter-api-2.0 { try_files $uri $uri/ /swagger-ui-index.html; } location / { try_files $uri $uri/ /index.html; } location = /index.html { add_header Cache-Control "no-store, no-cache, must-revalidate"; } } } ``` 2. 建立 portal container ``` sudo podman run --name=portal \ --detach --network=harbor \ --cap-add=setuid --cap-add=setgid \ --cap-add=net_bind_service --cap-drop=all \ --volume=/etc/harbor/portal.conf:/etc/nginx/nginx.conf:ro \ goharbor/harbor-portal:${HARBOR_VERSION} ``` ### 3.5. 部署 registryctl 1. 編輯設定檔 ``` sudo nano /etc/harbor/registryctl.yml ``` 檔案內容如下 : ``` --- protocol: "http" port: 8080 log_level: info registry_config: "/etc/registry/config.yml" ``` 2. 建立密碼 ``` # 產生 harbor 私鑰: 用於建立 token openssl genrsa 4096 | openssl pkey -traditional | sudo podman secret create harbor-key - # 產生 harbor secretkey openssl rand -base64 12 | head -c 16| sudo podman secret create harbor-secretkey - # harbor 密碼：用於其他元件與 harbor 通訊認證 openssl rand -base64 12 | head -c 16 | sudo podman secret create harbor-secret - # jobservice 密碼：用於與 jobservice 通訊認證 openssl rand -base64 12 | head -c 16 | sudo podman secret create jobservice-secret - # registry 密碼: 主要給 harbor 和 jobservice 使用 printf "password" | sudo podman secret create registry-passwd - ``` 3. 建立 registryctl container ``` sudo podman run --name=registryctl \ --detach --network=harbor \ --cap-add=setuid --cap-add=setgid --cap-drop=all \ --secret=harbor-secret,type=env,target=CORE_SECRET \ --secret=jobservice-secret,type=env,target=JOBSERVICE_SECRET \ --volume=${REGISTRY_PV}:/storage:z \ --volume=/etc/harbor/registry.yml:/etc/registry/config.yml:ro \ --volume=/etc/harbor/registryctl.yml:/etc/registryctl/config.yml:ro \ --volume=$(find /etc/ssl/certs/ -name ca-*.crt ! -name "*trust*"):/etc/registry/root.crt:ro \ goharbor/harbor-registryctl:${HARBOR_VERSION} ``` ### 3.6. 部署 harbor 1. 編輯設定檔 ``` sudo nano /etc/harbor/harbor.conf ``` 檔案內容如下 : ``` appname = Harbor runmode = prod enablegzip = true [prod] httpport = 8080 ``` 2. 編輯 Harbor 運作所需之環境變數 ``` sudo nano /etc/harbor/coreenv ``` 檔案內容如下 : ``` CONFIG_PATH=/etc/core/app.conf UAA_CA_ROOT=/etc/core/certificates/uaa_ca.pem _REDIS_URL_CORE=redis://redis:6379?idle_timeout_seconds=30 SYNC_QUOTA=true _REDIS_URL_REG=redis://redis:6379/1?idle_timeout_seconds=30 LOG_LEVEL=info DATABASE_TYPE=postgresql POSTGRESQL_HOST=postgresql POSTGRESQL_PORT=5432 POSTGRESQL_USERNAME=postgres POSTGRESQL_DATABASE=registry POSTGRESQL_SSLMODE=disable POSTGRESQL_MAX_IDLE_CONNS=100 POSTGRESQL_MAX_OPEN_CONNS=900 POSTGRESQL_CONN_MAX_LIFETIME=5m POSTGRESQL_CONN_MAX_IDLE_TIME=0 REGISTRY_URL=http://registry:5000 PORTAL_URL=http://portal:8080 TOKEN_SERVICE_URL=http://core:8080/service/token HARBOR_ADMIN_PASSWORD=Harbor12345 MAX_JOB_WORKERS=10 WITH_TRIVY=True CORE_URL=http://core:8080 CORE_LOCAL_URL=http://127.0.0.1:8080 JOBSERVICE_URL=http://jobservice:8080 TRIVY_ADAPTER_URL=http://trivy-adapter:8080 REGISTRY_STORAGE_PROVIDER_NAME=filesystem READ_ONLY=false RELOAD_KEY= REGISTRY_CONTROLLER_URL=http://registryctl:8080 REGISTRY_CREDENTIAL_USERNAME=harbor_registry_user CSRF_KEY=jPcpPYw2HU4V0r2LG2kyHWWSIwRE4Ln8 ROBOT_SCANNER_NAME_PREFIX=2GA78XLs PERMITTED_REGISTRY_TYPES_FOR_PROXY_CACHE=docker-hub,harbor,azure-acr,ali-acr,aws-ecr,google-gcr,quay,docker-registry,github-ghcr,jfrog-artifactory HTTP_PROXY= HTTPS_PROXY= NO_PROXY=localhost,postgresql,portal,trivy-adapter,jobservice,127.0.0.1,.local,.internal,core,registryctl,redis,registry,nginx,db,log,exporter PORT=8080 METRIC_ENABLE=true METRIC_PATH=/metrics METRIC_PORT=9090 METRIC_NAMESPACE=harbor METRIC_SUBSYSTEM=core QUOTA_UPDATE_PROVIDER=db ``` 4. 建立 Harbor container ``` # 定義 Harbor 對外的 url HARBOR_FQDN="pharbor.example.com" # 定義 Harbor 資料永存目錄區 HARBOR_PV="/data" [ ! -d ${HARBOR_PV} ] && sudo mkdir -p ${HARBOR_PV} sudo chown 10000:10000 ${HARBOR_PV} # Run Harbor Container sudo podman run --name=core --detach \ --network=harbor \ --cap-add=setuid \ --cap-add=setgid \ --cap-drop=all \ --env-file=/etc/harbor/coreenv \ --env=EXT_ENDPOINT=https://${HARBOR_FQDN} \ --secret=harbor-secret,type=env,target=CORE_SECRET \ --secret=db-secret,type=env,target=POSTGRESQL_PASSWORD \ --secret=jobservice-secret,type=env,target=JOBSERVICE_SECRET \ --secret=registry-passwd,type=env,target=REGISTRY_CREDENTIAL_PASSWORD \ --secret=harbor-secretkey,type=mount,target=/etc/core/key \ --secret=harbor-key,type=mount,target=/etc/core/private_key.pem \ --volume=${HARBOR_PV}:/data:z \ --volume=/etc/harbor/harbor.conf:/etc/core/app.conf:ro \ --volume=$(find /etc/ssl/certs/ -name ca-*.crt ! -name "*trust*"):/etc/core/certificates/root.crt:ro \ --volume=/usr/share/zoneinfo/Asia/Taipei:/etc/localtime:ro \ goharbor/harbor-core:${HARBOR_VERSION} ``` ### 3.7. 部署 jobservice 1. 編輯設定檔 ``` sudo nano /etc/harbor/jobservice.yml ``` 檔案內容如下 : ``` --- #Protocol used to serve protocol: "http" #Server listening port port: 8080 #Worker pool worker_pool: #Worker concurrency workers: 10 backend: "redis" #Additional config if use 'redis' backend redis_pool: #redis://[arbitrary_username:password@]ipaddress:port/database_index redis_url: redis://redis:6379/2?idle_timeout_seconds=30 namespace: "harbor_job_service_namespace" idle_timeout_second: 3600 #Loggers for the running job job_loggers: # The jobLoggers backend name, only support "STD_OUTPUT", "FILE" and/or "DB" - name: "STD_OUTPUT" level: "INFO" # INFO/DEBUG/WARNING/ERROR/FATAL - name: "FILE" level: "INFO" settings: # Customized settings of logger base_dir: "/var/log/jobs" sweeper: duration: 1 #days settings: # Customized settings of sweeper work_dir: "/var/log/jobs" #Loggers for the job service loggers: - name: "STD_OUTPUT" # Same with above level: "INFO" metric: enabled: true path: /metrics port: 9090 reaper: # the max time to wait for a task to finish, if unfinished after max_update_hours, the task will be mark as error, but the task will continue to run, default value is 24, max_update_hours: 24 # the max time for execution in running state without new task created max_dangling_hours: 168 # the max size of job log returned by API, default is 10M max_retrieve_size_mb: 10 ``` 2. 編輯環境變數 ``` sudo nano /etc/harbor/jobservice.env ``` 檔案內容 : ``` REGISTRY_URL=http://registry:5000 CORE_URL=http://core:8080 REGISTRY_CONTROLLER_URL=http://registryctl:8080 JOBSERVICE_WEBHOOK_JOB_MAX_RETRY=3 JOBSERVICE_WEBHOOK_JOB_HTTP_CLIENT_TIMEOUT=3 LOG_LEVEL=info HTTP_PROXY= HTTPS_PROXY= NO_PROXY=localhost,postgresql,portal,trivy-adapter,jobservice,127.0.0.1,.local,.internal,core,registryctl,redis,registry,nginx,db,log,exporter REGISTRY_CREDENTIAL_USERNAME=harbor_registry_user MAX_JOB_DURATION_SECONDS=86400 METRIC_NAMESPACE=harbor METRIC_SUBSYSTEM=jobservice ``` 3. 建立 jobservice container ``` # 定義 Jobservice 資料永存目錄區 JOBSVC_PV="/data/job_logs" [ ! -d ${JOBSVC_PV} ] && sudo mkdir -p ${JOBSVC_PV} sudo chown 10000:10000 ${JOBSVC_PV} sudo podman run --name=jobservice --detach \ --network=harbor \ --cap-add=setuid \ --cap-add=setgid \ --cap-add=chown \ --cap-drop=all \ --env-file=/etc/harbor/jobservice.env \ --secret=harbor-secret,type=env,target=CORE_SECRET \ --secret=jobservice-secret,type=env,target=JOBSERVICE_SECRET \ --secret=registry-passwd,type=env,target=REGISTRY_CREDENTIAL_PASSWORD \ --volume=/etc/harbor/jobservice.yml:/etc/jobservice/config.yml \ --volume=${JOBSVC_PV}:/var/log/jobs:z \ --volume=$(realpath $(find /etc/ssl/certs/ -name ca-*.crt ! -name "*trust*")):/harbor_cust_cert/root.crt:ro,Z \ goharbor/harbor-jobservice:${HARBOR_VERSION} ``` ### 3.8. 部署 exporter 1. 編輯環境變數檔 ``` sudo nano /etc/harbor/exporter.env ``` 檔案內容如下 : ``` LOG_LEVEL=info HARBOR_EXPORTER_PORT=8080 HARBOR_EXPORTER_METRICS_PATH=/metrics HARBOR_EXPORTER_METRICS_ENABLED=true HARBOR_EXPORTER_MAX_REQUESTS=30 HARBOR_EXPORTER_CACHE_TIME=23 HARBOR_EXPORTER_CACHE_CLEAN_INTERVAL=14400 HARBOR_METRIC_NAMESPACE=harbor HARBOR_METRIC_SUBSYSTEM=exporter HARBOR_SERVICE_HOST=core HARBOR_REDIS_URL=redis://redis:6379/2?idle_timeout_seconds=30 HARBOR_REDIS_NAMESPACE=harbor_job_service_namespace HARBOR_REDIS_TIMEOUT=3600 HARBOR_SERVICE_PORT=8080 HARBOR_SERVICE_SCHEME=http HARBOR_DATABASE_HOST=postgresql HARBOR_DATABASE_PORT=5432 HARBOR_DATABASE_USERNAME=postgres HARBOR_DATABASE_DBNAME=registry HARBOR_DATABASE_SSLMODE=disable HARBOR_DATABASE_MAX_IDLE_CONNS=100 HARBOR_DATABASE_MAX_OPEN_CONNS=900 HARBOR_DATABASE_CONN_MAX_LIFETIME=5m HARBOR_DATABASE_CONN_MAX_IDLE_TIME=0 ``` 2. 建立 exporter container ``` sudo podman run -d \ --name exporter --network harbor \ --env-file /etc/harbor/exporter.env \ --secret=db-secret,type=env,target=HARBOR_DATABASE_PASSWORD \ --volume=$(realpath $(find /etc/ssl/certs/ -name ca-*.crt ! -name "*trust*")):/harbor_cust_cert/root.crt:ro,Z \ goharbor/harbor-exporter:${HARBOR_VERSION} ``` ### 3.9. 部署 Nginx 1. 產生自簽憑證 ``` git clone https://github.com/braveantony/mkcert.git # change for your base domain nano mkcert/config # Server 相關設定 DOMAIN_NAME="example.com" ./mkcert/certctl sudo mkdir -p /data/secret/cert sudo cp mkcert/example.com/example.com.crt /data/secret/cert/server.crt sudo cp mkcert/example.com/example.com.key /data/secret/cert/server.key sudo chown -R 10000:10000 /data/secret/cert ``` 2. 建立設定檔 ``` sudo nano /etc/harbor/nginx.conf ``` 檔案內容 : ``` worker_processes auto; pid /tmp/nginx.pid; events { worker_connections 3096; use epoll; multi_accept on; } http { client_body_temp_path /tmp/client_body_temp; proxy_temp_path /tmp/proxy_temp; fastcgi_temp_path /tmp/fastcgi_temp; uwsgi_temp_path /tmp/uwsgi_temp; scgi_temp_path /tmp/scgi_temp; tcp_nodelay on; include /etc/nginx/conf.d/*.upstream.conf; # this is necessary for us to be able to disable request buffering in all cases proxy_http_version 1.1; upstream core { server core:8080; } upstream portal { server portal:8080; } log_format timed_combined '$remote_addr - ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time $upstream_response_time $pipe'; access_log /dev/stdout timed_combined; map $http_x_forwarded_proto $x_forwarded_proto { default $http_x_forwarded_proto; "" $scheme; } include /etc/nginx/conf.d/*.server.conf; server { listen 8443 ssl; # server_name harbordomain.com; server_tokens off; # SSL ssl_certificate /etc/cert/server.crt; ssl_certificate_key /etc/cert/server.key; # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers '!aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES:'; ssl_prefer_server_ciphers on; ssl_session_cache shared:SSL:10m; # disable any limits to avoid HTTP 413 for large image uploads client_max_body_size 0; # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486) chunked_transfer_encoding on; # Add extra headers add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"; add_header X-Frame-Options DENY; add_header Content-Security-Policy "frame-ancestors 'none'"; # customized location config file can place to /etc/nginx dir with prefix harbor.https. and suffix .conf include /etc/nginx/conf.d/harbor.https.*.conf; location / { proxy_pass http://portal/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $x_forwarded_proto; proxy_cookie_path / "/; HttpOnly; Secure"; proxy_buffering off; proxy_request_buffering off; } location /c/ { proxy_pass http://core/c/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $x_forwarded_proto; proxy_cookie_path / "/; Secure"; proxy_buffering off; proxy_request_buffering off; } location /api/ { proxy_pass http://core/api/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $x_forwarded_proto; proxy_cookie_path / "/; Secure"; proxy_buffering off; proxy_request_buffering off; } location /v1/ { return 404; } location /v2/ { proxy_pass http://core/v2/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $x_forwarded_proto; proxy_buffering off; proxy_request_buffering off; proxy_send_timeout 900; proxy_read_timeout 900; } location /service/ { proxy_pass http://core/service/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $x_forwarded_proto; proxy_cookie_path / "/; Secure"; proxy_buffering off; proxy_request_buffering off; } location /service/notifications { return 404; } } server { listen 8080; #server_name harbordomain.com; return 308 https://$host:443$request_uri; } upstream core_metrics { server core:9090; } upstream js_metrics { server jobservice:9090; } upstream registry_metrics { server registry:9090; } upstream harbor_exporter { server exporter:8080; } server { listen 9090; location = /metrics { if ($arg_comp = core) { proxy_pass http://core_metrics; } if ($arg_comp = jobservice) { proxy_pass http://js_metrics; } if ($arg_comp = registry) { proxy_pass http://registry_metrics; } proxy_pass http://harbor_exporter; } } } ``` 3. 建立 nginx container ``` sudo podman run --name=proxy \ --detach --network=harbor \ --cap-drop=all \ --cap-add=setuid --cap-add=setgid --cap-add=chown \ --cap-add=net_bind_service --cap-add=dac_override \ --volume=/etc/harbor/jobservice.yml:/etc/jobservice/config.yml:z \ --volume=/etc/harbor/nginx.conf:/etc/nginx/nginx.conf:ro,z \ --volume=/data/secret/cert:/etc/cert:ro,Z \ --publish=80:8080/tcp \ --publish=443:8443/tcp \ --publish=9090:9090/tcp \ goharbor/nginx-photon:${HARBOR_VERSION} ``` ### 3.10. 部署 trivy 1. 編輯環境變數檔 ``` sudo nano /etc/harbor/trivy.env ``` 檔案內容如下 : ``` REGISTRY_URL=http://registry:5000 SCANNER_LOG_LEVEL=info SCANNER_REDIS_URL=redis://redis:6379/5?idle_timeout_seconds=30 SCANNER_STORE_REDIS_URL=redis://redis:6379/5?idle_timeout_seconds=30 SCANNER_STORE_REDIS_NAMESPACE=harbor.scanner.trivy:store SCANNER_STORE_REDIS_NAMESPACE=harbor.scanner.trivy:store SCANNER_JOB_QUEUE_REDIS_URL=redis://redis:6379/5?idle_timeout_seconds=30 SCANNER_JOB_QUEUE_REDIS_NAMESPACE=harbor.scanner.trivy:job-queue SCANNER_TRIVY_CACHE_DIR=/home/scanner/.cache/trivy SCANNER_TRIVY_REPORTS_DIR=/home/scanner/.cache/reports SCANNER_TRIVY_VULN_TYPE=os,library SCANNER_TRIVY_SEVERITY=UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL SCANNER_TRIVY_IGNORE_UNFIXED=False SCANNER_TRIVY_SKIP_UPDATE=False SCANNER_TRIVY_SKIP_JAVA_DB_UPDATE=False SCANNER_TRIVY_OFFLINE_SCAN=True SCANNER_TRIVY_SECURITY_CHECKS=vuln SCANNER_TRIVY_GITHUB_TOKEN= SCANNER_TRIVY_INSECURE=False SCANNER_TRIVY_TIMEOUT=5m0s ``` 2. 建立 trivy container ``` sudo podman run --name=trivy-adapter --detach \ --network=harbor \ --cap-drop=all \ --env-file=/etc/harbor/trivy.env \ goharbor/trivy-adapter-photon:${HARBOR_VERSION} ``` ### 3.11. 檢查所有服務是否異常 1. 執行以下命令檢查 container 運作狀態 ``` sudo podman ps -a ``` 正確執行結果 : ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ad033894240e localhost/goharbor/redis-photon:v2.14.2 redis-server /etc... 2 hours ago Up 2 hours (healthy) redis 15a84e395302 localhost/goharbor/harbor-db:v2.14.2 2 hours ago Up 2 hours (healthy) postgresql a2ac4a2928d0 localhost/goharbor/harbor-portal:v2.14.2 nginx -g daemon o... About an hour ago Up About an hour (healthy) portal 5dd1ea9c53d8 localhost/goharbor/registry-photon:v2.14.2 About an hour ago Up About an hour (healthy) registry 7ad9c1c36d54 localhost/goharbor/harbor-registryctl:v2.14.2 52 minutes ago Up 52 minutes (healthy) registryctl 1ecd13d98484 localhost/goharbor/harbor-core:v2.14.2 42 minutes ago Up 42 minutes (healthy) core c7fc40df1b30 localhost/goharbor/harbor-jobservice:v2.14.2 28 minutes ago Up 28 minutes (healthy) jobservice 05c09539108d localhost/goharbor/harbor-exporter:v2.14.2 21 minutes ago Up 21 minutes exporter 243817e09c05 localhost/goharbor/nginx-photon:v2.14.2 nginx -g daemon o... 36 seconds ago Up 36 seconds (healthy) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp proxy 8c381495220b localhost/goharbor/trivy-adapter-photon:v2.14.2 3 seconds ago Up 4 seconds (healthy) trivy ``` ### 3.12. 連線 Harbor UI 打開瀏覽器，連線至 ``` https://$HARBOR_FQDN ``` ![image](https://hackmd.io/_uploads/SkJ22UQDel.png) 輸入身分資訊 : - 帳號 : `admin` - 密碼 : `Harbor12345` ![image](https://hackmd.io/_uploads/SkFEaLmvxg.png) ### 3.13. 設定 Harbor 開機自動啟動 :::info 目前只適用 systemd 的 OS，如果是 OpenRC 就不適用 ::: 1. 產生 Systemd Unit 並控制順序 ``` sudo podman generate systemd --name redis --restart-policy=always --after network-online.target --files sudo podman generate systemd --name postgresql --restart-policy=always --after container-redis.service --files sudo podman generate systemd --name registry --restart-policy=always --after container-postgresql.service --files sudo podman generate systemd --name portal --restart-policy=always --after container-registry.service --files sudo podman generate systemd --name registryctl --restart-policy=always --after container-portal.service --files sudo podman generate systemd --name core --restart-policy=always --after container-registryctl.service --files sudo podman generate systemd --name jobservice --restart-policy=always --after container-core.service --files sudo podman generate systemd --name exporter --restart-policy=always --after container-jobservice.service --files sudo podman generate systemd --name proxy --restart-policy=always --after container-exporter.service --files sudo podman generate systemd --name trivy-adapter --restart-policy=always --after container-proxy.service --files ``` 2. 將這些 `.service` 檔案複製到 systemd 系統目錄 ``` sudo cp container-*.service /etc/systemd/system/ ``` 3. 重新載入 daemon ``` sudo systemctl daemon-reload ``` 4. 設定服務開機自動啟動 ``` sudo systemctl enable \ container-redis.service \ container-postgresql.service \ container-registry.service \ container-portal.service \ container-registryctl.service \ container-core.service \ container-jobservice.service \ container-exporter.service \ container-proxy.service \ container-trivy-adapter.service ``` 5. 重開機測試 ``` sudo reboot ``` 6. ssh 連線進 podman host 主機 7. 確認 harbor 各元件的 containers 運作狀態 ``` sudo podman ps -a ``` 執行結果 : ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ad033894240e localhost/goharbor/redis-photon:v2.14.2 redis-server /etc... 7 hours ago Up 50 seconds (healthy) redis 15a84e395302 localhost/goharbor/harbor-db:v2.14.2 7 hours ago Up 50 seconds (healthy) postgresql a2ac4a2928d0 localhost/goharbor/harbor-portal:v2.14.2 nginx -g daemon o... 6 hours ago Up 49 seconds (healthy) portal 7ad9c1c36d54 localhost/goharbor/harbor-registryctl:v2.14.2 6 hours ago Up 49 seconds (healthy) registryctl 1ecd13d98484 localhost/goharbor/harbor-core:v2.14.2 6 hours ago Up 49 seconds (healthy) core c7fc40df1b30 localhost/goharbor/harbor-jobservice:v2.14.2 6 hours ago Up 49 seconds (healthy) jobservice 05c09539108d localhost/goharbor/harbor-exporter:v2.14.2 5 hours ago Up 49 seconds exporter 8df8d211ff56 localhost/goharbor/registry-photon:v2.14.2 3 hours ago Up 49 seconds (healthy) registry a5c4a370ab25 localhost/goharbor/trivy-adapter-photon:v2.14.2 8 minutes ago Up 49 seconds (healthy) trivy-adapter 479f0bd782bf localhost/goharbor/nginx-photon:v2.14.2 nginx -g daemon o... 5 minutes ago Up 48 seconds (healthy) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp, 0.0.0.0:9090->9090/tcp proxy ``` ## Step4：驗證 1. 設定 podman 信任 harbor 自簽 root CA 憑證 ``` HARBOR_FQDN="pharbor.example.com" sudo mkdir -p /etc/containers/certs.d/${HARBOR_FQDN} sudo cp ${HOME}/mkcert/example.com/example.com.crt /etc/containers/certs.d/${HARBOR_FQDN}/ca.crt ``` 2. podman 登入 harbor ``` podman login ${HARBOR_FQDN} -u admin -p "Harbor12345" ``` 執行結果： ``` Login Succeeded! ``` 3. 下載 Nginx container image ``` podman pull docker.io/library/nginx:stable ``` 4. 修改 image tag ``` podman tag docker.io/library/nginx:stable ${HARBOR_FQDN}/library/nginx:stable ``` 5. push image 至 harbor ``` podman push ${HARBOR_FQDN}/library/nginx:stable ``` 執行結果： ``` Getting image source signatures Copying blob e50a58335e13 done | Copying blob 5c00c19cd4b3 done | Copying blob 7dfc0854a228 done | Copying blob 4b43bb8fd938 done | Copying blob 5f3758e9c329 done | Copying blob ac61f062f9a7 done | Copying blob 16b60074fa6f done | Copying config 94a30ed18e done | Writing manifest to image destination ``` 6. 在 UI 檢查 library 專案是否多了一片 nginx image ![image](https://hackmd.io/_uploads/H1uxasNLWx.png) 8. 刪除原始的 image ``` podman rmi docker.io/library/nginx:stable ${HARBOR_FQDN}/library/nginx:stable ``` 執行結果： ``` Untagged: docker.io/library/nginx:stable Untagged: pharbor.example.com/library/nginx:stable Deleted: 94a30ed18e45363486a55379bd1fbb8c479f36524ae816779f3553afe6b787ed ``` 9. 從 harbor 下載 nginx image ``` podman pull ${HARBOR_FQDN}/library/nginx:stable ``` 執行結果： ``` Trying to pull pharbor.example.com/library/nginx:stable... Getting image source signatures Copying blob 2efbe32a5480 done | Copying blob 6996c0be3403 done | Copying blob ea368c811a9e done | Copying blob 765e9812de72 done | Copying blob b94f38f9c422 done | Copying blob 88e560633873 done | Copying blob f72edb929581 done | Copying config 94a30ed18e done | Writing manifest to image destination 94a30ed18e45363486a55379bd1fbb8c479f36524ae816779f3553afe6b787ed ``` 10. 掃描弱點和產出 SBOM 點進 `library/nginx` ![image](https://hackmd.io/_uploads/Skmy7nE8bx.png) 點選 “開始掃描” 按鈕和 "GENERATE SBOM" 按鈕 ![image](https://hackmd.io/_uploads/rkWqGhNUbl.png) 點選 "SBOM detials" ![image](https://hackmd.io/_uploads/BJ0M7248be.png) 檢視掃描節點 ![image](https://hackmd.io/_uploads/ry4G42N8Zl.png) --- ## 參考資料 - [Podman 安裝 harbor - zggzcgy blog](https://www.cnblogs.com/zggzcgy/p/18639254)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.