GitLab Runner === ###### tags: `III` `DevOps` `CI/CD` ## Add Runnger ![](https://i.imgur.com/kpFUhMa.png) --- ## Docker Container ### Install ``` docker run -d --name vmme-gitlab-runner --restart always \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest ``` ``` # tree /srv /srv └── gitlab-runner └── config └── config.toml ``` ### Register 進入gitlab-runner container ``` gitlab-runner register -n \ --url http://gitvepc/ \ --registration-token Pyiyjywx3yqgMo6_QCMT \ --executor docker \ --config /etc/gitlab-runner/config.toml \ --description "5GC Docker Runner" \ --docker-image "docker:19.03.8" \ --tag-list "5gc" \ --docker-volumes /var/run/docker.sock:/var/run/docker.sock \ --docker-disable-cache=true ``` 每 register 一次就會自動附加到 config.toml --- ## Helm Chart :::danger 不要用 GitLab GUI 安裝，它會把 Helm Tiller 和 GitLab Runner 放在 gitlab-managed-apps namespace 底下，這個 K8s 就不能拿來做 deploy。 安裝順序： 1. 透過 GitLab GUI 先裝 Helm Tiller 2. 透過 GitLab GUI 再裝 GitLab Runner ![](https://i.imgur.com/KmV2ghI.png) ::: ### 官方 Repos 官方沒有 expose docker.sock ``` helm repo add gitlab https://charts.gitlab.io helm search repo -l gitlab/gitlab-runner ``` package 存在 ~/.cache/helm/repository/ ### III Chart 我有 expose docker.sock ``` git clone http://gitvepc/abby/gitlab-runner --branch iii --single-branch ``` ### Install Release 以 chart 為模板，佈署 release，相應的 K8s resource 就此生成 ``` helm install --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> --name <RELEASE_NAME> <gitlab/gitlab-runner或./gitlab-runner> ``` ``` # <CONFIG_VALUES_FILE> gitlabUrl: http://gitvepc/ runnerRegistrationToken: "tqkgDDxsVAJ4VJY6ukjN" concurrent: 10 runners: image: docker:18.09 privileged: true volumes: - type: "host_path" name: "docker" path: "/var/run/docker.sock" mount_path: "/var/run/docker.sock" # IPs=$(kubectl get pod -o wide -n 5gc | awk 'NR > 1 {print $6}') # curl $IPs:9252/metrics metrics: enabled: true rbac: create: true hpa: minReplicas: 1 maxReplicas: 10 metrics: - type: Pods pods: metricName: gitlab_runner_jobs targetAverageValue: 150m ``` ### Upgrade Release ``` helm upgrade --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> --name <RELEASE_NAME> <gitlab/gitlab-runner或./gitlab-runner> ``` --- ## K8s 環境設定 ### kube-dns 要有 ```nameserver 8.8.8.8``` 跟 ```search nmsi.pri``` * CoreDNS 1. 直接永久加在改本地 /etc/resolv.conf 2. 重寫 ```gitvepc``` 為``` gitvepc.nmsi.pri``` - ```kubectl edit cm coredns -n kube-system``` ![](https://i.imgur.com/fa2VFv4.png) 3. 清掉 cache - ```kubectl get pods -n kube-system -o name | grep coredns | xargs kubectl delete -n kube-system``` ### cluster 至少要有一個 node 允許裝 pod ``` kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule- ``` ### enable remote tcp access to Docker daemons on all machines :::info 當對同一個 container 分不同的 dind job 做 ``docker run``, ``docker cp``, ``docker rm``, ``docker rmi``，有可能會使 job 分散在不同機器上，就會找不到要被操作的 container，所以要 remote access 不同機器上的 Docker daemon。 ::: * enable [live-restore](https://docs.docker.com/config/containers/live-restore/) ``` # vi /etc/docker/daemon.json { "live-restore": true } # systemctl reload docker ``` * edit docker.service ``` # vi /lib/systemd/system/docker.service ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock # systemctl daemon-reload # service docker restart ``` * remote access ``` docker -H tcp://10.22.100.233:2375 ps ``` ### Assign K8s credential variables to enable kubectl ![](https://i.imgur.com/L3RnkhT.png) --- ## Troubleshooting * 進入 gitlab-runner container ``` docker exec -it gitlab-runner bash ``` * 進入 runner-gitlab-runner pod ``` kubectl exec -it runner-gitlab-runner-7fc7d55f4b-6wvp2 -c runner-gitlab-runner -n gitlab-managed-apps -- sh ``` * 列出 repo ``` helm repo list ``` * 列出 release ``` helm list -n <NAMESPACE> ``` * 查看 release 的設定現狀 ``` helm get all <RELEASE_NAME> -n <NAMESPACE> helm get values <RELEASE_NAME> -n <NAMESPACE> ``` * 除錯 template 格式 ``` helm template --debug <CHART_DIR> ``` * config.toml 在 gitlab-runner pod 裡面 ``` # /home/gitlab-runner/.gitlab-runner/config.toml listen_address = ":9252" concurrent = 10 check_interval = 30 log_level = "info" [session_server] session_timeout = 1800 [[runners]] name = "buildenv-gitlab-runner-779d55d76c-kphqg" output_limit = 4096 request_concurrency = 1 url = "http://gitvepc/" token = "dQRerLmwh2j1E6wBLXyy" executor = "kubernetes" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.kubernetes] host = "" bearer_token_overwrite_allowed = false image = "docker:18.09" namespace = "5gc" namespace_overwrite_allowed = "" privileged = true poll_timeout = 180 service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.pod_security_context] [runners.kubernetes.volumes] [[runners.kubernetes.volumes.host_path]] mount_path = "/var/run/docker.sock" name = "docker" path = "/var/run/docker.sock" ``` 在 gitlab-runner container 裡面 ``` # /etc/gitlab-runner/config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "Abby Docker Runner" url = "http://gitvepc/" token = "mX-FxXdVkshksUSDvDiw" executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.docker] tls_verify = false image = "docker:19.03.8" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = true volumes = ["/var/run/docker.sock:/var/run/docker.sock"] shm_size = 0 ``` --- ## K8s Monitoring ### Metrics 因為 gitlab-runner pod 的 values.yml 預設已開啟 9252 port ``` metrics: enabled: true ``` 可以確認 pod 有提供 metric ``` IPs=$(kubectl get pod -o wide -n 5gc | awk 'NR > 1 {print $6}') curl $IPs:9252/metrics ``` 所以 Prometheus 可以取得 metrics，以下是 PromQL :::info ``gitlab_runner_jobs``: 某 gitlab-runner 目前**正在執行** job 的數量 ``gitlab_runner_version_info``: 有哪些 gitlab-runner pod ``gitlab_runner_concurrent``: 某 gitlab-runner 允許接到 job 的最大數量 ::: ### HPA 觀察點 ``` sum(gitlab_runner_jobs{namespace="abby-ns"}) sum(gitlab_runner_version_info{namespace="abby-ns"}) (sum(gitlab_runner_jobs{namespace="abby-ns"}) / sum(gitlab_runner_concurrent{namespace="abby-ns"})) / sum(gitlab_runner_version_info{namespace="abby-ns"}) ```