GitLab Runner

tags: `III` `DevOps` `CI/CD`

Add Runnger

Docker Container

Install

docker run -d --name vmme-gitlab-runner --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

# tree /srv
/srv
└── gitlab-runner
    └── config
        └── config.toml

Register

進入gitlab-runner container

gitlab-runner register -n \
  --url http://gitvepc/ \
  --registration-token Pyiyjywx3yqgMo6_QCMT \
  --executor docker \
  --config /etc/gitlab-runner/config.toml \
  --description "5GC Docker Runner" \
  --docker-image "docker:19.03.8" \
  --tag-list "5gc" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
  --docker-disable-cache=true

每 register 一次就會自動附加到 config.toml

Helm Chart

不要用 GitLab GUI 安裝，它會把 Helm Tiller 和 GitLab Runner 放在 gitlab-managed-apps namespace 底下，這個 K8s 就不能拿來做 deploy。

安裝順序：

透過 GitLab GUI 先裝 Helm Tiller
透過 GitLab GUI 再裝 GitLab Runner

官方 Repos

官方沒有 expose docker.sock

helm repo add gitlab https://charts.gitlab.io
helm search repo -l gitlab/gitlab-runner

package 存在 ~/.cache/helm/repository/

III Chart

我有 expose docker.sock

git clone http://gitvepc/abby/gitlab-runner --branch iii --single-branch

Install Release

以 chart 為模板，佈署 release，相應的 K8s resource 就此生成

helm install --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> --name <RELEASE_NAME> <gitlab/gitlab-runner或./gitlab-runner>

# <CONFIG_VALUES_FILE>
gitlabUrl: http://gitvepc/
runnerRegistrationToken: "tqkgDDxsVAJ4VJY6ukjN"
concurrent: 10

runners:
  image: docker:18.09
  privileged: true
  volumes:
    - type: "host_path"
      name: "docker"
      path: "/var/run/docker.sock"
      mount_path: "/var/run/docker.sock"

# IPs=$(kubectl get pod -o wide -n 5gc | awk 'NR > 1 {print $6}')
# curl $IPs:9252/metrics
metrics:
  enabled: true

rbac:
  create: true

hpa:
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metricName: gitlab_runner_jobs
        targetAverageValue: 150m

Upgrade Release

helm upgrade --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> --name <RELEASE_NAME> <gitlab/gitlab-runner或./gitlab-runner>

K8s 環境設定

kube-dns 要有 `nameserver 8.8.8.8` 跟 `search nmsi.pri`

CoreDNS
1. 直接永久加在改本地 /etc/resolv.conf
2. 重寫 gitvepc 為 gitvepc.nmsi.pri - kubectl edit cm coredns -n kube-system
3. 清掉 cache - kubectl get pods -n kube-system -o name | grep coredns | xargs kubectl delete -n kube-system

cluster 至少要有一個 node 允許裝 pod

kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-

enable remote tcp access to Docker daemons on all machines

當對同一個 container 分不同的 dind job 做 docker run, docker cp, docker rm, docker rmi，有可能會使 job 分散在不同機器上，就會找不到要被操作的 container，所以要 remote access 不同機器上的 Docker daemon。

enable live-restore

# vi /etc/docker/daemon.json
{
  "live-restore": true
}
# systemctl reload docker

edit docker.service

# vi /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock
# systemctl daemon-reload
# service docker restart

remote access

docker -H tcp://10.22.100.233:2375 ps

Assign K8s credential variables to enable kubectl

Troubleshooting

進入 gitlab-runner container

docker exec -it gitlab-runner bash

進入 runner-gitlab-runner pod

kubectl exec -it runner-gitlab-runner-7fc7d55f4b-6wvp2 -c runner-gitlab-runner -n gitlab-managed-apps -- sh

列出 repo

helm repo list

列出 release

helm list -n <NAMESPACE>

查看 release 的設定現狀

helm get all <RELEASE_NAME> -n <NAMESPACE>
helm get values <RELEASE_NAME> -n <NAMESPACE>

除錯 template 格式

helm template --debug <CHART_DIR>

config.toml

在 gitlab-runner pod 裡面

# /home/gitlab-runner/.gitlab-runner/config.toml
listen_address = ":9252"
concurrent = 10
check_interval = 30
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "buildenv-gitlab-runner-779d55d76c-kphqg"
  output_limit = 4096
  request_concurrency = 1
  url = "http://gitvepc/"
  token = "dQRerLmwh2j1E6wBLXyy"
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "docker:18.09"
    namespace = "5gc"
    namespace_overwrite_allowed = ""
    privileged = true
    poll_timeout = 180
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]
  [[runners.kubernetes.volumes.host_path]]
    mount_path = "/var/run/docker.sock"
    name = "docker"
    path = "/var/run/docker.sock"

在 gitlab-runner container 裡面

# /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Abby Docker Runner"
  url = "http://gitvepc/"
  token = "mX-FxXdVkshksUSDvDiw"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.8"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

K8s Monitoring

Metrics

因為 gitlab-runner pod 的 values.yml 預設已開啟 9252 port

metrics:
  enabled: true

可以確認 pod 有提供 metric

IPs=$(kubectl get pod -o wide -n 5gc | awk 'NR > 1 {print $6}')
curl $IPs:9252/metrics

所以 Prometheus 可以取得 metrics，以下是 PromQL

gitlab_runner_jobs: 某 gitlab-runner 目前正在執行 job 的數量
gitlab_runner_version_info: 有哪些 gitlab-runner pod
gitlab_runner_concurrent: 某 gitlab-runner 允許接到 job 的最大數量

HPA 觀察點

sum(gitlab_runner_jobs{namespace="abby-ns"})
sum(gitlab_runner_version_info{namespace="abby-ns"})
(sum(gitlab_runner_jobs{namespace="abby-ns"}) / sum(gitlab_runner_concurrent{namespace="abby-ns"}))  /  sum(gitlab_runner_version_info{namespace="abby-ns"})