### Use Bash shell script to make new resource type for Concourse CI ##### https://hackmd.io/@fourdollars/MakeResourceType --- ## Who am I? FourDollars aka $4 - Debian Developer / Ubuntu Contributing Developer / GNOME Emeritus Member / Canonical Employee - Vim :heart: [Plurk](https://www.plurk.com/fourdollars) ---- [LinkedIn](https://www.linkedin.com/in/fourdollars/) --- ## What is Concourse CI? https://concourse-ci.org/ ---- Use YAML to define the pipelines <iframe width="100%" height="500" src="https://ci.concourse-ci.org/" frameborder="0"></iframe> https://ci.concourse-ci.org/ ---- ## 一些使用上的條件限制 * 只支援 Docker container * 只支援 amd64/x86_64 架構 * pipeline 不支援 if else 分支 * 要熟悉 YAML 及 JSON 語法 * 學習曲線高 ([官方自己說的](https://concourse-ci.org/docs.html)) ---- 簡介請參考 [Introducing Concourse CI](https://hackmd.io/@fourdollars/introducing-concourse-ci) by $4 --- All following examples are available on [](https://github.com/fourdollars/concourse-demo/) --- ## Introduce WebDAV resource [](https://github.com/fourdollars/webdav-resource/)  ---- pipeline.yml ```yaml= resource_types: - name: webdav type: registry-image source: repository: ghcr.io/fourdollars/webdav-resource defaults: domain: webdav username: webdav password: webdav ssl: false resources: - name: demo-storage type: webdav icon: nas source: path: demo - name: ubuntu-focal type: registry-image icon: ubuntu source: repository: ubuntu tag: focal jobs: - name: check-demo-message plan: - get: ubuntu-focal - get: demo-storage trigger: true - task: check-message image: ubuntu-focal config: platform: linux inputs: - name: demo-storage run: path: cat args: - demo-storage/msg.log ``` ---- # Live Demo ---- 入門請參考 TOSSUG [2021/05/04 Concourse CI 幼幼班](https://hackmd.io/@tossug/rJNzwsFPd) --- ## Make a simple HTTP resource ---- ### Goal: Print the content of HTTP file when it changed. ```shell $ python3 -m http.server 9527 $ date > msg.log $ curl http://0.0.0.0:9527/msg.log ``` ```yaml resources: - name: message type: http icon: nas source: url: http://0.0.0.0:9527/msg.log ``` ---- ## How to generate the digest? ```shell $ curl --head http://0.0.0.0:9527/msg.log HTTP/1.0 200 OK Server: SimpleHTTP/0.6 Python/3.8.10 Date: Tue, 13 Jul 2021 14:12:18 GMT Content-type: application/octet-stream Content-Length: 53 Last-Modified: Tue, 13 Jul 2021 13:59:01 GMT $ curl --head http://0.0.0.0:9527/msg.log | sha256sum ec8af3a926d2fe396d9b8d1d0e7a077bfa064937a9d02158321d1... ``` ---- ## This digest is unreliable. ```shell $ curl --head http://0.0.0.0:9527/msg.log | sha256sum efdaa9fc9f4b5f5d6d0106500621e7c5193efd1288785040809e0... $ curl --head http://0.0.0.0:9527/msg.log | sha256sum a326c45b04b5a756906882cdac6477689a72fed551db8afb6d565... $ curl --head http://0.0.0.0:9527/msg.log | sha256sum 8ab03fab7a6d5a86a290f54aa5b32978e0240517f757ae50a9cd6... ``` ---- ## Generate the reliable digest Find out the **immutable** parts ```shell $ curl --head http://0.0.0.0:9527/msg.log | \ grep -e ^Last-Modified -e ^Content-Length Content-Length: 53 Last-Modified: Tue, 13 Jul 2021 13:59:01 GMT $ curl --head http://0.0.0.0:9527/msg.log | \ grep -e ^Last-Modified -e ^Content-Length | sha256sum c3327a0c5bccfb7524f320bea52861c56542c9fd5e56603e3825c83... ``` --- ## Let's make the simple HTTP resource ---- ### Dockerfile ```dockerfile FROM alpine:latest RUN apk update RUN apk upgrade RUN apk add bash jq curl # check step ADD /check /opt/resource/check # put step ADD /out /opt/resource/out # get step ADD /in /opt/resource/in # make sure they are executable RUN chmod +x /opt/resource/* ``` * bash: deal with shell script * jq: deal with JSON * curl: deal with HTTP ---- ### Trinity: check / in / out ```shell $ ln -s in check $ ln -s in out $ vim in ``` Template ```bash= #!/bin/bash # Bash strict mode set -euo pipefail IFS=$'\n\t' # make stdout available as fd 3 for the result exec 3>&1 # redirect all output to stderr for logging exec 1>&2 # read the json from /dev/stdin and save to /tmp/input.json jq -M -S . < /dev/stdin > /tmp/input.json # '-M' monochrome (don't colorize JSON); # '-S' sort keys of objects on output; case "$0" in ('/opt/resource/check') json='[]' ;; ('/opt/resource/in') cd "$1" # /tmp/build/get json='{}' ;; ('/opt/resource/out') cd "$1" # /tmp/build/put json='{}' # noop ;; esac # output the json to fd 3 jq -n "$json" >&3 # '-n' Don't read from stdin ``` ---- ### Read the url from json ```bash # '-S' sort keys of objects on output; url=$(jq -r .source.url < /tmp/input.json) case "$0" in ``` ```yaml resources: - name: message type: http icon: nas source: url: http://0.0.0.0:9527/msg.log ``` ---- ### Generate the reliable digest in check step ```bash=17 case "$0" in ('/opt/resource/check') sha256=$(curl --head "$url" \ | grep -i -e ^Last-Modified -e ^Content-Length \ | sha256sum | awk '{print $1}') json=$(cat <<ENDLINE [ { "digest": "sha256:$sha256" } ] ENDLINE ) ;; ``` ---- ### Download the file in get step ```bash=21 ('/opt/resource/in') cd "$1" # /tmp/build/get curl --dump-header /tmp/header.log "$url" > "$(basename "$url")" sha256=$(grep -i -e ^Last-Modified -e ^Content-Length /tmp/header.log \ | sha256sum | awk '{print $1}') length=$(grep -i -e ^Content-Length /tmp/header.log \ | awk '{print $2}' | tr -d '\r') modified=$(grep -i -e ^Last-Modified /tmp/header.log \ | cut -d ' ' -f 2- | tr -d '\r') json=$(cat <<ENDLINE { "version": { "digest": "sha256:$sha256" }, "metadata": [ { "name": "url", "value": "$url" }, { "name": "Last-Modified", "value": "$modified" }, { "name": "Content-Length", "value": "$length" } ] } ENDLINE ) ;; ``` ---- 實作請參考 [1.9.1 Implementing a Resource Type](https://concourse-ci.org/implementing-resource-types.html) --- ## Let's build the Docker image for the simple HTTP resource  ---- #### pipeline.yml ```yaml= resources: - name: concourse-demo type: git source: uri: https://github.com/fourdollars/concourse-demo branch: coscup - name: resource-image type: docker-image source: repository: registry:5000/simple-http-resource username: registry password: registry insecure_registries: [ "registry:5000" ] jobs: - name: build-simple-http-resource plan: - get: concourse-demo trigger: true - put: resource-image params: build: concourse-demo/simple-http-resource ``` ---- # Live Demo --- ## Let's use the simple HTTP resource  ---- #### pipeline.yml ```yaml= resource_types: - name: simple-http type: docker-image source: repository: registry:5000/simple-http-resource username: registry password: registry insecure_registries: [ "registry:5000" ] resources: - name: message icon: nas type: simple-http source: url: https://people.debian.org/~fourdollars/coscup-2021.log - name: ubuntu-focal type: registry-image icon: ubuntu source: repository: ubuntu tag: focal jobs: - name: check-message plan: - get: ubuntu-focal - get: message trigger: true - task: check image: ubuntu-focal config: platform: linux inputs: - name: message run: path: cat args: - message/coscup-2021.log ``` ---- # Live Demo --- # The End ## Q&A