---
title: '整合PKS CI/CD與Spring Boot APP'
disqus: hackmd
---

整合PKS CI/CD與Spring Boot APP
===
本文件主要說明如何佈署最新一版的PKS CI/CD環境，以及和之前版本的差異，並以Spring Boot APP為例，執行於建置完的CI/CD流程

[TOC]

## 流程架構與先前版本的差異
藍色框：PKS Cluster
紫色框：Jenkins-Slave Pod
紅色框：Tomcat Image
1. 開發者Push Code到Gitlab
2. Gitlab觸發Webhook告知Jenkins發生Push Event
3. Jenkins在PKS上建立臨時的Jenkins-Slave Pod處理CI/CD
4. Jenkins-Slave下載CI/CD流程所須的Images，以及Gitlab上的Code
5. Jenkins-Slave利用Maven將Source Code包裝產生WAR檔
6. Jenkins-Slave將War檔及相關檔案利用Maven傳到SonarQube作分析
7. Jenkins-Slave以Tomcat為Base Image，利用Docker將第5步產生的WAR檔包裝進新的Tomcat Image
8. 將第7步Build完的Tomcat Image Push到Harbor
9. Jenkins-Slave透過K8S RESTful API佈署Tomcat Deployment
10. Tomcat Deployment Pull Harbor上對應的Tomcat Image並完成佈署

#### 版本差異

[第1版](https://drive.google.com/open?id=1M_w9Z1H-fWiijIQieCzV0ZRa_YoK0mOd)
流程沒什麼不一樣，較大差異在Maven內建於Jenkins Pod，因此為了要讓Jenkins內的WAR檔可被Jenkins-Slave共用，需要讓Jenkins-Slave掛跟Jenkins一樣的PV，但是vSphere的PV僅支援ReadWriteOnce，因此需要額外佈署NFS Server來提供ReadWriteMany的PV

[第2版](https://drive.google.com/open?id=1l66xhqM9Sgr7MuQUPbdB5JnDL6wTa3su)
將Maven移出Jenkins，因此環境不需額外佈署NFS並能支援較多Job同時執行

加入GPG加密，加密敏感性資料，但Jenkins log一樣能看到明碼

CD部份將原本使用CLI的方式改為透過K8S RESTful API來執行，減少懂K8s的開發人員可透過Jenkinsfile操作PKS Cluster的風險

[第3版](https://drive.google.com/drive/folders/1QE-Y2bjHdPJTnkTWhqA9Ukg6mBbFZhMK?usp=sharing)
將前兩版的Static PV改為Dynamic PV，方便環境佈署。

前兩版的所有服務並無考慮TLS，所有服務都是跑http，但一般環境都是跑https，因此第3版將所有服務都加上TLS進行整合

由於GPG對客戶的使用體驗太麻煩，因此將GPG拿掉，改由在Jenkins設定敏感性資料(不可視)，並使Jenkins log顯示遮閉後的結果

在了解Maven的Lifecycle後，拿掉前兩版多餘的步驟，減少CI/CD時間

加入JaCoCo，使SonarQube可顯示單元測試後的覆蓋率

#### Jenkinsfile
可從Jenkinsfile對照上述的流程與差異說明

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, cloud: 'kubernetes',
  containers: [
    containerTemplate(name: 'maven', image: 'argonhiisi/cicd-mvn-openjdk8', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'docker', image: 'docker:18.06', ttyEnabled: true),
    containerTemplate(name: 'curl', image: 'argonhiisi/curl-jq', ttyEnabled: true, command: 'cat')
  ],
  volumes: [
    hostPathVolume(hostPath: '/var/vcap/sys/run/docker/docker.sock', mountPath: '/var/run/docker.sock'),
    hostPathVolume(hostPath: '/etc/hosts', mountPath: '/etc/hosts')
  ]
) {
  node(label) {
    checkout scm
    def commitID = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

    stage('Maven project package') {
      container('maven') {
        sh "mvn org.jacoco:jacoco-maven-plugin:prepare-agent clean  package  -Dautoconfig.skip=true   -Dmaven.test.skip=false  -Dmaven.test.failure.ignore=true"
      }
    }

    stage('SonarQube') {
      container('maven') {
        wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: '${SONAR_TOKEN}', var: 'MASK_SONAR'],[password: '${SONAR_URL}', var: 'WORKER']]]){
        sh "mvn -U sonar:sonar -Dsonar.branch.name=master -Dsonar.host.url=${SONAR_URL} -Dsonar.login=${SONAR_TOKEN}"}
      }
    }


    stage('Build Image and Push to Harbor') {
      container('docker'){
        wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: '${HARBOR_PASS}', var: 'HARBOR']]]){
        sh """
        mkdir -p /etc/war/test && cp $WORKSPACE/target/postgres.war /etc/war/test && cp $WORKSPACE/server.xml /etc/war/test && cp $WORKSPACE/Dockerfile /etc/war
        cd /etc/war && docker build -t harbor.inwinstack.com/cicd/tomcatapp:${commitID} -f /etc/war/Dockerfile .
        docker login harbor.inwinstack.com -u admin -p ${HARBOR_PASS}
        docker push harbor.inwinstack.com/cicd/tomcatapp:${commitID}
        """}
      }
    }

    stage('Depoloy yaml to PKS') {
      container('curl'){
        sh "sed -i 's/version/${commitID}/g' $WORKSPACE/k8s/yaml/web-app.yaml"

        def deploymentName = "myweb"
        wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password: '${PKS_TOKEN}', var: 'TOKEN']]]){
        def WORKLOAD_EXIST = sh (
          script: """
          curl -X GET \
               -H "Authorization: Bearer ${PKS_TOKEN}" \
               --cacert secret/ca.crt \
               https://my-cluster2:8443/apis/apps/v1/namespaces/default/deployments/${deploymentName} | jq -r '.status'
          """,
          returnStdout: true
        ).trim()

        sh """
        sed -i 's/deploymentName/${deploymentName}/g' $WORKSPACE/k8s/api/update.sh
        if [ "${WORKLOAD_EXIST}" = "Failure" ]
        then
          echo "Workload not exist, create the workload"
          cd $WORKSPACE/k8s && sh api/create.sh ${PKS_TOKEN}
        else
          echo "Workload is exist, update workload"
          cd $WORKSPACE/k8s && sh api/update.sh ${PKS_TOKEN}
        fi
        """}

      }
    }
  }
}

環境佈署
---

#### 事前準備
佈署Gitlab、PKS、Harbor（在此使用公司內部DT2環境）

#### Image Build
下載[第3版](https://drive.google.com/drive/folders/1QE-Y2bjHdPJTnkTWhqA9Ukg6mBbFZhMK?usp=sharing)內的image.tar並解包

tar xvf image.tar

資料夾對應範例Image

| File | Image
| -------- | --------
| curl-base | appropriate/curl:latest
|curl-jq|argonhiisi/curl-jq
|jenkins|argonhiisi/jenkins-cert
|jnlp|argonhiisi/jnlp-slave
|maven|argonhiisi/cicd-mvn-openjdk8

其中jenkins和jnlp需要加入Gitlab的憑證，以下指令取得gitlab.example.com的憑證並覆蓋原資料夾下的mycert.crt

echo -n | openssl s_client -showcerts -connect gitlab.inwinstack.com:443 2>/dev/null| sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > mycert.crt

各Image有重新加入新憑證的需要重新Build Image並上傳至需要的映像倉庫

#### 佈署流程說明
下載[第3版](https://drive.google.com/drive/folders/1QE-Y2bjHdPJTnkTWhqA9Ukg6mBbFZhMK?usp=sharing)內的deploy.tar並解包

tar xvf deploy.tar

首先佈署Nginx Ingress

kubectl apply -f deploy/nginx-ingress/mandatory.yaml

kubectl apply -f deploy/nginx-ingress/lb-svc.yaml

準備SonarQube的憑證（若有需要Jenkins憑證，則同以下方式創建，在此創建環境為ubuntu）

執行以下指令創建工作環境

mkdir -m 0755 /etc/pki/mypbxCA

cd /etc/pki/mypbxCA

mkdir -m 0755 private/ certs/ newcerts/ crl/ 

建立 openssl 設定檔

cp /etc/ssl/openssl.cnf /etc/pki/mypbxCA/openssl.my.cnf

chmod 0600 /etc/pki/mypbxCA/openssl.my.cnf

touch /etc/pki/mypbxCA/index.txt

echo '01' > /etc/pki/mypbxCA/serial 

修改 openssl 設定，在 openssl.my.cnf 內增加以下設定(v3_ca是原本就有的)

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:true
subjectAltName = @alt_names

[alt_names]
DNS.1 = sonar.inwinstack.com

執行以下指令產生CA的憑證和私鑰
> 執行後會需要輸入CA資訊，重點在Common Name (eg, your name or your server's hostname)部份不要輸入要簽署的網站FQDN，在此個人輸入是Inwinstack CA

openssl req -config openssl.my.cnf -new -x509 -extensions v3_ca -keyout private/CA_ROOT.key -out certs/CA_ROOT.crt -days 36500

修改 openssl 設定
> 主要修改dir、certificate、private_key路徑和檔名，這是最後一個簽發步驟會用到的，在此順便先改

[ CA_default ]

dir             = .             # Where everything is kept
certs           = $dir/certs            # Where the issued certs are kept
crl_dir         = $dir/crl              # Where the issued crl are kept
database        = $dir/index.txt        # database index file.
#unique_subject = no                    # Set to 'no' to allow creation of
                                        # several ctificates with same subject.
new_certs_dir   = $dir/newcerts         # default place for new certs.

certificate     = $dir/certs/CA_ROOT.crt        # The CA certificate
serial          = $dir/serial           # The current serial number
crlnumber       = $dir/crlnumber        # the current crl number
                                        # must be commented out to leave a V1 CRL
crl             = $dir/crl.pem          # The current CRL
private_key     = $dir/private/CA_ROOT.key # The private key
RANDFILE        = $dir/private/.rand    # private random number file

x509_extensions = usr_cert              # The extentions to add to the cert

# Comment out the following two lines for the "traditional"
# (and highly broken) format.
name_opt        = ca_default            # Subject Name options
cert_opt        = ca_default            # Certificate field options

在 openssl.my.cnf 內增加以下設定（少的補上，原本就有的不用理他）

[ req ]
req_extetions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = sonar.inwinstack.com

建立sonarqube的私鑰和憑證簽署要求檔(CSR)
> 1. Country, State, Locality, Organization <這些基本資訊必須與 CARoot 憑證檔相同
> 2. Common Name：<輸入要簽發的網站名稱，必須是正確的 FQDN>
> 3. A challenge password []: <這裡不要輸入密碼，否則服務會無法自動重啟>

openssl req -config openssl.my.cnf -new -nodes -keyout private/sonar.key -out sonar.csr

簽署包含subjectAltName的sonarqube憑證
> 1. 簽署時，會要求輸入 CARoot 的金鑰密碼。
> 2. 建立 CSR 檔時所輸入的基本資訊必須與 CARoot 憑證的基本資訊相同，例如 Country, State, Locality,......，如果有一項不同，會無法簽署。
> 3. Certificate is to be certified until Jul 29 03:40:03 2013 GMT (365 days)
Sign the certificate? [y/n]: <輸入y，如果沒看此項，表示簽署不成功>
> 4. 1 out of 1 certificate requests certified, commit? [y/n] <輸入 y>

openssl ca -config openssl.my.cnf -in sonar.csr -out certs/sonar.crt -extensions v3_req

檢驗完成後的 Server 憑證檔

openssl x509 -in certs/sonar.crt -noout -text

結果如下（確認要有Subject Alternative Name）

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=TW, ST=Taiwan, L=Taipei, O=Inwinstack, OU=DT2, CN=Inwinstack CA/emailAddress=[e-mail]
        Validity
            Not Before: Jul 12 08:22:27 2019 GMT
            Not After : Jul 11 08:22:27 2020 GMT
        Subject: C=TW, ST=Taiwan, O=Inwinstack, OU=DT2, CN=sonar.inwinstack.com/emailAddress=[e-mail]
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d1:2b:6e:5f:ea:0a:e0:6e:8e:7a:dc:7f:ce:60:
                    df:ec:f9:cd:ce:8d:4a:8d:f5:fa:dd:a7:b5:05:87:
                    f4:da:6c:e2:cb:22:c3:77:b1:e0:4f:30:ba:cc:4c:
                    09:5a:52:44:c5:46:a3:45:e2:29:3c:f5:cd:97:3c:
                    6f:99:4b:d7:6d:a5:3d:b9:a4:70:b9:c2:be:2c:5d:
                    74:f8:75:67:97:27:1b:2b:2a:80:cc:47:c9:55:7b:
                    ed:33:de:cf:65:c7:fb:cc:e7:8f:da:0a:87:52:d7:
                    fe:1b:3c:4e:0d:f6:4a:dc:bb:5b:27:88:a9:d9:3f:
                    de:83:40:e7:17:9a:84:dc:7a:de:eb:84:29:67:5a:
                    62:ea:a7:25:07:bd:62:0e:b1:8a:29:eb:5a:6e:ee:
                    16:43:ea:67:77:89:c1:0c:93:23:80:69:9f:ad:05:
                    39:de:d1:a0:74:14:a2:d6:0e:2d:f6:ce:b0:da:34:
                    82:b5:a5:d8:2e:60:78:65:ce:1c:a3:2b:b0:31:a1:
                    71:c9:54:f6:6a:c6:d3:a2:98:77:1b:0f:67:73:c3:
                    33:08:d4:12:ad:58:60:7e:89:26:86:a4:26:72:93:
                    d8:d3:0e:d9:5b:d1:a7:0c:9a:ec:84:46:3f:4c:f2:
                    df:92:6f:7f:6c:bf:44:8e:6a:95:14:f3:0e:9c:63:
                    e5:e1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:sonar.inwinstack.com
    Signature Algorithm: sha256WithRSAEncryption
         82:12:5e:c6:e9:de:2e:41:b5:aa:a9:94:2a:55:a3:a5:19:b3:
         86:cf:d6:e1:62:97:cb:07:d2:5c:eb:7b:5f:16:9a:8a:77:03:
         ca:1d:7b:06:3b:b9:5c:6a:4b:64:15:07:05:97:ac:53:f9:ab:
         2e:b0:e3:77:56:00:d0:a0:90:d6:4c:75:38:e4:30:85:3d:3f:
         4e:3c:69:67:23:86:93:11:1c:64:1c:bf:12:22:16:7a:45:41:
         7f:42:38:db:fb:d6:e5:10:57:19:75:12:b2:83:b5:44:ab:a7:
         5e:36:c0:44:0a:18:3c:ae:46:14:e2:0a:eb:88:44:b7:35:b0:
         08:08:62:75:bd:8b:a5:bb:17:fe:f1:c4:a9:fc:ae:dc:cc:c3:
         88:9b:fc:eb:ed:cd:65:08:34:a7:9a:f8:13:16:30:98:32:bc:
         ca:f0:f0:aa:bf:d2:25:6b:0e:95:ba:a9:c9:5f:64:fa:f5:8e:
         31:5a:76:cf:4c:91:8e:8a:32:1f:76:ab:86:87:87:4a:95:76:
         65:2a:5b:3e:40:35:2a:59:e2:df:55:18:d4:c0:a8:84:23:d5:
         e2:42:e4:23:3f:f8:be:c4:10:a8:db:a1:25:70:66:08:40:0c:
         cd:c0:f0:48:85:18:3e:6c:21:85:2f:a9:55:61:7b:99:38:d2:
         3f:a0:83:14

建立憑證Secret

kubectl create secret tls jenkins-ssl-secret --key=deploy/jenkins-pks/ssl/server.key --cert=deploy/jenkins-pks/ssl/server.crt

kubectl create secret tls sonar-ssl-secret --key=deploy/sonar-pks/ssl/server.key --cert=deploy/sonar-pks/ssl/server.crt

佈署SonarQube

kubectl apply -f deploy/sonar-pks/pv

kubectl apply -f deploy/sonar-pks/k8s-yaml/postgres

kubectl apply -f deploy/sonar-pks/k8s-yaml/sonar

kubectl apply -f deploy/sonar-pks/ingress

完成後進去sonarqube pod（以下pod名稱自行修改），將branch plug-in移到extensions/plugins內

kubectl exec -it sonarqube-6ff66d767b-rdpn6 bash

mv branch-2.0.0.jar extensions/plugins/

exit

kubectl delete po sonarqube-6ff66d767b-rdpn6

完成後開啟SonarQube UI建立Token、User和Password，點選 *Administration* 確定有以下畫面，不然Pipeline會無法正確執行check SCM

佈署Jenkins
> 記得要先修改deploy/jenkins-pks/k8s-yaml/jenkins-deployment.yaml內的hostAliases為正確的對應

kubectl apply -f deploy/jenkins-pks/pv

kubectl apply -f deploy/jenkins-pks/k8s-yaml

kubectl apply -f deploy/jenkins-pks/ingress

完成後開啟UI，內部URL最後要設定為pks_worker_ip:NodePort（或是進入設定系統內修改Jenkins URL）

Jenkins須額外下載3個plug-in，下載方法之前的文件都有，在此不另外說明

1. Gitlab
2. Kubernetes
3. Mask Passwords Plugin

點選 *管理 Jenkins* 在 *設定全域安全性* 內取消勾選 *防範 CSRF 入侵*

點選 *管理 Jenkins* 進入 *設定系統* 在雲的部份新增k8s連線資訊，Name對應到Jenkinsfile的cloud名稱，可自行修改，設定如下(要加入SSL憑證請另外[參考](https://docs.google.com/document/d/1ZzIqkjbqxyKHO98AiW8AcVdqPLiZuivNCF-f2F-HAM0/edit?usp=sharing))


點選 *新增作業* 選取 *Pipeline*


勾選 *參數化建置* 點選 *新增參數* 選取 *密碼參數*

| 參數 | 說明 |
| -------- | -------- |
| SONAR_TOKEN | SonarQube產生的Token |
|SONAR_URL|http://pks_worker_ip:31002
|HARBOR_PASS | harbor的密碼
|PKS_TOKEN|PKS的Token，取得方法請[參考](https://docs.google.com/document/d/1VJakRVg0bn1Kf1H6Vp6GY3i2YiYC2q2L4l1YTvJey_M/edit?usp=sharing)


最後請參考該說明內的[Jenkins Configuration](https://hackmd.io/OCumh2jsTD-C7pdMSpJP6g?view#Jenkins-Configuration)在Jenkins產生Token讓Gitlab連接即可

範例說明
---
下載[第3版](https://drive.google.com/drive/folders/1QE-Y2bjHdPJTnkTWhqA9Ukg6mBbFZhMK?usp=sharing)內的mvntest.tar並解包

tar xvf mvntest.tar

詳細範例說明請[參考](https://docs.google.com/document/d/1cLiIYsS7BGaSjrhaTupda3Vw-e3aP8ubA-ah48imOWk/edit?usp=sharing)

檔案結構，紅框部份是在原始範例上新增的檔案


| 檔案 | 說明 |
| -------- | -------- |
| Dockerfile|Tomcat Image的Dockerfile|
|Jenkinsfile|Jenkins-slave執行的Pipeline Script
|create.sh|kubectl create的RESTful API
|update.sh|kubectl replace的RESTful API
|web-app.yaml|要佈署Tomcat Image的Deployment yaml
|web-svc.yaml|Tomcat deployment的service yaml
|ca.crt|從PKS取得的ca.crt，取得方法同樣[參考](https://docs.google.com/document/d/1VJakRVg0bn1Kf1H6Vp6GY3i2YiYC2q2L4l1YTvJey_M/edit?usp=sharing)
|server.xml|Tomcat的設定檔

> 一般的Spring Boot範例在佈署於Tomcat後無法正常顯示，主因在於mvn package後會將網頁及設定包在WEB-INF下，但Tomcat無法直接讀取WEB-INF內的資料，需修改專案初始產生的主程式(有@SpringBootApplication的檔案)，以範例程式為例，該檔案為PostgresPcfApplication.java

原檔案如下

package com.inwinstack;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class PostgresPcfApplication {

	public static void main(String[] args) {
		SpringApplication.run(PostgresPcfApplication.class, args);
	}

}

修改後如下，主要繼承SpringBootServletInitializer，另外覆寫 configure方法，加入java configuration，更多細節可[參考](https://coffee0127.github.io/blog/2017/04/26/convert-spring-boot-jar-to-war/)

package com.inwinstack;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;

@SpringBootApplication
public class PostgresPcfApplication extends SpringBootServletInitializer {
  
    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
        return application.sources(PostgresPcfApplication.class);
    }
    
	public static void main(String[] args) {
		SpringApplication.run(PostgresPcfApplication.class, args);
	}

}

範例佈署測試結果
---
全流程請參考[測試影片](https://drive.google.com/file/d/1wf4YdbCopEwh6MVj0bu38SpZsdJJWr6e/view?usp=sharing)

確認最終Jenkins log有遮閉重要資料

> 以下有兩個警告，第1個警告是建議密碼存在檔案內再Load進，不然有明碼洩漏問題，但由於在此我們使用Mask password plugin，因此沒有明碼問題，第2個警告由於該jenkins slave是暫時的pod且沒有掛PV，在Job完成後就會刪除，因此明碼洩漏問題風險很低


###### tags: `DevOps` `PKS` `CI/CD` `GitLab` `Jenkins` `Tomcat` `Demo` `操作流程` `Spring Boot` `SonarQube` `PostgreSQL` `Maven` `Java`