:::success
# OT Project - DevSecOps in GitLab CI/CD platform
**Authors: Ivan Okhotnikov, F P**
:::

## Annotation

For this project, we decided to revise the third DevSecOps lab using the GitLab platform, instead of Jenkins. First of all, we are interested in comparing the capabilities of GitLab CI/CD with the capabilities that we previously used in Jenkins. And we want to check if they are all available in GitLab CI/CD. Considering the fact that Jenkins and GitLab CI/CD are the most popular and competing continuous integration projects, we understand that none of these tools can definitely replace the other. Everything will depend on the needs of the company and the system being developed.

Thus, during this project, we will work with the GitLab CI/CD tool on the goals set in the third laboratory work, namely:

1. To consider, based on the laboratory work, the possibility of using similar tools for the Gitlab CI/CD:
1.1. Explore visualization modules for tests in GitLab CI/CD.
1.2. Integrate Snyk with Gitlab CI/CD and find project vulnerabilities.
1.3 Make deploy with Ansible Role.
2. Create notifications via Telegram about the result of pipeline execution.

## Jenkins VS GitLab

For a more complete understanding of the purpose of the project, we will first determine why GitLab CI/CD is a direct competitor to Jenkins. Below you can see a comparison table with the different characteristics of these two projects:

[Jenkins and Git Comparison Table](https://docs.google.com/spreadsheets/d/1922-VAi6O21Oo50KfhI03VilZNnHkMQZ4IWLoQAGvQM/edit?usp=sharing)

There are also alternative CI/CD projects, but they are much less common in practice than GitLab CI/CD and Jenkins, so we do not consider them in this project.

## General Part ot DevSecOps lab

Using your own runner is much safer than using a public one (there is a high probability that there may be critical vulnerabilities in it). Therefore, we will start by registering our runner:

sudo gitlab-runner register --url https://gitlab.com/ --registration-token $REGISTRATION_TOKEN


Next, a custom "lichi" tag was created for the runner, now we can start writing the pipeline in gitlab-ci.yaml


variables:
  DOCKER_REPO_NAME: "ot-project"
  ANSIBLE_IMAGE: "ansible/container-conductor-alpine-3.5:0.9.3rc4"
  KANIKO_IMAGE: "gcr.io/kaniko-project/executor:debug"

.build-only: &build-only
  only:
    - test
    - main

.deploy-image: &deploy-image
  only:
    - main

.setup_ssh: &setup_ssh >
  echo "Setup SSH" &&
  mkdir -p ~/.ssh &&
  echo "$DEPLOY_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/deploy &&
  chmod 600 ~/.ssh/deploy &&
  echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config

.deploy: &deploy >
  echo "Deploy application" &&
  chmod 700 .deploy &&
  cd .deploy/ &&
  ansible-playbook deploy.yml
  --private-key="~/.ssh/deploy"
  -i "environments/$ANSIBLE_INVENTORY"
  -e docker_registry_url="\"$DOCKER_REGESTRY_URL\""
  -e docker_registry_project="\"$DOCKER_REGESTRY_PROJECT\""
  -e docker_backend_repo_name="\"DOCKER_REPO_NAME\""
  -e docker_registry_creds_username="\"$DOCKER_REGESTRY_USERNAME\""
  -e docker_registry_creds_password="\"$DOCKER_REGESTRY_PASSWORD\""
  -e app_version="\"$APP_VERSION\""

Initially, we determine the path of collecting artifacts (path `/target`, all the results are sent there). And we define the main stages. We use caching so that in the future it will be possible to access artifacts that were created by previous jobs.

For example, Apache Tomcat is used as a web server to run a web application, so we will build a docker image based on it. At the build step, an executable WAR file is created that will be used in Apache Tomcat. This file will be an artifact that will be cached.

Or when we run tests, for example, integration tests. To use these tests later in static analysis, first we need to perform an analysis, get an artifact, convert it: first to an `xml` file, then to a `json` file - therefore, all this is combined into one analysis stage. This stage will convert the received artifact and send it for dockerization. Then we proceed to the deployment step to deploy it to the appropriate environment.

cache:
  paths:
    - target/

stages:
  - build
  - test
  - analyze
  - docker-build
  - deploy

In this picture you can see the process of writing artifacts to the cache from the target folder.


Having defined the main stages, we can now describe them. Here, in code below, you can see that the image is being used `maven:3.8-openjdk-11`, on the basis of which the corresponding Docker container will be launched, inside which the designated scripts will be executed (for example, installation without tests).

Now let's follow the tasks of the main part of the laboratory work:

> 1. Create the first job to fetch and build the Maven project. Note: skip the testing in this job

build:
  tags:
    - "lichi"
  <<: *build-only
  image: maven:3.8-openjdk-11
  stage: build
  script:
    - mvn install

> 2. Create the next job to perform the unit tests of the project.

unit-tests:
  tags:
    - "lichi"
  <<: *build-only
  image: maven:3.8-openjdk-11
  stage: test
  script:
    - mvn test
  artifacts:
    paths:
      - target/jacoco.exec
    reports:
      junit:
        - target/surefire-reports/TEST-*.xml

In this screenshot you can see a report on the execution of unit tests. For the visualization of which the JUnit module was used, which is built into the gitlab.


> 3. Create the next job to perform the integration test on the project.

integration-tests:
  tags:
    - "lichi"
  <<: *build-only
  image: maven:3.8-openjdk-11
  stage: test
  script:
    - mvn verify -Psurefire

> 4. Create the next job to perform static code analysis on the project

To solve this problem, a custom script was used (it is presented ain the end of this subtask), which converts an artifact created after executing `checkstyle` from xml format to json specifically for the CodeQuality module.

stat-analyze:
  tags:
    - "lichi"
  <<: *build-only
  image: maven:3.8-openjdk-11
  stage: analyze
  script:
    - mvn checkstyle:checkstyle
    - bash checkstyleconvert.sh ./target/checkstyle-result.xml > checkstyle-report.json
  artifacts:
    reports:
      codequality: checkstyle-report.json

After we perform a static analysis of the code, with the checkstyle tool, it will be possible to visualize the result.


It can be concluded that it is not necessary to have Jenkins to solve this problem, it is enough just to have your own image. Each company sets up alerts and errors independently, you can specify this in the secrets of the project itself and not dive into the code. The resulting file with errors is the same xml, json, which together are artifacts. And based on what errors occur, it will not be a problem for developers to write their own analyzer for their specific case, it will suit much more than a universal module. You can visually see the number of errors that have occurred in the pipeline itself.

> 5. Create the next job to dockerize your artifact from target/****.war and push it to the docker hub.

docker-build:
  tags:
    - "lichi"
  <<: *deploy-image
  stage: docker-build
  image:
    name: $KANIKO_IMAGE
    entrypoint: [""]
  script:
    - echo "Run image backend building process"
    - echo "{\"auths\":{\"$DOCKER_REGESTRY_URL\":{\"username\":\"$DOCKER_REGESTRY_USERNAME\",\"password\":\"$DOCKER_REGESTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - >
      /kaniko/executor
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/$DOCKERFILE
      --destination $DOCKER_REGESTRY_URL/$DOCKER_REGESTRY_PROJECT/$REPO_NAME:latest
      --cache=true;
  variables:
    DOCKERFILE: Dockerfile
    REPO_NAME: $DOCKER_REPO_NAME

Variables created for secret data transfer during deployment. Since our repository is public, this is an additional security measure.


> 6. Create the next job to deploy the docker image from the docker hub to the Jenkins agent and validate/show that you can access the web app.

deploy:
  tags:
    - "lichi"
  <<: *deploy-image
  stage: deploy
  image: $ANSIBLE_IMAGE
  before_script:
    - *setup_ssh
  script:
    - *deploy
  environment:
    name: staging
  variables:
    ANSIBLE_INVENTORY: test
    DEPLOY_SERVER_PRIVATE_KEY: $SSH_DEPLOY_KEY
    APP_VERSION: $CI_COMMIT_SHORT_SHA

> 7. Finally, chain all the jobs such that when you run the 1st job it should execute the rest.

This point can be counted by definition, since this is the main function of Gitlab CI/CD. Below you can see its visualization:


In addition to the lab work, ansible role was used for deployment (ansible image can be found in `gitlab-ci.yaml`), so we were able to deploy the web application on a web server with a public address, it can be accessed by any client at the following address:

http://95.214.62.65:8081/vprofile-v2/login

A special script was used to convert the checkstyle report to json:

checkstyleconvert.sh

#!/bin/bash

function mapSeverity() {
    severity="$1"

    if [ "$severity" == "error" ]; then
        printf "critical"
    elif [ "$severity" == "warning" ]; then
        printf "major"
    elif [ "$severity" == "info" ]; then
        printf "minor"
    else
        printf "info"
    fi
}

function err() {
    printf "%s\n" "$*" >&2
}

file=""
first=1
path="$(pwd)"
printf "[\n"
tail -n +3 $1 | while read p; do
    if printf "$p" | grep -q "<file name.*" -; then
        file="$(realpath --relative-to "${PWD}" "$(expr "$p" : '<file name=\"\(.*\)\".*')")"
        err Processing checkstyle results for "$file"
    fi
    if printf "$p" | grep -q "<error.*" -; then
        line="$(expr "$p" : '.*line=\"\([0-9]*\)\".*')"
        message="$(expr "$p" : '.*message=\"\(.*\)\" source.*' | sed -e 's/&apos;/`/g' -e 's/&lt;/</g' -e 's/&gt;/>/g' -e 's/&quot;/\\\"/g' -e 's/&amp;/\&/g')"
        severityCheckstyle="$(expr "$p" : '.*severity=\"\(.*\)\" message.*')"
        severity=$(mapSeverity "$severityCheckstyle")
        checksum=$(printf "%s %s %s\n" "$file" "$line" "$message" | sha1sum | awk '{print $1}')
        if [ "$first" == 1 ]; then
            printf '{ "description": "%s", "severity": "%s", "fingerprint": "%s", "location": { "path": "%s", "lines": { "begin": %d } } }\n' "$message" "$severity" "$checksum" "$file" "$line"
            first=0
        else
            printf ',{ "description": "%s", "severity": "%s", "fingerprint": "%s", "location": { "path": "%s", "lines": { "begin": %d } } }\n' "$message" "$severity" "$checksum" "$file" "$line"
        fi
    fi
done
printf "]\n"

Thus, at the end of the first part, we can conclude that this laboratory work (its main part) is really quite simple when using GitLab CI/CD, whereas Jenkins requires much more time to install and create jobs. Gitlab is a version control system that contains the possibility of implementing CI/CD on board, while Jenkins is not for version control, but for automating assemblies, so its configuration takes longer than connecting the GitLab module. A similar analogue is GitHub Action. Despite the fact that Jenkins is incredibly flexible, Gitlab is much simpler.

By this we want to say that DevSecOps is a story not only about big companies, but also about small ones. At the same time, GitLab is a story of how to give security to companies that do not have sufficient resources incliding to hiring developers.

## "Snyk" implementation

> Snyk is a tool that checks dependencies for many ecosystems based on the author's database. It also provides opportunities to search for vulnerabilities, check GitHub repositories, check open source packages before use, integrate into CI processes, update or fix vulnerable dependencies, and much more.

The screenshots below show in detail the process of integrating Snyk into the application/repository. The tool itself is written in Node.JS, and you can log in to the site through several services — GitHub, Google, Bitbucket, Azure AD & Docker ID.

Generate a Personal Access Token in GitLab. Далее необходимо выбрать GitLab в качестве Source control.


Added our account credentials and the token we just generated to the GitLab integration settings area in Snyk.


To integrate with GitLab, as a Snyk admin user, created a personal access token that allows us to use the api area for access.


Now we can select the necessary project that will work with Snyk:



Now Snyk is working with our project. The main idea of Snyk is precisely the warning of the developer about dependencies containing vulnerabilities. So, Snyk can offer two options for action:
* Opening a fix Merge Request: generating a Snyk merge request with the minimal changes needed to fix the vulnerabilities affecting the project.
* Fixing this vulnerability: generating a Snyk merge request that fixes only this vulnerability.

However, it seems important to mention here that Snyk does not assume the elimination of vulnerabilities in the application itself, it explores (based on its database) specifically unrecognizable dependencies and offers to fix/update them. So the main idea of the reports received from Snyk is to keep all dependencies updated.


Here you can see an example of one of these vulnerable dependencies:
[org.springframework:spring-beans](https://snyk.io/vuln/maven:org.springframework%3Aspring-beans)

It is very convenient that an open database allows you to investigate what exactly is wrong with this version of the dependency. In our case, outdated Spring Beans can cause attacks such as Denial of Service (DoS), Remote Code Execution, Arbitrary Code Execution.


Next, we tried to fix the existing vulnerabilities. We want to say right away that not all Snyk fixes were workable, some caused critical errors in the operation of the papline.


Forced fix with code edits:


Here you can also see the result after executing the merge request with the fix. Initially, there were 12 critically vulnerable dependencies that could cause arbitrary or remote code execution or a DOS attack.


After the fix, there were other 2 Docker File dependencies recommended for elimination, but we did not consider them so critical.


It can be concluded that Snyk is a simple tool with good potential, now it is also quite popular. This is due to the fact that it allows you to monitor the appearance of new vulnerabilities in open source packages, this is especially useful for large projects where developers do not physically have the ability to monitor the status of dozens of dependencies.

Additionally, I would also like to mention the new Snyk Code tool, which allows you to test the application source code directly for vulnerabilities. This is a very convenient solution largely due to the fact that it uses SAST as part of the development process, enabling developers to build software securely during development, and not trying to find and fix problems after the code is compiled. However, there is a risk that after such automatic edits, the work of the project will be disrupted.

> Snyk Code utilizes a semantic analysis AI engine that learns from millions of open-source commits and is paired with Snyk’s Security Intelligence database--this creates a continually growing code security knowledge base, which reduces false positives to near-zero and provide actionable findings that matter.

As you can see, Snyk immediately displays both the vulnerable line of code and recommendations for its changes.


Among other things, it finds other types of code vulnerabilities, for example, here we found the credentials of a test user in open form.


Snyk can also be integrated into the pipeline as one of the stages. As a result, we could have access to a command line from which we could manage Snyk and its "roles":

1. Exposing vulnerabilities (snyk monitor)
1. Using Snyk as a gatekeeper (snyk test)
1. Continuous monitoring (snyk test and snyk monitor)

In our project, we missed the opportunity to do this, however, thanks to the research of this topic after a comment during the presentation, we realized that Snyk is not as simple as it seems. It really has a lot of possibilities beyond the banal updating of package versions.

## Additional feature implementation

As an additional feature, we chose to add a Webhook, with which we can quickly receive notifications about changes in the CI assembly. The [Intergram](https://integram.org/) chatbot provides a link to a webhook.


Then we select the events that will be noticed by the bot. By default, web interceptors subscribe only to a push event, but in the future we can configure them as you like.


In case there is any errors. For example, an error occurred at the docker-build step, and we can quickly access the pipeline using the link:


And here is the result of a successful pipeline:


## Conclusion

During this project, we realized that it is really easier to do most of the lab work on the GitLab platform, however, in the case of integrating third-party projects such as Snyk, additional authorization and import of the repository is necessary. Whereas in Jenkins, Snyk is represented simply by an additional plugin. Thus, gaining time in setting up the tool itself and creating a pipeline, we lose it on additional integrations, and they are often no less important parts of the project.

However, intermediate conclusions were made in the body of the report itself, which also hint that not everything is so clear.