:::success
# OT Lab 3 - DevSecOps
:::

## Task 1: Set up your environment

:::info
1. Create two VMs, for the Jenkins server and its agent.
a. Install Docker Engine on both VMs.
:::

I will use two simplest machines with Ubuntu 20.04. To begin with, we will install what is necessary for both machines:

#java
sudo apt-get update
sudo apt-get install openjdk-11-jdk

#docker
sudo apt-get update
sudo apt install docker.io
sudo apt-get install ca-certificates 

#maven
sudo apt install maven

Now what is needed on the **server**:

wget –q –O – https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add jenkins.io.key
sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins
sudo systemctl enable jenkins
sudo systemctl start jenkins

<center>

Figure 2 - Versions

Figure 3 - Jenkins home page
</center>

Now you need to add an agent (client) to Jenkins. To begin with, I will create SSH keys on the client side, so that after adding them to Jenkins for authorization:

cd ~/.ssh
ssh-keygen -t rsa -m PEM -C "Jenkins Agent Key" -f "jenkinsAgent_rsa"
cat ~/.ssh/jenkinsAgent_rsa.pub >> ~/.ssh/authorized_keys

To create a client, go to Jenkins Management, then select Manage Nodes and Clouds and add a new node. It will require the IP address of the host and a private key.

<center>

Figure 4 - Log records about adding a new node
</center>

<center>

Figure 5 - Checking Docker "Hello World"
</center>

:::info
2. Isolate your Jenkins server behind a reverse proxy and access it through port 80 (e.g Nginx Reverse Proxy).
:::
We need to install the nginx server on the server, then refer to the example of the configuration file in the Jenkins documentation. I will use it by removing unnecessary comments.

#on server
sudo apt install nginx

#sudo nano /etc/nginx/sites-enabled/default


upstream jenkins-server {
  keepalive 32;
  server 127.0.0.1:8080;
}

server {
  listen          80;
  
  access_log      /var/log/nginx/jenkins.access.log;
  error_log       /var/log/nginx/jenkins.error.log;

  location / {
      proxy_pass         http://jenkins-server;
      proxy_redirect     default;

      proxy_max_temp_file_size 0;

    
      client_max_body_size       10m;
      client_body_buffer_size    128k;

      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      proxy_buffering            off;
      proxy_request_buffering    off;
      proxy_set_header Connection "";
  }
}

Now you need to add the appropriate restrictions in the firewall rules.

sudo ufw deny 8080
sudo ufw allow 80
sudo ufw enable

As you can see in the screenshot below, Jenkins is no longer available via port 8080.

<center>

Figure 6 - Result
</center>

:::info
3. You will be working on the following Maven project. Go through it and familiarize yourself (`git clone https://github.com/HamidullahMuslih/ot-lab-sdlc.git`).
Note: the project is written in Springboot (Backend) and Bootstrap (Frontend) and already contains unit test cases.
:::

I have already initially installed maven on both virtual machines. Now I need to add the appropriate plugin.

<center>


Figure 7 - Maven plugin
</center>

:::info
4. Read and explain in one line sentence the Maven lifecycle. Understand commands for each phase (e.g build, unit test, integration test, packaging and etc.) since you will need to use them during the pipeline tasks.
:::

Maven will first validate the project, then will try to compile the sources, run those against the tests, package the binaries, run integration tests against that package, verify the integration tests, install the verified package to the local repository, then deploy the installed package to a remote repository:

1. **validate** - validate the project is correct and all necessary information is available
1. **compile** - compile the source code of the project
1. **test** - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed
1. **package** - take the compiled code and package it in its distributable format, such as a JAR.
1. **verify** - run any checks on results of integration tests to ensure quality criteria are met
1. **install** - install the package into the local repository, for use as a dependency in other projects locally
1. **deploy** - done in the build environment, copies the final package to the remote repository for sharing with other developers and projects.

## Task 2: Implement DevSecOps Pipeline

:::info
1. Create the first job to fetch and build the Maven project. Note: skip the testing in this job.
:::

To create the first job, you need to create a new item, specify its name and select **Maven project** as the source. Next, you need to specify the appropriate settings (thanks for the branch `/main` :D).

<center>


Figures 8, 9 - Settings
</center>

**mvn clean install** - as a basic startup, it essentially launches my project from scratch. After that, I run the build and get the following positive result. Although this command does many times more than is necessary for the first project, but it saves my project to a local repository. ([source](https://stackoverflow.com/questions/16602017/how-are-mvn-clean-package-and-mvn-clean-install-different))

<center>

Figure 10 - Success
</center>

:::info
2. Create the next job to perform the unit tests of the project.
:::

Since the last task will be to collect the entire chain of tasks, I decided that it would be convenient to use pipeline for these purposes. I tried to study the post-build options in the Maven project, but I didn't quite understand how to implement them as separate tasks, rather than full-fledged projects.

<center>

Figure 11 - New Pipeline
</center>

To work with the pipeline, I will use examples of scripts from the [Jenkins documentation](https://plugins.jenkins.io/pipeline-maven/).

pipeline{
    agent any
    stages{
        stage("Getting a repository"){
            steps{
                git branch: 'main', url: 'https://github.com/HamidullahMuslih/ot-lab-sdlc.git'
            }
        }
        stage("Building"){
            steps{
                sh "mvn package -Dmaven.test.skip"
            }
        }
        stage("Testing"){
            steps{
                sh "mvn test"
            }
        }
    }
}

<center>

Figure 12 - Dashboard of 2.2 task
</center>

For example, in the console output section, I looked at how tests are skipped at one of the build stages.

<center>

Figure 13 - Example of "-Dmaven.test.skip"
</center>

:::info
3. Create the next job to perform the integration test on the project.
:::
To be honest, most of the information has to be taken from the forums, since the Maven documentation is more descriptive than practical. As I understand it, I need a verify phase, which is part of a dual lifecycle. [source](https://stackoverflow.com/questions/66876221/what-is-the-difference-between-mvn-verify-vs-mvn-test)

> Conclusion: if you want to run your integration tests and check it, use verify. If you only want to run unit tests, use test.

> My personal advice: if in doubt, use verify.

pipeline{
    agent any
    stages{
        stage("Getting a repository"){
            steps{
                git branch: 'main', url: 'https://github.com/HamidullahMuslih/ot-lab-sdlc.git'
            }
        }
       stage('Build & Unit test'){
         steps{
        // Perform only build and unit test with skip the integration testing
         sh 'mvn clean verify -DskipITs=true';
              }
            }
      stage ('Integration Test'){
        steps{
        // Perform integration testing and skip unit testing
        sh 'mvn clean verify -Dsurefire.skip=true';
            }
      }
  }
}

You can see the logs of each step of the assembly below.
<center>

FIgure 15 - Result
</center>

:::info
4. Create the next job to perform static code analysis on the project (e.g with the CheckStyle, PMD, FindBugs tools).
Note: Read about Warnings Next Generation Plugin.
a. Show the reports and share your understanding.
:::

First, install the **Warnings Next Generation Plugin**. The [plugin](https://www.jenkins.io/doc/pipeline/steps/warnings-ng/) collects compiler warnings or problems reported by static analysis tools and visualizes the results.

pipeline{
    agent any
    stages{
        stage("Getting the repository"){
            steps{
                git branch: 'main', url: 'https://github.com/HamidullahMuslih/ot-lab-sdlc.git'
            }
        }
        stage("Building without tests"){
            steps{
                sh "mvn package -DskipTests"
            }
        }
        stage("WNGP scanForIssues"){
            steps{
                sh " mvn checkstyle:checkstyle findbugs:findbugs"
            }
        }
      stage("WNGP recordIssues"){
            steps{
                recordIssues(tools: [checkStyle(), findBugs()])
                         }
        }
    }
}

While I was looking for the rules for the gate, I also installed the Git Forensics plugin, thanks to it, you can see who the author of all the displayed errors is. `checkstyle:checkstyle` is a reporting goal that performs Checkstyle analysis and generates a report on violations. FindBugs looks for bugs in Java programs.

<center>

FIgure 16 - General view
</center>

Here you can see statistics on errors, warnings, and other things that plugins could find. Also, it is separately visible who wrote them in the git code. The Git Forensics plugin will collect commit statistics for all repository files in the style of Code as a Crime Scene. For all files an additional detail view is available, that shows the added and deleted lines by each commit.

<center>



FIgures 17, 18, 19 - Dashboards
</center>

:::info
4. Create the next job to perform static code analysis on the project
b. Feed the static code analysis reports to the Violations Plugin and set the quality gates for the number of bugs in one of the reports, validate your work to fail/unstable the job.
Note: after validation remove the quality gate restriction since it will fail the last task of the lab.
:::

For this task in the maven project, I created a goal in the form of "`clean install checkstyle:checkstyle`" to build a project with pre-deletion. I added the Violations plugin, in the settings of which you can select the threshold for the checkstyle plugin. At a value of 500, the plugin will stop the build and give a failure.

<center>

Figure 20 - Violations settings
</center>

Thus, the number of elements detected by checkstyle exceeded the maximum possible limit.

<center>

Figure 21 - Result
</center>

:::info
5. Create the next job to dockerize your artifact from target/****.war and push it to the docker hub.
:::

For this task, I created fake accounts in DockerHub and GitHub, so it will be called just a set of letters. To add it to the Jenkins credentials, you need to go to the credentials configuration, select global and go to create new credentials. Next, I entered my password and login from the docker hub, and Jenkins generated an ID code for me.

<center>

Figure 22 - New credentials
</center>

Below you can see the standard pipeline for launching the image.

pipeline{
    environment {
        REGISTRY = "xdcfgvhbjnk/lab-ot3"
        DOCKERHUB_CREDENTIALS=credentials('85a691e7-fe8a-4109-8ed0-f7e23c15f2e7')
      }
    agent any
    stages{
        stage("Getting the repository"){
            steps{
                git branch: 'main', url: 'https://github.com/xdcfgvhbjnk/ot-lab-sdlc'
            }
        }
        stage("Building without tests"){
            steps{
                sh "mvn package -DskipTests"
            }
        }
        stage('Building image') {
            steps{
                sh 'docker build -t $REGISTRY:latest .'
            }
        }
        
        stage('Docker login') {
          steps{
            sh 'echo $DOCKERHUB_CREDENTIALS_PSW | docker login -u $DOCKERHUB_CREDENTIALS_USR --password-stdin'
          }
        }
        stage('Deploy Image') {
          steps{
            sh 'docker push $REGISTRY:latest'
          }
        }
    }
    
    post {
		always {
			sh 'docker logout'
		}
	}
}

Before that, I needed to create a fork to the source repository, and then create a new Dockerfile with the following content into it:

FROM tomcat:9.0
COPY target/*.war /usr/local/tomcat/webapps
CMD ["catalina.sh", "run"]

Also we need to give the agent user access:

sudo chown jenkins:jenkins /var/run/docker.sock

After that, you can run the pipeline and see the build result. Similarly, after updating the dockerhub, you can see that we have done a push in the docker registry.

<center>

Figure 23 - Result
</center>

<center>

Figure 24 - Changes in the repository
</center>

:::info
6. Create the next job to deploy the docker image from the docker hub to the Jenkins agent and validate/show that you can access the web app.
:::

We use the same config, except that now we need to call Agent Jenkins himself.

pipeline{
    environment {
        REGISTRY = "xdcfgvhbjnk/lab-ot3"
        DOCKERHUB_CREDENTIALS=credentials('85a691e7-fe8a-4109-8ed0-f7e23c15f2e7')
      }
    agent {
        label 'jenkins-client'
    }
    stages{
        stage('Docker login') {
          steps{
            sh 'echo $DOCKERHUB_CREDENTIALS_PSW | docker login -u $DOCKERHUB_CREDENTIALS_USR --password-stdin'
          }
        }
        stage('Docker Image Pull') {
          steps{
            sh 'docker pull $REGISTRY:latest'
          }
        }
        stage('Deploy docker project') {
            steps{
                sh 'docker run -p 8086:8080 --name otlab3 -itd $REGISTRY:latest'
            }
        }
    }
}

Also we need to give the agent user access to docker:

sudo chown ubuntu:ubuntu /var/run/docker.sock

Now you can start building the pipeline.

<center>

Figure 25 - The build was completed successfully
</center>

In order to check the operability of the task, I need to connect to the agent itself, through the port I specified in the redirect. And another feature is the path /vprofile-v2, I haven't been able to find it for a very long time. I had to study the entire project repository. And hooray! We see the login page!

<center>

Figure 26 - Login Page
</center>

:::info
7. Finally, chain all the jobs such that when you run the 1st job it should execute the rest.
Note: Use Post-Build Actions>Build other projects.
:::

In general, initially the easiest way was to install the Build Pipeline plugin, which beautifully showed all the stages. However, as I understand it, this is only possible for projects that are no longer a pipeline. And my cars do just that. Therefore, to call each of them one by one, I need to set up triggers.

<center>

Figure 27 - Post-Build Actions for 2.4
</center>

This is what I do until the very last project 2.6. However, it's a pity, this is not the most beautiful way to show that everything works. You can see that I clicked the build of the first stage and my cursor remained on this icon, while the status of the build of the third task and its stages is displayed below.

<center>

Figure 28 - One by one
</center>

It took a lot of processes and additionally enable ignoring errors on already deployed tasks, but in general, the sequence chain looks exactly like this.

<center>

Figure 29 - Finally!
</center>

## References:
1. [Introduction to the Maven's lifecycle](https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html)
2. [Jenkins: Reverse Proxy configuration Nginx](https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-nginx/)