---
tags: OT
---
:::success
# OT Final Project: Malware analysis pipelines using OSS
* **Mikhail Syropyatov**
* **Rail Iaushev**
:::
## 0. Abstract
This project aims at looking at CI/CD pipelines from a security point of view. We will compare popular open source tools and try to identify the best solution for creating malware analysis pipelines based on our research.
**Tech stack:**
> - Jenkins and integrated plugins
> - VMWare
> - Docker
> - SonarQube
> - OWASP Dependency Check
> - OWASP ZAP
**Contents:**
[TOC]
## 1. Introduction and Background Research:
In this section we provide the background necessary for our project. First, in _1.1 Malware analysis in the field of offensive technologies_ we introduce the general understanding of malware analysis. Then, in _1.2 Comparison of CI/CD platforms_, we compare best CI/CD platforms that can be used for pipeline creation. Finally, _1.3 Comparison of top analysis tools_ section focuses on the best malware analysis tools and their comparison.
### 1.1. Malware analysis in the field of offensive technologies
Malware analysis is a critical process in the field of offensive technologies that involves examining and understanding malware. Malware refers to any software or code designed with malicious intent, such as viruses, worms, Trojans, ransomware, spyware, and botnets.
The significance of malware analysis lies in its role in identifying, dissecting, and comprehending the inner workings of malware. Knowledges that we get from studying malware allows us to develop effective defensive measures, detection mechanisms, and countermeasures against evolving cyber threats.
Why it is important in the field of ofeensive technologies?
1. **Threat Intelligence**: Malware analysis provides valuable threat intelligence by uncovering the methodologies, capabilities, and objectives of attackers. This information helps security professionals stay one step ahead of cybercriminals and proactively protect systems and networks.
1. **Incident Response**: When a security incident occurs, malware analysis plays a crucial role in identifying the malware responsible for the breach.
1. **Vulnerability Assessment**: Malware analysis helps identify vulnerabilities in software and systems.
1. **Signature Generation**: Malware analysis enables the creation of signatures or patterns that can be used in antivirus and intrusion detection systems.
1. **Reverse Engineering**: Malware analysis involves reverse engineering techniques to understand the inner workings of malware. By reverse engineering malware, we can identify the command and control infrastructure, data exfiltration methods, encryption mechanisms, and other crucial aspects that aid in devising countermeasures.
1. **Malware Attribution**: Analyzing malware can help attribute attacks to specific threat actors or hacking groups.
1. **Enhancing Cybersecurity Tools**: Through malware analysis, security department can identify weaknesses and limitations in existing security tools.
### 1.2. Comparison of CI/CD platforms
First we need to decide on which platform that has the ability to use CI/CD pipelines will be convenient to implement malware analysis pipelines and integrate OSS tools, which will be used in different stages of the pipeline.
Some popular CI/CD systems that are known for their ease of integration and flexibility include:
1. **Jenkins**: Jenkins is one of the most widely used open-source CI/CD systems. It offers a vast ecosystem of plugins that can be leveraged to integrate various stages of the malware analysis pipeline. Jenkins provides a high level of customization and flexibility, making it suitable for integrating different tools and processes.
1. **GitLab CI/CD**: GitLab provides a built-in CI/CD system that tightly integrates with its version control features. It offers an intuitive YAML-based configuration, making it easy to define pipelines and stages for malware analysis. GitLab also provides a container registry for storing and managing containerized analysis tools.
1. **Travis CI**: Travis CI is a popular cloud-based CI/CD system primarily used for open-source projects. It has a straightforward configuration format and integrates well with GitHub repositories. Travis CI supports running analysis tools and custom scripts within the build environment.
1. **CircleCI**: CircleCI is a cloud-based CI/CD platform that offers a simple and intuitive configuration. It supports parallel builds, provides a wide range of built-in integrations, and can be easily extended to incorporate the malware analysis pipeline stages.
1. **GitHub Actions**: GitHub Actions is tightly integrated with GitHub repositories and offers a seamless CI/CD experience. It allows you to define custom workflows using YAML files and provides a marketplace of pre-built actions for integrating various tools and processes.
In this case, **Jenkins** is the best choice for us, as it is the most specific platform for creating pipelines, which is convenient to work with and customize for specific purposes.
### 1.3. Comparison of top analysis tools
In the realm of cybersecurity, selecting the right tools for code analysis is crucial. These tools not only help in maintaining code quality but also ensure that the application is secure from potential vulnerabilities. It's important to use a combination of **Static Application Security Testing** (**SAST**), **Dynamic Application Security Testing** (**DAST**), and **Software Composition Analysis** (**SCA**) tools to have comprehensive coverage of all potential risks.
In this regard, we compare several open-source tools widely used in the industry. **SAST** tools that perform automatic reviews with static analysis of code to detect bugs and security vulnerabilities. **DAST** tools that provides automated scanning and a set of tools for manual vulnerability testing of web applications. **SCA** tools used to identify outdated libraries with known vulnerabilities in project dependencies.
|Tool Name| Type |Pros| Cons|
|---|---|---|---|
|SonarQube| SAST |Extensive language support, ease of integration with CI/CD pipelines, large user community| Setting up rules can be complex, some advanced features are available only in the paid version|
|FindSecBugs| SAST| High detection rate for common vulnerability patterns in Java| Limited to Java, needs to be combined with FindBugs for a comprehensive analysis|
|PMD| SAST| It supports multiple programming languages, including Java, JavaScript, and XML, making it versatile for various projects. PMD helps identify potential code issues, such as complex code, unused variables, or inefficient constructs, enabling developers to improve code quality and maintainability. It provides a wide range of built-in rules and allows custom rule configurations, allowing for flexibility and customization based on project-specific requirements. PMD can be integrated into build processes and IDEs, providing developers with real-time feedback and enhancing their coding practices.| It requires configuration and rule tuning to align with specific project needs, which can be time-consuming initially. The learning curve of PMD, especially for beginners, may be steep due to the complexity and wide range of available rules. Additionally, the performance of PMD on large codebases or complex projects might be slower compared to more specialized static analysis tools.|
|Agnitio| SAST| It provides support for a wide range of programming languages, making it versatile for various projects. Agnitio's analysis is known for its precision and accuracy, minimizing false positives and false negatives and enabling developers to focus on genuine security vulnerabilities. The tool integrates seamlessly with popular development tools and workflows, facilitating smooth integration into the software development lifecycle. Agnitio's performance is optimized for speed and scalability, allowing it to handle large codebases efficiently and deliver fast results.| The tool's coverage for certain niche or less popular programming languages may be limited compared to other SAST tools with broader language support. Additionally, the specific customization and extensibility options of Agnitio may be more limited compared to some other SAST tools that offer extensive customization capabilities.|
|OWASP ZAP| DAST| Comprehensive coverage of web app vulnerabilities, active community, many integrations |Can be complex to set up for full automation, scanning large applications can be time-consuming|
|Vega| DAST| Vega includes an automated scanner for quick tests and an intercepting proxy for tactical inspection. The Vega scanner finds XSS (cross-site scripting), SQL injection, and other vulnerabilities. Vega can be extended using a powerful API in the language of the web: Javascript. |Vega requires some familiarity with web security concepts and techniques to utilize it effectively. Additionally, Vega's scanning speed may be slower compared to some commercial alternatives, especially when dealing with complex or large-scale web applications. Another consideration is that Vega primarily focuses on web application scanning and may not provide extensive coverage for other types of vulnerabilities or network-level security assessments.|
|OpenVAS Scanner| DAST| It is a full-featured scan engine that executes a continuously updated and extended feed of Vulnerability Tests (VTs) |The scanning process can be resource-intensive, especially when scanning large networks or complex systems, which may impact network performance during the scanning period. Additionally, OpenVAS Scanner may not have the same level of comprehensive coverage and frequent updates as some commercial vulnerability scanning tools, potentially resulting in delayed detection of new vulnerabilities.|
|Nuclei| DAST| Powerful and flexible open-source tool designed for security scanning and fingerprinting of web applications. It allows users to define custom templates or use pre-existing templates to identify various types of vulnerabilities, misconfigurations, and security issues. | May require more manual effort in terms of template creation, maintenance, and customization compared to some commercial vulnerability scanning tools. Users need to stay up-to-date with the latest vulnerabilities and security issues to ensure the effectiveness of the templates used with Nuclei|
|OWASP Dependency-Check| SCA| Detects outdated libraries with known vulnerabilities, integrates with NIST's vulnerability database| False positives can occur as it sometimes relies on heuristic methods for detection|
|SOOS| SCA| Platform agnostic with active community support, seamless integration capabilities, real-time vulnerability analysis, and customization options.| Limited features compared to commercial tools, potentially less support and documentation, varying maturity and stability, and usability may require technical expertise.|
|MegaLinter| SCA| It supports a wide range of programming languages, allowing developers to enforce consistent code quality across different projects. By integrating multiple linters and static analysis tools into a unified toolchain, MegaLinter provides a centralized and streamlined approach to code analysis. It helps identify potential code issues, enforce coding best practices, and improve code quality and maintainability. MegaLinter's open-source nature allows for community contributions, customization, and the ability to adapt it to specific development workflows and requirements.| The initial setup and configuration of MegaLinter can be complex, especially for projects with specific requirements or custom configurations for individual linters. The performance of MegaLinter may be affected when dealing with large codebases or complex projects due to the execution of multiple linters and tools.|
In conclusion, each of these tools has its strengths and weaknesses, the key to effective cybersecurity is using them in combination to cover all potential points of vulnerability. It's also important to keep in mind that automated tools are just one part of a robust security approach and human expertise remains irreplaceable for accurate vulnerability assessment and mitigation.
## 2. Implementation
Now we are ready to describe our practical work. We will descibe Jenkins server and agent deployment in _2.1. Jenkins server and agent deployment_. Then we proceed to describe security scheme sample in _2.2. Security scheme in pipeline_. The next section, _Preparing Jenkins infrastucture_ walks through our configuration of the Jenkins server and neccessary tools. Section _2.4. Implementation of malware analysis pipeline_ shows the main steps of the work done to create a malware analysis pipeline. Finally, in _2.5. Problems faced_, we describe main problems, that we faced during project implementation.
### 2.1. Jenkins server and agent deployment
We created Virtual Machine with Ubuntu 22.04, updated the system using "sudo apt update" and then installed Docker Engine using following commands:
```bash!
1. sudo apt-get install ca-certificates curl gnupg
2. sudo install -m 0755 -d /etc/apt/keyrings
3. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
4. sudo chmod a+r /etc/apt/keyrings/docker.gpg
5. echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
6. sudo apt-get update
7. sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
After that cloning this virtual machine and getting one for Jenkins server, second for agent:
<center>
Figure 1: Created virtual machines
</center>
Now in Jenkins server VM we installing Jenkins with the following commands:
```bash!
sudo apt install openjdk-17-jre-headless
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
/usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins
```
In agent we also installed Java Development Kit with "_sudo apt install openjdk-17-jre-headless_" command.
<center>
Figure 2: Jenkins installed on server
</center>
On the server we installing nginx with "_sudo apt update && sudo apt install -y nginx_" command.
Then created new config file to isolate Jenkins behind the reverse proxy /etc/nginx/sites-available/jenkins with following content:
```nginx!
server {
listen 80;
server_name 192.168.204.130;
location / {
proxy_pass http://192.168.204.130:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
Applying changes and restarting nginx:
<center>
Figure 3: Including Jenkins config file to global nginx config
</center>
First of all, going into the Jenkins controller, we see a warning that it is not safe to run work on the node in which the controller itself is installed, so we add an agent cliking on "Set up agent". After adding agent we setting up the number of executors on the built-in node to 0.
<center>
Figure 4: Agent added to Jenkins
</center>
Now our Jenkins server is ready to start jobs, however, we need to use additional tools to ensure security in the process of running the pipeline and define different stages.
### 2.2. Security scheme in pipeline
Now we need to figure out at what stages of the pipeline what checks should be carried out.
Good example of how it can be implemented shown in Figure 5:
<center>
Figure 5: Apply security early in the software development life cycle to mitigate risks
</center>
In our case we will implement the holy trifecta of security testing: Source Composition Analysis (SCA), Static Application Security Testing (SAST), and Dynamic Application Security Testing (DAST).
As an SCA tool we will use OWASP Dependency-Check.
As an SAST tool we will use SonarQube.
As an DAST tool OWASP Zed Attack Proxy (ZAP).
### 2.3. Preparing Jenkins infrastucture
As a sample project, that will be used as test we will take following: https://github.com/HamidullahMuslih/ot-lab-sdlc.git
Adding DockerHub credentials:
<center>
Figure 6: DockerHub credentials
</center>
Adding github credentials:
<center>
Figure 7: DockerHub credentials
</center>
Adding maven installation:
<center>
Figure 8: Maven installation
</center>
Installing Pipeline Maven Integration plugin:
<center>
Figure 9: Pipeline Maven Integration plugin
</center>
Installing HTMLPublisher plugin:
<center>
Figure 10: Installed HTMLPublisher plugin
</center>
Creating SonarQube server on the VM with Jenkins using "*docker run -d --name sonarqube -e SONARESBOOTSTRAPCHECKSDISABLE=true -p 9000:9000 sonarqube:latest*" command.
Installing Sonarqube Scanner plugin:
<center>
Figure 11: Sonarqube Scanner plugin
</center>
Generating new token in SonarQube and adding it to Jenkins credentials as a secret text:
<center>
Figure 12: Sonarqube token
</center>
Token: sqa_4b4a9a555fae35785a556d3996e87ec19412a0da
After that adding our SonarQube server in Jenkins "System Configure" part.
Adding SonarQube scanner installation:
<center>
Figure 13: SonarQube scanner
</center>
Installing node.js on agent with
"*curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - && \
sudo apt-get install -y nodejs*" commands.
Adding jenkins user to the docker group with "*sudo usermod -aG docker jenkins*" command.
Installing Docker plugin:
<center>
Figure 14: Docker plugin
</center>
Installing OWASP ZAP plugin:
<center>
Figure 15: OWASP ZAP plugin
</center>
### 2.4. Implementation of malware analysis pipeline
Now, when the infrastructure is ready to use, we can start working on the pipeline itself.
Creating pipeline with malware-analysis name:
<center>
Figure 16: Creating a new pipeline
</center>
In the course of all the work, we created a script for this pipeline and it is shown below. In it, we indicate all the stages of the assembly, as well as the necessary parameters:
```groovy!
def scan_type
def target
pipeline {
agent any
tools {
maven 'mvn-3.9.2'
}
parameters {
choice choices: ["Baseline", "APIS", "Full"],
description: 'Type of scan that is going to perform inside the container',
name: 'SCAN_TYPE'
string defaultValue: "http://192.168.204.129:8080/login",
description: 'Target URL to scan',
name: 'TARGET'
}
stages {
stage('source') {
steps {
sh '''git clone https://github.com/HamidullahMuslih/ot-lab-sdlc.git
cd ot-lab-sdlc'''
}
}
stage('build') {
steps {
withMaven(maven : 'mvn-3.9.2') {
dir('/home/jenkins/jenkins_slave/workspace/malware-analysis/ot-lab-sdlc') {
sh "mvn clean package"
}
}
}
}
stage('integration-test') {
steps {
withMaven(maven: 'mvn-3.9.2') {
dir('/home/jenkins/jenkins_slave/workspace/malware-analysis/ot-lab-sdlc') {
sh "mvn clean test"
}
}
}
}
stage('Dependency check') {
steps {
sh '''VERSION=$(curl -s https://jeremylong.github.io/DependencyCheck/current.txt)
wget "https://github.com/jeremylong/DependencyCheck/releases/download/v8.2.1/dependency-check-8.2.1-release.zip"
unzip dependency-check-8.2.1-release.zip
cd dependency-check/bin/
./dependency-check.sh --project Maven --scan '/home/jenkins/jenkins_slave/workspace/malware-analysis/ot-lab-sdlc' -o '/home/jenkins/jenkins_slave/workspace/malware-analysis/ot-lab-sdlc/dependency-check.html' -f HTML --enableExperimental'''
publishHTML([allowMissing: false, alwaysLinkToLastBuild: false, keepAll: false, reportDir: '/home/jenkins/jenkins_slave/workspace/malware-analysis/ot-lab-sdlc/', reportFiles: 'dependency-check.html', reportName: 'Dependency Check Report', reportTitles: '', useWrapperFileDirectly: true])
}
}
stage('SAST with SonarQube') {
steps {
withSonarQubeEnv(credentialsId: 'sonar', installationName:'SonarQube') {
withMaven(maven: 'mvn-3.9.2') {
dir('/home/jenkins/jenkins_slave/workspace/malware-analysis/ot-lab-sdlc') {
sh '''mvn sonar:sonar -Dsonar.host.url="http://192.168.204.130:9000"'''
}
}
}
}
}
stage('Creating Docker container with webapp') {
steps {
sh '''docker pull hamidullahmuslih/vproapp
docker run -d -p 8080:8080 --name vproapp hamidullahmuslih/vproapp'''
}
}
stage('Creating OWASP ZAP docker container') {
steps {
script {
sh 'docker pull owasp/zap2docker-stable'
sh """docker run -dt --name owasp \
owasp/zap2docker-stable \
/bin/bash"""
}
}
}
stage('Preparing wrk directory') {
steps {
script {
sh """
docker exec owasp \
mkdir /zap/wrk
"""
}
}
}
stage('DAST with ZAP') {
steps {
script {
scan_type = "${params.SCAN_TYPE}"
target = "${params.TARGET}"
if(scan_type == "Baseline"){
sh """docker exec owasp \
zap-baseline.py \
-t $target \
-r zap_report.html \
-I"""
}
else if(scan_type == "APIS"){
sh """docker exec owasp \
zap-api-scan.py \
-t $target \
-r zap_report.html \
-I"""
}
else if(scan_type == "Full"){
sh """docker exec owasp \
zap-full-scan.py \
-t $target \
-r zap_report.html \
-I \
2> /dev/null && echo 'Done' >&2"""
}
}
}
}
stage('Copy Report to Workspace'){
steps {
script {
sh '''docker cp owasp:/zap/wrk/zap_report.html ${WORKSPACE}/zap_report.html'''
publishHTML([
allowMissing: false,
alwaysLinkToLastBuild: false,
keepAll: false,
reportDir: '/home/jenkins/jenkins_slave/workspace/malware-analysis/',
reportFiles: 'zap_report.html',
reportName: 'ZAP DAST Report',
reportTitles: ''
])
}
}
}
}
post {
always {
echo "Removing containers"
sh '''docker stop owasp
docker rm owasp
docker stop vproapp
docker rm vproapp'''
}
}
}
```
On the created pipeline we using this script:
<center>
Figure 17: Script used for pipeline
</center>
Now I can build the pipeline with parameters "SCAN_TYPE" and "TARGET", as shown in Figure below:
<center>
Figure 18: Building pipeline with parameters
</center>
Let's try to run this pipeline with Baseline type ZAP scan:
<center>
Figure 19: Successful pipeline build
</center>
When pipeline is finished to execute we can find reports of every malware analysis step, that we integrated to our pipeline:
<center>
Figure 20: Reports from malware analysis tools
</center>
Let's get a closer look to every report generated:
<center>
Figure 21: Dependency Check Report
Figure 22: SonarQube scanner Report
Figure 23: ZAP scanner Report
</center>
At the post actions of the pipeline created containers are deleted, so this pipiline is focused on malware analysis, but not pushing ready project to productions, because after all we need to analyze reports and fix possible problems appeared.
### 2.5. Problems faced
We tried to build the project, but faced problem unable to connect to web application, that's why we used Hamid's dockerhub project for this application, that can be found by following link: https://hub.docker.com/r/hamidullahmuslih/vproapp
Dockerfile used in ot-lab-sdlc directory:
```dockerfile!
FROM tomcat
COPY . .
RUN apt-get update ; apt-get install default-jdk -y ; update-alternatives --config javac
RUN rm -rf /usr/local/tomcat/webapps/*
ENV MAVEN_VERSION 3.9.2
ENV MAVEN_HOME /usr/local/apache-maven
ENV MAVEN_URL https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz
RUN mkdir -p ${MAVEN_HOME} \
&& curl -fsSL ${MAVEN_URL} | tar -xzC ${MAVEN_HOME} --strip-components=1
ENV PATH ${MAVEN_HOME}/bin:${PATH}
RUN mvn clean install -DSkipTests ; cp target/*.war /usr/local/tomcat/webapps/ROOT.war
CMD ["catalina.sh","run"]
```
Another problem was discovered in the process of working with a full ZAP scan script. Upon completion of the scan, the program issued an exit code 2, despite the fact that the analysis was successfully completed. Because of that the work was marked as failed and the next stages of the pipeline were not performed "due to an error that occurred in the previous steps".
This problem was solved by adding the following line to the block of code for the full scan, which was executed after the sh command: "2> /dev/null && echo 'Done' >&2".
## 3. Results and discussion
While working with that project, we deeply studied the process of creating a malware analysis pipeline, reviewed the best tools and tried to choose the most relevant today, worked with scripts in Jenkins, and gained important DevSecOps experience.
We found that most of the tools presented on the malware analysis market are commercial projects and there is not a lot open-source tools.
Finally, we managed to create a pipeline template that can be adjusted to any maven project, for this it is only necessary to make small changes to the pipeline script.
## 4. References
1.) https://devopscube.com/setup-slaves-on-jenkins-2/
2.) https://github.com/rshipp/awesome-malware-analysis
3.) https://github.com/ytisf/theZoo/tree/master/malware/Source/Original
4.) https://github.com/screetsec/TheFatRat
5.) https://www.opswat.com/blog/metadefender-for-jenkins-and-teamcity-secure-your-software-builds
6.) https://github.com/we45/Vulnerable-Flask-App
7.) https://www.appsecengineer.com/blog/how-to-build-a-devsecops-pipeline-in-jenkins
8.) https://www.comparitech.com/net-admin/best-static-code-analysis-tools/
9.) https://github.com/MaartenSmeets/spring-boot-demo
10.) https://blogs.appsecworld.com/2022/11/integrate-owasp-dependency-check-with-jenkins.html
11.) https://medium.com/globant/owasp-zap-integration-with-jenkins-795d65991404
12.) https://groups.google.com/g/zaproxy-users/c/v_QSoUiAtyc
13.) https://www.zaproxy.org/docs/docker/baseline-scan/
14.) https://nullsweep.com/dynamic-security-scanning-in-a-ci-zap-scanning-with-jenkins/
15.) https://hub.docker.com/r/hamidullahmuslih/vproapp
16.) https://www.janbasktraining.com/blog/jenkins-code-analysis/
17.) https://owasp.org/www-community/Free_for_Open_Source_Application_Security_Tools
Sign in with Wallet
Connect another wallet