First Steps to Full Lifecycle Security with Open Source Tools

Introduction

  • Anaïs Urlichs

    • Open Source Developer Advocate @ Aqua
    • CNCF Ambassador
    • YouTube AnaisUrlichs

  • Rory McCune

    • Cloud Native Security Advocate @ Aqua
    • Twitter/Github - @raesene

High Level Workshop Objectives

  • Scanning Containers and IaC in Development
  • Testing in the CI/CD pipeline
  • Security in Production

Course Pre-requisites

  • Ability to run a local Kubernetes cluster
    • KinD
    • minikube
    • microk8s
    • …
  • Ability to download and run binaries in :-
    • Linux
    • MacOS
    • FreeBSD?!

Note:

We can't cover all the options here. The second bit is for Trivy which has binaries for Linux MacOS and FreeBSD.

Course Logistics

Note:
Main ground rule is for people to put phones on silent

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Security in Development

  • Vulnerability Scanning
  • IaC Mis-configuration Scanning

Security Scanning Process

  • Before using any third-party resources
  • During Develpment
  • Before Deployment

Vulnerability Scanning

  • Using a container image vulnerability scanning tool is a useful way of assessing base images and also built container images.
  • We'll demonstrate this with Trivy

How do Container Vulnerability Scanners work?

  • Generally look at two types of information
    • OS packages (e.g.debian, alpine, RHEL)
    • Programming language packages (e.g. npm, rubygems)
  • Assess whether there are known vulnerabilities in the installed versions

Open Source Container Vulnerability Scanners

Note:

These are the main ones to mention, I think. With Snyk as we're talking Open source we should say that whilst the CLI is open source the database/server side isn't (although it is free)

Installing Trivy

Note:

Here we'll get everyone to install Trivy, let's expect this to take a couple of minutes (~5)

Using Trivy to scan Images

trivy i ubuntu:20.04

trivy i public.ecr.aws/docker/library/ubuntu:20.04

Note:

This scan is a typical one for a base image you might be considering using. It will return a decent number of vulnerabilities and this is something we'll talk about on the next slide

Ignore unfixed

trivy i --ignore-unfixed ubuntu:20.04

trivy i --ignore-unfixed public.ecr.aws/docker/library/ubuntu:20.04

Note:

This demonstrates the ignore-unfixed option which is useful in Debian and Ubuntu images to avoid showing vulnerabilities for which there is no patch. Whilst (for high security environments) these might matter, in most cases you'll just want to see things that can be fixed.

Looking for High/Critical OS Issues

trivy image --severity HIGH,CRITICAL --vuln-type os postgres:10.6

trivy image --severity HIGH,CRITICAL --vuln-type os public.ecr.aws/docker/library/postgres:10.15

Note:

This is a good demonstration of restricting the number of vulnerabilities to be looked at by restricting to high and critical severities.

Looking for High/Critical Library Issues

trivy image --severity HIGH,CRITICAL --vuln-type library node:10.6

trivy image --severity HIGH,CRITICAL --vuln-type library public.ecr.aws/docker/library/node:10.23-slim

Note:

We're doing this scan to show the differentiation on scanning libraries against scanning for OS vulnerabilities.

Scanning GitHub repositories

trivy repo --vuln-type library https://github.com/raesene/sycamore

Note:

This is a good example of scanning a respository, which can be done before cloning it. we'll get a good number of vulns here as this is an un-maintained rails app I wrote a while back.

Scanning the filesystem

  • Pick a project on your machine or
git clone https://github.com/raesene/sycamore
  • Scan a whole directory
trivy fs ./sycamore/
  • Scan a file
trivy fs ./sycamore/yarn.lock

JSON output

trivy i --format json raesene/spring4shelldemo:latest

Note:

This is useful both to show the output formats, but also to show that there's a lot of additional information in the JSON output that isn't in the default table.

Using jq to find a specific issue

trivy -q i --format json raesene/spring4shelldemo:latest | jq '.Results[].Vulnerabilities[] | select(.VulnerabilityID == "CVE-2022-22965")'

Note:

This one has a couple of pieces we should explain. Firstly we're using -q to ensure we get pure JSON out, then we're using jq to pick out the details of a specific vulnerability. N.B. we're not using shell format for this one as it doesn't show all the text on screen and escaping the CRLFs is tricky inside a jq expression

Configuration Scanning

  • A good practice during development or when using 3rd party projects
  • Can flag up where good security practices aren't being followed
  • Rulesets vary by tool, although some can be based on standards (e.g. Kubernetes PSS, CIS Benchmarks)

Configuration Scanning - Docker

git clone https://github.com/AnaisUrlichs/trivy-demo.git
cd trivy-demo

trivy config bad_iac/docker/

Fixing a Docker issue

  • Uncomment the USER line in the Dockerfile
trivy config bad_iac/docker/

Note:

Here we can demonstrate how to fix one of the issues and show how that is removed from the report.

Configuration Scanning - Kubernetes

trivy config bad_iac/kubernetes/

Note:

Here we'll want to walk through the output and some of the issues

Fixing a Kubernetes Issue

  • Pick an issue from the ones flagged up and see if you can fix it, then re-scan
trivy config bad_iac/kubernetes/

Note:

There's several issues in this manifest that can be fixed, so we can let people choose one (or more) to resolve.

Configuration Scanning - Terraform

trivy config bad_iac/terraform/

Note:

We can probably skip doing a fix on this one as we've demonstrated the process with the others

Trivy SBOM

trivy sbom ubuntu:20.04

*also available as Docker Desktop Extension

Security Scanning in CI/CD

  • Applying the same checks as are available in development, in CI/CD pipelines provides an additional layer of security.
  • Using GitHub Actions and SARIF, we can automate security checks either periodically or as code is checked in

Using a Github Action to build and scan a Docker image

Note:

What we'll do here is walk through some key elements of using Trivy by extracting sections from the Action

Permissions to build+scan an image

  permissions:
      contents: read
      packages: write
      # This is used to complete the identity challenge
      # with sigstore/fulcio when running outside of PRs.
      id-token: write
      security-events: write   # To upload sarif files

Note:

The key elements here are that we need rights to write the package to GHCR and we need security-events write permissions to output the results of the Trivy scan

Running Trivy

- name: Run trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}
          format: sarif
          output: 'trivy-results.sarif'

Note:

Here there's a couple of points to emphasise. First the use of our trivy-action then the image ref which is built up from environment variables in the workflow, so it's not static and then the output format, which we'll use in the next step

Uploading results to GitHub Security

 - name: Upload Trivy scan results to GitHub Security tab
        uses: github/codeql-action/upload-sarif@v1
        with:
          sarif_file: 'trivy-results.sarif'

Note:

Here's where we upload to GitHub security picking up the sarif file from the previous step.

Config scanning in CI/CD with Trivy

      - name: Run Trivy in Config mode to generate SARIF
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'config'
          hide-progress: false
          format: 'sarif'
          output: 'trivy-results.sarif'

Note:

this is a good illustration of changing a vulnerability scan for a configuration scan, which also works for SARIF output.

Open Source Security in Production

  • Once our workloads are deployed to clusters we need on-going security
  • Regular scans for compliance and assurance
  • Runtime security to detect attacks

Note:

We're mentioning runtime security here although we won't have time to get into it in practice (also tricky one to demonstrate) as it is relevant to production cluster security.

raesene We'll need to come up with a strategy for this, honestly not too sure of the best approach if Starboard is getting removed before kubecon

Starboard

  • Starboard Operator.

    • Used to automatically scan new resources in a cluster
    • Also compliance scans for cluster configuration

  • Main Installation options

    • kubectl
    • Helm
    • Operator Lifecycle Manager (OLM)

Kubernetes Operator

Automating human behaviour through controllers

https://www.cncf.io/wp-content/uploads/2021/07/CNCF_Operator_WhitePaper.pdf

Starboard inside your Kubernetes cluster

Installing Starboard Operator

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/starboard/v0.15.4/deploy/static/starboard.yaml

Note:

Worth discussing here, first the variety of objects created in the manifest and also how this install method is fine for a workshop with throwaway clusters, but we'd do things differently for production.

CRDs

  • Starboard creates a number of CRDs to hold scan data.
ciskubebenchreports.aquasecurity.github.io
clustercompliancedetailreports.aquasecurity.github.io
clustercompliancereports.aquasecurity.github.io
clusterconfigauditreports.aquasecurity.github.io
configauditreports.aquasecurity.github.io
vulnerabilityreports.aquasecurity.github.io

Confirming all is well

  • Checking the deployment of the operator tells us if the install was successful
kubectl get deployment -n starboard-system

Creating a Deployment

kubectl create ns app

kubectl apply -f https://raw.githubusercontent.com/AnaisUrlichs/trivy-demo/main/manifests/kubernetes.yaml -n app

VulnerabilityReport

Automatically scans the containers that are used inside of your cluster.

Deployment-scoped

kubectl get vulnerabilityreports -o wide -n app

Cluster-scoped

kubectl get clustervulnerabilityreports -o wide -n app

Configuration Auditing

Kubernetes configurations are checked against built-in policies

Deployment-scoped

kubectl get configauditreports -o wide -n app

Cluster-scoped

kubectl get clusterconfigauditreports -o wide -n app

Infrastructure Scanning

  • CIS benchmark for Kubernetes nodes provided by kube-bench.
  • Penetration test results for a Kubernetes cluster provided by kube-hunter.

Note:

https://aquasecurity.github.io/starboard/v0.15.4/configuration-auditing/built-in-policies/

CISKubeBenchReport

Maps CIS Benchmarks against Kubernetes version

kubectl get nodes

kubectl get ciskubebenchreports -o wide

kubectl describe ciskubebenchreports/<insert report name>

Note:

One report per node, does not have access to the main nodes, only to the worker nodes.

Other tool to produce CIS benchmarks

Cluster Compliance Report

NSA report

ClusterCompliance and ClusterComplianceDetail Report

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/starboard/main/deploy/specs/nsa-1.0.yaml

kubectl get clustercompliancereport -o wide

kubectl describe clustercompliancereport nsa

kubectl get clustercompliancedetailreport -o wide

NSA reports produced by other tools

kubescape NSA report

https://github.com/armosec/kubescape

Custom Policies

  • Trivy/Starboard: Write custom policies using Rego
  • tfsec: Custom policies in JSON/YAML
trivy config --policy ./custom-policies --namespaces user ./manifests

What is next?

  • Integrate Starboard metrics into your observability stack
  • Try out Trivy's functionality in your own projects
  • Let us know what use cases you would like to see

Repository https://github.com/AnaisUrlichs/trivy-demo
Aqua GitHub https://github.com/aquasecurity
Rory's Twitter https://twitter.com/raesene
Anais' Twitter https://twitter.com/urlichsanais

Thank you!

  • Rory's Twitter/GitHub: @raesene
  • Anais' Twitter: @urlichsanais
Select a repo