## First Steps to Full Lifecycle Security with Open Source Tools --- ## Introduction - Anaïs Urlichs - Open Source Developer Advocate @ Aqua - CNCF Ambassador - YouTube AnaisUrlichs - Rory McCune - Cloud Native Security Advocate @ Aqua - **Twitter/Github** - @raesene --- ## High Level Workshop Objectives - Scanning Containers and IaC in Development - Testing in the CI/CD pipeline - Security in Production --- ## Course Pre-requisites - Ability to run a local Kubernetes cluster - KinD - minikube - microk8s - ... - Ability to download and run binaries in :- - Linux - MacOS - FreeBSD?! Note: We can't cover all the options here. The second bit is for Trivy which has binaries for Linux MacOS and FreeBSD. --- ## Course Logistics - Ground rules - Materials - Slides - http://slides.pwndland.uk - commands - http://commands.pwndland.uk - setup notes - http://setup.pwndland.uk - Questions? Just Ask! Note: Main ground rule is for people to put phones on silent :smile: --- ## Security in Development - Vulnerability Scanning - IaC Mis-configuration Scanning --- ## Security Scanning Process - Before using any third-party resources - During Develpment - Before Deployment --- ## Vulnerability Scanning - Using a container image vulnerability scanning tool is a useful way of assessing base images and also built container images. - We'll demonstrate this with [Trivy](https://github.com/aquasecurity/trivy) --- ### How do Container Vulnerability Scanners work? - Generally look at two types of information - OS packages (e.g.debian, alpine, RHEL) - Programming language packages (e.g. npm, rubygems) - Assess whether there are known vulnerabilities in the installed versions --- ### Open Source Container Vulnerability Scanners - [Trivy](https://github.com/aquasecurity/trivy) - [Grype](https://github.com/anchore/grype) - [Clair](https://github.com/quay/clair) - [Snyk](https://github.com/snyk/cli) **CLI Only** Note: These are the main ones to mention, I think. With Snyk as we're talking Open source we should say that whilst the CLI is open source the database/server side isn't (although it is free) --- ## Installing Trivy - Several options on the install page - https://aquasecurity.github.io/trivy/v0.28.0/getting-started/installation/ - Homebrew for MacOS - APT repository for Debian/Ubuntu - YUM repo for RHEL/CentOS - Let's install Trivy! Note: Here we'll get everyone to install Trivy, let's expect this to take a couple of minutes (~5) --- ## Using Trivy to scan Images ```shell trivy i ubuntu:20.04 ``` ```shell trivy i public.ecr.aws/docker/library/ubuntu:20.04 ``` Note: This scan is a typical one for a base image you might be considering using. It will return a decent number of vulnerabilities and this is something we'll talk about on the next slide --- ## Ignore unfixed ```shell trivy i --ignore-unfixed ubuntu:20.04 ``` ```shell trivy i --ignore-unfixed public.ecr.aws/docker/library/ubuntu:20.04 ``` Note: This demonstrates the ignore-unfixed option which is useful in Debian and Ubuntu images to avoid showing vulnerabilities for which there is no patch. Whilst (for high security environments) these might matter, in most cases you'll just want to see things that can be fixed. --- ## Looking for High/Critical OS Issues ```shell trivy image --severity HIGH,CRITICAL --vuln-type os postgres:10.6 ``` ```shell trivy image --severity HIGH,CRITICAL --vuln-type os public.ecr.aws/docker/library/postgres:10.15 ``` Note: This is a good demonstration of restricting the number of vulnerabilities to be looked at by restricting to high and critical severities. --- ## Looking for High/Critical Library Issues ```shell trivy image --severity HIGH,CRITICAL --vuln-type library node:10.6 ``` ```shell trivy image --severity HIGH,CRITICAL --vuln-type library public.ecr.aws/docker/library/node:10.23-slim ``` Note: We're doing this scan to show the differentiation on scanning libraries against scanning for OS vulnerabilities. --- ## Scanning GitHub repositories ```shell trivy repo --vuln-type library https://github.com/raesene/sycamore ``` Note: This is a good example of scanning a respository, which can be done before cloning it. we'll get a good number of vulns here as this is an un-maintained rails app I wrote a while back. --- ## Scanning the filesystem - Pick a project on your machine or ```shell git clone https://github.com/raesene/sycamore ``` - Scan a whole directory ```shell trivy fs ./sycamore/ ``` - Scan a file ```shell trivy fs ./sycamore/yarn.lock ``` --- ## JSON output ```shell trivy i --format json raesene/spring4shelldemo:latest ``` Note: This is useful both to show the output formats, but also to show that there's a lot of additional information in the JSON output that isn't in the default table. --- ## Using jq to find a specific issue ```shell trivy -q i --format json raesene/spring4shelldemo:latest | jq '.Results[].Vulnerabilities[] | select(.VulnerabilityID == "CVE-2022-22965")' ``` Note: This one has a couple of pieces we should explain. Firstly we're using `-q` to ensure we get pure JSON out, then we're using `jq` to pick out the details of a specific vulnerability. N.B. we're not using shell format for this one as it doesn't show all the text on screen and escaping the CRLFs is tricky inside a jq expression --- ## Configuration Scanning - A good practice during development or when using 3rd party projects - Can flag up where good security practices aren't being followed - Rulesets vary by tool, although some can be based on standards (e.g. Kubernetes PSS, CIS Benchmarks) --- ## Configuration Scanning - Docker ```shell git clone https://github.com/AnaisUrlichs/trivy-demo.git cd trivy-demo ``` ```shell trivy config bad_iac/docker/ ``` --- ## Fixing a Docker issue - Uncomment the USER line in the Dockerfile ```shell trivy config bad_iac/docker/ ``` Note: Here we can demonstrate how to fix one of the issues and show how that is removed from the report. --- ## Configuration Scanning - Kubernetes ```shell trivy config bad_iac/kubernetes/ ``` Note: Here we'll want to walk through the output and some of the issues --- ## Fixing a Kubernetes Issue - Pick an issue from the ones flagged up and see if you can fix it, then re-scan ```shell trivy config bad_iac/kubernetes/ ``` Note: There's several issues in this manifest that can be fixed, so we can let people choose one (or more) to resolve. --- ## Configuration Scanning - Terraform ```shell trivy config bad_iac/terraform/ ``` Note: We can probably skip doing a fix on this one as we've demonstrated the process with the others --- ## Trivy SBOM ```shell trivy sbom ubuntu:20.04 ``` *also available as Docker Desktop Extension --- ## Security Scanning in CI/CD - Applying the same checks as are available in development, in CI/CD pipelines provides an additional layer of security. - Using GitHub Actions and SARIF, we can automate security checks either periodically or as code is checked in --- ### Using a Github Action to build and scan a Docker image - Here's an example https://github.com/raesene/sycamore/blob/main/.github/workflows/docker-publish.yml - It's a modified version of GitHub's basic Docker+cosign action Note: What we'll do here is walk through some key elements of using Trivy by extracting sections from the Action --- ### Permissions to build+scan an image ```yaml permissions: contents: read packages: write # This is used to complete the identity challenge # with sigstore/fulcio when running outside of PRs. id-token: write security-events: write # To upload sarif files ``` Note: The key elements here are that we need rights to write the package to GHCR and we need security-events write permissions to output the results of the Trivy scan --- ### Running Trivy ```yaml - name: Run trivy uses: aquasecurity/trivy-action@master with: image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }} format: sarif output: 'trivy-results.sarif' ``` Note: Here there's a couple of points to emphasise. First the use of our trivy-action then the image ref which is built up from environment variables in the workflow, so it's not static and then the output format, which we'll use in the next step --- ### Uploading results to GitHub Security ```yaml - name: Upload Trivy scan results to GitHub Security tab uses: github/codeql-action/upload-sarif@v1 with: sarif_file: 'trivy-results.sarif' ``` Note: Here's where we upload to GitHub security picking up the sarif file from the previous step. --- ### Config scanning in CI/CD with Trivy ```yaml - name: Run Trivy in Config mode to generate SARIF uses: aquasecurity/trivy-action@master with: scan-type: 'config' hide-progress: false format: 'sarif' output: 'trivy-results.sarif' ``` Note: this is a good illustration of changing a vulnerability scan for a configuration scan, which also works for SARIF output. --- ### Open Source Security in Production - Once our workloads are deployed to clusters we need on-going security - Regular scans for compliance and assurance - Runtime security to detect attacks Note: We're mentioning runtime security here although we won't have time to get into it in practice (also tricky one to demonstrate) as it is relevant to production cluster security. --- > [name=raesene] We'll need to come up with a strategy for this, honestly not too sure of the best approach if Starboard is getting removed before kubecon ### Starboard - Starboard Operator. - Used to automatically scan new resources in a cluster - Also compliance scans for cluster configuration - Main Installation options - kubectl - Helm - Operator Lifecycle Manager (OLM) --- ## Kubernetes Operator Automating human behaviour through controllers https://www.cncf.io/wp-content/uploads/2021/07/CNCF_Operator_WhitePaper.pdf ---  --- ### Installing Starboard Operator ```shell kubectl apply -f https://raw.githubusercontent.com/aquasecurity/starboard/v0.15.4/deploy/static/starboard.yaml ``` Note: Worth discussing here, first the variety of objects created in the manifest and also how this install method is fine for a workshop with throwaway clusters, but we'd do things differently for production. --- ### CRDs - Starboard creates a number of CRDs to hold scan data. ```shell ciskubebenchreports.aquasecurity.github.io clustercompliancedetailreports.aquasecurity.github.io clustercompliancereports.aquasecurity.github.io clusterconfigauditreports.aquasecurity.github.io configauditreports.aquasecurity.github.io vulnerabilityreports.aquasecurity.github.io ``` --- ### Confirming all is well - Checking the deployment of the operator tells us if the install was successful ```shell kubectl get deployment -n starboard-system ``` --- ## Creating a Deployment ```shell kubectl create ns app kubectl apply -f https://raw.githubusercontent.com/AnaisUrlichs/trivy-demo/main/manifests/kubernetes.yaml -n app ``` --- ## VulnerabilityReport Automatically scans the containers that are used inside of your cluster. **Deployment-scoped** ```shell kubectl get vulnerabilityreports -o wide -n app ``` **Cluster-scoped** ```shell kubectl get clustervulnerabilityreports -o wide -n app ``` --- ## Configuration Auditing Kubernetes configurations are checked against built-in policies **Deployment-scoped** ```shell kubectl get configauditreports -o wide -n app ``` **Cluster-scoped** ```shell kubectl get clusterconfigauditreports -o wide -n app ``` --- ## Infrastructure Scanning - CIS benchmark for Kubernetes nodes provided by kube-bench. - Penetration test results for a Kubernetes cluster provided by kube-hunter. Note: https://aquasecurity.github.io/starboard/v0.15.4/configuration-auditing/built-in-policies/ --- ## CISKubeBenchReport Maps CIS Benchmarks against Kubernetes version ```shell kubectl get nodes kubectl get ciskubebenchreports -o wide kubectl describe ciskubebenchreports/<insert report name> ``` Note: One report per node, does not have access to the main nodes, only to the worker nodes. --- ## Other tool to produce CIS benchmarks * https://github.com/chen-keinan/kube-beacon --- ## Cluster Compliance Report NSA report ClusterCompliance and ClusterComplianceDetail Report ```shell kubectl apply -f https://raw.githubusercontent.com/aquasecurity/starboard/main/deploy/specs/nsa-1.0.yaml kubectl get clustercompliancereport -o wide kubectl describe clustercompliancereport nsa kubectl get clustercompliancedetailreport -o wide ``` --- ## NSA reports produced by other tools  https://github.com/armosec/kubescape --- # Custom Policies - Trivy/Starboard: Write custom policies using Rego - tfsec: Custom policies in JSON/YAML ```shell trivy config --policy ./custom-policies --namespaces user ./manifests ``` --- ## What is next? - Integrate Starboard metrics into your observability stack - Try out Trivy's functionality in your own projects - Let us know what use cases you would like to see --- ## Links Repository https://github.com/AnaisUrlichs/trivy-demo Aqua GitHub https://github.com/aquasecurity Rory's Twitter https://twitter.com/raesene Anais' Twitter https://twitter.com/urlichsanais --- ## Thank you! * Rory's Twitter/GitHub: @raesene * Anais' Twitter: @urlichsanais