<!-- The sections in this template are highly-suggested, but optional. They exist to
provide an idea of what an in-depth proposal **might** include. If a different
format fits your proposal, feel free to use it, with the expectation that:

1. You must link to the proposal issue (github) under the title
1. If you omit sections that are relevant, you may be asked by maintainers to
add them in order to best review the proposal. -->

# Image Vulnerability Scanning for Tanzu Packages

* Proposal: https://github.com/vmware-tanzu/community-edition/issues/2967
* See Also:
* [k/k image scanning](https://github.com/kubernetes/sig-security/blob/main/sig-security-tooling/vulnerability-mgmt/container-images.md)
* [k/k scanning job on testgrid](https://testgrid.k8s.io/sig-security-snyk-scan#ci-kubernetes-snyk-master)

## Summary

Create a periodically running prow job that:
- Identifies all the images that are part of available tanzu packages
- Runs a vulnerability scan on each image
- Fails when an actionable vulnerability is found

## Motivation

Tanzu Community Edition is shipped prebaked with multiple packages that can be installed in a cluster. In line with having a secure supply chain, community edition maintainers lack visibility into container image vulnerabilities in these packages. Scanning them on a periodic basis will allow us to do just that. It will also help create a positive feedback loop where package owners will continually update their container images when new vulnerabilities are detected by us.

### Goals

* Identify a repeatable method to detect images that are part of default tanzu packages
* Run a vulnerability scan on these images, to detect fixable CVEs
* Fail the prow job discretely based on pre-configured rules
* Alert on failures to identified members of the community
* Run the vulnerability scan as a periodically running job

### Future Work

* Measuring mean time to detect, triage, fix and patch
* Creating automated Github Issues requesting image bumps from package repos
* Committing to any SLOs for security patches
* Build time dependency scanning for tanzu-framework and community-edition
* Secure Code scanning for community edition and tanzu-framework
* CIS benchmark scanning for kubernetes clusters created by community-edition

Note: These are all great ideas, that we hope to discuss in future design proposals

## Proposal

The implementation is derived heavily from what is done upstream in k/k. More details about image vulnerability scanning upstream are available [here](https://github.com/kubernetes/sig-security/blob/main/sig-security-tooling/vulnerability-mgmt/container-images.md)

### Detecting images that are part of default tanzu packages

Each tanzu package is built using `imgpkg` and is _packaged_ together into an OCI image. It uses a concept of _nested_ packages that allow creating a hierarchy of OCI images, where each package image can contain more than one OCI image.

For example, this is how we can get a list of packages from the _latest_ tce main package:

turtle@turtle-a01 ~ % imgpkg pull --recursive -b projects.registry.vmware.com/tce/main:latest -o /tmp/tce-main

<!-- markdownlint-disable MD033 -->
<details><summary>Click to view result</summary>
<!-- markdownlint-enable MD033 -->

Pulling bundle 'projects.registry.vmware.com/tce/main@sha256:0602f6591deb8a209b1d70a7460be14df6d9376238db0a9ae39d93bcd0a496a7'
  Extracting layer 'sha256:e45a923b4322b6ada9a4ba59d68ff3779bdf60e9326f5943d3691cce06c57300' (1/1)

Nested bundles
  Pulling nested bundle 'projects.registry.vmware.com/tce/cert-manager@sha256:31cdadff5b576fb04622b15c9c8fc22006bde0a4da21b9fce0dbf0b99254e2bb'
    Extracting layer 'sha256:0e849e1c9aff93b56c9ee297371ea93e5cd463b0078a28cfacb9366df446c559' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/cert-manager@sha256:43b4ba060a8a074d57d75d63ab0ddc1fb2265b5cf57c9b6856dcf3ce1f1c2a76'
    Extracting layer 'sha256:4d71f041ec782ada54842ee4529145f43b26c366ecb091ae90f4674d400853c2' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/cert-manager@sha256:c087c02ac3957486baeb91f9e0713a2d672dd7532c739ed836e9795e4160aeb6'
    Extracting layer 'sha256:e7c3eba858dcb144db64f96a984e9ea11c49c66f4284b53a3d5cc381dd9b62d1' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/cert-manager@sha256:ca4c551c1e9c5bc0e2b554f20651c9538c97a1159ccf9c9b640457e18cdec039'
    Extracting layer 'sha256:0bd8e44ad92aa15fee5e4a43c06531cc646deca55c61fa0dbd298e263fa3f077' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/contour@sha256:630f7a1a73b1c4912a983a08aceadb35cc6af118ef0b35e01ee5f26cd884a4da'
    Extracting layer 'sha256:1410f6c9bac830a244af883267f9be462737cdeb8ecc2ad29ce7411ec16a7fb8' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/contour@sha256:73cea143220db395b5bc17f6b7f72f206eea206b29eb31f800f31ec2b23f352b'
    Extracting layer 'sha256:eacc997a73b8d6fb6f1aca9488b36e5763f1b57c72de273eee2b2dbeb7ae710d' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/contour@sha256:be540433d503c870a589570f541eb40482271f3e27ac3563c33d42c1bd3b0647'
    Extracting layer 'sha256:7e64f5ac6716f3179de8840851830cc93f205268e0846f15d56e8023b31a719f' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/contour@sha256:df21fcabacaf247a01ae4ae75ee227d181060a7e981962224467d2222f765054'
    Extracting layer 'sha256:5dfe9ca710e9be6bbd8102a40866711d81591eb008490ca9ddc9878e78906cb7' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/external-dns@sha256:3eb57df9d359782f14173219a19f545139789f7de9e83f25e2779c07958e40f9'
    Extracting layer 'sha256:569f32d9295498c2e8780c5e12383ea85cdfa5a117e5220ab7dd31762a8b3513' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/external-dns@sha256:9dbe4f8744c07f3113bc80d169503299a24f0651c154b671b2ad8430297a5b17'
    Extracting layer 'sha256:02e5bf9f610cb757719346113f5f0a7edc1baa27ada4eb9f8d5b3d271827838c' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/fluent-bit@sha256:e8c0873ef8e1ea04f32b88afbab64fc4338d0c0536ef2f91b9dd54bee2c59ed7'
    Extracting layer 'sha256:77fc07635988d164b9c00f3ff2002096b587a67efa5edbef45a6be42adb3bde6' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/gatekeeper@sha256:508b518a00956f8cf365666541db34023461913937d93c4492fb585be8d5de20'
    Extracting layer 'sha256:a50d65daad1da6e19d718d1c0f80018e9a7602d80b7aeae04d6872523a3c69c9' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/gatekeeper@sha256:af58b5e95bf53dde9c82c7112a2f62dca749e9f2adf3b2488e3ae0b5ba80852a'
    Extracting layer 'sha256:a7b519c44117a793dd0d947394e19d701f7124ac0654c2d26d792a225f7e1e5b' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/grafana@sha256:53d41d9ac1534fc381efa5bb181aa4cac1ec26fc77c7ffadb34550930112e193'
    Extracting layer 'sha256:2fad530bfed104be9c02c469b0cc7ba02e23adb85ac90ca15e5ecb75c08e1ce2' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/harbor@sha256:32237a6fefd6e19f9d2b52508276a8480d47c21464c47846d0cff2fd68426764'
    Extracting layer 'sha256:422fb14ae50a2a945c584d4e5d55efc3ed97a7be398075f178653e82a961ddb4' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/harbor@sha256:610c0ec78afd7095eed05624976ab83bd05c41b1f926fb2ccb7cf1bdfc239f37'
    Extracting layer 'sha256:0cb50a5586381fe80afa1555c2174af3d183804eb6e853393f4d4a3ce6c550c2' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/knative-serving@sha256:3e1ea2dd010f93c62ee07815afe830a3347648814322638ed9662781a0afdf85'
    Extracting layer 'sha256:64a94a494cc8a8e866bd38dba80ce69be11fcbabb4cae9b7d44948c94a664ce2' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/knative-serving@sha256:648b8eb811806513e9681912899be3bdd086b8b7d99d0b7d52dbf44024ee858f'
    Extracting layer 'sha256:7e9045cd11d97c4db55e1c267841c5c9b303e43c80d869de9c4b23aa597caed5' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/knative-serving@sha256:e99da49f3a7515e787ca5278a5d99600e9ee6eaa863e5283e0f6477a7c3d8bc1'
    Extracting layer 'sha256:213fbf27607ab16493fc1c9f4a3003c3016e2648a347d37c22fa99dc563e2b5a' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/kpack@sha256:8e83dca61e3574ea1bb0ea7a6aabec1cb317eebbedd212439f70b698df6a4ba5'
    Extracting layer 'sha256:d3b757296f6169898361d3e956eecc779c3c45e92cd03579497c76dd7f3ba496' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/local-path-storage@sha256:336f49fb455dae12a03507c1874fdbee38240f4243c512539a2bb1835580462f'
    Extracting layer 'sha256:5901fc65949c0eecafa81af2617120a29db73137c054935b92422e9028c5f968' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/local-path-storage@sha256:e0db08cc6e83efb1f772ab9714d78900b5634146c266954abc805461a005beb1'
    Extracting layer 'sha256:273aa5cd8aa9b16b57f5ac952edc37e56350f25cf46569327744907c1c7e2e53' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/multus-cni@sha256:222728c4313e7f5015f5f59fdb8be114e1929b5b1349eaec59d6e710e1bf1395'
    Extracting layer 'sha256:c216f2d71abe0c60cbc1dcb6c16bd283ee37940f17f6158e9c1e67a14661f0a4' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/prometheus@sha256:78ccd19454aa309b4e43e529e787d9e15d7ec95e469bcfe9e2f42c05973ae4c5'
    Extracting layer 'sha256:d55eb5451011af714cdf4f8b005a55b4f8b12c440b67e774afd0a1aec5197dfe' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/velero@sha256:d1334e22e7c3e4e3e8e1b9b7a728f5f4b00c885f5963a74e20eac95484d82313'
    Extracting layer 'sha256:c4fda0e2428beb7a65c8c8dce4476e7721945f9386245edbb39afc2c5e86903c' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/velero@sha256:fe89fb0897df4f9df2665c305909c6cc5cda42bacf2fbfc2e111764e0431baf8'
    Extracting layer 'sha256:bf87f0fb5d0e9a423b732571736736e35a390018fd2dd6873c30aefdcad1ee6c' (1/1)
  Pulling nested bundle 'projects.registry.vmware.com/tce/whereabouts@sha256:a070b08ad08e787c718ef0b7cc2586afe9b1655d5b50406d3801c986d23f51d9'
    Extracting layer 'sha256:9b69dc26f2a3d3f82986b652a5b199ecf05d1c9dedbf7c6956a94aaab056b162' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded
turtle@turtle-a01 ~ % cd /tmp/tce-main 

</details>

Now that we have pulled all the nested bundles including the package images, let's use `harbor` package as an example:

turtle@turtle-a01 tce-main % cat /tmp/tce-main/.imgpkg/images.yml | grep "image:" | grep harbor 
  image: projects.registry.vmware.com/tce/harbor@sha256:32237a6fefd6e19f9d2b52508276a8480d47c21464c47846d0cff2fd68426764
  image: projects.registry.vmware.com/tce/harbor@sha256:610c0ec78afd7095eed05624976ab83bd05c41b1f926fb2ccb7cf1bdfc239f37

These are the two packages of `harbor` which are supported [tanzu packages](https://tanzucommunityedition.io/docs/latest/package-management/)

Let's get a list of images from one of these packages

turtle@turtle-a01 bundles % pwd       
/tmp/tce-main/.imgpkg/bundles
turtle@turtle-a01 bundles % cd sha256-32237a6fefd6e19f9d2b52508276a8480d47c21464c47846d0cff2fd68426764


turtle@turtle-a01 sha256-32237a6fefd6e19f9d2b52508276a8480d47c21464c47846d0cff2fd68426764 % cat .imgpkg/images.yml| grep "image:"
  image: index.docker.io/goharbor/harbor-core@sha256:b4703fd32a9dc4de208649405ac88de36126122f5d30ea0e7e2eff5626f28397
  image: index.docker.io/goharbor/harbor-db@sha256:4fac4598a1066611a7aaca000a56f81b951b5d047d6cefc8643fc4ecc638da67
  image: index.docker.io/goharbor/harbor-exporter@sha256:17b33dc8816db4fa214ce1765d5cc9e6ea6e2f7b80f0f64182aa84c9575bc3d2
  image: index.docker.io/goharbor/harbor-jobservice@sha256:e555c6736db6383e8147c847c733a36a16400f292e78ee4040c3396bae486da5
  image: index.docker.io/goharbor/harbor-portal@sha256:bd5103391fc37ecfa84cb07a4a7ff0dbed88b72f02d92e38d730da7141518b6c
  image: index.docker.io/goharbor/harbor-registryctl@sha256:f6764ff0ee618c916568d7bfbf144a9f9ec70c23589985efa451614aa735a371
  image: index.docker.io/goharbor/notary-server-photon@sha256:b1f41cfc903ecdf1e5b4efe7e640fe858f70a1c5cd0d3c51d4cc4ab34eb568fc
  image: index.docker.io/goharbor/notary-signer-photon@sha256:a0b375fa2e7c27de1cf41e23800e97878adcdeb236ecbf94dea16d03ed4449ef
  image: index.docker.io/goharbor/redis-photon@sha256:dc34c1f49c0bffe5fff540d1902bd45f3b6231f08d7abc244885744066085781
  image: index.docker.io/goharbor/registry-photon@sha256:c5c65ec7c78ca3965f75536d83d8252780191819b45e8ab5a7e9ff07f9e11046
  image: index.docker.io/goharbor/trivy-adapter-photon@sha256:42649438283db6db7e2086058d690be776266a25bea7255b67d24be0352b85e7

This list can then be used to scan all the relevant images one by one that are shipped with harbor packages.

Similarly, to generalize for all packages using a simple yaml parser coupled with directory traversals of nested bundles, will give us a list of images that are part of each package

### Run a vulnerability scan on these images to detect fixable CVEs

Using the list of images, we can then run a container image scanner to get a list of CVEs that can be fixed in these images. In k/k vulnerability scanning, we used snyk scanning for scanning all k/k images in a release. However, in case of Tanzu Community Edition package images, snyk has a limitation. Snyk scanning today does not support photon base images, whereas many tanzu packages like harbor use photon as their base image.

Trivy is another image vulnerability scanner from Aqua that is well known. It supports photon, ubuntu, debian and redhat base images, which should cover all the possible base images used by tanzu packages.

It also supports multiple filtering options, such as selecting output to be in JSON format, filtering out any vulnerabilities that do _not_ have fixes available upstream.

An example command with following filters could look like this:

trivy image --ignore-unfixed -f json -o results.json golang:1.15

It might also be reasonable to focus and hence fail only on vulnerabilities that are critical and fixable in the beginning. Once we have a better handle on these vulnerabilties, we can start assessing and fixing other vulnerabilities that are of lower severity (e.g. high, medium, and low in that order). A command with all of the above filtering that looks for `critical` severity vulnerabilities would look like this:

trivy image --ignore-unfixed --severity CRITICAL -f json -o golang-1.15.json golang:1.15

Example output of this command i.e. the contents of golang-1.15.json looks something like this:

<!-- markdownlint-disable MD033 -->
<details><summary>Click to view result</summary>
<!-- markdownlint-enable MD033 -->

{
  "SchemaVersion": 2,
  "ArtifactName": "golang:1.15",
  "ArtifactType": "container_image",
  "Metadata": {
    "OS": {
      "Family": "debian",
      "Name": "10.10"
    },
    "ImageID": "sha256:40349a2425efb3650843103390e2ace030c4288c96f3619cafb16320ab021efb",
    "DiffIDs": [
      "sha256:afa3e488a0ee76983343f8aa759e4b7b898db65b715eb90abc81c181388374e3",
      "sha256:4b0edb23340c111e75557748161eed3ca159584871569ce7ec9b659e1db201b4",
      "sha256:5a9a65095453efb15a9b378a3c1e7699e8004744ecd6dd519bdbabd0ca9d2efc",
      "sha256:ad83f0aa5c0abe35b7711f226037a5557292d479c21635b3522e1e5a41e3ce23",
      "sha256:d1c59e37fbfc7294184d6fbe4ff8e1690d9119b6233f91af5ad0a4b36e45dff7",
      "sha256:a5e37ec5a23acf75fddf2646a3cdbd88dccd38f5d26faed0333836c66511706c",
      "sha256:09d4db84ee6b83c3cf1df95024cdfa8f12d0b2b09d6c990b69c66767750c5485"
    ],
    "RepoTags": [
      "golang:1.15"
    ],
    "RepoDigests": [
      "golang@sha256:ea080cc817b02a946461d42c02891bf750e3916c52f7ea8187bccde8f312b59f"
    ],
    "ImageConfig": {
      "architecture": "amd64",
      "container": "bd6c6af94cead4d10509117797499c99a9a05979f9b96b6c6286929534303e4a",
      "created": "2021-08-05T21:24:52.946335381Z",
      "docker_version": "20.10.7",
      "history": [
        {
          "created": "2021-07-22T00:45:30.581499538Z",
          "created_by": "/bin/sh -c #(nop) ADD file:e952f6979e4b0ead00b6906db1dd70eb9beb564a04e2f02e2e0cff8614920216 in / "
        },
        {
          "created": "2021-07-22T00:45:31.247048715Z",
          "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
          "empty_layer": true
        },
        {
          "created": "2021-07-22T01:12:14.174675844Z",
          "created_by": "/bin/sh -c set -eux; \tapt-get update; \tapt-get install -y --no-install-recommends \t\tca-certificates \t\tcurl \t\tnetbase \t\twget \t; \trm -rf /var/lib/apt/lists/*"
        },
        {
          "created": "2021-07-22T01:12:20.292596966Z",
          "created_by": "/bin/sh -c set -ex; \tif ! command -v gpg \u003e /dev/null; then \t\tapt-get update; \t\tapt-get install -y --no-install-recommends \t\t\tgnupg \t\t\tdirmngr \t\t; \t\trm -rf /var/lib/apt/lists/*; \tfi"
        },
        {
          "created": "2021-07-22T01:12:38.897001154Z",
          "created_by": "/bin/sh -c apt-get update \u0026\u0026 apt-get install -y --no-install-recommends \t\tgit \t\tmercurial \t\topenssh-client \t\tsubversion \t\t\t\tprocps \t\u0026\u0026 rm -rf /var/lib/apt/lists/*"
        },
        {
          "created": "2021-07-23T02:01:18.317859294Z",
          "created_by": "/bin/sh -c apt-get update \u0026\u0026 apt-get install -y --no-install-recommends \t\tg++ \t\tgcc \t\tlibc6-dev \t\tmake \t\tpkg-config \t\u0026\u0026 rm -rf /var/lib/apt/lists/*"
        },
        {
          "created": "2021-07-23T02:01:18.934692655Z",
          "created_by": "/bin/sh -c #(nop)  ENV PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "empty_layer": true
        },
        {
          "created": "2021-08-05T21:24:38.110663502Z",
          "created_by": "/bin/sh -c #(nop)  ENV GOLANG_VERSION=1.15.15",
          "empty_layer": true
        },
        {
          "created": "2021-08-05T21:24:50.840607806Z",
          "created_by": "/bin/sh -c set -eux; \t\tdpkgArch=\"$(dpkg --print-architecture)\"; \turl=; \tcase \"${dpkgArch##*-}\" in \t\t'amd64') \t\t\turl='https://dl.google.com/go/go1.15.15.linux-amd64.tar.gz'; \t\t\tsha256='0885cf046a9f099e260d98d9ec5d19ea9328f34c8dc4956e1d3cd87daaddb345'; \t\t\t;; \t\t'armel') \t\t\texport GOARCH='arm' GOARM='5' GOOS='linux'; \t\t\t;; \t\t'armhf') \t\t\turl='https://dl.google.com/go/go1.15.15.linux-armv6l.tar.gz'; \t\t\tsha256='7192603af50afb23c9d8cd14d2b2c19e0985a34d3eca685fa098df7893000d19'; \t\t\t;; \t\t'arm64') \t\t\turl='https://dl.google.com/go/go1.15.15.linux-arm64.tar.gz'; \t\t\tsha256='714abb01af210473dd6af331094ad6847162eff81a7fc7241d24f5a85496c9fa'; \t\t\t;; \t\t'i386') \t\t\turl='https://dl.google.com/go/go1.15.15.linux-386.tar.gz'; \t\t\tsha256='3310fb0e48b0907bb520f6e3c6dcff63cc0913b92a76456f12980d0eb13b77d4'; \t\t\t;; \t\t'mips64el') \t\t\texport GOARCH='mips64le' GOOS='linux'; \t\t\t;; \t\t'ppc64el') \t\t\turl='https://dl.google.com/go/go1.15.15.linux-ppc64le.tar.gz'; \t\t\tsha256='37f3b99e21d0324a6583159e14e42e57e56561abbf7bf68bef3d8f57b29e39c0'; \t\t\t;; \t\t's390x') \t\t\turl='https://dl.google.com/go/go1.15.15.linux-s390x.tar.gz'; \t\t\tsha256='eae39d97df6b758636d5427be0b083dbf9d49007b302825ac6c8645de039aaab'; \t\t\t;; \t\t*) echo \u003e\u00262 \"error: unsupported architecture '$dpkgArch' (likely packaging update needed)\"; exit 1 ;; \tesac; \tbuild=; \tif [ -z \"$url\" ]; then \t\tbuild=1; \t\turl='https://dl.google.com/go/go1.15.15.src.tar.gz'; \t\tsha256='0662ae3813330280d5f1a97a2ee23bbdbe3a5a7cfa6001b24a9873a19a0dc7ec'; \t\techo \u003e\u00262; \t\techo \u003e\u00262 \"warning: current architecture ($dpkgArch) does not have a corresponding Go binary release; will be building from source\"; \t\techo \u003e\u00262; \tfi; \t\twget -O go.tgz.asc \"$url.asc\" --progress=dot:giga; \twget -O go.tgz \"$url\" --progress=dot:giga; \techo \"$sha256 *go.tgz\" | sha256sum --strict --check -; \t\texport GNUPGHOME=\"$(mktemp -d)\"; \tgpg --batch --keyserver keyserver.ubuntu.com --recv-keys 'EB4C 1BFD 4F04 2F6D DDCC EC91 7721 F63B D38B 4796'; \tgpg --batch --verify go.tgz.asc go.tgz; \tgpgconf --kill all; \trm -rf \"$GNUPGHOME\" go.tgz.asc; \t\ttar -C /usr/local -xzf go.tgz; \trm go.tgz; \t\tif [ -n \"$build\" ]; then \t\tsavedAptMark=\"$(apt-mark showmanual)\"; \t\tapt-get update; \t\tapt-get install -y --no-install-recommends golang-go; \t\t\t\t( \t\t\tcd /usr/local/go/src; \t\t\texport GOROOT_BOOTSTRAP=\"$(go env GOROOT)\" GOHOSTOS=\"$GOOS\" GOHOSTARCH=\"$GOARCH\"; \t\t\t./make.bash; \t\t); \t\t\t\tapt-mark auto '.*' \u003e /dev/null; \t\tapt-mark manual $savedAptMark \u003e /dev/null; \t\tapt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \t\trm -rf /var/lib/apt/lists/*; \t\t\t\tgo install std; \t\t\t\trm -rf \t\t\t/usr/local/go/pkg/*/cmd \t\t\t/usr/local/go/pkg/bootstrap \t\t\t/usr/local/go/pkg/obj \t\t\t/usr/local/go/pkg/tool/*/api \t\t\t/usr/local/go/pkg/tool/*/go_bootstrap \t\t\t/usr/local/go/src/cmd/dist/dist \t\t; \tfi; \t\tgo version"
        },
        {
          "created": "2021-08-05T21:24:51.802428526Z",
          "created_by": "/bin/sh -c #(nop)  ENV GOPATH=/go",
          "empty_layer": true
        },
        {
          "created": "2021-08-05T21:24:52.00134059Z",
          "created_by": "/bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "empty_layer": true
        },
        {
          "created": "2021-08-05T21:24:52.743035633Z",
          "created_by": "/bin/sh -c mkdir -p \"$GOPATH/src\" \"$GOPATH/bin\" \u0026\u0026 chmod -R 777 \"$GOPATH\""
        },
        {
          "created": "2021-08-05T21:24:52.946335381Z",
          "created_by": "/bin/sh -c #(nop) WORKDIR /go",
          "empty_layer": true
        }
      ],
      "os": "linux",
      "rootfs": {
        "type": "layers",
        "diff_ids": [
          "sha256:afa3e488a0ee76983343f8aa759e4b7b898db65b715eb90abc81c181388374e3",
          "sha256:4b0edb23340c111e75557748161eed3ca159584871569ce7ec9b659e1db201b4",
          "sha256:5a9a65095453efb15a9b378a3c1e7699e8004744ecd6dd519bdbabd0ca9d2efc",
          "sha256:ad83f0aa5c0abe35b7711f226037a5557292d479c21635b3522e1e5a41e3ce23",
          "sha256:d1c59e37fbfc7294184d6fbe4ff8e1690d9119b6233f91af5ad0a4b36e45dff7",
          "sha256:a5e37ec5a23acf75fddf2646a3cdbd88dccd38f5d26faed0333836c66511706c",
          "sha256:09d4db84ee6b83c3cf1df95024cdfa8f12d0b2b09d6c990b69c66767750c5485"
        ]
      },
      "config": {
        "Cmd": [
          "bash"
        ],
        "Env": [
          "PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "GOLANG_VERSION=1.15.15",
          "GOPATH=/go"
        ],
        "Image": "sha256:d84514ab6bb0dd5c64ad829d8520bcd23b2c9f4e65690d60d3ea5f9ea19cac33",
        "WorkingDir": "/go"
      }
    }
  },
  "Results": [
    {
      "Target": "golang:1.15 (debian 10.10)",
      "Class": "os-pkgs",
      "Type": "debian",
      "Vulnerabilities": [
        {
          "VulnerabilityID": "CVE-2021-3711",
          "VendorIDs": [
            "DSA-4963-1"
          ],
          "PkgName": "libssl1.1",
          "InstalledVersion": "1.1.1d-0+deb10u6",
          "FixedVersion": "1.1.1d-0+deb10u7",
          "Layer": {
            "Digest": "sha256:c040670e5e559fd936db175530ad4c1dd014bd25b2bf25ea19fa20554fe2d736",
            "DiffID": "sha256:4b0edb23340c111e75557748161eed3ca159584871569ce7ec9b659e1db201b4"
          },
          "SeveritySource": "nvd",
          "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2021-3711",
          "Title": "openssl: SM2 Decryption Buffer Overflow",
          "Description": "In order to decrypt SM2 encrypted data an application is expected to call the API function EVP_PKEY_decrypt(). Typically an application will call this function twice. The first time, on entry, the \"out\" parameter can be NULL and, on exit, the \"outlen\" parameter is populated with the buffer size required to hold the decrypted plaintext. The application can then allocate a sufficiently sized buffer and call EVP_PKEY_decrypt() again, but this time passing a non-NULL value for the \"out\" parameter. A bug in the implementation of the SM2 decryption code means that the calculation of the buffer size required to hold the plaintext returned by the first call to EVP_PKEY_decrypt() can be smaller than the actual size required by the second call. This can lead to a buffer overflow when EVP_PKEY_decrypt() is called by the application a second time with a buffer that is too small. A malicious attacker who is able present SM2 content for decryption to an application could cause attacker chosen data to overflow the buffer by up to a maximum of 62 bytes altering the contents of other data held after the buffer, possibly changing application behaviour or causing the application to crash. The location of the buffer is application dependent but is typically heap allocated. Fixed in OpenSSL 1.1.1l (Affected 1.1.1-1.1.1k).",
          "Severity": "CRITICAL",
          "CweIDs": [
            "CWE-120"
          ],
          "CVSS": {
            "nvd": {
              "V2Vector": "AV:N/AC:L/Au:N/C:P/I:P/A:P",
              "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
              "V2Score": 7.5,
              "V3Score": 9.8
            },
            "redhat": {
              "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
              "V3Score": 9.8
            }
          },
          "References": [
            "http://www.openwall.com/lists/oss-security/2021/08/26/2",
            "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3711",
            "https://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=59f5e75f3bced8fc0e130d72a3f582cf7b480b46",
            "https://lists.apache.org/thread.html/r18995de860f0e63635f3008fd2a6aca82394249476d21691e7c59c9e@%3Cdev.tomcat.apache.org%3E",
            "https://lists.apache.org/thread.html/rad5d9f83f0d11fb3f8bb148d179b8a9ad7c6a17f18d70e5805a713d1@%3Cdev.tomcat.apache.org%3E",
            "https://security.netapp.com/advisory/ntap-20210827-0010/",
            "https://security.netapp.com/advisory/ntap-20211022-0003/",
            "https://ubuntu.com/security/notices/USN-5051-1",
            "https://www.debian.org/security/2021/dsa-4963",
            "https://www.openssl.org/news/secadv/20210824.txt",
            "https://www.oracle.com/security-alerts/cpuoct2021.html",
            "https://www.tenable.com/security/tns-2021-16",
            "https://www.tenable.com/security/tns-2022-02"
          ],
          "PublishedDate": "2021-08-24T15:15:00Z",
          "LastModifiedDate": "2022-01-06T09:15:00Z"
        },
        {
          "VulnerabilityID": "CVE-2021-3711",
          "VendorIDs": [
            "DSA-4963-1"
          ],
          "PkgName": "openssl",
          "InstalledVersion": "1.1.1d-0+deb10u6",
          "FixedVersion": "1.1.1d-0+deb10u7",
          "Layer": {
            "Digest": "sha256:c040670e5e559fd936db175530ad4c1dd014bd25b2bf25ea19fa20554fe2d736",
            "DiffID": "sha256:4b0edb23340c111e75557748161eed3ca159584871569ce7ec9b659e1db201b4"
          },
          "SeveritySource": "nvd",
          "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2021-3711",
          "Title": "openssl: SM2 Decryption Buffer Overflow",
          "Description": "In order to decrypt SM2 encrypted data an application is expected to call the API function EVP_PKEY_decrypt(). Typically an application will call this function twice. The first time, on entry, the \"out\" parameter can be NULL and, on exit, the \"outlen\" parameter is populated with the buffer size required to hold the decrypted plaintext. The application can then allocate a sufficiently sized buffer and call EVP_PKEY_decrypt() again, but this time passing a non-NULL value for the \"out\" parameter. A bug in the implementation of the SM2 decryption code means that the calculation of the buffer size required to hold the plaintext returned by the first call to EVP_PKEY_decrypt() can be smaller than the actual size required by the second call. This can lead to a buffer overflow when EVP_PKEY_decrypt() is called by the application a second time with a buffer that is too small. A malicious attacker who is able present SM2 content for decryption to an application could cause attacker chosen data to overflow the buffer by up to a maximum of 62 bytes altering the contents of other data held after the buffer, possibly changing application behaviour or causing the application to crash. The location of the buffer is application dependent but is typically heap allocated. Fixed in OpenSSL 1.1.1l (Affected 1.1.1-1.1.1k).",
          "Severity": "CRITICAL",
          "CweIDs": [
            "CWE-120"
          ],
          "CVSS": {
            "nvd": {
              "V2Vector": "AV:N/AC:L/Au:N/C:P/I:P/A:P",
              "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
              "V2Score": 7.5,
              "V3Score": 9.8
            },
            "redhat": {
              "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
              "V3Score": 9.8
            }
          },
          "References": [
            "http://www.openwall.com/lists/oss-security/2021/08/26/2",
            "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3711",
            "https://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=59f5e75f3bced8fc0e130d72a3f582cf7b480b46",
            "https://lists.apache.org/thread.html/r18995de860f0e63635f3008fd2a6aca82394249476d21691e7c59c9e@%3Cdev.tomcat.apache.org%3E",
            "https://lists.apache.org/thread.html/rad5d9f83f0d11fb3f8bb148d179b8a9ad7c6a17f18d70e5805a713d1@%3Cdev.tomcat.apache.org%3E",
            "https://security.netapp.com/advisory/ntap-20210827-0010/",
            "https://security.netapp.com/advisory/ntap-20211022-0003/",
            "https://ubuntu.com/security/notices/USN-5051-1",
            "https://www.debian.org/security/2021/dsa-4963",
            "https://www.openssl.org/news/secadv/20210824.txt",
            "https://www.oracle.com/security-alerts/cpuoct2021.html",
            "https://www.tenable.com/security/tns-2021-16",
            "https://www.tenable.com/security/tns-2022-02"
          ],
          "PublishedDate": "2021-08-24T15:15:00Z",
          "LastModifiedDate": "2022-01-06T09:15:00Z"
        }
      ]
    }
  ]
}

</details>

It is expected that this image will have some vulnerabilities that can be fixed as this is an old golang image.

After the scans are complete, we can parse the JSON files one by one to check if either of them have non-zero number of CVEs. If we find one or more such files, we fail the job discretly. The reason to fail it discretely is to not have a public list of images in testgrid that have critical vulnerabilities. Having such a list will reduce the work and effort an attacker needs to exert to carry out a successful exploit. So in a game of cat and mouse we do not want to give the attacker this advantage. This strategy of finding a middle ground in doing security publicly and not sharing critical vulnerability publicly was used in k/k vulnerability scanning after discussions between SIG release, SIG architecture and SIG Security.

Other potential option could be limiting access to testgrid dashboard to maintainers similar to dependabot. This will give us a chance to be more verbose on sharing which image failed exactly and what CVE(s) was detected.

### Alerting on job failures

The scanning job can fail when a vulnerability is found or when it fails for some other reason, e.g. scanner failed for some unknown reason, filtering failed or scanner timed out. In both cases, it is useful to get alerted on it. For this, we should create a private google group to get alerted on using prow's built-in mail based alerting functionality, with members who play the following roles:

- Community Edition Maintainers
- Release Engineering team
- Security Liasons

For the sake of redundancy and to limit exposure to sensitive data like fixable vulnerabilties, it is recommended that only 2 or 3 people (across different timezones) in each role should be added. Ideally having an alias for each of these roles, will help in managing / rotating members from this google group.

### Running the prow job periodically

Prow has built-in support to configure via yaml the exact interval for successive job runs. In k/k a similar vulnerability scan is run every 6 hours. So we could use the same time duration for running this job too.

### User Stories

#### Story 1

As a community-edition maintainer, I want to be aware of vulnerabilities in tanzu package images, so that I can bump the package versions that fix the vulnerabilities or reach out to repo maintainer of the package to get it fixed

#### Story 2

As a community-edition maintainer, I want to restrict the information about known vulnerabilities in package images, so that an attacker does not have access to this information at the same time as me

#### Story 3

As a community edition maintainer, I want information about vulnerabilities close to the code and testing environment, so that it can be fixed with the same level of ease as bug fixes instead of having to triage another out of band means of tracking vulnerabilities

#### Story 4

As a community edition maintainer, I want to be alerted only on vulnerabilities that are actionable, so that I can focus on the ones that need action instead of getting bombarded with poor quality vulnerability alerts that consume time with limited benefits to overall security of the project

### Constraints

- Selection of vulnerability scanner should support all the base images
- No licensing issues are encountered (i.e. not exceeding the limit on free version)
- Community edition jobs can be run on prow and test-grid. Note: This is not a hard requirement, but will simplify and encourage reuse of code and methodology as k/k container image vulnerability scanning also uses the same set of tools.
- Fixes to images are consumed on a best effort basis
- The vulnerability scanning process will be reasonably tool agnostic, such that in future when needed, the vulnerability scanner can be swapped with something else with minimal effort or change in process

### Security Model

## Access control

- Changes to code for vulnerability scanning job, will follow the same contribution guidelines and review workflow
- The vulnerability dashboard on testgrid can be publicly accessible (read-only) if the output of the job run does not include the reason for failure. A maintainer / security liaison would run the script locally (in private) to identify which image is vulnerable and then triage appropriately.
- The vulnerability dashboard on testgrid should not be visible publicly if the output of the job run does include the reason for failure.

### Risks and Mitigations

Most of them covered in [Constraints](https://hackmd.io/uUzc5C8ESv6g2WRnd-4H1A#Constraints) section. Happy to get feedback on this and add more that were missed.

## Compatibility

N/A

## Alternatives

We would be limited to what we have today, where we are reactive to any end user requests to fix vulnerabilities, which results in lack of visibility on vulnerabilities in tanzu packages and a risk of supporting packages that have known fixable vulnerabilties across different community-edition releases.

## Upgrade Strategy

- Vulnerability scanner tool (e.g. trivy) will be downloaded and installed on each job run so that only the latest tool version is used
- Any rare breaking changes between versions will result in job failure, which can be fixed as needed

## Additional Details

Keeping this section blank as this is not targeting a release (yet)
<!---
### Test Plan

**Note:** *Section not required until targeted at a release.*

Consider the following in developing a test plan for this enhancement:

* Will there be e2e and integration tests, in addition to unit tests?
* How will it be tested in isolation vs with other components?

No need to outline all of the test cases, just the general strategy. Anything
that would count as tricky in the implementation and anything particularly
challenging to test should be called out.

All code is expected to have adequate tests (eventually with coverage
expectations).

### Graduation Criteria

**Note:** *Section not required until targeted at a release.*

Define graduation milestones.

These may be defined in terms of API maturity, or as something else. Initial
proposal should keep this high-level with a focus on what signals will be looked
at to determine graduation.

Consider the following in developing the graduation criteria for this
enhancement:

* [Maturity levels (`alpha`, `beta`, `stable`)][maturity-levels]
* [Deprecation policy][deprecation-policy]

Clearly define what graduation means by either linking to the [API doc
definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning),
or by redefining what graduation means.

In general, we try to use the same stages (alpha, beta, GA), regardless how the
functionality is accessed.

[maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions
[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/

### Version Skew Strategy

If applicable, how will the component handle version skew with other components?
What are the guarantees? Make sure this is in the test plan.
-->

<!-- Links -->
[community meeting]: https://hackmd.io/CiuO4V0AT6WL_TgA47MXBA