# Verifying images and attestations using AWS Signer, Notation, and Kyverno

We are excited to announce the launch of [kyverno-notation-aws](https://github.com/nirmata/kyverno-notation-aws), a Nirmata extension service that executes AWS Signer plugin for Notation to verify image signatures and attestations.

AWS Signer, announced by Amazon AWS a few months ago is a new capability that gives us native AWS support for signing and verifying container images stored in container registries like Amazon Elastic Container Registry (Amazon ECR). It supports features like cross-account signing, signature validity duration, and profile lifecycle management with cancellation and revocation operations.

AWS Signer supports signing and verifying container images and is integrated with Notation, an open-source Notary project within the Cloud Native Computing Foundation (CNCF). With contributions from AWS and others, Notary is an open standard and client implementation that allows for vendor-specific plugins for key management and other integrations. AWS Signer manages signing keys, key rotation, and PKI management for you and is integrated with Notation through a curated plugin that provides a simple client-based workflow.

The Nirmata extension service is invoked from a Kyverno policy which then calls the notation service configured with AWS Signer to provide image signature and attestation verification as well as digest mutation.

# Features

## **Image Signature Verification**

The Nirmata extension service provides image signature verification capabilities. Verifying images establishes authenticity and provenance, giving you the ability to determine if content comes from a particular party. This allows you to only allow images from trusted parties to be used in your container image builds and deployments

The Nirmata extension service is invoked from a Kyverno policy which passes it a list of images to be verified. The service then verifies [notation](https://notaryproject.dev/) format signatures for container images using the [AWS Signer](https://docs.aws.amazon.com/signer/index.html) notation plugin and returns responses back to Kyverno.

The service manages Notation [trust policies](https://github.com/notaryproject/specifications/blob/main/specs/trust-store-trust-policy.md#trust-policy) and [trust stores](https://github.com/notaryproject/specifications/blob/main/specs/trust-store-trust-policy.md#trust-store) as Kubernetes resources.

## **Image Digest Mutation**

Image tags are mutable in nature and can be spoofed. However, we can use image digests that are immutable to ensure better security, and image integrity and prevent tampering. The service allows users to replace image tags with digests.

The Kyverno policy passes the `images` variable to the services' `/checkimages` endpoint. The result returns a list of images with their JSON path and digests so Kyverno can mutate each image in the admission payload.

Here is an example:

**Response object structure**

{
  "verified": true,
  "message": "...",
  "results": [
    {
       "name": "container1",
       "path":  "/spec/containers/0",
       "image":  "ghcr.io/kyverno/test-verify-image@sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105"
    } 
  ]
}

* `verified: true` means that all the images provided to the extensions were verified. We can use this variable when we only want to validate image verification.
* `message`: “” returns the error message when verification fails.
* `results` is an array of objects containing the name of the container image, the path of the container image, and the mutated image.

**Kyverno policy fragment**

mutate:
   foreach:
   - list: "response.results"
     patchesJson6902: |-
       - path: {{ element.path }}
         op: replace
         value: {{ element.image }}

This policy fragment describes a mutation where we are looping through the array of results we have received from the extension and do a replace operation, where we replace the current image with the same image containing digests.

## **Attestation Verification**

In addition to verifying signatures, the extension service can verify signed metadata i.e. [attestations](https://nirmata.com/2022/03/15/a-map-for-kubernetes-supply-chain-security/).

To verify attestations, the Kyverno policy can optionally pass a variable called `attestations` in the request:

- key: attestations
  value:
    - imageReference: "*"
        - name: sbom/cyclone-dx
          conditions:
            all:
            - key: \{{creationInfo.licenseListVersion}}
              operator: Equals
              value: "3.17"message: invalid license version
        - name: application/sarif+json
          conditions:
            all:
            - key: \{{ element.components[].licenses[].expression }}
              operator: AllNotIn
              value: ["GPL-2.0", "GPL-3.0"]
    - imageReference: "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo*"type:
        - name: application/vnd.cyclonedx
          conditions:
            all:
            - key: \{{ element.components[].licenses[].expression }}
              operator: AllNotIn
              value: ["GPL-2.0", "GPL-3.0"]

The `attestations` variable is a JSON array of where each entry has:

1. An `imageReference` to match images;
2. A type that specifies the name of the attestation; and
3. A list of conditions we want to verify the attestation data

In the example above we are verifying the following:

1. The attestations `sbom/cyclone-dx` and `application/sarif+json` exist for all images.
2. The `creationInfo.licenseListVersion` is equal to 3.17 in the SBOM and GPL licenses are not present.
3. The attestation `application/vnd.cyclonedx` is available for all versions of the `844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo` image and does not contain GPL licenses.

**NOTE:** The conditions key in the attestations must be escaped with `\` so Kyverno does not substitute them before executing the extension service.

## Caching

To prevent repeated lookups for verified images, the Nirmata extension has a built-in cache.

Caching is enabled by default and can be managed using the `--cacheEnabled` flag. The cache is a TTL-based cache, i.e., entries expire automatically after some time and the value of TTL can be customized using `--cacheTTLDurationSeconds` (default is 3600) and the max number of entries in the cache can be configured using `--cacheMaxSize` (default is 1000).

The cache stores the verification outcomes of images for the trust policy and verification outcomes of attestations with the trust policy and conditions. The cache is an in-memory cache that gets cleared when the pod is recreated. The cache will also be cleared when there is any change in trust policies and trust stores.

## Multi-Tenancy

In a shared cluster, each team may have different signatures and trust policies. To support such use cases, the extension allows configuring multiple [trust policies](https://github.com/notaryproject/specifications/blob/main/specs/trust-store-trust-policy.md#trust-policy) and [trust stores](https://github.com/notaryproject/specifications/blob/main/specs/trust-store-trust-policy.md#trust-store) as Kubernetes custom resources.

The extension service allows specifying what trust policy they want to use for verification thus enabling multi-tenancy. Multiple teams can share one cluster and have different trust policies separate from each other. To specify the trust policy, we can pass the `trustPolicy` variable in the request:

 - key: trustPolicy
   value: "tp-{{request.namespace}}"

or we can set the `DEFAULT_TRUST_POLICY` environmental variable. In the above example, we are dynamically using the trust policy for the namespace of the request.

## High Availability

Kyverno-notation-aws can be installed in a highly-available manner where additional replicas can be deployed for the plugin. The plugin does not use leader election for inbound API requests which means verification requests can be distributed and processed by all available replicas. Leader election is required for certificate management so therefore only one replica will handle these tasks at a given time.

Multiple replicas configured for the plugin can be used for both availability and scale. Vertical scaling of the individual replicas’ resources may also be performed to increase combined throughput.

# Demo

To install `kyverno-notation-aws` plugin, follow the instructions in [kyverno-notation-aws repository’s readme](https://github.com/nirmata/kyverno-notation-aws#install).

After installation, `kyverno-notation-aws` namespace should look as follows:

$ kubectl get all,secret,sa,lease,rolebindings -n kyverno-notation-aws

NAME                                        READY   STATUS    RESTARTS   AGE
pod/kyverno-notation-aws-5b49875b94-ftntr   1/1     Running   0          16m

NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/svc   ClusterIP   172.20.43.47   <none>        443/TCP   33d

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kyverno-notation-aws   1/1     1            1           33d

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/kyverno-notation-aws-5b49875b94   1         1         1       16m

NAME                                           TYPE                DATA   AGE
secret/root-secret                             kubernetes.io/tls   3      33d
secret/svc.kyverno-notation-aws.svc.tls-ca     kubernetes.io/tls   2      8d
secret/svc.kyverno-notation-aws.svc.tls-pair   kubernetes.io/tls   2      8d

NAME                                  SECRETS   AGE
serviceaccount/default                0         33d
serviceaccount/kyverno-notation-aws   0         33d

NAME                                             HOLDER                                  AGE
lease.coordination.k8s.io/kyverno-notation-aws   kyverno-notation-aws-5b49875b94-ftntr   8d

NAME                                                                     ROLE                             AGE
rolebinding.rbac.authorization.k8s.io/kyverno-notation-aws-rolebinding   Role/kyverno-notation-aws-role   33d

Next, we have to create a test namespace to test our plugin:

$ kubectl create ns test-notation

namespace/test-notation created

Now, we have to apply a Kyverno policy so Kyverno can interact with the plugin to verify an image:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-images     
spec:
  validationFailureAction: Enforce
  failurePolicy: Fail
  webhookTimeoutSeconds: 30
  schemaValidation: false
  rules:
  - name: call-aws-signer-extension
    match:
      any:
      - resources:
          namespaces:
          - test-notation
          kinds:
          - Pod
          operations:
            - CREATE
            - UPDATE
    context:
    - name: tlscerts
      apiCall:
        urlPath: "/api/v1/namespaces/kyverno-notation-aws/secrets/svc.kyverno-notation-aws.svc.tls-pair"
        jmesPath: "base64_decode( data.\"tls.crt\" )"
    - name: response
      apiCall:
        method: POST
        data:
        - key: images
          value: "{{images}}"
        - key: trustPolicy
          value: "tp-{{request.namespace}}"
        - key: attestations
          value: 
          - imageReference: "*"
            type: 
            - name: sbom/example
              conditions:
                all:
                - key: \{{creationInfo.licenseListVersion}}
                  operator: Equals
                  value: "3.17"
                  message: invalid license version
        service:
          url: https://svc.kyverno-notation-aws/checkimages
          caBundle: '{{ tlscerts }}'
    mutate:
      foreach:
      - list: "response.results"
        patchesJson6902: |-
            - path: {{ element.path }}
              op: replace
              value: {{ element.image }}

This policy does the following:

- Checks for Pod creation and update requests in `test-notation` namespace
- Sends an API call to `https://svc.kyverno-notation-aws/checkimages` with TLS certs extracted from the secret using an API call. The API call request is a POS request that contains the following
- Kyverno’s `images` variable which sends the image info of all the images used in the given resource.
- Trust policy name, to specify what trust policy to use for this request. Optional if `DEFAULT_TRUST_POLICY` environmental variable is set in the plugin.
- An attestations list as described in [attestation verification section](#Attestation-Verification).

Here, we have an unsigned image in AWS ECR `844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned`:

$ notation inspect 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned

Warning: Always inspect the artifact using digest(@sha256:...) rather than a tag(:v1-unsigned) because resolved digest may not point to the same signed artifact, as tags are mutable.
844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5 has no associated signature

When we run an unsigned image, it gives an error saying the image was not signed:

$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned --dry-run=server

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy check-images error: failed to apply policy check-images rules [call-aws-signer-extension: failed to load context: failed to fetch data for APICall: HTTP 406 Not Acceptable: failed to verify container kyverno-demo: failed to verify image {{844333597536.dkr.ecr.us-west-2.amazonaws.com kyverno-demo kyverno-demo v1-unsigned } /spec/containers/0/image}: no signature is associated with "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5", make sure the artifact was signed successfully]

When using an image that has an SBOM attached to it and is signed using notary `844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1`:

$ oras discover -o tree 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1

844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
├── application/vnd.cncf.notary.signature
│   └── sha256:04eed4edc7a549399648e9fdd6d34fb46ff79cd9f8b155827d6c569c7c3c732f
└── sbom/example
    └── sha256:352c1a77a9c635b9b2e9bc6b26ccedd9321088e103668ab935a0b6aa3a622aa4
        └── application/vnd.cncf.notary.signature
            └── sha256:032720a8b067f7cb25b5eb5cecd2b663fa05bc352c8206003a334856ce7b8a4b

The output from `notation inspect` shows that the image was signed using AWS signer and has signing profile information attached:

$ notation inspect 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1

Warning: Always inspect the artifact using digest(@sha256:...) rather than a tag(:v1) because resolved digest may not point to the same signed artifact, as tags are mutable.
Inspecting all signatures for signed artifact
844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
└── application/vnd.cncf.notary.signature
    └── sha256:04eed4edc7a549399648e9fdd6d34fb46ff79cd9f8b155827d6c569c7c3c732f
        ...
        │   ├── com.amazonaws.signer.signingJob: arn:aws:signer:us-west-2:844333597536:/signing-jobs/006bb1ca-1380-4b9d-8dba-19a605df8fbb
        │   ├── com.amazonaws.signer.signingProfileVersion: arn:aws:signer:us-west-2:844333597536:/signing-profiles/kyvernodemo/Kq390zhFlj
        │   └── io.cncf.notary.verificationPlugin: com.amazonaws.signer.notation.plugin
        ...
        ├── certificates
        │   ├── SHA256 fingerprint: bb0727fbd9a9688f5257ab229fc370b382a034d47552fec272bdd7f912fc1751
        │   │   ├── issued to: CN=AWS Signer,OU=AWS Cryptography,O=AWS,L=Seattle,ST=WA,C=US
        │   │   ├── issued by: CN=AWS Signer us-west-2 Code Signing CA G1,OU=Cryptography,O=AWS,ST=WA,C=US
        │   │   └── expiry: Mon Aug  7 04:31:29 2023
        ...
        └── signed artifact
            ├── media type: application/vnd.docker.distribution.manifest.v2+json
            ├── digest: sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
            └── size: 526

The plugin's trust policy uses that signer profile as a trusted identity:

apiVersion: notation.nirmata.io/v1alpha1
kind: TrustPolicy
spec:
  ...
    trustStores:
    - signingAuthority:aws-signer-ts
    trustedIdentities:
    - "arn:aws:signer:us-west-2:844333597536:/signing-profiles/kyvernodemo"

The attached SBOM `sbom/example` has the license version set to `3.17`:

{
 "SPDXID": "SPDXRef-DOCUMENT",
 "name": "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo-v1",
 "spdxVersion": "SPDX-2.2",
 "creationInfo": {
  "created": "2023-08-01T08:33:48.450737Z",
  "creators": [
   "Organization: Anchore, Inc",
   "Tool: syft-v0.46.3"
  ],
  "licenseListVersion": "3.17"
 },
 "dataLicense": "CC0-1.0",
 "documentNamespace": "https://anchore.com/syft/image/844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo-v1-2fe0c3fb-75ce-4c3c-9dee-9c87e8b5e688",
 "packages": []
}

When we run this image, verification succeeds:

$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1 --dry-run=server

pod/test1 created (server dry run)

If we update the ClusterPolicy to check for `"licenseListVersion": "3.18"`:

type: 
  - name: sbom/example
    conditions:
      all:
      - key: \{{creationInfo.licenseListVersion}}
        operator: Equals
        value: "3.18"
        message: invalid license version

Image verification fails with an `invalid license version` error:

$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1 --dry-run=server\

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy check-images error: failed to apply policy check-images rules [call-aws-signer-extension: failed to load context: failed to fetch data for APICall: HTTP 406 Not Acceptable: failed to verify attestatations: failed to verify attestations: failed to verify conditions 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1 sha256:352c1a77a9c635b9b2e9bc6b26ccedd9321088e103668ab935a0b6aa3a622aa4: failed to evaluate conditions: invalid license version

# Conclusion

In this post, we showed you the details of kyverno-notation-aws, a Nirmata extension service that executes the AWS Signer plugin for Notation to verify image signatures and attestations. The Nirmata extension service is invoked from a Kyverno policy which then calls a notation service configured with AWS Signer to provide image signature and attestation verification and digest mutation. We’ll continue to enhance and expand integrations for this extension, and we welcome your feedback. Please visit [kyverno-notation-aws repository on GitHub](https://github.com/nirmata/kyverno-notation-aws) to check on progress, and please open an issue to tell us about changes you’d like us to work on next.