This bug bash focuses mainly o

This bug bash focuses on verify with CRL

Overview

[Ratify v1.4.0-rc]

Supports CRL based on notation-core-go v1.2.0-rc1 and notation-go v1.3.0-rc1

Intro

Certificate validation is an essential step during signature validation. Currently Ratify supports checking for revoked certificates through OCSP supported by notation-go library. However, OCSP validation requires internet connection for each validation while CRL could be cached for better performance. As notary-project added the CRL support for notation signature validation, Ratify utilized it.

CRL Download

CRL download location (URL) can be obtained from the certificate's CRL Distribution Point (CDP) extension. If the certificate contains multiple CDP locations then each location download is attempted in sequential order, until a 200 response is received for any of the location. For each CDP location, Notary Project verification workflow will try to download the CRL. If the CRL cannot be downloaded within the timeout threshold the revocation result will be "revocation unavailable".

Prerequisite

This instruction based on Ubuntu 22.04 LTS

Please install notation following the instruction here
Please install docker engine following the instruction here
Pleaes install kubectl following the instruction here
Please install minikube following the instruction here
Please install helm and helmfile from here helmfile with the following commands:

curl -LO https://github.com/helmfile/helmfile/releases/download/v0.169.2/helmfile_0.169.2_linux_amd64.tar.gz

tar xvzf <ARCHIVE_FILE> -C /usr/bin/ helmfile

Prepare the Test Scripts

Script A is used to generate testing certificates for CRL testing in Ratify CLI and the Ratify K8s.
https://raw.githubusercontent.com/ratify-project/ratify/refs/heads/dev/scripts/generate-crl-testing-certs.sh
The generated files are:

certchain_with_crl.pem: the fullchain file that includes the leaf
certificate with CRL, intermediate certificate with invalid OCSP and valid
CRL, and the root certificate.
leaf.crl: the CRL file that includes the leaf certificate.
leaf.key: the private key of the leaf certificate.
leaf_revoked.crl: the CRL file that includes the revoked leaf certificate.
intermediate.crl: the CRL file that includes the intermediate certificate.
intermediate_revoked.crl: the CRL file that includes the revoked intermediate
root.crt: the root certificate.

Script B is used to start a CRL server:
https://raw.githubusercontent.com/ratify-project/ratify/refs/heads/dev/scripts/crl_server.py

Prepare an Image

docker build -t YOURREGISTY/net-monitor:v1 https://github.com/wabbit-networks/net-monitor.git#main
docker push YOURREGISTY/net-monitor:v1

Sign the Image with the Generated Cert and Prepare CRL Server

# Generate NOTATION SIGNING KEY
mkdir -p .staging/notation/crl-test
. ./generate-crl-testing-certs.sh .staging/notation/crl-test

# Create signingkeys.json
notation cert generate-test --default "wabbit-networks.io"


jq '.keys += [{"name":"crl-test","keyPath":"$HOME/.staging/notation/crl-test/leaf.key","certPath":"$HOME/.staging/notation/crl-test/certchain_with_crl.pem"}]' ~/.config/notation/signingkeys.json > tmp && mv tmp ~/.config/notation/signingkeys.json 

notation sign --key "crl-test" YOURREGISTY/net-monitor:v1

# Set yourhost to local ip address
sudo sed -i '$a 127.0.0.1 yourhost' /etc/hosts

# Following command can be use to check if there is any other python script running in the background

ps aux | grep python 

# Start Serving
python3 ./crl_server.py & echo "started crl server"

Reference: https://notaryproject.dev/docs/user-guides/how-to/notation-config-file/#sample-of-signingkeysjson

Ratify CLI

Install Ratify CLI

Download the latest Ratify code base and build CLI

git clone https://github.com/ratify-project/ratify.git
cd ratify
make build-cli

Prepare Configuration file

cat <<EOF > ./config.json
{
    "store": {
        "version": "1.0.0",
        "plugins": [
            {
                "name": "oras",
                "cosignEnabled": true,
                "useHttp": true
            }
        ]
    },
    "policy": {
        "version": "1.0.0",
        "plugin": {
            "name": "configPolicy"
        }
    },
    "verifier": {
        "version": "1.0.0",
        "plugins": [
            {
                "name": "notation",
                "artifactTypes": "application/vnd.cncf.notary.signature",
                "verificationCerts": [
                    "~/.staging/notation/crl-test/root.crt"
                ],
                "trustPolicyDoc": {
                    "version": "1.0",
                    "trustPolicies": [
                        {
                            "name": "default",
                            "registryScopes": [
                                "*"
                            ],
                            "signatureVerification": {
                                "level": "strict"
                            },
                            "trustStores": [
                                "ca:certs"
                            ],
                            "trustedIdentities": [
                                "*"
                            ]
                        }
                    ]
                }
            }
        ]
    }
}
EOF

Scenario 1: Verify Image with Revocation Check

# Please make sure that you are using the newly build CLI
./bin/ratify verify -c config.json -s YOURREGISTY/net-monitor:v1

Scenario 2: Verify Image with Revocation Check with Leaf Cert Revoked - Expected Failure

URL_LEAF="http://localhost:10086/leaf/revoke"
curl -X POST "$URL_LEAF" -H "Content-Type: application/json"

URL_INTER=http://localhost:10086/intermediate/unrevoke
curl -X POST "$URL_INTER" -H "Content-Type: application/json"

rm -rf ~/.cache/notation/crl
ratify verify -c config.json -s YOURREGISTY/net-monitor:v1

Reset the CRL Server

URL_LEAF="http://localhost:10086/leaf/unrevoke"
curl -X POST "$URL_LEAF" -H "Content-Type: application/json"

URL_INTER=http://localhost:10086/intermediate/unrevoke
curl -X POST "$URL_INTER" -H "Content-Type: application/json"

Ratify K8S

Deploy using Dev Helmfile

Development charts + images are published weekly and latest versions are tagged with rolling tags referenced in dev helmfile.

Deploy to cluster:

minikube start

helmfile sync -f git::https://github.com/ratify-project/ratify.git@dev.helmfile.yaml

Update hostAliases with following commands. The default deployment name is ratify and RATIFY_NAMESPACE is gatekeeper-system

TARGET_IP=$(ip -4 addr show "eth0" | awk '/inet/ {print $2}' | cut -d'/' -f1)

kubectl patch deployment ${RATIFY_NAME} -n ${RATIFY_NAMESPACE} --type='merge' -p '{"spec":{"template":{"spec":{"hostAliases":[{"ip":"'"${TARGET_IP}"'","hostnames":["yourhost"]}]}}}}'

Update KeymanagedProvider and Trust Policy

Upgrade ratify-notation-inline-cert-0 to load root.crt created from the script.

apiVersion: config.ratify.deislabs.io/v1beta1
kind: KeyManagementProvider
metadata:
  name: ratify-notation-inline-cert-0
spec:
  type: inline
  parameters:
    contentType: certificate
    value: |
        -----BEGIN CERTIFICATE-----
        PLEASE INPUT THE ROOT CERT
        -----END CERTIFICATE-----

kubectl replace -f kmp.yaml --namespace gatekeeper-system

Update Trust Policy and Apply

apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
spec:
  name: notation
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      ca:
        ca-certs:
          - ratify-notation-inline-cert-0
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: default
          registryScopes:
            - "*"
          signatureVerification:
            level: permissive
          trustStores:
            - ca:ca-certs
          trustedIdentities:
            - "*"

kubectl replace -f verifier.yaml --namespace gatekeeper-system

See Ratify in action

Test Case 1. Successfully create the pod demo and pass revocation check

kubectl run demo1 --image=YOURREGISTY/net-monitor:v1 -n default
kubectl get pods demo1 -n default


# clean resource to delete cache
kubectl delete pod <ratify-pod-name> -n gatekeeper-system

Test Case 2. Deploy fail due to revocation check

# revoke certificate by update CRL in CRL server. These commands are running in local env.

URL_LEAF="http://localhost:10086/leaf/revoke"
curl -X POST "$URL_LEAF" -H "Content-Type: application/json"

URL_INTER=http://localhost:10086/intermediate/unrevoke
curl -X POST "$URL_INTER" -H "Content-Type: application/json"

kubectl run demo2 --image=YOURREGISTY/net-monitor:v1 -n default

Test Case 3. Shut down CRL server revocation check fail by checking cache

# stop CRL server
# DefaultCacheTTL for verifier result is 10s
# Deploy image again

kubectl run demo3 --image=YOURREGISTY/net-monitor:v1 -n default

CRL (Cache) Configuration

Ratify added support for caching CRLs response to improve availability, latency and avoid network overhead.

To enable or disable CRL cache with CLI, simply edit the crl.cache.enabled in [config.json].

For Kubernetes scenarios, update the crl.cache.enabled in values.yaml of Ratify Helm Chart.

Reference

For bug bash focuses on Helm chart rendering of the Notation verifier: https://hackmd.io/@-Kl8GpDUT7WroKlFboALHw/HktQaDX8kl

We have the dev image ready for test: The ratify image and ratify-crds image for dev builds exist as separate packages on Github here and here
dev helm chart: Package ratify-chart-dev/ratify

Shiwei Zhang

2024/12/30 03:27:44

jq '.keys += [{"name":"crl-test","keyPath":"$PWD/.staging/notation/crl-test/leaf.key","certPath":"$PWD/.staging/notation/crl-test/certchain_with_crl.pem"}]' ~/.config/notation/signingkeys.json > tmp && mv tmp ~/.config/notation/signingkeys.json

should use notation policy command sets

Juncheng

2025/01/09 13:34:31

reference added

Susan Shi

2025/01/06 02:38:02

ratify

./bin/ratify to ensure to run the just built binary

shahram kalantari

2025/01/06 05:42:02

~/.config

For MAC users, this needs to change to `~/Library/Application Support`

Feynman Zhou

2025/01/07 04:42:05

I am thinking that we could write a blog post based on this bug bash doc. Some users might be interested in having a PoC before using Ratify. This could be a complete reference for them, but not an official guidance.

2025/01/12 14:05:53

Added more user-friendly hints