***This bug bash focuses on verify with CRL***

## Overview

[Ratify v1.4.0-rc]

Supports CRL based on notation-core-go v1.2.0-rc1 and notation-go v1.3.0-rc1

## Intro

Certificate validation is an essential step during signature validation. Currently Ratify supports checking for revoked certificates through OCSP supported by notation-go library. However, OCSP validation requires internet connection for each validation while CRL could be cached for better performance. As notary-project added the CRL support for notation signature validation, Ratify utilized it.

### CRL Download

CRL download location (URL) can be obtained from the certificate's CRL Distribution Point (CDP) extension. If the certificate contains multiple CDP locations then each location download is attempted in sequential order, until a 200 response is received for any of the location. For each CDP location, Notary Project verification workflow will try to download the CRL. If the CRL cannot be downloaded within the timeout threshold the revocation result will be "revocation unavailable".

### Prerequisite

This instruction based on Ubuntu 22.04 LTS

Please install `notation` following the instruction [here](https://notaryproject.dev/docs/user-guides/installation/cli/)
Please install `docker engine` following the instruction [here](https://docs.docker.com/engine/install/ubuntu/)
Pleaes install `kubectl` following the instruction [here](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
Please install `minikube` following the instruction [here](https://minikube.sigs.k8s.io/docs/start/?arch=%2Flinux%2Fx86-64%2Fstable%2Fbinary+download)
Please install helm and helmfile from [here](https://helm.sh/docs/intro/install/#from-script) helmfile with the following commands:

curl -LO https://github.com/helmfile/helmfile/releases/download/v0.169.2/helmfile_0.169.2_linux_amd64.tar.gz

tar xvzf <ARCHIVE_FILE> -C /usr/bin/ helmfile

### Prepare the Test Scripts

1. Script A is used to generate testing certificates for CRL testing in Ratify CLI and the Ratify K8s.
https://raw.githubusercontent.com/ratify-project/ratify/refs/heads/dev/scripts/generate-crl-testing-certs.sh
The generated files are:
- `certchain_with_crl.pem`: the fullchain file that includes the leaf
certificate with CRL, intermediate certificate with invalid OCSP and valid
CRL, and the root certificate.
- `leaf.crl`: the CRL file that includes the leaf certificate.
- `leaf.key`: the private key of the leaf certificate.
- `leaf_revoked.crl`: the CRL file that includes the revoked leaf certificate.
- `intermediate.crl`: the CRL file that includes the intermediate certificate.
- `intermediate_revoked.crl`: the CRL file that includes the revoked intermediate
- `root.crt`: the root certificate.

2. Script B is used to start a CRL server:
https://raw.githubusercontent.com/ratify-project/ratify/refs/heads/dev/scripts/crl_server.py

### Prepare an Image

docker build -t YOURREGISTY/net-monitor:v1 https://github.com/wabbit-networks/net-monitor.git#main
docker push YOURREGISTY/net-monitor:v1

### Sign the Image with the Generated Cert and Prepare CRL Server

# Generate NOTATION SIGNING KEY
mkdir -p .staging/notation/crl-test
. ./generate-crl-testing-certs.sh .staging/notation/crl-test

# Create signingkeys.json
notation cert generate-test --default "wabbit-networks.io"


jq '.keys += [{"name":"crl-test","keyPath":"$HOME/.staging/notation/crl-test/leaf.key","certPath":"$HOME/.staging/notation/crl-test/certchain_with_crl.pem"}]' ~/.config/notation/signingkeys.json > tmp && mv tmp ~/.config/notation/signingkeys.json 

notation sign --key "crl-test" YOURREGISTY/net-monitor:v1

# Set yourhost to local ip address
sudo sed -i '$a 127.0.0.1 yourhost' /etc/hosts

# Following command can be use to check if there is any other python script running in the background

ps aux | grep python 

# Start Serving
python3 ./crl_server.py & echo "started crl server"

Reference: https://notaryproject.dev/docs/user-guides/how-to/notation-config-file/#sample-of-signingkeysjson

## Ratify CLI

### Install Ratify CLI

Download the latest Ratify code base and build CLI

git clone https://github.com/ratify-project/ratify.git
cd ratify
make build-cli

### Prepare Configuration file

cat <<EOF > ./config.json
{
    "store": {
        "version": "1.0.0",
        "plugins": [
            {
                "name": "oras",
                "cosignEnabled": true,
                "useHttp": true
            }
        ]
    },
    "policy": {
        "version": "1.0.0",
        "plugin": {
            "name": "configPolicy"
        }
    },
    "verifier": {
        "version": "1.0.0",
        "plugins": [
            {
                "name": "notation",
                "artifactTypes": "application/vnd.cncf.notary.signature",
                "verificationCerts": [
                    "~/.staging/notation/crl-test/root.crt"
                ],
                "trustPolicyDoc": {
                    "version": "1.0",
                    "trustPolicies": [
                        {
                            "name": "default",
                            "registryScopes": [
                                "*"
                            ],
                            "signatureVerification": {
                                "level": "strict"
                            },
                            "trustStores": [
                                "ca:certs"
                            ],
                            "trustedIdentities": [
                                "*"
                            ]
                        }
                    ]
                }
            }
        ]
    }
}
EOF

### Scenario 1: Verify Image with Revocation Check

# Please make sure that you are using the newly build CLI
./bin/ratify verify -c config.json -s YOURREGISTY/net-monitor:v1

### Scenario 2: Verify Image with Revocation Check with Leaf Cert Revoked - Expected Failure

URL_LEAF="http://localhost:10086/leaf/revoke"
curl -X POST "$URL_LEAF" -H "Content-Type: application/json"

URL_INTER=http://localhost:10086/intermediate/unrevoke
curl -X POST "$URL_INTER" -H "Content-Type: application/json"

rm -rf ~/.cache/notation/crl
ratify verify -c config.json -s YOURREGISTY/net-monitor:v1

### Reset the CRL Server

URL_LEAF="http://localhost:10086/leaf/unrevoke"
curl -X POST "$URL_LEAF" -H "Content-Type: application/json"

URL_INTER=http://localhost:10086/intermediate/unrevoke
curl -X POST "$URL_INTER" -H "Content-Type: application/json"

## Ratify K8S

### Deploy using Dev Helmfile

Development charts + images are published weekly and latest versions are tagged with rolling tags referenced in dev helmfile.

Deploy to cluster:

minikube start

helmfile sync -f git::https://github.com/ratify-project/[e-mail]

Update `hostAliases` with following commands. The default deployment name is `ratify` and `RATIFY_NAMESPACE` is `gatekeeper-system`

TARGET_IP=$(ip -4 addr show "eth0" | awk '/inet/ {print $2}' | cut -d'/' -f1)

kubectl patch deployment ${RATIFY_NAME} -n ${RATIFY_NAMESPACE} --type='merge' -p '{"spec":{"template":{"spec":{"hostAliases":[{"ip":"'"${TARGET_IP}"'","hostnames":["yourhost"]}]}}}}'

### Update KeymanagedProvider and Trust Policy

Upgrade `ratify-notation-inline-cert-0` to load `root.crt` created from the script.

apiVersion: config.ratify.deislabs.io/v1beta1
kind: KeyManagementProvider
metadata:
  name: ratify-notation-inline-cert-0
spec:
  type: inline
  parameters:
    contentType: certificate
    value: |
        -----BEGIN CERTIFICATE-----
        PLEASE INPUT THE ROOT CERT
        -----END CERTIFICATE-----

kubectl replace -f kmp.yaml --namespace gatekeeper-system

Update Trust Policy and Apply

apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
spec:
  name: notation
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      ca:
        ca-certs:
          - ratify-notation-inline-cert-0
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: default
          registryScopes:
            - "*"
          signatureVerification:
            level: permissive
          trustStores:
            - ca:ca-certs
          trustedIdentities:
            - "*"

kubectl replace -f verifier.yaml --namespace gatekeeper-system

### See Ratify in action

Test Case 1. Successfully create the pod demo and pass revocation check

kubectl run demo1 --image=YOURREGISTY/net-monitor:v1 -n default
kubectl get pods demo1 -n default


# clean resource to delete cache
kubectl delete pod <ratify-pod-name> -n gatekeeper-system

Test Case 2. Deploy fail due to revocation check

# revoke certificate by update CRL in CRL server. These commands are running in local env.

URL_LEAF="http://localhost:10086/leaf/revoke"
curl -X POST "$URL_LEAF" -H "Content-Type: application/json"

URL_INTER=http://localhost:10086/intermediate/unrevoke
curl -X POST "$URL_INTER" -H "Content-Type: application/json"

kubectl run demo2 --image=YOURREGISTY/net-monitor:v1 -n default

Test Case 3. Shut down CRL server revocation check fail by checking cache

# stop CRL server
# DefaultCacheTTL for verifier result is 10s
# Deploy image again

kubectl run demo3 --image=YOURREGISTY/net-monitor:v1 -n default

## CRL (Cache) Configuration

Ratify added support for caching CRLs response to improve availability, latency and avoid network overhead.

1. CLI

To enable or disable CRL cache with CLI, simply edit the `crl.cache.enabled` in [config.json].

2. K8s

For Kubernetes scenarios, update the `crl.cache.enabled` in `values.yaml` of Ratify Helm Chart.

## Reference

For bug bash focuses on Helm chart rendering of the Notation verifier: https://hackmd.io/@-Kl8GpDUT7WroKlFboALHw/HktQaDX8kl

We have the dev image ready for test: The ratify image and ratify-crds image for dev builds exist as separate packages on Github [here](https://github.com/ratify-project/ratify/pkgs/container/ratify-dev) and [here](https://github.com/ratify-project/ratify/pkgs/container/ratify-crds-dev)
dev helm chart: Package [ratify-chart-dev/ratify](https://github.com/ratify-project/ratify/pkgs/container/ratify-chart-dev%2Fratify)