This bug bash focuses on Helm chart rendering of the Notation verifier.

# Overview
[Ratify v1.4.0]
Supports installation of Ratify with custom configuration for the Notation verifier's trust policies.

# Intro
Ratify needs to be ready to work out of the box after helm installatioin and it should not expect users to configure any custom resources. Ratify helm chart should provide the ability to configure necessary parameters to set up resources: ORAS Stores, KMP and Notation Verifier. In this release, Ratify is ecxpected to provide out-of-the-box experience in the context of Notary Project signature validation on K8s clusters.

### What's changed
Ratify helm chart has added more parameters to values.yaml for the notation verifier, so that the user can configure them before installation. The verifier logic has not been touched though.
In the current version (1.3), the notation verifier defines a `certs` item for `verificationCertStores` that uses AzureKeyVault or inline certs depending on AzureKeyVault and notationCerts settings. This `cert` is referenced in the `trustStores` attribute of the default `trustPolicy` of the verifier that is rendered by default during the installation. Other attributes of this trustPolicy are also set by default, among which, `trustedIdentities` and `registryScopes` are set to `*`. Here is the rendered notation verifier with one inline certificate after the installation of Ratify v1.3:

apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "5"
spec:
  name: notation
  version: 1.0.0
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      certs:
        - ratify-notation-inline-cert-0
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: default  
          registryScopes:
            - "*"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:certs
          trustedIdentities:
            - "*"

In the new version (v1.4), the user will be able to define multiple trustPolicies by providing registryScopes, trustStores, and trustedIdentities for each trustPolicy, and for each item in each trustStore of each trustPolicy, a new cert will need to be created in verificationCertStores.

# Testing helm template

Download the latest Ratify code base

git clone https://github.com/ratify-project/ratify.git
cd ratify

create a few `.crt` files for testing purposes:

touch notation-file1.crt
echo "fake cert 1" > notation-file1.crt
touch notation-file2.crt
echo "fake cert 2" > notation-file2.crt
touch notation-file3.crt
echo "fake cert 3" > notation-file3.crt

Since the verifier logic is not changed, instead of installing Ratify, `helm template` command will be used accross this bug bash for testing.

## Happy path
### The default behavior
By default, values.yaml defines the following setting for notation:

notation:
  enabled: true
  trustPolicies:
    - name: default
      verificatonLevel: strict
      registryScopes:
        - "*"
      trustedIdentities: ["*"]
      trustStores: []

run `helm template multiple-trust-policies ./charts/ratify --set featureFlags.RATIFY_CERT_ROTATION=true`. For backward compatiblity, it should render the same notation verifier as the current version, so the rendered template should look like this:

# Source: ratify/templates/verifier.yaml
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "5"
spec:
  name: notation
  version: 1.0.0
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      certs:
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: default  
          registryScopes:
            - "*"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:certs
          trustedIdentities:
            - "*"

### Setting parameters through values.yaml

1. update `./charts/ratify/values.yaml` with the following values for `notationCerts` and `notation`:

notationCerts:
  ["notation-file1.crt", "notation-file2.crt", "notation-file3.crt"]
notation:
  enabled: true
  trustPolicies:
    - name: trustPolicy-0
      registryScopes:
        - "registry1.azurecr.io/"
      trustedIdentities: ["cert identity 1"]
      trustStores:
        - ca:notationCerts[0]
        - tsa:notationCerts[1]
        - signingAuthority:notationCerts[2]
    - name: trustPolicy-1
      registryScopes:
        - "registry2.azurecr.io/"
      trustedIdentities: ["cert identity 2"]
      trustStores:
        - ca:AzureKeyVault

Note that the config above defines two trustPolicies, with the first one having 3 trustStores and the second one having one trustStore. So there will need to be 4 certs in total created in verificationCertStores of the rendered template.

2. run the follwoing command:

helm template multiple-trust-policies ./charts/ratify \
    --set featureFlags.RATIFY_CERT_ROTATION=true

The rendered template for verifier-notation should look like the following. Particularly, there should be two trustPolicies, with the first one having a trustStore with three items, each referencing a specific cert item in the verificationCertStores as described in values.yaml. Inline certificates should be named as `multiple-trust-policies-ratify-notation-inline-cert-[i]` and AzureKeyValut is named as `kmprovider-akv` inside `verificationCertStores`.

# Source: ratify/templates/verifier.yaml
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "5"
spec:
  name: notation
  version: 1.0.0
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      ca:
        cert-0:
          - multiple-trust-policies-ratify-notation-inline-cert-0
        cert-3:
          - kmprovider-akv
      signingAuthority:
        cert-2:
          - multiple-trust-policies-ratify-notation-inline-cert-2
      tsa:
        cert-1:
          - multiple-trust-policies-ratify-notation-inline-cert-1
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: trustPolicy-0  
          registryScopes:
            - "registry1.azurecr.io/"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:cert-0
            - tsa:cert-1
            - signingAuthority:cert-2
          trustedIdentities:
            - "x509.subject: cert identity 1"
        - name: trustPolicy-1  
          registryScopes:
            - "registry2.azurecr.io/"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:cert-3
          trustedIdentities:
            - "x509.subject: cert identity 2"

### Setting parameters through command line
1. revert `./charts/ratify/values.yaml` to its original values:

notationCerts: []
notation:
  enabled: true
  trustPolicies:
    - name: default
      verificatonLevel: strict
      registryScopes:
        - "*"
      trustedIdentities: ["*"]
      trustStores: []

2. run the follwoing command:

helm template multiple-trust-policies ./charts/ratify \
    --set featureFlags.RATIFY_CERT_ROTATION=true \
    --set AzureKeyVault.enabled=true \
    --set-file notationCerts[0]="notation-file1.crt" \
    --set-file notationCerts[1]="notation-file2.crt" \
    --set-file notationCerts[2]="notation-file3.crt" \
    --set notation.trustPolicies[0].registryScopes[0]="registry1.azurecr.io/" \
    --set notation.trustPolicies[0].trustedIdentities[0]="cert identity 1" \
    --set notation.trustPolicies[0].trustStores[0]=ca:notationCerts[0] \
    --set notation.trustPolicies[0].trustStores[1]=tsa:AzureKeyVault \
    --set notation.trustPolicies[0].trustStores[2]=signingAuthority:notationCerts[2] \
    --set notation.trustPolicies[1].registryScopes[0]="registry2.azurecr.io/" \
    --set notation.trustPolicies[1].trustStores[0]=ca:notationCerts[1]

Note that in order for the rendering to work properly, when providing a comma-separated string in the comman line as input parameters (in this case, for the first trustedIdentities), the comma character should be escaped with backslash. The rendered template for verifier-notation should look like the following.

# Source: ratify/templates/verifier.yaml
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "5"
spec:
  name: notation
  version: 1.0.0
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      ca:
        cert-0:
          - multiple-trust-policies-ratify-notation-inline-cert-0
        cert-3:
          - multiple-trust-policies-ratify-notation-inline-cert-1
      signingAuthority:
        cert-2:
          - multiple-trust-policies-ratify-notation-inline-cert-2
      tsa:
        cert-1:
          - kmprovider-akv
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: trustPolicy-0  
          registryScopes:
            - "registry1.azurecr.io/"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:cert-0
            - tsa:cert-1
            - signingAuthority:cert-2
          trustedIdentities:
            - "x509.subject: CN=Microsoft SCD Products RSA Signing,O=Microsoft Corporation,L=Redmond,ST=Washington,C=US"
        - name: trustPolicy-1  
          registryScopes:
            - "registry2.azurecr.io/"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:cert-3
          trustedIdentities:
            - "x509.subject: identity 2"

### Testing optional parameters
trustedIdentities is optional. In this test we don't set it and will make sure it will be set to `*` in the rendered template.
1. update `./charts/ratify/values.yaml` with the following values:

notationCerts:
  ["notation-file1.crt", "notation-file2.crt", "notation-file3.crt"]
notation:
  enabled: true
  trustPolicies:
    - name: trustPolicy-0
      registryScopes:
        - "registry1.azurecr.io/"
      trustStores:
        - ca:notationCerts[0]
        - tsa:notationCerts[1]
        - signingAuthority:notationCerts[2]
    - name: trustPolicy-1
      registryScopes:
        - "registry2.azurecr.io/"
      trustStores:
        - ca:AzureKeyVault

2. run the follwoing command:

helm template multiple-trust-policies ./charts/ratify \
    --set featureFlags.RATIFY_CERT_ROTATION=true

The rendered template for verifier-notation should look like the following. Particularly, trustedIdentities in both trustPolicies should be set to `*`

# Source: ratify/templates/verifier.yaml
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notation
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "5"
spec:
  name: notation
  version: 1.0.0
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:
      ca:
        cert-0:
          - multiple-trust-policies-ratify-notation-inline-cert-0
        cert-3:
          - kmprovider-akv
      signingAuthority:
        cert-2:
          - multiple-trust-policies-ratify-notation-inline-cert-2
      tsa:
        cert-1:
          - multiple-trust-policies-ratify-notation-inline-cert-1
    trustPolicyDoc:
      version: "1.0"
      trustPolicies:
        - name: trustPolicy-0  
          registryScopes:
            - "registry1.azurecr.io/"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:cert-0
            - tsa:cert-1
            - signingAuthority:cert-2
          trustedIdentities:
            - "*"
        - name: trustPolicy-1  
          registryScopes:
            - "registry2.azurecr.io/"
          signatureVerification:
            level: strict  
          trustStores:
            - ca:cert-3
          trustedIdentities:
            - "*"

## Failing scenario
Referencing an unknown certificate should fail.

1. update `./charts/ratify/values.yaml` with the following:

notationCerts:
  ["notation-file1.crt", "notation-file2.crt", "notation-file3.crt"]
notation:
  enabled: true
  trustPolicies:
    - name: trustPolicy-0
      registryScopes:
        - "registry1.azurecr.io/"
      trustStores:
        - ca:notationCerts[0]
        - tsa:notationCerts[1]
        - signingAuthority:notationCerts[2]
    - name: trustPolicy-1
      registryScopes:
        - "registry2.azurecr.io/"
      trustStores:
        - ca:unknown

Note that in the config above, the trustStore of the second trustPolicy is referencing an unknown cert.

2. run `helm template multiple-trust-policies ./charts/ratify --set featureFlags.RATIFY_CERT_ROTATION=true`.
It should fail with this error message:
`Error: execution error at (ratify/templates/verifier.yaml:46:17): Unknown trust store reference: unknown`

3. revert `./charts/ratify/values.yaml` to its original values:

notationCerts: []
notation:
  enabled: true
  trustPolicies:
    - name: default
      verificatonLevel: strict
      registryScopes:
        - "*"
      trustedIdentities: ["*"]
      trustStores: []

4. run the following command to test the same behavior through passing command line args:

helm template multiple-trust-policies ./charts/ratify \
    --set featureFlags.RATIFY_CERT_ROTATION=true \
    --set AzureKeyVault.enabled=true \
    --set-file notationCerts[0]="notation-file1.crt" \
    --set-file notationCerts[1]="notation-file2.crt" \
    --set-file notationCerts[2]="notation-file3.crt" \
    --set notation.trustPolicies[0].registryScopes[0]="registry1.azurecr.io/" \
    --set notation.trustPolicies[0].trustedIdentities[0]="cert identity 1" \
    --set notation.trustPolicies[0].trustStores[0]=ca:notationCerts[0] \
    --set notation.trustPolicies[0].trustStores[1]=tsa:AzureKeyVault \
    --set notation.trustPolicies[0].trustStores[2]=signingAuthority:notationCerts[2] \
    --set notation.trustPolicies[1].registryScopes[0]="registry2.azurecr.io/" \
    --set notation.trustPolicies[1].trustStores[0]=ca:unknown

It should also fail with the following error message:
`Error: execution error at (ratify/templates/verifier.yaml:46:17): Unknown trust store reference: unknown`

## clean up
Delete the temporary .crt files:
`rm -fr notation-file*.crt`

# Testing helm install
The goal is to make sure that Ratify will work out of the box with a configured notation verifier after installation. To achieve a complete e2e test, from installing the new helm chart with full notation parameters to creating deployments in a k8s cluster, a few prerequisites are needed:
1. An AkS cluster is needed where Ratify will be installed and tested
2. An AKV certificate is needed to create a certificate and sign an image with it. The certificate will then be used by the notation verifier when Ratify is installed to verify the image at the time a pod is being created.
3. An ACR registry is needed to host a signed and an unsigned image. The signed image will be the image that is signed by the AKV certificate, and the unsigned image will be any arbitrary image. Ratify will verify the signed image, but will reject the unsigned image.
4. Root certificates of some MCR image, so that it can be set as Notation certs when installing Ratify, and later verify MCR images when creating pods in the AKS cluster

The notation verifier that we will test will have two trust policies:
1. The first trust policy is configured to take care of public MCR/MAR images, so that any unsigned image will not pass verification. We will download a microsoft cert (the one that is used for signing MCR images), and will convert it to .pem format to feed the notation verifier during the installation of Ratify.
2. The second trust policy is configured to take care of private images. We will create an AKV certificate, will sign an image by that certificate and push it to a private ACR, and will use the AKV certificate in the notation verifier during the installation of Ratify.

This section provides instructions for the above prerequisites, and then testing Ratify by trying to create signed/unsigned ACR/MCR images. This document assumes that you already have an AKS instace (make sure azure policy is disabled), an AKV instance, and an ACR instace, so please create one in azure if you don't already have one.

Login to azure (`az login`) to prepare for testing

## Setting environment varibales

export AKS_RG=<your aks resource group name>
export AKS_NAME=<your aks name>
export ACR_RG=<your acr resource group name>
export ACR_NAME=<your acr name>
export AKV_RG=<your akv resource group name>
export AKV_NAME=<your akv name>

export AKS_OIDC_ISSUER="$(az aks show -n $AKS_NAME -g $AKS_RG --query "oidcIssuerProfile.issuerUrl" -otsv)"

export SUB_ID=$(az account show --query id --output tsv)
export IDENTITY_RG=<your identity resource group name>
export IDENTITY_NAME=<your identity name>
az identity create --name "${IDENTITY_NAME}" --resource-group "${IDENTITY_RG}" --location "eastus" --subscription "${SUB_ID}"
export IDENTITY_CLIENT_ID=$(az identity show --name  $IDENTITY_NAME --resource-group $IDENTITY_RG --query 'clientId' -o tsv)
export IDENTITY_OBJECT_ID=$(az identity show --name $IDENTITY_NAME --resource-group $IDENTITY_RG --query 'principalId' -otsv)

export TENANT_ID=$(az account show --query tenantId --output tsv)
export CERT_NAME=wabbit-networks-io
export CERT_SUBJECT="CN=wabbit-networks.io,O=Notation,L=Seattle,ST=WA,C=US"
export REGISTRY=$ACR_NAME.azurecr.io
export REPO=demo-ratify/net-monitor
export SIGNED_TAG=signed-akv
export UNSIGNED_TAG=unsigned
export ACR_REPO_URI=$REGISTRY/$REPO
export SIGNED_IMAGE=$REGISTRY/${REPO}:$SIGNED_TAG
export UNSIGNED_IMAGE=$REGISTRY/${REPO}:$UNSIGNED_TAG
export IMAGE_SOURCE=https://github.com/wabbit-networks/net-monitor.git
export MCR_REPO_URI="mcr.microsoft.com/windows-cssc/python"

export USER_ID=$(az ad signed-in-user show --query id -o tsv)

export RATIFY_NAMESPACE="gatekeeper-system"
export RATIFY_SA_NAME="ratify-admin"

## Grant required permissions

az acr login -n $ACR_NAME

# create a federated credential for your managed identity
az identity federated-credential create \
--name ratify-federated-credential \
--identity-name "$IDENTITY_NAME" \
--resource-group "$IDENTITY_RG" \
--issuer "$AKS_OIDC_ISSUER" \
--subject system:serviceaccount:"$RATIFY_NAMESPACE":"$RATIFY_SA_NAME"

# Assign acrpull role to the managed identity.
az role assignment create \
    --assignee-object-id $IDENTITY_OBJECT_ID \
    --role acrpull \
    --scope subscriptions/${SUB_ID}/resourceGroups/${ACR_RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}

# Grant access to AKV to your managed identity
az role assignment create --role "Key Vault Secrets User" --assignee $IDENTITY_OBJECT_ID \
--scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"

# The AcrPull and AcrPush roles are required for the logged in user to sign container images in ACR.
az role assignment create --role "AcrPull" --role "AcrPush" --assignee $USER_ID --scope "/subscriptions/$SUB_ID/resourceGroups/$ACR_RG/providers/Microsoft.ContainerRegistry/registries/$ACR_NAME"
az role assignment create --role "Key Vault Certificates Officer" --role "Key Vault Crypto User" --assignee $USER_ID --scope "/subscriptions/$SUB_ID/resourceGroups/$AKV_RG/providers/Microsoft.KeyVault/vaults/$AKV_NAME"

## Create an AKV certificate
This section provides the necessary commands to generate a self signed cert in AKV. For full instructions, please refer to [this page](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-sign-build-push)

# Create a certificate policy file.
cat <<EOF > ./my_policy.json
{
    "issuerParameters": {
    "certificateTransparency": null,
    "name": "Self"
    },
    "keyProperties": {
      "exportable": false,
      "keySize": 2048,
      "keyType": "RSA",
      "reuseKey": true
    },
    "secretProperties": {
      "contentType": "application/x-pem-file"
    },
    "x509CertificateProperties": {
    "ekus": [
        "1.3.6.1.5.5.7.3.3"
    ],
    "keyUsage": [
        "digitalSignature"
    ],
    "subject": "$CERT_SUBJECT",
    "validityInMonths": 12
    }
}
EOF

# create the cert 
az keyvault certificate create -n $CERT_NAME --vault-name $AKV_NAME -p @my_policy.json

## Push images to ACR

az acr build -r $ACR_NAME -t $REGISTRY/${REPO}:$SIGNED_TAG $IMAGE_SOURCE --no-logs --query "outputImages[0].digest" -o tsv
notation sign --signature-format cose --id $KEY_ID --plugin azure-kv --plugin-config self_signed=true $IMAGE

az acr build -r $ACR_NAME -t $REGISTRY/${REPO}:$UNSIGNED_TAG $IMAGE_SOURCE --no-logs --query "outputImages[0].digest" -o tsv

## create root certificate for MAR images
1. Download ca cert from: https://www.microsoft.com/pkiops/certs/Microsoft%20Supply%20Chain%20RSA%20Root%20CA%202022.crt
2. Download tsa cert: https://www.microsoft.com/pki/certs/MicRooCerAut_2010-06-23.crt
Convert both of the cert files to .pem format:
openssl x509 -inform der -in <PATH_TO_CRT_FILE_1> -out <PATH_TO_PEM_FILE_1>
openssl x509 -inform der -in <PATH_TO_CRT_FILE_2> -out <PATH_TO_PEM_FILE_2>

Note that you will use `<PATH_TO_PEM_FILE_1>` and `<PATH_TO_PEM_FILE_2>` later when installing Ratify.

## Install Gatekeeper if not already installed

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper/gatekeeper  \
--name-template=gatekeeper \
--namespace gatekeeper-system --create-namespace \
--set enableExternalData=true \
--set validatingWebhookTimeoutSeconds=5 \
--set mutatingWebhookTimeoutSeconds=2 \
--set externaldataProviderResponseCacheTTL=10s

## Install Ratify
Note: Don't forget to replace `<PATH_TO_PEM_FILE_1>` and `<PATH_TO_PEM_FILE_1>` with the right strings in the below section:

# update helm chart ./charts/ratify/values.yaml to point to the new dev image in your cloned repo
image:
  repository: ghcr.io/ratify-project/ratify-dev
  crdRepository: ghcr.io/ratify-project/ratify-crds-dev
  tag: dev.20250108.f04f768
  pullPolicy: Always
  
# First, uninstall Ratify if already installed:
kubectl delete -f https://ratify-project.github.io/ratify/library/default/template.yaml
kubectl delete -f https://ratify-project.github.io/ratify/library/default/samples/constraint.yaml
helm delete ratify --namespace gatekeeper-system
kubectl delete crd stores.config.ratify.deislabs.io verifiers.config.ratify.deislabs.io certificatestores.config.ratify.deislabs.io policies.config.ratify.deislabs.io

# Install Ratify with the configration that was described in the beginning of this e2e test:
helm install ratify ./charts/ratify \
    --atomic \
    --namespace gatekeeper-system \
    --set provider.enableMutation=false \
    --set featureFlags.RATIFY_CERT_ROTATION=true \
    --set-file notationCerts[0]=<PATH_TO_PEM_FILE_1> \
    --set-file notationCerts[1]=<PATH_TO_PEM_FILE_2> \
    --set oras.authProviders.azureWorkloadIdentityEnabled=true \
    --set azureWorkloadIdentity.clientId=$IDENTITY_CLIENT_ID \
    --set azurekeyvault.enabled=true \
    --set azurekeyvault.vaultURI=https://${AKV_NAME}.vault.azure.net/ \
    --set azurekeyvault.certificates[0].name=$CERT_NAME \
    --set azurekeyvault.tenantId=$TENANT_ID \
    --set notation.trustPolicies[0].registryScopes[0]=$MCR_REPO_URI \
    --set notation.trustPolicies[0].trustStores[0]=ca:notationCerts[0] \
    --set notation.trustPolicies[0].trustStores[1]=tsa:notationCerts[1] \
    --set notation.trustPolicies[0].trustedIdentities[0]="CN=Microsoft SCD Products RSA Signing\,O=Microsoft Corporation\,L=Redmond\,ST=Washington\,C=US" \
    --set notation.trustPolicies[1].registryScopes[0]=$ACR_REPO_URI \
    --set notation.trustPolicies[1].trustStores[0]=ca:azurekeyvault
    
# Enforce Gatekeeper policy to allow only signed images can be deployed on AKS:
kubectl apply -f https://ratify-project.github.io/ratify/library/default/template.yaml
kubectl apply -f https://ratify-project.github.io/ratify/library/default/samples/constraint.yaml

## Test creating pods

# Ratify should reject the following because the image is unsigned:
kubectl run demo-acr-unsigned --image=$REGISTRY/$REPO:$UNSIGNED_TAG 

# Ratify should pass verification of the following because the image is signed:
kubectl run demo-acr-signed --image=$REGISTRY/$REPO:$SIGNED_TAG

# Ratify should reject the following because the image is unsigned:
kubectl run demo-mar-unsigned --image=mcr.microsoft.com/azureml/mlflow-ubuntu18.04-py37-cpu-inference:20230227.v2
# Note: "oras discover mcr.microsoft.com/azureml/mlflow-ubuntu18.04-py37-cpu-inference:20230227.v2 --artifact-type application/vnd.cncf.notary.signature" will verify that the image has notation signature

# Ratify should pass verification of the following because the image is signed:
kubectl run demo-mar-signed --image=mcr.microsoft.com/windows-cssc/python:3.9-nanoserver-ltsc2019-202310
# Note: "oras discover mcr.microsoft.com/windows-cssc/python:3.9-nanoserver-ltsc2019-202310 --artifact-type application/vnd.cncf.notary.signature" will verify that the image has notation signature