# Ratify Version 1.2.0 Scenario Validations

## Testing Setup

### Installing Ratify

Install Ratify from dev build with a single cosign Inline KMP

helm install ratify \
  oci://ghcr.io/deislabs/ratify-chart-dev/ratify --atomic \
  --version 0-dev \
  --namespace gatekeeper-system \
  --set featureFlags.RATIFY_CERT_ROTATION=true \
  --set image.pullPolicy=Always \
  --set logger.level=debug \
  --set-file cosignKeys[0]=<PUBLIC KEY PATH>

Install Ratify from dev build with a key from AKV

helm install ratify \
  oci://ghcr.io/deislabs/ratify-chart-dev/ratify --atomic \
  --version 0-dev \
  --namespace gatekeeper-system \
  --set featureFlags.RATIFY_CERT_ROTATION=true \
  --set image.pullPolicy=Always \
  --set logger.level=debug \
  --set azureWorkloadIdentity.clientId=<CLIENT_ID> \
  --set azurekeyvault.enabled=true \
  --set azurekeyvault.tenantID=<TENANT_ID> \
  --set azurekeyvault.vaultURI=<VAULT_URI> \
  --set azurekeyvault.keys[0].name=<KEY_NAME> \
  --set azurekeyvault.keys[0].version=<KEY_VERSION>

Install Ratify from dev build with Workload Identity ACR + AKV

helm install ratify \
  oci://ghcr.io/deislabs/ratify-chart-dev/ratify --atomic \
  --version 0-dev \
  --namespace gatekeeper-system \
  --set featureFlags.RATIFY_CERT_ROTATION=true \
  --set image.pullPolicy=Always \
  --set logger.level=debug \
  --set oras.authProviders.azureWorkloadIdentityEnabled=true \
  --set azureWorkloadIdentity.clientId=<CLIENT_ID> \
  --set azurekeyvault.enabled=true \
  --set azurekeyvault.tenantID=<TENANT_ID> \
  --set azurekeyvault.vaultURI=<VAULT_URI> \
  --set azurekeyvault.keys[0].name=<KEY_NAME> \
  --set azurekeyvault.keys[0].version=<KEY_VERSION>

### Generating test Keys

> NOTE: Cosign requires that the signing key be encoded in a particular format. This means you CANNOT directly use openssl (or equivalent tool) to generate the keypair and then sign. You must import the cosign keypair first using Cosign, which generates an encoded private key. Cosign will also generate an accompanying public key but that has no special properties and can be used interchangeably with openssl outputted public key.

Generating a local RSA 2048 key pair

openssl genrsa -out priv_rsa.pem 2048
cosign import-key-pair --key priv_rsa.pem --output-key-prefix cosign-rsa

Generating a local EC P256 key pair

openssl ecparam -name prime256v1 -genkey -noout -out priv_ec.pem
cosign import-key-pair --key priv_ec.pem --output-key-prefix cosign-ec

Generating a cosign sample key pair

cosign generate-key-pair --output-key-prefix cosign-ed25519

### Signing using Cosign CLI
Using cosign to sign with local key

cosign sign --key <private key path> --tlog-upload=false --registry-username <username> --registry-password <password> myregistry.io/hello-world:v1

Using cosign to sign with AKV key reference

cosign sign --key azurekms://<keyvault_name>.vault.azure.net/<key_name>/<key_version> --tlog-upload=false --registry-username <username> --registry-password <password> myregistry.io/hello-world:v1

### Sample Ratify CRD templates
Sample Cosign Verifier with Multiple Trust Policies:

apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-cosign
spec:
  name: cosign
  artifactTypes: application/vnd.dev.cosign.artifact.sig.v1+json
  parameters:
    trustPolicies:
      - name: default
        scopes:
          - "*"
        keys:
          - provider: <KMP NAME>
      - name: second-policy
        scopes:
          - "myregistry.io/*"
        keys:
          - provider: <KMP NAME 2>

Sample Inline Key Management Provider (KMP)

apiVersion: config.ratify.deislabs.io/v1beta1
kind: KeyManagementProvider
metadata:
  name: keymanagementprovider-inline
spec:
  type: inline
  parameters:
    contentType: key
    value: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEb3Q7aXODW+bHM1ALN2T51yvZv4UH
      yEHyuUX4NORgFTk+IYfBFWtCL28D5Df5TRMJKOwKoNGiImWxQHv9U3iIuQ==
      -----END PUBLIC KEY-----

Sample AKV KMP

apiVersion: config.ratify.deislabs.io/v1beta1
kind: KeyManagementProvider
metadata:
  name: keymanagementprovider-inline
spec:
  type: azurekeyvault
  parameters:
    vaultURI: https://<KEY VAULT NAME>.vault.azure.net/
    keys:
      - name: <KEY NAME>
        version: <KEY VERSION> # Optional, fetch latest version if empty 
    tenantID: <TENANT ID>
    clientID: <CLIENT ID>

## Key Management Providers + Cosign Scenarios

For testing, most of the scenarios below will only require Ratify to be installed via helm chart values (change values accordingly). The [Installing Ratify](#Installing-Ratify) guide above has common scenarios with helm install commands.

Sample keys of different type need to be generated and then used to sign sample images using `cosign` cli. Please refer to [Generate test Keys](#Generating-test-Keys) & [Signing using Cosign CLI](#Signing-using-Cosign-CLI) for sample commands.

Some scenarios will require manually deletion/editing of a resource. Please use `kubectl` accordingly. Please refer to [CRD Template](#Sample-Ratify-CRD-templates) for example templates for KMP + Cosign verifier.

### Inline provider

#### Test 1: Generate a local RSA 2048 key pair. Sign sample image wth Cosign. Inline Provider KMP
- Sub 1: Reconfigure Inline KMP with a different key and make sure validation fails (check extensions field of verifier report)
- Sub 2: Delete Inline KMP. Make sure verification returns error since matching KMP will NOT be found

#### Test 2: Generate a local EC P256 key pair. Sign sample image wth Cosign. Inline Provider KMP

#### Test 3: Create a cosign sample key pair using cosign cli `generate-key-pair` command and then sign image. Verify using Inline Provider KMP

### AKV provider

#### Test 4: Create a 2048 RSA key in Azure Key Vault. Sign sample image with `azurekms`. Azure Key Vault KMP

#### Test 5: Create an EC P256 key in Azure Key Vault. Sign sample image with `azurekms`. Azure Key Vault KMP

#### Test 6: Generate multiple keys. Sign sample image with each key. Multiple Inline KMP providers. Verify extensions field shows multiple key verification success.

#### Test 7: Generate 2 keys. Sign 2 different images with 2 different keys. Push image to 2 different repositories. Configure 2 different trust policies with individual repository scope. Verify both images succeed.

### Registry scopes

#### Test 8: Generate a local RSA 2048 key pair. Sign sample image with Cosign. Inline Provider KMP. Set 2 trust policies for cosign verifier: '\*' and a scope that will not match the image. Make sure verification succeeds and uses the '\*' scoped trust policy

### Advanced Scenarios

#### Imported Keys in AKV

##### Test 9: Generate a local RSA 2048 key pair. Sign sample image with Cosign. Upload public key to AKV. AKV KMP setup to reference AKV key. Make sure verification succeeds**

#### Key rotation

##### Test 10: Verify images after key rotation in AKV (by Yi)

Normally, a key stored in AKV will be rotated regularly for security, in this case, a new version of the key is created with the same key name. Thus, some images were signed with old version of key, and some were signed new version of key. Signature verification should pass for all these images.

NOTE:
- There is [an issue](https://github.com/deislabs/ratify/issues/1131) for Ratify to automatically fetch the latest version of key, so that users only need to configure the key name. In this case, Ratify needs to regularly pull all the versions of keys so that Ratify can verify images that some were signed with old keys and some were signed with new keys. Before this issue is resolved, users have to update the CR manually each time the key is rotated.

Steps:
1. Users created a key in AKV
2. Users sign the image-1 with this key
3. Users deploy this image-1 which should fail the signature verification
4. Users configure Ratify for the current version of key
5. Users deploy this image-1 which should succeed
6. Users rotate the key with a new version
7. Users sign a new image-2 wit the new version
8. Users deploy the image-2 which should fail the signature verfication
9. Users add the new version of key to Ratify configuration
10. Users deploy the image-2 which should succeed
11. Users deploy the image-1 again which should succeed as well.

#### Key disabling

##### Test 11: Verify images after disabling the specific version of key stored in AKV (by Yi)

Users may disble a key in AKV. The reason could be the specific version of key was compromised or other reasons. Images signed with disabled key should fail the signature verification.

NOTE:
- There is [an issue](https://github.com/deislabs/ratify/issues/1131) for Ratify to automatically fetch the latest version of key, so that users only need to configure the key name. In this case, Ratify needs to regularly pull all the versions of keys excluding the version that are disabled. Before this issue is resolved, users have to specify the version of key to be used, and remove the version of key that should not be used.

Steps:
1. Continue from **Test 10**
2. Users disable the first version of key in AKV
3. Users deploy the image-1 again, it should succeed as uses does not update the configuration yet
4. Users remove the first version of key in Ratify
5. Users deploy the image-1 again, it should fail the signature verification.
6. Users deploy the image-2 again, it should succeed, as the new version of key should work.

#### Upgrade from v1.1.1
**Test 12: Install Ratify from v1.1.1 with cosign verification configured. Upgrade using helm to cosign v1.2.0 and see that 1. warning message in logs for cert store being configured. 2. New KMP configured and cosign verifier uses KMP instead. 3. Verification still succeeds even though cosign verifier using legacy "key" field.**

### Negative Test Cases

#### NT 1: Generate a local RSA 2048 key pair. Sign sample image with Cosign. Do not set up inline provider. Verification should fail.

#### NT 2: Generate a local RSA 2048 key pair. Sign sample image with Cosign. Setup Inline Provider KMP. Change `scopes` to use a scope that will NOT match. Verification should fail.

#### NT 3: The identity is not allowed for pulling images from ACR (by Yi)

#### NT 4: The identity is not allowed for reading keys from AKV (by Yi)

### Test-Matrix

Please provide any detailed feedback inline with the actual test description above. Use 'details' section in table below for brief comments.

| Test Name | Assignee | Result (PASS/FAIL) | Details | | | | |
| --------- | -------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | --- | -------------- | --------------------------------------------------------------------------------------------------- |
| Test 1 | Susan | 1.Passed | Had some issue specifying multiple keys | | | | |
| | | | | | | | |
| Test 2 | Susan | Passed | | | | | |
| Test 3 | Susan | Passed | Others scenarios to validate: multiple keys specified at install, specifying diferrent scopes, Ratify in useRego=true mode | | | | |
| Test 4 | Yi | Passed | RSA 3072 key was used | | | | |
| Test 5 | Akash | Passed | | | | | |
| Test 6 | Yi/Susan | Passed | Not exactly test 6, Yi tested one images signed with two keys, the verification succeeded as long as one signature passed verification. Yi was unsure how to set up the test to require both signatures to be valid. | | | | |
| Test 7 | Yi | Passed | | | | | |
| Test 8 | Yi | Passed | Not exactly the same as Test 8, but test various combinations on scopes: one policy with `*` or `registry` scope; one policy with scope not covering the signed image; one policy playing with regex pattern;two policies with one using `*`, another with wrong key provider; two policies with one using `*`, another for specific registry; | | | | |
| Test 9 | Susan | Passed | | | | | |
| Test 10 | Yi | Passed | Not desired experience, see note in Test 10
| Test 11 | Yi | Partial passed | Disabled key can still be fetched, [issue reported](https://github.com/deislabs/ratify/issues/1449) |
| Test 12 | Akash | Failed (passed after fix) | After upgrade, the cosign verifier's key field contains the value of the key instead of the filepath of the key which is what it expects. | | | | |
| NT 1 | Susan | Passed | | | | | |
| NT 2 | Susan | Passed | | | | | |
| NT 3 | Yi | Passed (with question) | How long does it take for the changes to apply when an identity loses a role? The error message does not indicate an auth issue clearly: "message": "verification failed: Error: referrer store failure, Code: REFERRER_STORE_FAILURE, Component Type: referrerStore, Detail: could not resolve descriptor for a subject from any stores" | | | | |
| NT 4 | Yi | Passed (with condition) | Failed if an identity was assigned the right role first, keymanagementprovider was applied successfully, then removed the role. |

## Multi-tenancy Scenarios

### Background
Presently, we have store/verifier/kmp/policy segmented into both clustered and namespaced scopes.

Users who don't want to opt into the multi-tenancy feature can continue utilizing the clustered store/verifier/policy, which is already supported in v1.1. Regarding key/cert management, users have the option to utilize either the clustered KMP or the namespaced CertStore. While we recommend users to opt for KMP over CertStore, if they still prefer CertStore, certStore resources within the "default" namespace will function similarly to the cluster-wide one.

And for users who wanna try multi-tenancy feature, they could apply both namespaced and clustered resoures per use case.
***e.g.*** users wanna deploy imageA to namespaceA. Ratify would fetch store/verifier/store/kmp resources applied in namespaceA. If any resource is not found in namespaceA, it will fetch cluster-wide resources instead.

### Test Scenarios
#### Cluster-wide use case (default use case since v1.1)
##### Test Preparation
1. Deploy Ratify
2. Use the default constraint template.
3. Apply all cluster-wide resources and certStore into `default` namespace.
4. Run tests.

##### Test cases
Bascially now we can run all above KMP test cases (including negative tests) to make sure it functions same.
Some additional test cases:
***Note1***: if users specify a certStore name in verifier config without namespace prefix, Ratify would append Ratify's namespace to it, which is `gatekeeper-system` by default.
***Note2***: for below test cases, we could try both default constraint templates(CT) and multi-tenancy CT. But if we want to test certStore CR usage, then we should adopt the default CT as the multi-tenancy is not designed for certStore.

| No. |Namespace image deployed to | Trust Store Type | Trust Store Name in Verifier Config | Expected Behavior of verifier fetching certs | Expeccted of image verify | Assignee |
| -- |--------------------------- | ---------------- | ----------------------------------- | ------------------------------------------------------ | ------------------------- | --- |
| Test 1 |namespace1 | certStore | cert1 | return certs from gatekeeper-system/cert1 if it exists | true if cert1 exists | |
| Test 2 |namespace1 | certStore | namespace1/cert1 | return certs from namespace1/cert1 if it exists | true if cert1 exists | |
| Test 3 |namespace1 | certStore | default/cert1 | return certs from default/cert1 if it exists | true if cert1 exists | |
| Test 4 |namespace1 | KMP | cert1 | return certs from cluster-wide cert1 | true if cert1 exists | |
| Test 5 |namespace1 | KMP | namespace1/cert1 | return certs from namespace1/cert1 if it exists | true if cert1 exists | |
| Test 6 |namespace1 | KMP | namespace2/cert1 | cannot fetch cert1 from different namespace | false | |
| Test 7 |Upgrade Ratify with v1.1.1 Verifier/CertStore/Store to Ratify1.1.2. ( backward compat test) | | | | | |

Test 8: Cluster admins set up cluster-wide resources (KMP, Verifier and Store), only image signed with keys in KMP can be deployed (Yi)

#### Namespaced use case
##### Test Preparation
1. Deploy Ratify
2. Use the [multi-tenancy constraint template](https://github.com/deislabs/ratify/pull/1429/files#diff-0a8612e0b463616fa986822f89171328655a98f4fce10aac06726eb65722541aR1).
3. Delete all cluster-wide resources.
4. Apply all required namespaced resources.
5. Run tests.

##### Test cases
Assume we applied all resources to namespace: `namespace1`. And we'll just use KMP for cert/key management.
| No. | Namespace image deployed to | Trust Store Type | Trust Store Name in Verifier Config | Expected Behavior of verifier fetching certs | Expected of image verify | Assignee |
| ----| --------------------------- | ---------------- | ----------------------------------- | ----------------------------------------------- | ------------------------ | --- |
| Test 9 | namespace1 | KMP | cert1 | return certs from cluster-wide cert1 | true if cert1 exists | |
| Test 10 | namespace1 | KMP | namespace1/cert1 | return certs from namespace1/cert1 if it exists | true if cert1 exists | |
| Test 11 | namespace1 | KMP | namespace2/cert1 | cannot fetch cert1 from different namespace | false | |

Test 12: Team-a set up namespaced ratify resources to ensure only images signed by Team-a can be deployed to namespace team-a, simliar for team-b

Test 13: Team-b cannot configure ratify resources in namespace team-b to access ratify resources in namespace team-a.

Test 14: The cluster admin set up cluster wide KMP resource for team-a and team-b, both team-a and team-b can access cluster-wide KMP in their own namespaced resource

Test 15: Team a set up namespaced policy in namespace team-a, which is different from team b's policy that inherits the default cluster-wide policy

##### Negative tests
1. NT-1: Remove certain kind of resource from both cluster and namespace. e.g. Remove all Cluster-wide policies and namespaced policies.

#### Test results

| Test Name | Assignee | Result (PASS/FAIL) | Details |
| --------- | -------- | ------------------ | ---------- |
| Test 1 | Binbin | Passed | |
| Test 2 | Binbin | Passed | |
| Test 3 | Binbin | Passed | |
| Test 4 | Binbin | Passed | |
| Test 5 | Binbin | Passed | |
| Test 6 | Binbin | Passed | |
| Test 7 | Binbin | Passed | |
| Test 8 | Yi | Passed | |
| Test 9 | Binbin | Passed | |
| Test 10 | Binbin | Passed | |
| Test 11 | Binbin | Passed | |
| Test 12 | Yi | Failed | need to configure cluster-wide store, see [issue](https://github.com/deislabs/ratify/issues/1458) |
| Test 13 | Yi | Passed | |
| Test 14 | Yi | Passed | |
| Test 15 | Yi | Passed | |
| NT 1 | Binbin | Passed | |