Negative test cases for Ratify

Overview

The purpose of negative testing is to ensure that Ratify can gracefully handle invalid input or unexpected user behavior. It also ensures Ratify can provide useful error logs for users when handling invalid inputs or unexpected server responses. Negative test cases can also help to improve the overall quality of Ratify.

This document summarizes necessary negative test cases for Ratify.

Areas

Registry (Assignee: Susan)
Verifier - Notation (Assignee: Binbin )
Certificate store (Assignee: Juncheng)
Policy (Assignee: @Feynman )
HA (Assignee: @akashsinghal )

OS Arch

Linux OS

AMD x86-64 (Major)
ARM64

Test cases

No	Category	Description	Results
TC-1	Registry	Users provide an mismatched registry credential in Ratify store CRD (authProvider)	pass
TC-2	Registry	Registry connection timeout	pass
TC-3	Registry	Interact with insecure registry without the right flag (useHTTP)	pass
TC-4	Registry	Verify a image signature without read permission to the registry.	pass
TC-5	Verifier	Trust Policy is not set	pass
TC-6	Verifier	`registryScope` is not set or misconfigured in Trust policy	pass
TC-7	Verifier	Trust policy matches incorrectly and fails due to wrong certificate	pass
TC-8	Verifier	The image signed by an untrusted identities or `trustedIdentities` was misconfigured	pass
TC-9	Cert Store	Certificate expired	pass
TC-10	Cert Store	Certificate revoked	pass
TC-11	Cert Store	Ratify can't access AKV	pass
TC-12	Cert Store	An invalid certificate is filled out in the cert store CRD (inline)	failed
TC-13	Cert Store	Certificate format/type is not supported by Ratify	pass
TC-14	Policy	`policyPath` is not existed	pass
TC-15	Policy	Syntax problems or misconfiguration in the REGO policy	pass
TC-16	Policy	Syntax problems or misconfiguration in the Config policy	failed
TC-17	HA	A Kubernetes node that Ratify instances are running on top of it is crashed	pass
TC-18	HA	Redis Pod is crashed or in outage	pass
TC-19	HA	Gatekeeper instance is crashed or disconnected	N/A
TC-20	HA	Dapr sidecar is not available	pass
TC-21	HA	Dapr is not configured on cluster properly	pass

Test results

Gather the Ratify logs for each test case.

Registry

TC1

Error from server: admission webhook "mutation.gatekeeper.sh" denied the request: failed to resolve external data placeholders: failed to retrieve external data item from provider ratify-mutation-provider: Original Error: (Original Error: (HEAD "http://registry:5000/v2/notation/manifests/signed": response status code 401: Unauthorized), Error: repository operation failure, Code: REPOSITORY_OPERATION_FAILURE, Plugin Name: oras), Error: get subject descriptor failure, Code: GET_SUBJECT_DESCRIPTOR_FAILURE, Plugin Name: oras, Component Type: referrerStore, Detail: failed to get subject descriptor for image registry:5000/notation:signed

TC2

Error from server: admission webhook "mutation.gatekeeper.sh" denied the request: failed to resolve external data placeholders: failed to retrieve external data item from provider ratify-mutation-provider: Original Error: (Original Error: (Head "https://huishwabbit2.azurecr.io/v2/net-monitor/manifests/v1": dial tcp: lookup huishwabbit2.azurecr.io on 10.0.0.10:53: no such host), Error: repository operation failure, Code: REPOSITORY_OPERATION_FAILURE, Plugin Name: oras), Error: get subject descriptor failure, Code: GET_SUBJECT_DESCRIPTOR_FAILURE, Plugin Name: oras, Component Type: referrerStore, Detail: failed to get subject descriptor for image huishwabbit2.azurecr.io/net-monitor:v1

TC3

Error from server: admission webhook "mutation.gatekeeper.sh" denied the request: failed to resolve external data placeholders: failed to retrieve external data item from provider ratify-mutation-provider: Original Error: (Original Error: (Head "https://registry:5000/v2/notation/manifests/signed": http: server gave HTTP response to HTTPS client), Error: repository operation failure, Code: REPOSITORY_OPERATION_FAILURE, Plugin Name: oras), Error: get subject descriptor failure, Code: GET_SUBJECT_DESCRIPTOR_FAILURE, Plugin Name: oras, Component Type: referrerStore, Detail: failed to get subject descriptor for image registry:5000/notation:signed

TC4

Error from server: admission webhook "mutation.gatekeeper.sh" denied the request: failed to resolve external data placeholders: failed to retrieve external data item from provider ratify-mutation-provider: Original Error: (Original Error: (HEAD "https://huishwabbit1.azurecr.io/v2/net-monitor/manifests/v1": response status code 401: Unauthorized), Error: repository operation failure, Code: REPOSITORY_OPERATION_FAILURE, Plugin Name: oras), Error: get subject descriptor failure, Code: GET_SUBJECT_DESCRIPTOR_FAILURE, Plugin Name: oras, Component Type: referrerStore, Detail: failed to get subject descriptor for image huishwabbit1.azurecr.io/net-monitor:v1

TC5

If users applied a CR without the trustPolicy, Ratify will log an error while reconciling it:

time=2023-09-20T12:47:29.879027432Z level=error msg=Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Plugin Name: notation, Component Type: verifierunable to create verifier from verifier crd

If the trustPolicy is not set in ConfigMap of charts, Ratify will fail deployment.

TC6

If users applied a CR without registryScope, Ratify will log an error while reconciling it:

time=2023-09-20T13:08:43.696147959Z level=error msg=Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Plugin Name: notation, Component Type: verifierunable to create verifier from verifier crd

If the registryScope is not set in ConfigMap of charts, Ratify will fail deployment.
If users applied a CR with misconfigured registryScope, the verification would fail:

"verifierReports": [
    {
      "subject": "ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b",
      "referenceDigest": "sha256:57be2c1c3d9c23ef7c964bba05c7aa23b525732e9c9af9652654ccc3f4babb0e",
      "artifactType": "application/vnd.cncf.notary.signature",
      "verifierReports": [
        {
          "isSuccess": false,
          "message": "Original Error: (Original Error: (artifact \"ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b\" has no applicable trust policy. Trust policy applicability for a given artifact is determined by registryScopes. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
          "name": "notation",
          "extensions": null
        }
      ],
      "nestedReports": []
    }
  ]
}

TC7

The image verification fails.

"verifierReports": [
    {
      "subject": "ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b",
      "referenceDigest": "sha256:57be2c1c3d9c23ef7c964bba05c7aa23b525732e9c9af9652654ccc3f4babb0e",
      "artifactType": "application/vnd.cncf.notary.signature",
      "verifierReports": [
        {
          "isSuccess": false,
          "message": "Original Error: (Original Error: (error while loading the trust store, valid certificates must be provided, only CA certificates or self-signed signing certificates are supported), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
          "name": "notation",
          "extensions": null
        }
      ],
      "nestedReports": []
    }
  ]

TC8

The image verification fails:

time=2023-09-22T13:50:40.440640495Z level=info msg=verify result for subject ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b: {
  "verifierReports": [
    {
      "isSuccess": false,
      "name": "notation",
      "message": "Original Error: (Original Error: (error while parsing the certificate subject from the digital signature. error : \"distinguished name (DN) \\\"CN=ratify.default\\\" has no mandatory RDN attribute for \\\"C\\\", it must contain 'C', 'ST', and 'O' RDN attributes at a minimum\"), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
}

TC10

time=2023-09-22T08:58:54.288013215Z level=error msg=Reconciler error CertificateStore=gatekeeper-system/certstore-akv controller=certificatestore controllerGroup=config.ratify.deislabs.io controllerKind=CertificateStore error=Error fetching certificates in store certstore-akv with azurekeyvault provider, error: failed to get secret objectName:wabbit-network-io, objectVersion:, error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=404 – Original Error: autorest/azure: Service returned an error. Status=404 Code="SecretNotFound" Message="A secret with (name/id) wabbit-network-io was not found in this key vault. If you recently deleted this secret you may be able to recover it using the correct recovery command. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125182" name=certstore-akv namespace=gatekeeper-system reconcileID=28eac175-4f3b-4353-88df-df5bdc3ae6de

TC12

The inline cert is invalid and has an error but Ratify logs doesn't mentioned the root cause.

time=2023-09-21T17:11:38.097617136Z level=info msg=received request POST /ratify/gatekeeper/v1/verify
time=2023-09-21T17:11:38.097677237Z level=info msg=start request POST /ratify/gatekeeper/v1/verify component-type=server go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.097999243Z level=info msg=verifying subject ghcr.io/feynmanzhou/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e component-type=server go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.098228648Z level=info msg=Resolve of the image completed successfully the digest is sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e component-type=executor go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.098545355Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.625152926Z level=info msg=Resolve of the image completed successfully the digest is sha256:f4c1e923d1f2a7b76513c889a0db548a093f422d06ac6b83ce7243e0c8fa7805 component-type=executor go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.625312829Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.625165626Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.625883641Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0
time=2023-09-21T17:11:38.904404237Z level=info msg=verify result for subject ghcr.io/feynmanzhou/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e: {
  "verifierReports": [
    {
      "subject": "ghcr.io/feynmanzhou/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e",
      "referenceDigest": "sha256:f4c1e923d1f2a7b76513c889a0db548a093f422d06ac6b83ce7243e0c8fa7805",
      "artifactType": "application/vnd.cncf.notary.signature",
      "verifierReports": [
        {
          "isSuccess": false,
          "message": "Original Error: (Original Error: (signature is not produced by a trusted signer), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
          "name": "notation",
          "extensions": null
        }
      ],
      "nestedReports": []
    }
  ]
} component-type=server go.version=go1.20.8 trace-id=8c82c679-037d-4973-9d3b-ba955c8a73a0

TC13

 Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Plugin Name: azurekeyvault, Component Type: certProvider, Documentation: https://learn.microsoft.com/en-us/azure/key-vault/general/overview, Detail: failed to get keyvault client name=certstore-akv namespace=default reconcileID=a68577aa-af1f-4e87-8f3a-2296d6680934

TC14

time=2023-09-21T16:49:39.101904574Z level=info msg=verify result for subject ghcr.io/feynmanzhou/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e: {
  "verifierReports": [
    {
      "isSuccess": false,
      "name": "notation",
      "message": "Original Error: (Original Error: (signature is not produced by a trusted signer), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
} component-type=server go.version=go1.20.8 trace-id=871c479e-4d1a-413b-b6b5-a4332873afc7
time=2023-09-21T16:49:39.649743359Z level=info msg=Reconciling Policy regopolicy
time=2023-09-21T16:49:39.652539916Z level=info msg=selected policy provider: regopolicy
time=2023-09-21T16:49:39.65272052Z level=info msg=Deleting policy configpolicy
time=2023-09-21T16:49:39.660718083Z level=info msg=Deleted policynameconfigpolicy
time=2023-09-21T16:49:39.660797685Z level=info msg=Reconciling Policy configpolicy
time=2023-09-21T16:49:39.660900987Z level=error msg=failed to get Policy: Policy.config.ratify.deislabs.io "configpolicy" not found
time=2023-09-21T16:49:49.266265821Z level=info msg=received request POST /ratify/gatekeeper/v1/mutate
time=2023-09-21T16:49:49.266514626Z level=info msg=start request POST /ratify/gatekeeper/v1/mutate component-type=server go.version=go1.20.8 trace-id=acb4f156-8132-4cfb-9df6-e73266c5c6c7
time=2023-09-21T16:49:49.26667593Z level=info msg=mutating image ghcr.io/deislabs/ratify/notary-image:signed component-type=server go.version=go1.20.8 trace-id=acb4f156-8132-4cfb-9df6-e73266c5c6c7
time=2023-09-21T16:49:49.266880934Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=acb4f156-8132-4cfb-9df6-e73266c5c6c7

TC15

time=2023-09-21T16:24:03.422952769Z level=error msg=Reconciler error Policy=regopolicy controller=policy controllerGroup=config.ratify.deislabs.io controllerKind=Policy error=failed to create policy enforcer: failed to create policy provider: Original Error: (Original Error: (failed to create policy engine: failed to create policy query, err: failed to prepare rego query, err: 1 error occurred: policy.rego:13: rego_unsafe_var_error: var fals is unsafe), Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Plugin Name: regopolicy, Component Type: policyProvider, Documentation: https://github.com/deislabs/ratify/blob/main/docs/reference/providers.md#policy-providers, Detail: failed to create OPA engine), Error: plugin init failure, Code: PLUGIN_INIT_FAILURE, Plugin Name: regopolicy, Component Type: policyProvider, Documentation: https://github.com/deislabs/ratify/blob/main/docs/reference/providers.md#policy-providers, Detail: failed to create policy provider name=regopolicy namespace= reconcileID=71adeaf9-a6f0-4974-88cf-34bd2be47a99

TC16

I misconfigured the config policy but the Ratify logs don't point this root cause out.

time=2023-09-21T16:42:38.410494337Z level=info msg=received request POST /ratify/gatekeeper/v1/verify
time=2023-09-21T16:42:38.410757342Z level=info msg=start request POST /ratify/gatekeeper/v1/verify component-type=server go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96
time=2023-09-21T16:42:38.411028347Z level=info msg=verifying subject ghcr.io/feynmanzhou/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e component-type=server go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96
time=2023-09-21T16:42:38.411223651Z level=info msg=Resolve of the image completed successfully the digest is sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e component-type=executor go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96
time=2023-09-21T16:42:38.411495657Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96
time=2023-09-21T16:42:38.876979537Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96
time=2023-09-21T16:42:38.877157441Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96
time=2023-09-21T16:42:38.878372465Z level=info msg=verify result for subject ghcr.io/feynmanzhou/net-monitor@sha256:27c0290c485140c3c998e92c6ef23fba2bd9f09c8a1c7adb24a1d2d274ce3e8e: {
  "verifierReports": [
    {
      "isSuccess": false,
      "name": "notation",
      "message": "Original Error: (Original Error: (signature is not produced by a trusted signer), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
} component-type=server go.version=go1.20.8 trace-id=96f67e50-2bbe-42c6-98ac-fb0c7e41be96

TC17

SSH into one of the nodes that has a ratify replica scheduled on it. Manually reboot the node. Immediately execute pod creation commands.

Error from server: admission webhook "mutation.gatekeeper.sh" denied the request: failed to resolve external data placeholders: failed to send external data request to provider ratify-mutation-provider: failed to send external data request: Post "https://ratify.gatekeeper-system:6001/ratify/gatekeeper/v1/mutate": dial tcp 10.0.254.84:6001: connect: no route to host

There's a short period of time when the node health is not known to be degraded yet. During this service will continue to route requests to dead pods. Once node health is degraded, the requests seem to all route to healthy instance.

TC18

The Redis stateful sets (master + replica) were drained to 0

Error from kubectl run output:

Error from server: admission webhook "mutation.gatekeeper.sh" denied the request: failed to resolve external data placeholders: failed to validate external data response from provider ratify-mutation-provider: non-empty system error: operation failed with error operation timed out after duration 1.95s

Ratify Logs:

time=2023-09-21T20:49:55.965807677Z level=info msg=received request POST /ratify/gatekeeper/v1/mutate 
time=2023-09-21T20:49:55.965917879Z level=info msg=start request POST /ratify/gatekeeper/v1/mutate component-type=server go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa
time=2023-09-21T20:49:55.966001082Z level=info msg=mutating image ghcr.io/deislabs/ratify/notary-image:signed component-type=server go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa
time=2023-09-21T20:49:57.916645653Z level=debug msg=subject descriptor cache miss for value: ghcr.io/deislabs/ratify/notary-image:signed component-type=referrerStore go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa
time=2023-09-21T20:49:57.916739655Z level=debug msg=auth cache miss component-type=referrerStore go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa
time=2023-09-21T20:49:57.917014862Z level=error msg=Error saving value to redis: error saving state: rpc error: code = DeadlineExceeded desc = context deadline exceeded component-type=cache go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa
time=2023-09-21T20:49:57.917058964Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa
time=2023-09-21T20:49:57.917228968Z level=debug msg=mutation: execution time for request: 1951ms component-type=server go.version=go1.20.8 trace-id=084fab3d-0f0d-4f32-9a9b-03db3c3df5fa

Audit also begins to fail with timeout of 4.9 second.

time=2023-09-21T20:52:45.703791311Z level=info msg=received request POST /ratify/gatekeeper/v1/verify 
time=2023-09-21T20:52:45.703840611Z level=info msg=start request POST /ratify/gatekeeper/v1/verify component-type=server go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:45.703937712Z level=info msg=verifying subject ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b component-type=server go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.604600679Z level=debug msg=subject descriptor cache miss for value: ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b component-type=referrerStore go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.604668279Z level=debug msg=auth cache miss component-type=referrerStore go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60472728Z level=error msg=Error saving value to redis: error saving state: rpc error: code = DeadlineExceeded desc = context deadline exceeded component-type=cache go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60474728Z level=warning msg=Error: cache not set, Code: CACHE_NOT_SET, Component Type: cache, Detail: failed to set auth cache for ghcr.io component-type=referrerStore go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60483898Z level=warning msg=Original Error: (Original Error: (Head "https://ghcr.io/v2/deislabs/ratify/notary-image/manifests/sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b": context deadline exceeded), Error: repository operation failure, Code: REPOSITORY_OPERATION_FAILURE, Plugin Name: oras), Error: get subject descriptor failure, Code: GET_SUBJECT_DESCRIPTOR_FAILURE, Plugin Name: oras, Component Type: referrerStore, Detail: failed to resolve the subject descriptor component-type=referrerStore go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60485408Z level=debug msg=cache miss for subject ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b component-type=server go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60487278Z level=error msg=Error saving value to redis: error saving state: rpc error: code = DeadlineExceeded desc = context deadline exceeded component-type=cache go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60488398Z level=warning msg=unable to insert cache entry for subject ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b component-type=server go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac
time=2023-09-21T20:52:50.60490018Z level=info msg=verify result for subject ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b: {
  "verifierReports": [
    {
      "subject": "ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b",
      "isSuccess": false,
      "message": "verification failed: Error: referrer store failure, Code: REFERRER_STORE_FAILURE, Component Type: referrerStore, Detail: could not resolve descriptor for a subject from any stores"
    }
  ]
} component-type=server go.version=go1.20.8 trace-id=384a4c5e-6654-47a8-bdaa-16823df527ac

The original mutation exceeded 1.95s error does not immediately point to what the root cause. Scanning through logs, you eventually see context deadline exceeded errors for cache writes which points to underlying cache being unavailable.

TC19

N/A

TC20

Scale dapr-side-car inject deployment to 0. Scale ratify deployment to 0. Scale ratify deployment back up to 2. Now the daprd sidecar containers will not be present on each pod.

Ratify pod will fail to startup if no daprd pod is enabled. This is due to cache intialization failure which is currently a blocking error.

Logs

dapr client initializing for: 127.0.0.1:50001
Error: error initializing cache of type dapr: error creating default client: error creating connection to '127.0.0.1:50001': context deadline exceeded

TC21

Force a Dapr installation misconfiguration by not applying the redis StateStore CR.

User will not see any errors when applying pods/deployment. Perf will be degraded an may lead to more timeouts.

From logs, cache errors:

time=2023-09-21T22:17:50.216846963Z level=error msg=Error saving value to redis: error saving state: rpc error: code = FailedPrecondition desc = state store is not configured component-type=cache go.version=go1.20.8 trace-id=99e8aed9-e944-4e6f-8588-d7be7d16f940

From Xinhe

level=error msg=Reconciler error CertificateStore=default/certstore-incorrect-cert controller=certificatestore controllerGroup=config.ratify.deislabs.io controllerKind=CertificateStore error=Error fetching certificates in store certstore-incorrect-cert with inline provider, error: Error: cert invalid, Code: CERT_INVALID, Component Type: certProvider name=certstore-incorrect-cert namespace=default reconcileID=6a444f61-fed0-4d0a-b6e1-08bedbe90712
time=2023-09-22T01:55:54.866028606Z level=warning msg=no certificate fetched for certStore certstore-incorrect-cert component-type=verifier go.version=go1.20.8 trace-id=4b6580f6-8b08-4c0b-a1ab-d4264298a6c9
time=2023-09-22T01:55:54.866327608Z level=info msg=verify result for subject ghcr.io/deislabs/ratify/notary-image@sha256:8e3d01113285a0e4aa574da8eb9c0f112a1eb979d72f73399d7175ba3cdb1c1b: {
  "verifierReports": [
    {
      "isSuccess": false,
      "name": "notation",
      "message": "Original Error: (Original Error: (error while loading the trust store, unable to fetch certificates for namedStore: certs), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
}

Susan Shi

2023/09/06 06:23:58

Negative te

Please also update some scenarios to be executed on arm64 machines, for Azure, we can create a node pool with with VM VM_SIZE="Standard_D2ps_v5"

2023/09/06 23:12:26

Results

Could we capture the result in this column to start our FAQ, TSG document?

Feynman Zhou

2023/09/06 23:42:48

We will capture the result here and see which cases are valuable to be organized in FAQ, TSG

2023/09/06 23:13:02

Registry connection timeout

any suggestions on how to simulate a registry timeout?

2023/09/06 23:16:34

trusted identities

Suggestion to add a scenarios where there is a mix of valid and invalid signature

2023/09/06 23:45:20

It sounds reasonable. We can add this case

Akash Singhal

2023/09/21 21:36:43

C-19

@Feynman could you provide more details on expected behavior and why this test is necessary? If GK is crashed, then Ratify won't even receive requests. And in flight requests will still be responded to from Ratify's perspective since it's not expecting a server response.

2023/09/21 23:39:16

Make sense. I think we can skip this case