A work in progress (WIP) document discussing the following open source [cert-manager issue](https://github.com/cert-manager/cert-manager/issues/2239), and how it might make sense to tackle this. All comments,ffeedback and edge cases welcomed. ## The Issue(s) Currently a user creating a certificate resource has many options and depending on the Issuer / CA they are using, and may have to fill out more YAML fields than they care about. I propose that the real issue to resolve here is one of user convenience / experience, in that a user ultimately wants a certificate and the vast majority of the configuration is of little concern. A second issue that I think we perhaps tackle sererately is the idea of what `Issuer` / `ClusterIssuer` should be used for the certificate. A tenant with access only to one namespace and no access to configure Issuers would use only the Issuer(s) present. In a lot of scenarios that would be a single issuer for that namespace or a default `ClusterIssuer`. In both of those cases, just having the certificate default to an `Issuer` rather than having to manually specify it would be a more optimal experience. The questions to me are: - What is defaultable? - What should a `Certificate` user have to specify? - How do we avoid / resolve collisions of default settings? - What is the relationship between `Certificates` and `Issuers`? ### Users / Use Cases For users who create their own `Issuers` or `ClusterIssuers` but do not manage a whole cluster (eg. the cert-manager installation), I think its acceptable that they have the ability to both: - Specify a specific `Issuer` for a certificate (Which anyone can do now) - Not have to specify an `Issuer` at all, but be able to specify the "default" `Issuer` which should be used, based on some criteria. (this enhancement) A cluster operator / platform team might want to set the appropriate values on a `Certificate` for a given namespace, or set of `Issuers`. This could be to encourage a default key sive of `4096` for example, or it could just be to select the correct `Issuer`. Presets / Defaults might be a method by which a platform team changes defaults in the future. An end user / tenant who just wants a certificate for `my-app.company.domain.tld` in the secret `my-app-cert`. A CA / `Issuer` might have ideas about the configuration options, eg. the Venafi Issuer (or policy) might enforce private key rotation, so a preset / default value for the `Issuer` would ideally match this backend policy, if not specified / overridden by the user. Perhaps in the future these presets could be synced from a CA into the Issuer resource? (as a stauts field) ## What is not in scope? Specifically we want to enable "default" values which cert-manager users can specify for their cluster / namespaces. These values will not be enforced, and if the user provides values, the user provided values would take preference. For those that want to enforce settings, you should check out cert-managers [approver-policy](https://github.com/cert-manager/approver-policy). ## What is defaultable? My working assumption is that we are only concerned with `certificate.spec` fields, so I figured it useful to output these high level fields to see: - What makes sense to be defaultable - What challenges might be presented by certain fields Here is a list of all certificate spec fields: ```yaml! apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: httpbin namespace: httpbin spec: additionalOutputFormats: {} commonName: "" dnsNames: [] duration: "" emailAddresses: [] encodeUsagesInRequest: false ipAddresses: [] isCA: false issuerRef: {} keystores: {} literalSubject: "" privateKey: {} renewBefore: "" revisionHistoryLimit: 10 secretName: httpbin-cert subject: {} uris: [] usages: [] ``` I'd suggest that the following keys would never be defaulted as the end user of the certificate would want to specify these (if used): ```yaml commonName: "" dnsNames: [] emailAddresses: [] ipAddresses: [] literalSubject: "" uris: [] secretName: "" # Unless we could use CEL for templating? Not massive value though ``` This leaves the following keys to potentially default, with current cert-manager defaults: ```yaml! additionalOutputFormats: {} # No default - alpha feature needs enabled duration: "" # 90 days unless issuer / CA says otherwise encodeUsagesInRequest: false # isCA: false # If true will auto add 'cert sign' to 'usages' below. issuerRef: {} # Currently needs specified keystores: {} # No defaults privateKey: # Some defaults provided within algorithm: # Unsure onb default here encoding: "PKCS1" rotationPolicy: "Never" size: # Depedant on the algorithm, either ignored (Ed25519), 2048 (RSA) & 256 (ECDSA) renewBefore: # Defaults 2/3rd of certificate life revisionHistoryLimit: nil # No default here for backwards compatability secretName: httpbin-cert # REQUIRED - no defaulting? Unless CEL helps? subject: {} # Possibly retrieved from Issuer? Some fields may be required depening on issuer usages: # Defaults to these values shown if not specified - "digital signature" - "key encipherment" ``` Some of the above keys still might not make sense to default as it very much might depend on the users application, eg, they need keystores in which case they need to pass the secret name for te keystore password. Pltaform teams might want to setup a pattern for this in their cluster, but right now, doing this with templating (outside of cert-maanger all together) is probably the best. ## Possible Solutions / Concepts ### Certificate Spec Value Presets (no Issuer) 1. One solution would be to have a `CertificatePreset` be tied to an `Issuer` or `ClusterIssuer` explicitly. So the default values provided in this case would apply to all certificate requests that use that `Issuer`. This might introduce the concept as a new CRD making a 1-1 connection between Issuer and Preset. This would avoid collisions of multiple Presets to an Issuer, but would mean the `certificate.spec.issuerRef` would still need to be specified on the certificate itself. ```yaml! apiVersion: presets.cert-manager.io/v1alpha1 kind: CertificatePreset metadata: name: example spec: issuerSelector: kind: ClusterIssuer group: cert-manager.io name: letsencrypt-prod presets: subject: organizations: - My Company privateKey: size: 4096 algorithm: RSA ``` This would introduce a new CRD resource but would give you options in how and when these defaults are applied. 1. If the above 1-1 relationship is assumed a good route, then perhaps a new CRD is not required, and instead, the presets could be specified within the issuer itself. For example, something like: ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-production spec: acme: {} # ususal configuration certificatePresets: subject: organizations: - My Company privateKey: size: 4096 algorithm: RSA ``` This expands an existing CRD definition but with know set of potential values, so it could be resonably well defined. For those that use many Issuer resources, this could mean defining a lot of extra YAML in all their issuers. 1. If the Preset and Issuer are not 1-1 mapping, then we could have a similar selection mechanism that [approver-policy]() has for `CertificateRequestPolicies` (CRP). Where an issuerRef is selected but could be a wildcard match adding the option to select multiple issuers with the same `CertificatePreset` resource: ```yaml! apiVersion: presets.cert-manager.io/v1alpha1 kind: CertificatePreset metadata: name: example spec: presets: subject: organizations: - My Company privateKey: size: 4096 algorithm: RSA selector: issuerRef: group: cert-manager.io kind: ClusterIssuer name: letsencrypt-production namespace: matchNames: - '*' ``` The limitation here is that we now have potentially conflicting settings or have to define an order of precedence, which becomes tricky adding more complexity. 1. There is of course the [original proposed solution](https://github.com/cert-manager/cert-manager/issues/2239#issue-507864728) where a preset is applied when the certificate is applied to the cluster as an admission webhook, for example: ```yaml! apiVersion: admission.cert-manager.io/v1alpha1 kind: CertificatePreset metadata: name: devops spec: selector: matchLabels: certificate-class: prod-edge organization: - Jetstack keySize: 2048 keyAlgorithm: RSA issuerRef: name: letsencrypt-prod ``` The original proposal stated a preset would be applied once and not re-applied if changed later. This would me re-applying certificates if presets were changed. A user may not be aware of presets changing so this might be a little disconnected from a user experience. 1. We could instead look to other tools such as Gatekeeper or Kyverno to apply mutations to a `Certificate` resource when no value is provided. Subtly this is not using them to enforce, but provide defaults. This would leverage open source solutions currently available to all that have been tried and tested, rather than developing a customer solution and / or resource specifically for `Certificate`s. ### Defaulting Issuers (Not Certificate details) 1. For setting a default `Issuer`, the solution here could be fairly simple in that a `Namespace` annotion is added to set a default issuer for a given namespace. For a cluster level default, the annotion(s) would have to be set on the cert-manager installation namespace. Using this mechanism a default could be set for the cluster without having to set the annotation in all namespaces, which is ok for 10 namespaces, but a little cumbersome for 500 namespaces. A namespace level setting should have precedence over the cluster default. ```yaml! --- apiVersion: v1 kind: Namespace metadata: annotations: cert-manager.io/default-issuer: my-issuer labels: kubernetes.io/metadata.name: app-team-1 name: app-team-1 spec: finalizers: - kubernetes --- apiVersion: v1 kind: Namespace metadata: annotations: cert-manager.io/default-cluster-issuer: a-default-cluster-issuer labels: kubernetes.io/metadata.name: cert-manager name: cert-manager spec: finalizers: - kubernetes ``` The order of application could be: - Default annotation set in `cert-manager` namespace - Default cluster-issuer annotation set in the tenant namesapce - Default issuer annotation set in the tenant namespace - Certificate `issuerRef` spec wins all. There might need to be a set of annotations to specify `group` and `kind` as well as the name. For example to support external issuers such as the [AWS Private CA](https://github.com/cert-manager/aws-privateca-issuer) and [Google CAS](https://github.com/jetstack/google-cas-issuer) issuers. 1. Another potential solution to automatically setting the `Issuer` for a certificate, might be an `IssuerPreset` resource. This resource would preset the Issuer details for a given selector. For example maybe something like ```yaml! apiVersion: presets.cert-manager.io/v1alpha1 kind: IssuerPreset metadata: name: example spec: issuerRef: group: cert-manager.io kind: ClusterIssuer name: inhouse-acme-solution selector: matchLabels: pki: internal matchExpressions: - { key: pki, operator: In, values: [internal, inhouse, custom] } namespace: matchNames: - '*' ``` With this type of setup it would be possible to set one preset top many certifcates from many namespaces. 1. An alternative to the above would be a much simpler `Preset` resource, that has Cluster and namespaced variants: ```yaml! apiVersion: presets.cert-manager.io/v1alpha1 kind: IssuerPreset metadata: name: example namespace: my-app spec: issuerRef: group: cert-manager.io kind: ClusterIssuer name: inhouse-acme-solution # Or a specific namespaced issuer # issuerRef: # group: cert-manager.io # kind: Issuer # name: my-awesome-issuer ``` And the cluster variant: ```yaml! apiVersion: presets.cert-manager.io/v1alpha1 kind: ClusterIssuerPreset metadata: name: example spec: issuerRef: group: cert-manager.io kind: ClusterIssuer name: inhouse-acme-solution ``` Having a dedicated resource might make it easier to permission and reason about. The above provides the 1-1 mapping I think is reasonble for setting this sort of default. 1. We could again use a policy tool to provide a default value for the `issuerRef`. Whilst this is slightly more challenging with these fields being required it it possible. 1. In k8s 1.28 it is fairly simple to enable [ValidatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) as it is now a beta feature behind a feature flag. It might be possible to create some rules in the future using CEL to set some defaults. This needs more investigation as at the moment this only supports validating webhooks but a mutating version is possibly in the works. (Link TBD) ### Proposed solutions From a non-maintainer perspective, to me it would seem best to have the following solution(s): 1. Not build anything into cert-manager specifically for this. Instead we document and leverage other open-source tools to set default values on `Certificate` resourcs. This solution can potentially cover both cases of certificate fields and the issuer being used. ### Risks - No default issuer is set for a namespace / certificate so the certificate resource is left in "limbo"? - Would make `issuerRef` a non required field in `Certificate` spec. Possibly breaking change? (This can be avoided) - Solutions that enable inheritence / precedence might cause confusion without clear documentation. - Adding this to the existing Issuer specification might impact External Issuers and create a maintenance headache - Adding any solution at all would be involved work and require long term maintenance. - Required use of other open-source tooling might not be valid for some users who cannot control their cluster environment and / or addons. ### Extension - ingresss-shim Maybe some consideration to the ingress-shim should be given here as well. If the proposed idea moves forwards, then in theory a given `Ingress` would in theory be tied to a default `Issuer` or `ClusterIssuer`. All other values could be defaulted with a preset value. Where the tenant / user wants to retain control, they could still use the annotationa available to override. It's worth noting here that with the ingress-shim it is possible to set a cluster wide default `ClusterIssuer` with these arguments to the cert-manger-controller: > --default-issuer-group string Group of the Issuer to use when the tls is requested but issuer group is not specified on the ingress resource. (default "cert-manager.io") > --default-issuer-kind string Kind of the Issuer to use when the tls is requested but issuer kind is not specified on the ingress resource. (default "Issuer") > --default-issuer-name string Name of the Issuer to use when the tls is requested but issuer name is not specified on the ingress resource. It is not possible to set a cluster wide default when using `Certificate` resources directly. I think this is an issue to be raised seperately as there is inconsistency in features here. ## Whats next? - [X] Discuss at a cert-manager bi-weekly meeting - [ ] Gather feedback on the proposed solution(s) - [ ] Check for better alternatives. - [X] Kyverno - [ ] OPA Gatekeeper - [ ] Pick a solution(s) to move forward with that covers the majority of use cases. - [ ] Document on the [issue](https://github.com/cert-manager/cert-manager/issues/2239) the intended direction and close / progress as appropriate. ### Future enhancements / comments / ideas / to discuss... - Perhaps in future the defaults could be interpretted from the given CA into the Issuer `status` field. - Maybe these presets are reflected in the `certificate.status` so a tenant has an easy way to see what has been defaulted, so they can determine if they need to override it. Talked over this with maintainers and found that an event on the certificate would be best here. - More complex defaulting use case, such as the `secretName` field. This could be `${cert-name}-cert` for example by default. - Changing presets - should a certificate be reissued automatically, or an evaluation ran to see if it does need re-issued? - Supporting external issuers is a stretch goal? ## Kyverno Alternative As part of "Check for better alternatives" I have taken a deep dive into Kyverno as a potential solution otuside of cert-manager code to set default values. ### Two example use cases I am targetting these two initial use cases covering both simple and required `Certificate` spec fields: 1. Default the `revisionHistoryLimit` in a Certificate resource 2. Default the `issuerRef` in a Certificate resource ### 1 - revisionHistoryLimit We can mutate this spec field and override the value for any Certificate resource easily with policy. We can also conditionally set this if the value is not provided by the user / tenant. This is done by using `+(<field>)` syntax [documented here](https://kyverno.io/docs/writing-policies/mutate/#add-if-not-present-anchor). This works really well and has no other impact really that I could see. ### 2 - issuerRef This field is more complex as it it an object (`{}`) and also "required". On initial testing I got this following error: ```log Error from server (NotAcceptable): error when creating "yamls/kyverno/cert.yaml": admission webhook "webhook.cert-manager.io" denied the request: spec.issuerRef.name: Required value: must be specified ``` This means the Certificate resource is failing based on the cert-manager validation webhook. Validation should be applied after mutating webhooks, so in theory the kyverno policy has already been processed at the point. One idea here is just to remove the "cert-manager-webhook" webhooks as I am unsure what they do. Deleting these did indeed make my Kyverno `ClusterPolicy` work in this case, but not an ideal solution. I attempted to look for ordering of webhooks, however this doesn't seem to be a feature. Infact the [documentation]() suggests webhooks should be idempotent and you shouldn't rely on ordering. I have found that the cert-manager `mutatingwebhookconfigurations.admissionregistration.k8s.io` actually will set an empty string for something like the `secretName` field if none is provided. The webhook called `cert-manager-webhook` will actually execute before the Kyverno webhooks. This means that Kyverno will see a value is set, even if it is `""` and will therefore not set the default it has in `ClusterPolicy`. A workaround to this issue is to rename the "cert-manager-webhook" validating and mutating webhook configurations to prefix them with `z-` to always run last, eg: ```shell > k get mutatingwebhookconfigurations.admissionregistration.k8s.io NAME WEBHOOKS AGE kyverno-policy-mutating-webhook-cfg 1 6d23h kyverno-resource-mutating-webhook-cfg 1 6d23h kyverno-verify-mutating-webhook-cfg 1 6d23h z-cert-manager-webhook 1 87m > k get validatingwebhookconfigurations.admissionregistration.k8s.io NAME WEBHOOKS AGE ingress-nginx-admission 1 28d kyverno-cleanup-validating-webhook-cfg 1 6d23h kyverno-exception-validating-webhook-cfg 1 6d23h kyverno-policy-validating-webhook-cfg 1 6d23h kyverno-resource-validating-webhook-cfg 0 6d23h z-cert-manager-webhook 1 89m ``` This does indeed work and the Kyverno policies will now execute before the cert-manager-webhook one. Another suggestion was to scope down the "cert-manager-webhook" from all cert-manager.io & acme.cert-manager.io resources, to only `CertificateRequest` resource under cert-manager.io. With this change there is no mutating of `Certificates` and this better enables the policy to work. This appraoch is being progressed [here](https://github.com/cert-manager/cert-manager/pull/6311). ### Generic Kyverno Policy Examples You might find it useful to browse some example policies [here](https://github.com/kyverno/policies/) from Kyverno. There are some example cert-manager examples also [here](https://github.com/kyverno/policies/blob/main/cert-manager/restrict-issuer/restrict-issuer.yaml) but the repo covers many others tools and use cases. ### Response to issue / demo I have been re-evaluating the "Certificate Preset" idea based on what could be done with other open source tools, rather than natively within cert-manager. Again I wanted to stay away from `enforced` policy here and stay with softly defaulting unless a user specifies something. Specifically I have been trying [kyverno](https://kyverno.io/) as an example tool you could use to provide "default" values to the Certificates. You could of course use [Gatekeeper](https://www.openpolicyagent.org/docs/latest/kubernetes-introduction/) here as well and welcome any examples from people if they are willing to share those. Essentially my findings are that within a Certificate resource, it is very quick and easy to setup a `Policy` / `ClusterPolicy` to default any non-required values. So this would include: ```yaml additionalOutputFormats: {} # No default - alpha feature needs enabled duration: "" # 90 days unless issuer / CA says otherwise encodeUsagesInRequest: false # isCA: false # If true will auto add 'cert sign' to 'usages' below. keystores: {} # No defaults privateKey: # Some defaults provided within algorithm: # Unsure onb default here encoding: "PKCS1" rotationPolicy: "Never" size: # Depedant on the algorithm, either ignored (Ed25519), 2048 (RSA) & 256 (ECDSA) renewBefore: # Defaults 2/3rd of certificate life revisionHistoryLimit: nil # No default here for backwards compatability subject: {} # Possibly retrieved from Issuer? Some fields may be required depening on issuer usages: # Defaults to these values shown if not specified - "digital signature" - "key encipherment" ``` You could of course default these values as well but I think most people would need to provide a value to at least one of these fields. So to me this makes little sense to default: ```yaml commonName: "" dnsNames: [] emailAddresses: [] ipAddresses: [] literalSubject: "" uris: [] ``` The two fields that I am going to exclude initially are: ```yaml issuerRef: {} # Currently needs specified secretName: "" # Unless we could use CEL for templating? Not massive value though ``` These are "required" fields in the CRD and therefore cert-manager default installation will validate the resource has these fields set appropriately before saving to the cluster. (This can be worked around, but more on this later under [Defaulting Required Keys](#Defaulting-Required-Keys)) #### Demo Env These instructions are really just for this demo, feel free to use for your own demo and testing. **NOTE**: This is **not** a production setup. ```shell= kind create cluster --name defaults --image "kindest/node@sha256:9f3ff58f19dcf1a0611d11e8ac989fdb30a28f40f236f59f0bea31fb956ccf5c" export CERT_MANAGER_CHART_VERSION="v1.12.3" KYVERNO_CHART_VERSION="3.0.5" helm upgrade --install kyverno kyverno/kyverno --version $KYVERNO_CHART_VERSION --namespace kyverno-system --create-namespace helm upgrade --install cert-manager jetstack/cert-manager --namespace cert-manager --version $CERT_MANAGER_CHART_VERSION --set installCRDs=true --create-namespace ``` #### Demo Resources There are only two resource to try here and we have not setup an `Issuer` because we don't actually need to get a certificate to test that the correct defaults have applied. Here's the `ClusterPolicy`: ```yaml= --- # Example policy to mutate a certificate resource if values are not provided # for certain fields that are not "required" fields in the Certificate CRD. apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: 0-mutate-certificate-defaults spec: # See here: https://kyverno.io/docs/writing-policies/policy-settings/ # Important if running multiple policies, otherwise Kyverno will try to # validate the Certificate resource before applying other policies. # Usually you should ignore this and set it to true schemaValidation: true # default failurePolicy: Fail rules: # Set a sane default for the history field if not already present - name: set-revisionHistoryLimit match: any: - resources: kinds: - Certificate mutate: patchStrategicMerge: spec: # +(...) This is the clever syntax for if not already set +(revisionHistoryLimit): 2 # Set rotation to always if not already set - name: set-privateKey-rotationPolicy match: any: - resources: kinds: - Certificate mutate: patchStrategicMerge: spec: privateKey: +(rotationPolicy): Always # Set private key details for algorithm an size - name: set-privateKey-details match: any: - resources: kinds: - Certificate mutate: patchStrategicMerge: spec: privateKey: +(algorithm): ECDSA +(size): 521 +(encoding): PKCS1 ``` And the `Certificate` I used for testing, called "test-revision": ```yaml= --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: test-revision namespace: default spec: dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer secretName: test-revision-cert ``` Apply the policy to the cluster and check the policy is ready: ```shell= > kubectl apply -f yamls/kyverno/cpol-mutate-certificate-defaults.yaml > kubectl get cpol NAME BACKGROUND VALIDATE ACTION READY AGE MESSAGE 0-mutate-certificate-defaults true Audit True 4s Ready ``` Then dry-run the certificate to validate the output is as expected: ```shell= > k apply -f yamls/kyverno/cert-test-revision.yaml --dry-run=server -o yaml ``` ```yaml= apiVersion: cert-manager.io/v1 kind: Certificate metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"cert-manager.io/v1","kind":"Certificate","metadata":{"annotations":{},"name":"test-revision","namespace":"default"},"spec":{"dnsNames":["example.com"],"issuerRef":{"group":"cert-manager.io","kind":"ClusterIssuer","name":"not-my-corp-issuer"},"secretName":"test-revision-cert"}} policies.kyverno.io/last-applied-patches: | set-privateKey-details.0-mutate-certificate-defaults.kyverno.io: added /spec/privateKey/size set-privateKey-rotationPolicy.0-mutate-certificate-defaults.kyverno.io: added /spec/privateKey set-revisionHistoryLimit.0-mutate-certificate-defaults.kyverno.io: added /spec/revisionHistoryLimit creationTimestamp: "2023-08-29T17:36:35Z" generation: 1 name: test-revision namespace: default uid: 11c2ccb0-e1a8-4b44-b09e-41c434c823fd spec: dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer privateKey: algorithm: ECDSA encoding: PKCS1 rotationPolicy: Always size: 521 revisionHistoryLimit: 2 secretName: test-revision-cert ``` We have successfully defaulted the `privateKey` and `revisionHistoryLimit` fields. Let's actualy override all of these now to show that we can still set what we want as an end user: ```yaml= --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: test-revision-override namespace: default spec: dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer secretName: test-revision-override-cert privateKey: algorithm: RSA encoding: PKCS8 rotationPolicy: Never size: 4096 revisionHistoryLimit: 44 ``` And now check you get all the settings you actually wanted: ```shell= > kubectl apply -f yamls/kyverno/cert-test-revision-override.yaml ``` ```yaml= apiVersion: cert-manager.io/v1 kind: Certificate metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"cert-manager.io/v1","kind":"Certificate","metadata":{"annotations":{},"name":"test-revision-override","namespace":"default"},"spec":{"dnsNames":["example.com"],"issuerRef":{"group":"cert-manager.io","kind":"ClusterIssuer","name":"not-my-corp-issuer"},"privateKey":{"algorithm":"RSA","encoding":"PKCS8","rotationPolicy":"Never","size":4096},"revisionHistoryLimit":44,"secretName":"test-revision-override-cert"}} creationTimestamp: "2023-08-29T17:39:55Z" generation: 1 name: test-revision-override namespace: default uid: 26637271-afb3-46b8-b4ce-eb3336ce1b9f spec: dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer privateKey: algorithm: RSA encoding: PKCS8 rotationPolicy: Never size: 4096 revisionHistoryLimit: 44 secretName: test-revision-override-cert ``` #### Defaulting Required Keys This gets trickier because of some implicit things going on. As discovered with help from @walljr & @erikgb, when a `Certificate` resource is applied to a kubernetes cluster, mutating webhooks are applied before validating webhooks. When the existing cert-manager mutating webhook runs, if there is no value in a required field, it will add the required field with an empty value such as: `secretName: ""`. The consequence of this action is that Kyverno's mutating webhook policy will not apply as an empty value is already present. There are two potential fixes for this: 1. Rename the cert-manager-webook mutating and validating webhooks with `z-<existing_name>` so that they execute last, after the Kyverno webhooks 2. We fix cert-manage mutatingwebhookconfiguration to not mess with `Certificate` resource as in [this PR](https://github.com/cert-manager/cert-manager/pull/6311). Let's go with option 2 as we can manually apply this fix to our environment. Replace the `resources: - '*/*'` with: `resources: -certificaterequests`, in the `cert-manager-webhook` mutating webhook resource, resulting in something like this: ```yaml= webhooks: - admissionReviewVersions: - v1 ... name: webhook.cert-manager.io ... rules: - apiGroups: - cert-manager.io - acme.cert-manager.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - certificaterequests scope: '*' ``` Now we can set a Kyverno `ClusterPolicy` to apply default values for `secretName` and `issuerRef` fields. In the example below we do two things: 1. Apply a default `secretName` that is the name of the Certificate object suffixed with `-cert`. 2. We set the relevant `issuerRef` fields to use our example 'our-corp-issuer' `ClusterIssuer`. ```yaml= --- apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: 1-mutate-certificate-required spec: rules: # Test if we can set a secretName when one is not provided - name: set-default-secret-name match: any: - resources: kinds: - Certificate mutate: patchStrategicMerge: spec: +(secretName): "{{request.object.metadata.name}}-cert" # Test if we can set a default issuerRef - name: set-default-issuer-ref match: any: - resources: kinds: - Certificate mutate: patchStrategicMerge: spec: +(issuerRef): name: our-corp-issuer kind: ClusterIssuer group: cert-manager.io ``` Apply this policy and also patch the previous policy we applied to disable schema validation after mutation: ```shell= kubectl apply -f yamls/kyverno/cpol-mutate-certificate-required.yaml kubectl patch cpol 0-mutate-certificate-defaults --type=merge -p '{"spec":{"schemaValidation": false}}' ``` > **NOTE**: Please read about the validation setting, we have use disabled it here because we have two policies both affecting certificates. Cert-manager will validate the actual resource with the `validatingwebhookconfigurations.admissionregistration.k8s.io` after all muttions are applied anyway. You could avoid this completely by using one policy with all required mutation rules. Now we can test with a new `Certificate` resource: ```yaml= --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: test-minimal namespace: default spec: dnsNames: - example.com ``` Apply this resource in dry run to validate all our rules have applied: ```shell= kubectl apply -f yamls/kyverno/cert-test-minimal.yaml --dry-run=server -o yaml ``` Example succesful output: ```yaml= apiVersion: cert-manager.io/v1 kind: Certificate metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"cert-manager.io/v1","kind":"Certificate","metadata":{"annotations":{},"name":"test-minimal","namespace":"default"},"spec":{"dnsNames":["example.com"]}} policies.kyverno.io/last-applied-patches: | set-default-issuer-ref.1-mutate-certificate-required.kyverno.io: added /spec/issuerRef set-default-secret-name.1-mutate-certificate-required.kyverno.io: added /spec/secretName set-privateKey-details.0-mutate-certificate-defaults.kyverno.io: added /spec/privateKey/encoding set-privateKey-rotationPolicy.0-mutate-certificate-defaults.kyverno.io: added /spec/privateKey set-revisionHistoryLimit.0-mutate-certificate-defaults.kyverno.io: added /spec/revisionHistoryLimit creationTimestamp: "2023-08-30T16:54:03Z" generation: 1 name: test-minimal namespace: default uid: c29c5d27-47d5-4431-bd2f-20660a9a18b5 spec: dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: our-corp-issuer privateKey: algorithm: ECDSA encoding: PKCS1 rotationPolicy: Always size: 521 revisionHistoryLimit: 2 secretName: test-minimal-cert ``` And just to be absolutely sure we have not enforced any settings let us try and set every value we have a policy rule for, and check they do indeed match the `Certificate` applied. ```yaml= --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: test-revision-override namespace: default spec: dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer privateKey: algorithm: RSA encoding: PKCS8 rotationPolicy: Never size: 4096 revisionHistoryLimit: 44 secretName: test-revision-override-cert ``` Apply this file, even just in dry run and use `yq` to extract the returned `spec` section: ```shell= kubectl apply -f yamls/kyverno/cert-test-revision-override.yaml --dry-run=server -o yaml | yq .spec ``` This should return: ```yaml= dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer privateKey: algorithm: RSA encoding: PKCS8 rotationPolicy: Never size: 4096 revisionHistoryLimit: 44 secretName: test-revision-override-cert ``` Which should match exactly the certificate resource we applied already: ```yaml= # yq '.spec' yamls/kyverno/cert-test-revision-override.yam dnsNames: - example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: not-my-corp-issuer privateKey: algorithm: RSA encoding: PKCS8 rotationPolicy: Never size: 4096 revisionHistoryLimit: 44 secretName: test-revision-override-cert ``` #### Summary With these example Kyverno policies we have shown that policies are a viable tool to set default values, even for required fields with a minor tweak. The quickstart for this is essentially 5 commands including installation: ```shell= export CERT_MANAGER_CHART_VERSION="v1.12.3" KYVERNO_CHART_VERSION="3.0.5" helm upgrade --install kyverno kyverno/kyverno --version $KYVERNO_CHART_VERSION --namespace kyverno-system --create-namespace helm upgrade --install cert-manager jetstack/cert-manager --namespace cert-manager --version $CERT_MANAGER_CHART_VERSION --set installCRDs=true --create-namespace kubectl apply -f <path-to-policy> kubectl apply -f <path-to-certificate> ``` In my opinion this is a vastly more flexible maintainable and open solution to setting cluster wide defualt values. It's so easy that anyone could get started now.