--- title: Kyverno generate rule notes description: An active set of notes for generate rule behavior changes proposed in Kyverno 1.10+ tags: kyverno --- # Kyverno generate rule notes This is a working notes section to document the "generate existing" behavior and other changes to generate rules. ## Definitions ### New "New" in the context of generation for existing means a couple things: 1. A resource which a user/process is proposing to be created (i.e., a `CREATE` AdmissionReview request) 2. An existing resource in the cluster which is being updated or deleted, thereby creating an AdmissionReview request (`UPDATE` or `DELETE`) These all have in common that they are encapsulated in an AdmissionReview request sent by the Kubernetes API server to listening webhooks. #### Examples 1. A user attempting to create a new Pod 2. a user attempting to add a label to a ConfigMap 3. a ServiceAccount attempting to delete a ClusterRoleBinding. ### Existing "Existing" in the context of generation for existing means a resource which pre-exists in the cluster but does not have an AdmissionReview request sent for it. #### Examples 1. A Secret which is sitting idle and not being manipulated in any way (reads, gets, lists do not trigger an AdmissionReview) 2. Several Namespaces which exist in the cluster and are not being manipulated ### Downstream Synonymous with "generated". A downstream/generated resource is one which Kyverno creates based upon a rule of type `generate`. ## Premise "Generate for existing" functions identically to standard generate rules except in one respect. With the field `generateExisting` set to a value of `true` causes a one-time matching of resources (determined by a union of the `match` and `exclude` blocks plus preconditions) when the policy is created. From that moment forward, it functions no different from a standard generate rule. The generated resources managed by Kyverno are identical regardless of the type of generate rule from which they emanate. ## Behavioral Flows ### Simple create `data`, sync ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: any: - resources: kinds: - Namespace generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true data: metadata: labels: created-by: kyverno also-created-by: chip spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Existing Namespaces `baz` and `bin`]-->n2[Install policy]-->n3[Generated Netpols in `baz` and `bin`] ``` ### Simple create `data`, sync plus new ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: any: - resources: kinds: - Namespace generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true data: metadata: labels: created-by: kyverno also-created-by: chip spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Existing Namespaces `baz` and `bin`]-->n2[Install policy]-->n3[Generated netpols in `baz` and `bin`]-->n4[Create new Namespace `foo`]-->n5[Generated netpol in `foo`] ``` ### Simple create `data`, sync plus new, plus delete ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: any: - resources: kinds: - Namespace generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true data: metadata: labels: created-by: kyverno also-created-by: chip spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Existing Namespaces `bin` and `baz`]-->n2[Install policy]-->n3[Generated netpols in `bin` and `baz`]-->n4[Create new Namespace `foo`]-->n5[Generated netpol in `foo`]-->n6[Delete policy]-->n7[Delete netpols in `bin`, `baz`, and `foo`] ``` ## Rule Update Behaviors There are a few different parts to a "generate existing" rule. Depending on which part(s) a user updates and the configuration of that rule, different behaviors may result or need to be blocked. This attempts to capture some of those. ### Data, sync: Change match (including preconditions) In the scenario, which probably isn't any different just because this is a "generate existing" rule, there are already downstream resources which exist based upon the original rule definition. #### Step A 1. Block such an update in the previous step. 2. Delete netpol in `bin` and generate in `baz` 3. Leave netpol in `bin` and generate in `baz` #### Step B Assuming the rule update in the previous step is allowed: 1. Update netpols in both `bin` and `baz` 2. Update netpol in `baz` only #### Questions 1. Would your answer be different if a new Namespace is created anywhere in this flow which matches on the current state of the rule? 2. Would your answer be different if `generate.data.synchronize: false`? ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: #### This and down is changed any: - resources: kinds: - Namespace selector: matchLabels: foo: bar #### This and up is changed generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true data: metadata: labels: created-by: kyverno also-created-by: chip spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Existing Namespaces `bin` and `baz`]-->n2[Namespace `bin` has label `foo=bar` while `baz` has label `foo=brown`]-->n3[Install policy]-->n4[Namespace `bin` receives netpol]-->n5[Change policy to match on `foo=brown`]-->n6[A: ??????]-->n7[Modify the rule's contents of `generate.data`]-->n8[B: ??????] ``` ### Data, sync: Change `kind`, `apiVersion`, `name`, or `namespace` under the `generate` object In the scenario, which probably isn't any different just because this is a "generate existing" rule, there are already downstream resources which exist based upon the original rule definition. Remember that the Kubernetes API will not allow changes to these fields on an existing resource. #### Step A 1. Block updates to `kind`, `apiVersion`, `name`, or `namespace` after the policy/rule is created. 2. Block updates to `kind`, `apiVersion`, `name`, or `namespace` only if there are any downstream resources which exist. 3. Delete `default-deny` netpol and generate `default-foo` ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: any: - resources: kinds: - Namespace selector: matchLabels: foo: bar generate: kind: NetworkPolicy #### This and down is changed apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" #### This and up is changed synchronize: true data: metadata: labels: created-by: kyverno also-created-by: chip spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Existing Namespaces `bin` and `baz`]-->n2[Namespace `bin` has label `foo=bar` while `baz` has label `foo=brown`]-->n3[Install policy]-->n4[Namespace `bin` receives netpol]-->n5[Change `generate.name` to `default-foo`]-->n6[A: ??????] ``` ### Clone, sync: Change `name` or `namespace` under `generate.clone` In this scenario, the rule is updated with respect to the clone source, either in the `name` or `namespace` field. Synchronization is enabled which presently only governs what happens when the source is changed. #### Step A 1. Block updates to `name` and `namespace` after the policy/rule is created. 2. Block updates to `name` and `namespace` only if there are any downstream resources which exist. 3. The netpol in `bin` is refreshed/recreated with the contents of the `bar` netpol. 4. The netpol in `bin` remains untouched. Only newly-generated resources from this point forward are cloned from `bar`. #### Step B 1. Nothing happens. 2. The netpol in `bin` is synchronized (assumes selection 4 from above). #### Considerations 1. What happens if the new name points to a resource which is not a netpol? Would Kyverno be expected to compare the apiVersion and kind of the new resource to ensure it matches what is specified under the `generate` object? 3. What happens if the new name points to a non-existent resource? ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: any: - resources: kinds: - Namespace selector: matchLabels: foo: bar generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true clone: name: foo #### This and down is changed namespace: platform #### This and up is changed ``` ```mermaid flowchart TD n1[Existing Namespaces `bin` and `baz`]-->n2[Namespace `bin` has label `foo=bar` while `baz` has label `foo=brown`]-->n3[Install policy]-->n4[Namespace `bin` receives netpol]-->n5[Change `generate.clone.name` to `bar`]-->n6[A: ??????]-->n7[User updates `foo` netpol]-->n8[B: ??????] ``` ## Trigger changes/deletes ### Trigger deleted Deletion of the trigger responsible for a corresponding generated resource, when `spec.generate.synchronize` is set to `true` will cause deletion of the generated resource. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: rules: - name: generate-networkpolicy match: any: - resources: kinds: - Service selector: matchLabels: foo: bar generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.namespace}}" synchronize: true data: metadata: labels: created-by: kyverno spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Create Service `yellow` with label `foo=bar` in `eng` Namespace]-->n2[Generate netpol `default-deny`]-->n3[Delete `yellow` Service]-->n4[`default-deny` netpol in `eng` is deleted] ``` ### Trigger changed, no match A change of the trigger responsible for a corresponding generated resource, when `spec.generate.synchronize` is set to `true` will cause deletion of the generated resource if it no longer matches the rule. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: rules: - name: generate-networkpolicy match: any: - resources: kinds: - Namespace selector: matchLabels: foo: bar generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true data: metadata: labels: created-by: kyverno spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Create Namespace `crane` with label `foo=bar`]-->n2[Generate netpol `default-deny` in `crane`]-->n3[Change label on `crane` to `foo=new`]-->n4[`default-deny` netpol in `crane` is deleted] ``` ### Trigger changed, new match When a triggering resource is changed in such a way that the changes result in a positive match based on an existing generate rule, a new resource will be generated. Note that this should not be new behavior and is not influenced by the value of the `spec.generate.synchronize` field. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: rules: - name: generate-networkpolicy match: any: - resources: kinds: - Namespace selector: matchLabels: foo: bar generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true data: metadata: labels: created-by: kyverno spec: podSelector: {} policyTypes: - Ingress - Egress ``` ```mermaid flowchart TD n1[Create Namespace `jupiter` with no label]-->n2[Add label `foo=bar` to `jupiter` Namespace]-->n3[`default-deny` netpol in `jupiter` is created] ``` ## Miscellaneous Behaviors ### Clone, sync: Clone source is deleted while rule/policy exists This scenario explores what-ifs after the point in time at which the clone source of a generate rule is deleted. Keep in mind that with the new behavioral changes, deletion of the original source, with synchronization enabled, will cause immediate deletion of all generated/downstream resources. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resources spec: generateExisting: true rules: - name: generate-existing-networkpolicy match: any: - resources: kinds: - Namespace selector: matchLabels: foo: bar generate: kind: NetworkPolicy apiVersion: networking.k8s.io/v1 name: default-deny namespace: "{{request.object.metadata.name}}" synchronize: true clone: name: foo #### This resource is deleted namespace: platform ``` ```mermaid flowchart TD n1[Existing Namespaces `bin` and `baz`]-->n2[Namespace `bin` has label `foo=bar` while `baz` has label `foo=brown`]-->n3[Install policy]-->n4[Namespace `bin` receives netpol]-->n5[The `foo` netpol is deleted]-->n6[New trigger comes in]-->n7[A: ????] n5-->n8[User re-creates the `foo` source with different contents]-->n9[New trigger comes in]-->n10[B: ????] ``` #### Step A 1. The only option here is failure, but would this result in an Update Request? If not, what's the desired behavior? #### Step B 1. Fail 2. Generate from new version of `foo` (how will label assignment work since new `foo` exists after the rule/policy is installed?) ## Outstanding Questions ### General 1. Which fields do or do not support variables in a rule? ### Clone 1. Should we warn upon creation of policy/rule if the named source cannot be found? 2. Do we (or, if not, should we) use the destination metadata as an input filter to locate the source? 1. Should we check that the source and destination type match? ## Total Change Proposals 1. Change the field `generateExistingOnPolicyUpdate` to `generateExisting`. 2. Update changes 1. Immutable fields 2. Behavioral changes 3. Removal of need for generate existing on policy update for classic behavior ### Change of field The field `spec.generateExistingOnPolicyUpdate` will be changed to `spec.generateExisting` to more accurately reflect its true purpose. No behavioral changes occur. ### Update Changes #### Immutable Fields When a `generate` rule is created, all subsequent updates to that rule in all fields will be denied unless one of the following: 1. `spec.generateExisting` 2. `spec.generate.synchronize` 3. `spec.generate.data{}` (in case of a data type generate rule) 4. `spec.rules[].name` (if the rule name is changed, this is seen as a deletion of the old rule and creation of a new rule...we might change this) #### Behavioral Changes | CREATE | | | | |---------------------------------|--------------------------|--------------------------|--------------------------| | Creates in `data` type | Trigger | Policy | Rule | | generateExisting: true | Generate resource | Generate resource | Generate resource | | synchronize: true | Generate resource | None | None | | CREATE | | | | | |---------------------------------|--------------------------|--------------------------|--------------------------|-----------------------------------------------------| | Creates in `clone` type | Trigger | Policy | Rule | Source | | generateExisting: true | Generate resource | Generate resource | Generate resource | Can be created before or after rule creation | | synchronize: true | Generate resource | None | None | Can be created before or after rule creation | | UPDATE | | | | | | | |------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|--------------------------------------------|----------------------------------------------------|------------------------|---------------------|------------------------------| | Changes in `data` type (excludes setting changes) | Trigger | Policy | Rule-Match | Rule-Dest. Meta | Rule-Dest | Dest | | generateExisting: true | None | Adding a rule causes gen existing | Block (NEW)* | Block (NEW) | None | None | | synchronize: true | Dest deleted if now non-matching; retained if still matching; created if newly matching (NEW) | None | Block (NEW)* | Block (NEW) | Sync to dest | Sync from rule-dest | | UPDATE | | | | | | | | |-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------|------------------------|-------------------------|------------------------------|-------------------------| | Changes in `clone` type (excludes setting changes) | Trigger | Policy | Rule-Match | Rule-Dest. Meta | Rule-Source Meta | Source | Dest | | generateExisting: true | None | Adding a rule causes gen existing | Block (NEW)* | Block (NEW) | Block (NEW) | None | None | | synchronize: true | Dest deleted if now non-matching; retained if still matching; created if newly matching (NEW) | None | Block (NEW)* | Block (NEW) | Block (NEW) | Sync source to dest | Sync from source | | DELETE | | | | |---------------------------------|----------------------------|---------------------|---------------------| | Deletes in `data` type | Trigger | Policy | Rule | | generateExisting: true | None | None | None | | synchronize: true | Dest. deleted (NEW) | Delete dest. | Delete dest. | | DELETE | | | | | |---------------------------------|----------------------------|---------------|-------------|-----------------------------| | Deletes in `clone` type | Trigger | Policy | Rule | Source | | generateExisting: true | None | None | None | None | | synchronize: true | Dest. deleted (NEW) | None | None | Delete dest. (NEW)** | ### Restoration of behavior To do this, which is required post-1.7.0 ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: zk-kafka-address spec: generateExistingOnPolicyUpdate: true rules: - name: k-kafka-address match: any: - resources: kinds: - Namespace exclude: any: - resources: namespaces: - kube-system - default - kube-public - kyverno generate: synchronize: true apiVersion: v1 kind: ConfigMap name: zk-kafka-address # generate the resource in the new namespace namespace: "{{request.object.metadata.name}}" data: kind: ConfigMap metadata: labels: somekey: somevalue data: ZK_ADDRESS: "192.168.10.10:2181,192.168.10.11:2181,192.168.10.12:2181" KAFKA_ADDRESS: "192.168.10.13:9092,192.168.10.14:9092,192.168.10.15:9092" ``` will revert to the pre-1.7.0 behavior and not require presence of the `spec.generateExistingOnPolicyUpdate` field or its new name.