# Cluster API: label propagation: debugging Telekom issue ## What we know * T0: After Machine create + subsequent reconcile which migrates managed fields * MachineSet * an annotation in .spec.template.metadata.annotations (this is continuously reconciled) * VSphereMachineTemplate * annotations in .spec.template.metadata.annotations (this is only added on create) * VSphereMachine Object * metadata annotations * annotation from MachineSet * annotations from VSphereMachineTemplate * managed fields * capi-machineset owns annotation from MachineSet * T1: User removes the annotation from MachineSet * MachineSet * no annotations in .spec.template.metadata.annotations (this is continuously reconciled) * VSphereMachineTemplate * annotations in .spec.template.metadata.annotations (this is only added on create) * VSphereMachine Object * metadata annotations * **no annotations** (the annotations from VSphereMachineTemplate got unexpectedly lost!!!) * managed fields * capi-machineset has an empty entry for metadata.annotations ## Additional data * Issue: https://github.com/kubernetes-sigs/cluster-api/issues/8522 * We saw this with Kubernetes 1.23.16 and 1.25.x with CAPI 1.4.1 and CAPV 1.6.0 * We couldn't reproduce it locally with Tilt with Kubernetes 1.23.17 with CAPI and CAPD on main and release-1.4 * tested both with legacy and classy Cluster * also tried it with a non-functional CAPV cluster in Tilt, also couldn't reproduce * We couldn't reproduce it with TKG Glasgow + CAPI v1.4.1 + CAPV v1.6.0 * We couldn't reproduce it locally with CAPI + CAPD (release-1.4) with HA control-plane * Setup: tilt up => create a HA control plane workload cluster, run tilt up against the workload cluster again * Debugging on the Telekom environment: * The issue occurs with legacy Cluster (without ClusterClass) * The patch we are sending at T1 looks correct (metadata.annotations is set to an empty map) * Seems like in the Telekom environment in some scenario a previously ran kubectl patch on a VSphereMachine annotation lead to preserved annotations * While annotations get deleted, label are not => we think the problem is pretty much that we send that empty annotations map * SSA call looks like this (that's one from Tilt) (resourceVersion is not set) * PATCH https://10.128.0.1:443/apis/infrastructure.cluster.x-k8s.io/v1beta1/namespaces/default/dockermachines/capi-quickstart-2-md * {"apiVersion":"infrastructure.cluster.x-k8s.io/v1beta1","kind":"DockerMachine","metadata":{"annotations":{},"labels":{"cluster.x-k8s.io/cluster-name":"capi-quickstart-2","cluster.x-k8s.io/deployment-name":"capi-quickstart-2-md-0","cluster.x-k8s.io/set-name":"capi-quickstart-2-md-0-7d4dc4b877x6c8cx","machine-template-hash":"3808706433-27j7c"},"name":"capi-quickstart-2-md-0-gvpq2","namespace":"default","uid":"e12b60d0-50d4-4676-816f-7c68d0f43fb3"}} ## Current working assumptions * It doesn't seem to be related to mgmt cluster Kubernetes version ## TODO * Telekom (Jakob) * More information about setup: * There's an off chance that this could be an issue with HA control plane / load balancers / caching * Differences in APIserver "setup": HA vs not, kind vs real environment, feature gate, flags, ... * Some external component: webhook, controller, IPAM, kyverno (?) * full list of versions (Kubernetes, CAPI, CAPV, IPAM, ...) * Experiment: * Does the same happen with another infra provider (CAPM3) * doesn't seem to be an issue with metal3machines, but we don't have labels from the template there. I'll try to test that as well. * Stage 1: T0 + pause Cluster + kubectl patch on VSphereMachine.annotations * Stage 2: Test the same with: * Cluster API-independent object, e.g. ServiceAccount, some cert-manager CRD (builtin & CRD) * Cluster API object but not reconciled by Cluster API * Data: * audit log * logs for requests to admission webhooks * Talk to Flux maintainers * Hidde told us they've seen something similar with 1.27 (which caused the very quick release of 1.27.1) and some 1.26.x version * Hidde recommended to try reaching out to Jordan Liggit once we are fairly certain that it is an apiserver issue as he's into SSA and helped them fix some issues they found very quickly * Stefan * Look into apiserver code * branch in the patch endpoint of apiserver that handles SSA: [here](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/patch.go#L636) * Creation of the merge manager in the api server: [here](https://github.com/kubernetes/kubernetes/blob/c1bbf791dd22e94aaa879f473ec9f10a24da0ec9/staging/src/k8s.io/apimachinery/pkg/util/managedfields/internal/structuredmerge.go#L43) * Entrypoint in the apply logic of structured-merge-diff, which actually performs the merge and handles the managed fields: [here](https://github.com/kubernetes-sigs/structured-merge-diff/blob/master/merge/update.go#L162) * the apply function first performs a [merge](https://github.com/kubernetes-sigs/structured-merge-diff/blob/master/merge/update.go#L174) (only adding fields), and then a [prune](https://github.com/kubernetes-sigs/structured-merge-diff/blob/master/merge/update.go#L199) (removing deleted fields). * Test with CAPA: run CAPA quickstart * Backlog * Reproduce issue with envtest * Reproduce first on our side, then could make sense to debug into apiserver
Last changed by  
 
Stefan Büringer
73

Read more

Cluster API: v1.4 Test failures in e2e tests

High priority - Low priority Rate limit for github api has been reached Let's ignore for now, known root cause and its not our code Fails because of rate limiting when downloading cert-manager.yaml. Found in: (multiple times)e2e-main: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-main/1625685968952496128When testing clusterctl upgrades (v0.4=>current) Also in other test cases

3/22/2023
Cluster API: Validating label and annotation propagation

Labels and annotations that are set by users The following labels and annotations could be set by users via ClusterClass or Cluster topology, and then they will propagate through the entire hierarchy. I think we can only implement something for the newly introduced hold-upgrade-sequence and defer-upgrade annotations in v1beta1. Everything else would be a breaking change as we consider labels and annotations part of our API. labels:cluster.x-k8s.io/watch-filter: is only supposed to be set top-level=> Proposal: no filtering annotations: topology.cluster.x-k8s.io/hold-upgrade-sequence & topology.cluster.x-k8s.io/defer-upgrade=> Proposal: do not propagate at all, seems bad to leak them over the entire hierarchy if we consider them just "experimental fields". Also requires a lot of API calls to propagate them across the entire hierarchy when they are set.

3/10/2023
ClusterAPI - ClusterClass struct names (hack session)

spec: variables: # []ClusterClassVariable - name: clusterIdentityName required: true schema: # VariableSchema openAPIV3Schema: # JSONSchemaProps type: string default: cluster-identity patches: # []ClusterClassPatch - name: clusterName

1/11/2022
ClusterAPI - Running e2e tests & controllers locally with Intellij

This document contains some additional notes for the corresponding Zoom session (link Passcode: N20+nkQ%). Overall we will: Run the CAPD-based quickstart e2e test via Intellij Run the capi-controller via Intellij and proxy webhook requests from the kind cluster to the locally running controller Note: There is already documentation about how to run e2e tests via an IDE in the CAPI book. The current doc is a concrete walkthrough and additionally contains instructions on how to run a controller locally.

10/22/2021
Read more from Stefan Büringer
Published on HackMD