# Spike [CPVYGR-484](https://itrack.web.att.com/browse/CPVYGR-484): Validate Node Label Synch between Resources BMH and Node
This document discuss CAPM3 **BareMetalHost** (BMH) resources driving the synchronization of labels with CAPI Node resources.This new capability was merged into CAPM3 in April, 8 2021 (v0.4.2).

Note that this document describes the test executed on clusters deployed using airshipctl (Ephemeral and Target clusters) based on CAPI v0.4.2 and CAPM3 v0.5.0 uplift patchsets.

## BMH/Node Label Synch Design & Implementation
As per the [design](https://github.com/metal3-io/metal3-docs/blob/master/design/sync-labels-bmh-to-node.md#metal3cluster) and [implementation](https://github.com/metal3-io/cluster-api-provider-metal3/pull/152) documents the labels added/deleted to/from **BMH** resources will be synchronized with their corresponding **Node** resources only if these labels are within the set of pre-defined prefixes, e.g., *bmh-test-prefix.metal3.io, foo.bar*.

Note that the action of adding and/or deleting labels within the pre-defined prefixes directly from the Node resources will result in reverting back these actions, i.e., deleting added labels and/or adding back deleted labels.

## Setting the List of Pre-Defined Prefixes
The list of pre-defined prefixes are set on the **Metal3Cluster** resources as annotations using the keyword "*metal3.io/metal3-label-sync-prefixes*".

kind: Metal3Cluster
metadata:
  name: target-cluster
  namespace: target-infra
  annotations:
    metal3.io/metal3-label-sync-prefixes: "bmh-test-prefix.metal3.io, foo.bar"

For the test, kubectl command was used to set the list of pre-defined prefixes as shown below:

$ kubectl --context ephemeral-cluster annotate metal3cluster target-cluster metal3.io/metal3-label-sync-prefixes=bmh-test-prefix.metal3.io,foo.bar -n=target-infra --overwrite

metal3cluster.infrastructure.cluster.x-k8s.io/target-cluster annotated

$ kubectl --context ephemeral-cluster get metal3cluster -n target-infra -o yaml

apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1alpha5
  kind: Metal3Cluster
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha5","kind":"Metal3Cluster","metadata":{"annotations":{},"name":"target-cluster","namespace":"target-infra"},"spec":{"controlPlaneEndpoint":{"host":"10.23.25.102","port":6443},"noCloudProvider":true}}
      metal3.io/metal3-label-sync-prefixes: bmh-test-prefix.metal3.io,foo.bar
    creationTimestamp: "2021-10-05T21:37:40Z"
    finalizers:
    - metal3cluster.infrastructure.cluster.x-k8s.io
...
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

## Testing the Label Synchronization
The label synchronization test was conducted with BMH resources hosted on the Ephemeral cluster and Node resources hosted on the Target cluster.

We are listing the labels set on BMH and Node resources before initiating the test so we have a baseline.

$ kubectl --context ephemeral-cluster get baremetalhosts --show-labels -n=target-infra

NAME     STATE         CONSUMER                     ONLINE   ERROR   LABELS
node01   provisioned   cluster-controlplane-k9smc   true             airshipit.org/example-label=label-bmh-like-this,airshipit.org/k8s-role=controlplane-host,cluster.x-k8s.io/cluster-name=target-cluster


$ kubectl --context target-cluster get nodes --show-labels

NAME     STATUS   ROLES    AGE   VERSION    LABELS
node01   Ready    master   61m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

Now we can add (pre-defined prefixed) labels to the BMH and verify the synchronization.

$ kubectl --context ephemeral-cluster label baremetalhosts node01 bmh-test-prefix.metal3.io/rack=xyz-123 -n=target-infra

$ kubectl --context ephemeral-cluster label baremetalhosts node01 foo.bar/xyz=abc -n=target-infra

In this test we added two prefixed labels: bmh-test-prefix.metal3.io/rack=xyz-123, foo.bar/xyz=abc and the commands below show the synchronization taking place.

$ kubectl --context ephemeral-cluster get baremetalhosts --show-labels -n=target-infra

NAME     STATE         CONSUMER                     ONLINE   ERROR   LABELS
node01   provisioned   cluster-controlplane-k9smc   true             airshipit.org/example-label=label-bmh-like-this,airshipit.org/k8s-role=controlplane-host,bmh-test-prefix.metal3.io/rack=xyz-123,cluster.x-k8s.io/cluster-name=target-cluster,foo.bar/xyz=abc


$ kubectl --context target-cluster get nodes --show-labels

NAME     STATUS   ROLES    AGE   VERSION    LABELS
node01   Ready    master   67m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

## Adding & Deleting Prefixed Labels directly to/from the Node
The following tests were executed to verify that the action of adding/deleting prefixed labels directly to/from the Node resource are reverted back.

First test was to add a prefixed label to the Node and verify the result.

$ kubectl --context target-cluster label node node01 bmh-test-prefix.metal3.io/color=blue
node/node01 labeled

Which resulted in the label not showing as added to the Node. In reality, the label was added to the Node but then deleted.

$ kubectl --context target-cluster get nodes --show-labels

NAME     STATUS   ROLES    AGE   VERSION    LABELS
node01   Ready    master   76m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

The second test was to delete a prefixed label from the Node and verify that the action was reverted back. In this example, the prefixed label "*bmh-test-prefix.metal3.io/rack*" is deleted.

$ kubectl --context target-cluster label node node01 bmh-test-prefix.metal3.io/rack-

node/node01 labeled

By inspecting immediatly after the list of labels of the Node, you will see that the label was in fact deleted.

$ kubectl --context target-cluster get nodes --show-labels

NAME     STATUS   ROLES    AGE   VERSION    LABELS
node01   Ready    master   79m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

But a few seconds later the prefixed label was added back.

$ kubectl --context target-cluster get nodes --show-labels
NAME     STATUS   ROLES    AGE   VERSION    LABELS
node01   Ready    master   80m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

## Testing After "*clusterctl move*" on Worker Node (Node03)
Just to ensure that same behavior is achieved in the Airship 2 context, similar test were executed after the "*airshipctl phase run clusterctl-move*" and worker node has been created with "*airshipctl phase run workers-target*".

Before executing the test the list of labels for BMH and Node resources are listed below.

$ kubectl --context target-cluster get baremetalhosts --show-labels -n=target-infra

NAME     STATE         CONSUMER                     ONLINE   ERROR                LABELS
node01                 cluster-controlplane-k9smc   true                          airshipit.org/example-label=label-bmh-like-this,airshipit.org/k8s-role=controlplane-host,bmh-test-prefix.metal3.io/rack=xyz-123,cluster.x-k8s.io/cluster-name=target-cluster,foo.bar/xyz=abc
node02   registering                                false    registration error   airshipit.org/k8s-role=controlplane-host,airshipit.org/stage=initinfra
node03   provisioned   worker-1-2r6mp               true                          airshipit.org/k8s-role=worker,cluster.x-k8s.io/cluster-name=target-cluster

$ kubectl --context target-cluster get nodes --show-labels
NAME     STATUS   ROLES    AGE     VERSION    LABELS
node01   Ready    master   132m    v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane
node03   Ready    <none>   8m29s   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node03,kubernetes.io/os=linux,metal3.io/uuid=384d6449-6cbe-40bc-affa-edf12733510e,node-type=worker

For this test, the prefixed label "*bmh-test-prefix.metal3.io/worker-rack=abc-456*" has been added to the correspoding worker BMH resource.

$ kubectl --context target-cluster label baremetalhosts node03 bmh-test-prefix.metal3.io/worker-rack=abc-456 -n=target-infra
baremetalhost.metal3.io/node03 labeled

As you can see below, the prefixed label was added to *"node03"*.

$ kubectl --context target-cluster get nodes --show-labels
NAME     STATUS   ROLES    AGE    VERSION    LABELS
node01   Ready    master   135m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane
node03   Ready    <none>   11m    v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/worker-rack=abc-456,kubernetes.io/arch=amd64,kubernetes.io/hostname=node03,kubernetes.io/os=linux,metal3.io/uuid=384d6449-6cbe-40bc-affa-edf12733510e,node-type=worker

Last but not least, delete the prefixed label "*bmh-test-prefix.metal3.io/worker-rack*" on the BMH and verify that it has also been deleted from the Node.

$ kubectl --context target-cluster label bmh -n target-infra node03 bmh-test-prefix.metal3.io/worker-rack-
baremetalhost.metal3.io/node03 labeled

$ kubectl --context target-cluster get nodes --show-labels
NAME     STATUS   ROLES    AGE     VERSION    LABELS
node01   Ready    master   5h10m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane
node03   Ready    <none>   3h6m    v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node03,kubernetes.io/os=linux,metal3.io/uuid=384d6449-6cbe-40bc-affa-edf12733510e,node-type=worker

## Deleting Pre-Defined Prefixes from Metal3Cluster
Removing pre-defined prefixes from the Metal3Cluster has two consequences:
1. Adding prefixed label post the removal from the Metal3Cluster are not synchronized
2. Any existing prefixed labels on BMH & Node will not be removed. It is your responsibility to delete them.

To demonstrate this assertion, the pre-determined prefix is deleted from the Metal3Cluster.

$ kubectl --context target-cluster annotate metal3cluster target-cluster metal3.io/metal3-label-sync-prefixes- -n=target-infra
metal3cluster.infrastructure.cluster.x-k8s.io/target-cluster annotated

$ kubectl --context target-cluster get metal3cluster -n target-infra -o yaml
apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1alpha5
  kind: Metal3Cluster
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha5","kind":"Metal3Cluster","metadata":{"annotations":{},"labels":{"airshipit.org/stage":"initinfra"},"name":"target-cluster","namespace":"target-infra"},"spec":{"controlPlaneEndpoint":{"host":"10.23.25.102","port":6443},"noCloudProvider":true}}
...

And, as shown below, the prefixed labels, "*bmh-test-prefix.metal3.io/rack=xyz-123*" and "*foo.bar/xyz=abc*", were not deleted from BMH and Node resources.

$ kubectl --context target-cluster get nodes --show-labels
NAME     STATUS   ROLES    AGE     VERSION    LABELS
node01   Ready    master   5h56m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane
node03   Ready    <none>   3h52m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node03,kubernetes.io/os=linux,metal3.io/uuid=384d6449-6cbe-40bc-affa-edf12733510e,node-type=worker

$ kubectl --context target-cluster get bmh -n target-infra --show-labels
NAME     STATE         CONSUMER                     ONLINE   ERROR                LABELS
node01                 cluster-controlplane-k9smc   true                          airshipit.org/example-label=label-bmh-like-this,airshipit.org/k8s-role=controlplane-host,bmh-test-prefix.metal3.io/rack=xyz-123,cluster.x-k8s.io/cluster-name=target-cluster,foo.bar/xyz=abc
node02   registering                                false    registration error   airshipit.org/k8s-role=controlplane-host,airshipit.org/stage=initinfra,bmh-test-prefix.metal3.io/worker-rack=abc-456
node03   provisioned   worker-1-2r6mp               true                          airshipit.org/k8s-role=worker,cluster.x-k8s.io/cluster-name=target-cluster

After the pre-determined prefix(es) is deleted from Metal3Cluster, you will have to explicitly delete the prefixed labels from BMH and Node, if not done yet.

As you can see below, the prefix label synchronization has been "disabled" so the label deletion on BMH did not propagated to the Node.

$ kubectl --context target-cluster label bmh -n target-infra node01 bmh-test-prefix.metal3.io/rack-
baremetalhost.metal3.io/node01 labeled

$ kubectl --context target-cluster get bmh node01 -n target-infra --show-labels
NAME     STATE   CONSUMER                     ONLINE   ERROR   LABELS
node01           cluster-controlplane-k9smc   true             airshipit.org/example-label=label-bmh-like-this,airshipit.org/k8s-role=controlplane-host,cluster.x-k8s.io/cluster-name=target-cluster,foo.bar/xyz=abc

$ kubectl --context target-cluster get node node01 --show-labels
NAME     STATUS   ROLES    AGE    VERSION    LABELS
node01   Ready    master   6h2m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bmh-test-prefix.metal3.io/rack=xyz-123,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

Therefore, you will need to manually delete the prefixed label from the Node, if needed.

$ kubectl --context target-cluster label node node01 bmh-test-prefix.metal3.io/rack-
node/node01 labeled

$ kubectl --context target-cluster get node node01 --show-labels
NAME     STATUS   ROLES    AGE    VERSION    LABELS
node01   Ready    master   6h4m   v1.19.14   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,foo.bar/xyz=abc,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux,metal3.io/uuid=8c75c121-de7b-4794-940a-7ada43fadc8d,node-role.kubernetes.io/master=,node-type=controlplane

# Conclusion
The BMH label synchronization with Node works as designed and implemented by CAPM3 on CAPI v1alpha4/CAPM3 v0.5.0 uplift (Airship 2) environment.