# SPLAT-391

As part of a [SPLAT-391](https://issues.redhat.com/browse/SPLAT-391) Spike, we're looking adding (into the installer, etc) vSphere anti-affinity rules for the control plane nodes to improve high availability of OpenShift. In other words, high availabilty would be improved if each OpenShift control plane node ran on a separate physical vSphere/ESXi host (if one physical ESXi host dies, you don't lost 2 OpenShift control plane nodes).

## TL;DR

I want to provide the conclusion first for readers that don't care as much about the details.

The conclusion of this spike is that **documentation** of the vSphere DRS anti-affinity rules for the OpenShift control plane would be best course of action at this time.

The addition of vSphere DRS anti-affinity rules into the OpenShift installer would not be trivial due to:

- Install-config schema additions
- Differences between VMC and normal vSphere
- Significant increase in privilege that would be required by the installer

The following command will create the necessary anti-affinity rule:

$ govc cluster.rule.create -name my-control-plane-group \
  -dc IBMCloud -cluster 
  -enable -anti-affinity my-master-0 my-master-1 my-master-2

This rule will need to be updated if a control plane node is recreated for any reason.

## Discourse
There's been questions about control plane anti-affinity DRS in the past:

Conversations:
- (Latest convo) https://coreos.slack.com/archives/CH06KMDRV/p1643034127071300
- https://coreos.slack.com/archives/CH06KMDRV/p1642419730003000
- https://coreos.slack.com/archives/CH06KMDRV/p1629100661117600
- https://coreos.slack.com/archives/CH06KMDRV/p1617206808168300

RFE's:
- https://issues.redhat.com/browse/RFE-1792
- Other potentially related RFE: https://issues.redhat.com/browse/RFE-1402 (it doesn't seem to be worded right).

## Installer

[Terraform resources](https://registry.terraform.io/providers/hashicorp/vsphere/latest/docs/resources/compute_cluster_vm_anti_affinity_rule
) exist for DRS rules which is great. The terraform *alone* seems like it would be straight forward *technically* with some **major caveats** outside of Terraform:

1. VMC and vSphere have [different configurations](https://docs.vmware.com/en/VMware-Cloud-on-AWS/solutions/VMware-Cloud-on-AWS.919a954a9b6ca17cdc719ec42cda1401/GUID-7A9A832FF741413D27FA2FE9E408FEF4.html) to accomplish anti-affinity.
2. CI would be a challenge given #1 above.
3. Would support only cover vSphere? Would a separate IPI solution need to be created for VMC?

Terraform is just piece of the problem. The `install-config.yaml` providers API would need to be updated to include an option for using anti-affinity or not. Would it need to cover both VMC and vSphere style rules? What would defaults be?

Given the findings and questions above, it would be challenging and time-consuming to update the installer for vSphere (and/or VMC) anti-affinity rules. To give a better idea of work, I'd estimate a 5 or 8 for [story points](https://docs.google.com/presentation/d/1IYzZZqm9abKDeVBwNmJo3wl90LkAf55Ed3G1X-swUg0/edit#slide=id.gd2a1541dc4_0_0).

Lastly, I could see this requiring an OpenShift enhancement to be submitted.

## Machine API

MAPI would have generally the same challenges as installer, CI and VMC vs vSphere configuration. Additionally, new fields would need to be added to Machine provider configurations.

These sorts of updates would need to be addressed in the same enhancement for installer mentioned above.

## New Permission Requirements

The following was mentioned [in Slack](https://coreos.slack.com/archives/CH06KMDRV/p1643034884075400?thread_ts=1643034127.071300&cid=CH06KMDRV). Also, here's the [VMWare doc reference](https://vdc-download.vmware.com/vmwb-repository/dcr-public/bf660c0a-f060-46e8-a94d-4b5e6ffc77ad/208bc706-e281-49b6-a0ce-b402ec19ef82/SDK/vsphere-ws/docs/ReferenceGuide/vim.cluster.ConfigSpecEx.html).

Host.Inventory.EditCluster

This privilege would significantly raise the amount of privilege that OpenShift has at the vSphere cluster level.

Current privileges can be [found here](https://docs.openshift.com/container-platform/4.9/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-installer-infra-requirements_installing-vsphere-installer-provisioned) and are quite limited at vSphere cluster level.

## Looking Forward to CAPI

Review shows that CAPI likely supports anti-affinity rules. A live test on how this works has not been completed.

https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/master/docs/proposal/20201103-failure-domain.md
https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/f7c3a7495d26e96fb50d83ff0d1994016e3b211e/pkg/services/govmomi/cluster/rule.go#L50-L65

## Other On-Prem Clouds

### OpenShift on RHV (oVirt)
[I spoke with the OpenShift-on-RHV team](https://coreos.slack.com/archives/CNSJG0ZED/p1644355661192359) and they mention that anti-affinity rules are configured at installation time but not after that (i.e. not when recreating a control plane node).

- [OpenShift-on-RHV Presentation](https://docs.google.com/presentation/d/19e_Zf2c74ofAwZ8jVeDR-RH7qi_C9IFnfmAvC2AIxvA/view#slide=id.gcf02f91d4f_0_0)
- There appears to be some work: https://github.com/openshift/cluster-api-provider-ovirt/pull/80??

### Azure Stack Hub
Patrick Dillon says nothing as far as he knows.

### OpenStack
OpenStack sets up control plane anti-affinity rules as part of installation. ([Source](https://docs.openshift.com/container-platform/4.9/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-setting-worker-affinity_installing-openstack-installer-custom)):