# Pre-enhancement: Using instance storage

This is a "pre-enhancement": Once we get some consensus on the broad outline we can convert this into a proper https://github.com/openshift/enhancements

🆕 Initial prototype code here: https://github.com/cgwalters/coreos-cloud-instance-store-provisioner

## Background: Instance local storage

Azure/GCP/AWS all support "instance local" disks:

- https://docs.microsoft.com/en-us/azure/virtual-machines/ephemeral-os-disks
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-instance-store-volumes.html
- https://cloud.google.com/compute/docs/disks/local-ssd

Azure even does this by default; in our default cluster install
instance type we have a 64GB disk that's NTFS formatted (of course)
that we just ignore today (they say it's free).

## Speed versus durability

Today default OpenShift 4 (IPI) installs use "maximum durability" storage for the root partition of the control plane and workers. Our default etcd writes to the root partition. Most cloud providers don't document exactly how they implement durable storage, but it clearly involves things like writing data to multiple distributed disk drives.

### Workers

On workers it's pretty silly to e.g. write downloaded container images to a durable store because we can just re-download them from the registry. Same for writes to the container overlayfs.

### "Less extremely durable" control plane

And a common problem in OpenShift is I/O performance for etcd - what we're doing by default is really having each write be durable to an extreme level - we multiply by 3 durable writes.

More on this below.

## Strawman proposal

In both machineAPI and the `install-config.yaml` we support provider-specific configuration that mounts `/var` as an instance-local volume. And this is very provider specific.

To start for example, for [AWS m5d instance types](https://aws.amazon.com/ec2/instance-types/m5/), adding a simple stanza into the install config like:

apiVersion: v1
baseDomain: example.com
controlPlane:
 ...
compute:
- name: worker
  platform:
    aws:
      instanceStorage: "os"
      type: m5d.xlarge
      zones:
      - us-west-2c
  replicas: 5

This would automatically format all instance-local drives and aggregate them into e.g. an LVM raid0 (striped), and set them up for some directories like `/var/lib/containers`.

The install config would just be sugar for an annotation on a `machineset` object to support day 2 changes (more on this below).

### Supporting instance store loss

It will be easier to implement if we don't try to support making the root volume instance-local. For exmaple, some cloud providers do automatic migration and that will lose the instance store.

Unfortunately we can't easily make all of `/var` instance local on the clouds that may lose data across migration. If we wipe out `/var` today that will also remove the kubelet pull secret which would break our ability to rejoin the cluster.

##### Proposed supported instance-local content dirs

- `/var/lib/containers`: The local container image and overlayfs - this is an obvious one.
- `/var/log`: In particular hyperkube *spams* the journal today (though there's inbound log level fixes)
- `/var/lib/etcd`: The etcd data store

#### etcd options

By simply *supporting* instance stores (even if gated under some sort of hidden `env OPENSHIFT_INSTALL_FEATURES=instance-store openshift-install`), we can experiment with more sophisticated approaches to etcd durability. Members of the etcd team or performance teams can gather numbers. We can simulate failures and determine the difficulty of recovery.

##### "reprovisionable" clusters

First, I suspect a number of users that are fully invested in a "GitOps" model where all critical API objects are committed to Git would likely be fine e.g. gaining 5-10x API server performance for e.g. 0.001% risk of concurrent failure of every control plane node per year (made up numbers). Particularly by splitting etcd across availability zones we're already quite reliable.

##### "one fully durable"

One approach would be to have have just one control plane node writing data to a fully durable store. In case of loss of that node, the etcd operator would detach that volume from the lost node and reattach it to a running control plane node, which would then switch to it and take over writing to it. Integration with machineAPI would cause a new control plane node to be spawned.

##### "automatic restore from backup"

Run with all nodes on instance storage, but ensure that etcd backup to e.g. S3 is integrated and that we can auto-recover from backup in case of concurrent loss of all control plane nodes.

#### cloud should match bare metal

We support bare metal clusters where the master nodes are using e.g. a single disk for etcd storage. Or possibly RAID1, but the point is that the disks on these nodes have much lower durability than most durable cloud storage.

## Implementation

Some discussion in https://github.com/openshift/enhancements/pull/368#issuecomment-733188597
and https://github.com/coreos/ignition/issues/1126

We should support changing this "day 2" as well. First we need a basic framework where
the MCO manages the pointer configuration. That was already implemented, then reverted;
but we are on track to bring it back.

Once that's done, we can configure things so that the MCO watches the machinesets, and
for any type that has an annotation `instancestorage.openshift.io = "os"`, the MCO would
update the provided pointer config to inject Ignition to configure the instance-local storage.