# PAO as part of core OCP

**POSTED AS https://github.com/openshift/enhancements/pull/867/**

This is a preliminary plan describing how to make the [Performance Addon Operator](https://github.com/openshift-kni/performance-addon-operators) part of OpenShift core.

## Introduction

The [Performance Addon Operator](https://github.com/openshift-kni/performance-addon-operators) is a component that makes it easier to configure an OCP cluster for low latency and real-time purposes [OCP docs](https://docs.openshift.com/container-platform/4.8/scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.html).

It is a high level orchestrator that consumes a [Performance Profile](https://github.com/openshift-kni/performance-addon-operators/blob/master/docs/interactions/performance-profile.yaml) and generates multiple manifests that are then processed by core OpenShift components like MCO and NTO.

We have a simplified interaction diagram here: https://github.com/openshift-kni/performance-addon-operators/blob/master/docs/interactions/diagram.png

A more in depth description of how a low latency tuned cluster works was presented at DevConf 2021: https://devconfcz2021.sched.com/event/gmJD/openshift-for-low-latency-and-real-time-workloads

## Current status

PAO is being developed under the OpenShift KNI organization (https://github.com/openshift-kni/performance-addon-operators), built by custom builder upstream and cPaaS downstream. PAO is installed via OLM as a second-day operator.

The current deliverables are:

- [performance-addon-operator](https://catalog.redhat.com/software/containers/openshift4/performance-addon-rhel8-operator/5e1f480fbed8bd66f81cbe23?container-tabs=overview&gti-tabs=registry-tokens)

The operator contains the high level orchestration logic + a [tool to simplify generation of Performance Profiles](https://docs.openshift.com/container-platform/4.8/scalability_and_performance/cnf-create-performance-profiles.html). This tool can be executed locally on admin's laptop using `podman run --entrypoint performance-profile-creator performance-addon-operator:ver`

- [performance-addon-must-gather](https://catalog.redhat.com/software/containers/openshift4/performance-addon-operator-must-gather-rhel8/5ee0bfa4d70cc56cea83827f?container-tabs=overview&gti-tabs=registry-tokens)

The must gather image [enhances](https://github.com/openshift-kni/performance-addon-operators/tree/master/must-gather) the main OCP one and [collects](https://docs.openshift.com/container-platform/4.8/scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.html#cnf-about-gathering-data_cnf-master) some latency tuning configuration and data that are needed for debugging.

- and the functional tests are also vendored into [cnf-tests](https://catalog.redhat.com/software/containers/openshift4/cnf-tests-rhel8/5e9d81f65a134668769d560b?container-tabs=overview&gti-tabs=registry-tokens) that can be used to [validate customer deployments](https://docs.openshift.com/container-platform/4.8/scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.html#cnf-performing-end-to-end-tests-for-platform-verification_cnf-master) (this includes other aspects like SCTP, PTP, SR-IOV as well).

## Current release mechanism

The Telco team maintains its own build and release pipelines and releases via a [Telco specific errata](https://red.ht/3boqjeS) that however targets the OpenShift product. So from a customer perspective PAO is part of OCP.

Continuous testing is done via [OpenShift CI jobs](https://github.com/openshift/release/tree/daaab71832999c072c60667955637fa3a535d4ba/ci-operator/config/openshift-kni/performance-addon-operators) as well as some downstream specific Jenkins instances (both virtual and bare metal).

## The issues

- More and more customers are using PAO to configure their clusters for low latency or are being told to do so. The Telco team has so far managed to release PAO on the same day as OCP to maintain the illusion it is part of the product, but it costs a LOT of time to maintain the downstream build process.

- The OCP [gcp-rt](https://prow.ci.openshift.org/?type=periodic&job=*gcp-rt*) lane tests OCP deployment on top of GCP with the real-time kernel enabled, but it only works [thanks to some pieces of PAO tuning](https://github.com/openshift/release/pull/18870). However the tuning is a pre-rendered snippet, not linked to the operator, because the deployment does not know about PAO.

- PAO is closely tied to some OCP components ([Cluster Node Tuning Operator](https://github.com/openshift/cluster-node-tuning-operator)) and depends on the OCP version. So we release PAO for every OCP minor version (since 4.4) and together with some of the OCP Z-streams. The upgrade procedure is tricky though, first OCP needs to be upgraded and then PAO via an OLM channel change. This is needed to make sure the necessary functionality is present in the running NTO.

- Even minor updates might be surprising for customers as new versions of PAO might fix something in the tuning logic and that usually ends up with multiple reboots of a cluster. OLM based upgrades are not synchronized with OCP upgrades and so might not happen in the same "downtime window". This might become expecially tricky if the user enables automatic operator upgrades.

- The new [Workload partitioning](https://github.com/openshift/enhancements/blob/master/enhancements/workload-partitioning/management-workload-partitioning.md) feature seeks to use PAO as the configuration tool. However Workload partitioning needs to be enabled at install phase and PAO CRDs are not yet available at that time. [A "hacky" procedure](https://github.com/openshift/enhancements/pull/753) is needed to overcome this.

## Proposal

We (the telco team) would like to make PAO part of OpenShift. There are multiple paths we can take to achieve this, I am going to describe the one we prefer: **PAO becoming a sub-controller in MCO or NTO.**

The initial plan is to keep all logic and custom resources intact, we want to start with simply moving the code over.

To be really clear here, the Telco team will handle all the necessary changes to both codebases, however we will need support in the form of PR reviews and design consultations.

### Code

PAO code would be moved into the MCO repository as a separate controller. The Telco team would keep ownership as it has the necessary domain knowledge needed to implement the low latency tuning.

The code currently uses Operator SDK libraries and this might need to be changed.

PAO also implements a migration webhook to migrate its custom resources during upgrade. The hook is currently installed via OLM mechanisms and this might need to be reimplemented or changed to use whatever CVO or ART support.

Logic changes or tighter integration with other MCO controllers is not expected to be part of this initial move. Just the delivery mechanism wrt OCP.

We should take into the account that we have a lot of infrastructure around our APIs:

1. Two version of APIs, we can drop the v1alpha1
2. Validating webhook
3. Conversion webhook

Now we get all the configuration for free from the OLM, we will need to provide a way to configure all these things without OLM usage.

An additional question, where should live the performance-profile-creator, should it be part of the OpenShift repository or should we leave it under the openshift-kni one.

[cnf-tests](https://github.com/openshift-kni/cnf-features-deploy/) are NOT part of this move either, the Telco team will keep maintaining the validation suite in OpenShift KNI.

### CI

PAO contains both unit and functional tests implemented using Gingko.

Integration with MCO is currently TBD, please advise if anything special is needed.

One of the test suites (4_latency) implements an opinionated latency testing suite to get comparable results from customers and to prevent configuration / command line invocation mistakes. Keeping it in PAO/MCO or moving it into cnf-tests is TBD.

### Build pipeline

PAO would inherit the MCO pipeline and all the images will be released via ART.

### Releases

PAO should be shipped as part of core OCP. ART again. PAO already obeys the OCP github PR rules and uses the same PR bot so nothing changes in the lifecycle.

### API stability

The `PerformanceProfile` CR uses an `performance.openshift.io/v2` API group which marks it as stable interface that needs to be maintained for about a year after official deprecation. We do not expect any change to the API as part of this proposal.

### Bugs

PAO bugs are currently tracked in The OpenShift Bugzilla component [Performance Addon Operator](https://bugzilla.redhat.com/buglist.cgi?component=Performance%20Addon%20Operator&list_id=12053428&product=OpenShift%20Container%20Platform)

This should be kept as-is, the Telco team will own all the bugs.

The bugzilla component is currently not part of OCP metrics and we can include it. TBD

## Open questions

### Upgrade

- PAO has been installed via OLM since 4.4. Once it moves to core we will have to figure out the upgrade procedure, because OLM will not remove the optional operator by itself.

One of the options is to keep using the same leadership election method and make sure the newest version always wins. The OLM operator will then stay on the cluster (can be release noted) as an inactive component.

## Alternatives

### PAO as a sub-controller of Cluster Node Tuning Operator

PAO could also live inside [NTO](https://github.com/openshift/cluster-node-tuning-operator) if MCO cannot accept it. The long term future might be to merge all of them together anyway. All other aspects of this design are the same for this approach.

NTO and PAO have working relationship and both share a common goal. So in one point of view this might even be a better idea than a merge with MCO.

### PAO as a full OCP operator

The alternative approach is to move PAO into OCP as a fully standalone operator. This would have no impact on the MCO team and would be simpler in terms of code.

However there are disadvantages to this approach:

- OCP is already heavy on infrastructure resource consumption and an additional operator would make that worse
- implementing a conditional start is possible though, similar to what machine-operator uses, but negates the simpler on coding advantage

## Future

PAO currently uses Ignite heavily and since MCO is [looking at dropping the support](https://hackmd.io/UzgrWuCLQ-u9bMh_JhRKOA#Proposal), it would make it easier to adapt to this change if PAO could directly contribute to the layered Machine Image instead of describing the necessary config changes as an MachineConfig first.