---
title: Operator Condition Improvement
authors:
- "@dinhxuanvu"
reviewers:
- "@ecordell"
approvers:
- "@ecordell"
creation-date: 2021-05-05
last-updated: 2021-05-05
status: provisional
---

# Operator Condition Improvement

## Release Signoff Checklist

- [ ] Enhancement is `implementable`
- [ ] Design details are appropriately documented from clear requirements
- [ ] Test plan is defined
- [ ] Graduation criteria for dev preview, tech preview, GA

## Summary

There is a potential race condition for the `Upgradeable` condition in OperatorCondition due to lacking of version control.

## Bugzilla

https://bugzilla.redhat.com/show_bug.cgi?id=1927340

## Motivation

### Problems with Current Operator Condition

Currently, operator is pushing changes to OperatorCondition's status to signal its `Upgradeable` condition and OLM is reading the status to determine if the operator is upgradeable. However, there is no mechanism for operator and OLM to acknowledge their actions. For example, operator is unable to know if OLM has read its Upgradeable condition or OLM is unable to know if this current Upgradeable condition is up-to-date or not. This leads to a potential race condition where the operator sets the deployment pod ready or OLM allows the upgrade to succeed prematurely.

## Proposal

### Technical Background

- Operator is currently expected to update `Upgradeable` condition to OperatorCondition's `status` to signal its condition during upgrade.
- By default, the `Upgradeable` condition is not set in the OperatorCondition's status which indicates the upgrade can happen freely.
- OLM (olm-operator) will allow the upgrade to succeed (moving CSV's status from `Pending` to `Succeeded`) if `Upgradeable` condition is set to `True`.
- Admin can set `Upgradeable` overrides in OperatorCondition's `spec` to supersede the `Upgradeable` condition in the `status`.

### Proposed Changes

OperatorCondition can use the `generation` and `observedCondition` mechanism to create feedback loop and version control for both operator and OLM to keep track of condition changes and acknowleage each other's action. Object's `generation` in `metadata` is automatically incremented when the spec is updated but not the status.

The `conditions` array will be added to OperatorCondition's spec and operator is now expected to only update the `conditions` in the spec to reflect its condition and no longer push changes to OperatorCondition's status.

As the spec is updated, the `generation` is incremented and OperatorCondition's controller will watch and acknowledge the changes by updating the OperatorCondition's status with `observedGeneration` to match the current `generation`.

The operator can read the OperatorCondition's status and check `observedGeneration` and compare with the recorded `generation`/`observedGeneration` that it has set previously in the spec to determine if OLM has reconciled the OperatorCondition's conditions.

apiVersion: operators.coreos.com/v1
kind: OperatorCondition
metadata:
  generation: 2
spec:
  conditions:
  - lastTransitionTime: "2021-05-05T16:56:44Z"
    lastUpdateTime: "2021-05-05T16:56:44Z"
    message: The operator is not ready
    reason: The operator is not upgradeable
    status: "False"
    observedGeneration: 1
    type: Upgradeable
status:
  conditions:
  - lastTransitionTime: "2021-05-05T16:56:50Z"
    lastUpdateTime: "2021-05-05T16:56:50"
    message: The operator is not ready
    reason: The operator is not upgradeable
    status: "False"
    observedGeneration: 2
    type: Upgradeable

Here is an example of the flow.

0. Operator pod does not report ready. The OperatorCondition object is created by the controller with generation = `1` for the operator.
1. Operator sets `Upgradeable` condition in the spec to `False`, increments generation to `2`. Ideally, the operator should set `observedGeneration` in the condition in the spec to the current `generation` that it has read which is `1`. In this way, when it reads the object again it can compare the `observedGeneration` in the spec and status.
2. OLM sees the `Upgradeable` to `False` spec change, and sets `Upgradeable` to `False` in the status along with `status.observedGeneration` to `2`.
3. The operator sees that `observedGeneration` is now `2`, which is >= what the operator set in step (1). It knows OLM has seen `Upgradeable=False` and realizes it’s now safe to report the pod as ready.
4. The operator sets `spec.conditions.Upgradeable` to `True` and OLM will reconcile this change to the OperatorCondition's status.
5. Finally, OLM (olm-operator) will pick this status and switch CSV status from `Pending` to `Succeeded`. Note that, OLM will not take action in term of moving CSV's status if `generation` =! `observedGeneration`.

## Actionable Items
1. Update OperatorCondition's CRD to add the `conditions` to the spec
- https://github.com/operator-framework/api/pull/119
- https://github.com/operator-framework/operator-lifecycle-manager/pull/2160
3. Publish a document to present the workflow on how to use OperatoCondition properly
4. Backport this to OCP 4.7 (which is when OperatorConditon is first introduced)