e2e strawman v0.6.2

OLM would be replaced by three projects:

  • RukPak, which deals with bundles and their application on a cluster
  • Deppy, which deals with indexes and resolving content from them
  • Operator Provisioner, which plugs into Depster/RukPak to give them support for OLM-operator bundles

The division between these components arose naturally when planning the concerns for the new APIs - giving them different names / subprojects highlights their independence (but may not be reflect the short-term implementation plan).

Each component builds on the other and is explained in detail below.

RukPak

RukPak is a pluggable solution for the packaging and distribution of cloud-native content and supports advanced strategies for installation, updates, and policy.

RukPak provides a content ecosystem for clusters:

  • Bundles contain remote or local content. This includes remote image bundles in the manifest bundle format, local volumes, git repos, etc.
  • Instances indicate that the content of a Bundle should be active in a cluster - think Pod but for arbitrary artifact types.

Different types of Bundles can be supported by adding provisioners to a cluster, much in the same way as persitstent volume types or CSI drivers. The unpacking and installation of bundles can be configured via ProvisionerClasses at runtime.

APIs and Components:

graph LR
subgraph RukPak APIs
     Class(ProvisionerClass)
     Bundle(Bundle) -.-> |has a| Class
     Instance(Instance) -.-> |has a| Class
end

subgraph Running Components
Provisioners[Provisioners<br>Operator,Helm,etc] --> Instance & Bundle & Class
end


classDef future fill:#ddd;

Bundle API

ABundle represents content that needs to be made available to other consumers in the cluster.

Much like the contents of a container image need to be pulled and unpacked in order for Pods to start using them, Bundles are used to reference content that may need to be pulled and should be unpacked. In this sense, Bundle is a generalization of theimage concept.

The specifics of how a Bundle get unpacked and consumed are defined by the ProvisionerClass and the provisioner that is configured to handle that ProvisionerClass.

kind: Bundle
metadata: 
  name: plumbus-operator.v0.9.3
spec:
  class: deppy.resolveset
  refs:
  - file://content
  volumeMounts:
  - mountPath: /content
    configMap:
      name: local 
      namespace: plumbus

Bundles do nothing on their own - they require a provisioner to unpack and make their content available.

Instance API

The Instance API points to a Bundle and indicates that it should be "active". This includes pivoting from older versions of a active bundles. Instance may also include an embedded spec for a desired Bundle.

Much like Pods stamp out instances of images, and Instance stamps out an instances of Bundles.Instance can be seen as a generaliztion of the Pod concept.

The specifics of how an Instance makes changes to a cluster based on a referenced Bundle is defined by the ProvisionerClass and the provisioner that is configured to handle that ProvisionerClass.

kind: Instance
metadata:
  name: resolved-654adh
spec:
  selector: 
    matchLabels:
      subscription: etcd-operator
  bundle:
    name: resolved-654adh
    spec:
      class: olm.resolveset
      refs:
      - file://content
      volumeMounts:
      - mountPath: /content
        configMap:
          name: resolved-654adh-content
          namespace: olm

ProvisionerClass API

ProvisionerClass defines a configuration for a provisioner. Provisioners are controllers that understand Instance, Bundle, and ProvisionerClass APIs and take action.

Each provisioner has a unique id. For example, the provisioner that ships with OLM and understand operator bundles has the id: operators.coreos.com/operators.

A ProvisionerClass specifies a specific configuration of provisioning (i.e. settings to interpret Instance and Bundle apis). A provisioner will only operate on Instance and Bundle objects that reference a ProvisionerClass that contain the provisioner's unique id.

If this seems familiar, it is the same pattern that is used by StorageClass / PeristentVolume / PersistentVolumeClaim in kubernetes.

Example ProvisionerClass:

kind: ProvisionerClass
apiVersion: rukpak.io/v1
metadata:
  name: <good name>
provisioner: <unique id>
# parameters has no schema, is provisioner-specific
parameters: {}

ProvisionerClass Permission

ProvisionerClass comes with a use verb.

Only users and serviceaccounts with use on a ProvisionerClass can create an object that references that ProvisionerClass.

A webhook rejects requests on Instance, and Bundle that reference ProvisionerClasses that the requesting user/serviceaccount does not have the use verb on.

Approval

Approval is an implementation detail of provisioners, which may choose to block rollout until certain conditions are met.

Instances have an Approved condition:

kind: Instance
metadata:
  name: resolved-654adh
spec:
  # omitted
status: 
  conditions:
  - lastUpdateTime: "2020-02-08T11:37:35Z"
    lastTransitionTime: "2020-02-08T11:37:35Z"
    message: Approved by my custom approver controller
    reason: ApprovedByMyPolicy
    status: True
    type: Approved

A webhook ships with RukPak that prevents any update to the Approved condition by any user or service account that lacks the approve verb on the Instance API.

The provisioner for the specified provisioner class is responsible for determining if approval is required or not. The provisioner may choose to auto-approve by writing the Approved condition into the status, or it may choose to leave the Instance unapproved. A user, or a separate approval plugin, may take over the approval from there.

kubectl instance approve <my-instance> will approve an instance that is awaitng approval, as long as the user has sufficient permission.




Deppy

Deppy runs on- or off-cluster to provide dependency resolution for catalogs of RukPak bundles.

Deppy provides an API for resolving constraints over catalog content (see: declarative index format).

The on-cluster installation of Deppy also includes:

  • Continual (optional) resolution against catalog content, to keep installed packages up-to-date.
  • A RukPak provisioner and default ProvisionerClass for deppy.resolveset bundles.
  • An API for defining constraints at runtime (Input)
  • A RukPak Approval plugin that provides Android-style update policy

ResolveSet Bundles

This is an example of a Bundle for a local deppy.resolveset bundle. It declares the the contents specified by refs are needed locally by other consumers in the cluster.

The controller watching this Bundle decides how to unpack and is configured via spec.class. In this case, the provisioner knows that the resolveset lists out other Bundles that should be created on the cluster, and begins to create them.

kind: Bundle
metadata: 
  name: etcd-operator.v0.9.3
spec:
  class: deppy.resolveset
  refs:
  - file://content
  volumeMounts:
  - mountPath: /content
    configMap:
      name: resolved-654adh-content
      namespace: olm
status:
  unpacked: NotStarted | InProgress | Done
  objects:
  - /objects/resolved.bundles.json
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: resolved-654adh-content
  namespace: default
data:
  resolved.bundles.json: |-
   {
    "schema": "bundle.v1",
    "packageName": "quay.io/operatorhubio/etcd",
    "path": "quay.io/operatorhubio/etcd:v0.6.1",
    "version": "0.6.1",
    "properties": [
        {
            "name": "pivotFrom",
            "value": "etcd-v0.6.0"
        },
        {
            "name": "olm.gvk",
            "value": {
              "group": "etcd.database.coreos.com", 
              "version": "v1beta2",
              "kind": "EtcdCluster"
            }
        },
    ],
    "channels": [
        "alpha"
    ],
   },
   {
      "schema": "bundle.v1",
      "packageName": "quay.io/operatorhubio/prometheus",
      "version": "1.0.0",
      "properties": [
        {
          "name": "olm.label",
          "value": "LTS",
        }
      ]
   }

ResolveSet Provisioner Configuration

Deppy ships with one provisioner, deppy.io/resolver, that understands resolveset bundles and knows how to unpack, apply, and pivot them.

The default ProvisionerClasses configure one of these provisioners to support common Instance and Bundle workflows.

kind: ProvisionerClass
apiVersion: rukpak.io/v1
metadata:
  name: olm.resolveset
provisioner: operators.coreos.com/olm
parameters:
  approval: AllowAll | DenyAll | Android
  # for a given bundle entry, determine what ProvisionerClass to use when stamping out a Bundle
  matchBundle:
  - selector: ".schema == olm.bundle"
    class: olm.bundle
  - selector: ".schema == helm.chart"
    class: helm.bundle
  # equivalent to above when picking ProvisionerClass based on schema
  matchSchema:
  - schema: olm.bundle
    class: olm.bundle

RukPak Approval Plugin

The resolveset provisioner understands the folliowing parameter in ProvisionerClass

kind: ProvisionerClass
parameters:
  approval: AllowAll | DenyAll | Android
  • AllowAll: Any Instance for a Bundle using a ProvisionerClass with this setting will be created with the Approved condition.
  • DenyAll: Any Instance for a Bundle using a ProvisionerClass with this parameter will be created without the Approved condition. Another actor (human, automation) can add Approved after the fact.
  • Android: Any Instance for a Bundle using a ProvisionerClass with this parameter will be created with the Approved condition if it can be determined that no bundle exceeds the permission of the previously installed bundle.

The value of approval for the default deppy.resolveset ProvisionerClass is Android.

Input API

The Input APIs are used to create inputs for the resolver. An instance of Input is more or less an Installable (in internal resolver parlance).

apiVersion: deppy.io/v1
kind: Input
metadata:
  name: plumbus
spec:
  inputClass: subscription
  constraints:
  # most common - equivalent to a Subscription
  - type: olm.catalog
    value: 
      name: community
      namespace: my-ns
      package: plumbus
      channel: stable

  # other examples of constraints - less common, but useful

  # lock to a specific version by package/version
  - type: olm.packageVersion
    value:
      package: "plumbus"
      version: v2.0.0

  # lock to a specific version by name
  - type: olm.name
    value: "plumbus.v2.0.0"

  # restrict to a range of versions
  - type: olm.packageVersion
    value:
      package: "plumbus"
      version: ">2.0.0 <3.0.0"

  # arbitrary rego queries
  - type: olm.rego
    value: "semver.Compare(minOCPVersion, 4.8.0) == 1"

status: 
  # instances with properties that match the constraints
  instances:
  - name: plumbus.v2.0.0-alpha
    # other instances satisfied by this instance
    satisfies: 
    - kind: Instance
      name: other-operator
      apiVersion: rukpak.io/v1
      meets:
      - {"olm.constraint": {"olm.name": "etcd-operator.v0.9.3"}}
    # dependencies introduced by this instance
    dependencies:
    - kind: Instance
      name: prometheus-operator-abc
      apiVersion: rukpak.io/v1
      meets:
      - {"olm.constraint": {"olm.gvk": {"group": "manufacturing.how.theydoit.com", "version": "v1alpha1", "kind": "Grumbo"}}
      - {"olm.constraint": {"olm.label": "LTS"}}
    # isntance-specific conditions
    conditions:
    - type: DependenciesMissing
      status: True
      reason: Only 2/3 dependency constraints are met.
      message: "etcd-operator has the following constraints: X,Y,Z. X and Z are satisfied by prometheus-operator-abc, but no instance satisfies Y"
      lastTransitionTime: "2019-09-16T22:26:29Z"
  
  # conditions about the input itself
  conditions:
  - type: ResolutionFailed
    message: "unable to find a solution that matched constraints: X requires Y, but Z requires !Y, X and Z are mandatory"
    status: True

InputClass API

InputClass defines a configuration for an Input.

End-users may select, but will likely never write, an InputClass.

Default InputClasses

Deppy ships with a set of default InputClasses

Simulate Subscription behavior

kind: InputClass
apiVersion: deppy.io/v1
metadata:
  name: subscription
parameters:
  # if force = true, all matching entries will be installed 
  # directly, and the resolver will not be engaged
  force: false
  # if true, new `Inputs` will be generated for any Instances
  # that have been installed as a dependency
  generateInputsForDependencies: true
  # if true, does not output an `Instance`
  dryRun: false
  # if true, the resolveset bundle includes a lot of data about the resolution process
  trace: false
  # any constraints defined here are included automatically in the `Input` constraints that reference this class
  constraints: 
  - type: minKubeVersion 
    value: >= 1.20
  - type: minOCPVersion
    value: >= 4.7

Force install operators

kind: InputClass
apiVersion: deppy.io/v1
metadata:
  name: force
parameters:
  force: true
  generateInputsForDependencies: false
  dryRun: false
  trace: false
  constraints: 
  - type: minKubeVersion 
    value: >= 1.20
  - type: minOCPVersion
    value: >= 4.7

Install dependencies, but don't keep them up-to-date independently (i.e. minimal version selection)

kind: InputClass
apiVersion: deppy.io/v1
metadata:
  name: minimal-version
parameters:
  force: true
  generateInputsForDependencies: false
  dryRun: false
  trace: false
  constraints: 
  - type: minKubeVersion 
    value: >= 1.20
  - type: minOCPVersion
    value: >= 4.7

Resolve but don't apply

kind: InputClass
apiVersion: deppy.io/v1
metadata:
  name: dry-run
parameters:
  force: false
  generateInputsForDependencies: false
  dryRun: true
  trace: false
  constraints: 
  - type: minKubeVersion 
    value: >= 1.20
  - type: minOCPVersion
    value: >= 4.7
kind: InputClass
apiVersion: deppy.io/v1
metadata:
  name: dry-run-next
parameters:
  force: false
  generateInputsForDependencies: false
  dryRun: true
  trace: false
  # set to the next minor release of kube/ocp 
  constraints: 
  - type: minKubeVersion 
    value: >= 1.21
  - type: minOCPVersion
    value: >= 4.8

CatalogSource API

Just the normal CatalogSource API. Cluster scoped?

Resolver API

Note: See Future Work notes at the end of the doc for caveats / considerations.

The resolver has an interactive (non-kube) API that supports:

Queries

Input:

  • a constraint
  • (or) a rego query

Output:

  • a list of matching bundles from the catalog cache

Interactive Resolution

Input:

  • a list of lists of constraints (i.e. a list of Inputs)

Output:

  • resolved set, or error



Operator Provisioner

The Operator provisioner understands registry+v1 manifest bundles and instances, providing support for olm.bundle entries in catalogs. It also comes with:

  • Default ProvisionerClasses for operator bundles
  • An approval plugin for operator bundles
  • A set of default InputClasses for use when installing operator bundles
  • A porcelein Operator API that can be used for installing operator bundles in lieu of writing Input directly.

Operator Bundle Example

This is an example of a Bundle for a remote registry+v1 bundle image. It declares the the contents specified by refs are needed locally by other consumers in the cluster.

The controller watching this Bundle decides how to unpack and is configured via spec.class. In this case, the provisioner knows that this is registry+v1 bundle, and unpacks the manifests held within.

The status is controlled by the provisioner configured on the class, but it lists out locations in the cluster that unpacked content can be found.

kind: Bundle
metadata: 
  name: etcd-operator.v0.9.3
spec:
  class: olm.bundle
  refs: 
  - docker://quay.io/etcd/bundle@sha256
status:
  unpacked: NotStarted | InProgress | Done
  objects:
  - /objects/csv
  - /objects/role

Operator Bundle Instance Example

This is an example of an Instance that declares that registry+v1 bundle image should be pulled down and installed on the cluster.

The selector may match other (older) bundles. The controller watching this instance decides how to pivot and is configured via spec.bundle.spec.class, or on spec.class of the Bundle object that Instance points to.

kind: Instance
spec: 
  # selector is used to find previous/current Bundles
  selector: 
    matchLabels:
      subscription: etcd-operator
  # a reference to the Bundle object that should be considered "Active".
  bundle: 
     name: etcd-operator.v0.9.3
     # optional, for direct application of a bundle definition
     spec:
       class: olm.bundle
       refs: 
       - docker://quay.io/etcd/bundle@sha256
       # or: volume-backed
status:
  conditions:
  - type: Approved
    status: True
    reason: ApprovedByMyPolicy
    message: Approved by my custom approver controller
    lastTransitionTime: "2020-02-08T11:37:35Z"

Default ProvisionerClasses

The default ProvisionerClasses configure one of these provisioners to support common Instance and Bundle workflows.

kind: ProvisionerClass
apiVersion: rukpak.io/v1
metadata:
  name: olm.bundle
provisioner: operators.coreos.com/operators
parameters:
  approval: AllowAll | DenyAll | UserPermission
  # options for unpacking bundles
  maxObjectSize: 10mb

Approval for the operator provisioner

The operator provisioner understands the folliowing parameter in ProvisionerClass

kind: ProvisionerClass
parameters:
  approval: AllowAll | DenyAll | UserPermission
  • AllowAll: Any Instance for a Bundle using a ProvisionerClass with this setting will be created with the Approved condition.
  • DenyAll: Any Instance for a Bundle using a ProvisionerClass with this parameter will be created without the Approved condition. Another actor (human, automation) can add Approved after the fact.
  • UserPermission: Any Instance for a Bundle using a ProvisionerClass with this parameter will be created with the Approved condition if it can be determined via SubjectAccessReview checks that the user that created it has sufficient permission to create all of the contents of the Bundle.

The value of approval for the default olm.bundle ProvisionerClass is UserPermission.

Operator API

The simplest way to install operators and subscribe to updates is via the Operator API. The current status and install progress, as well as references to all components of an operator, can be found on the status block of the Operator object.

Example:

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: subscription
  
  # optional
  # this field is translated directly to an `Input` constraint
  # that looks like:
  # 
  # type: olm.catalog
  # value: 
  #   name: community
  #   namespace: my-ns
  #   package: plumbus
  #   channel: stable
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  # optional, lock to a specific version
  # this field is translated directly to an `Input` constraint
  # that looks like:
  # 
  # type: olm.packageVersion
  # value: 
  #   package: plumbus
  #   version: v2.0.0
  # 
  version: "2.0.0"
  
  # version may also be a semver range, like ">2.0.0 <3.0.0" 
  version: ">2.0.0 <3.0.0"
  
  # optional
  # additional arbitrary constraints can be added 
  constraints:
  - type: olm.rego
    value: "semver.Compare(minOCPVersion, 4.8.0) == 1"

status:
  updates:
  - options:
    - plumbus.v2.0.8
    - plumbus.v2.0.7
    - plumbus.v2.0.6
    catalogSourceRef:
      name: community
      namespace: my-ns
    package: plumbus
    channel: beta

  conditions:
  - kind: UpdateAvailable
    status: True
    reason: CrossChannelUpdateFound
    message: updates have been found that require a change in the operator spec to apply
    lastTransitionTime: "2019-09-16T22:26:29Z"

  components:
   matchLabels:
      operators.coreos.com/plumbus: ""
   refs:
    - kind: Input
      name: plumbus
      apiVersion: rukpak.io/v1
    - kind: Instance
      name: plumbus.v2.0.0-alpha
      apiVersion: rukpak.io/v1
      conditions:
      - type: Installing
        status: true
    - kind: Bundle
      name: plumbus.v2.0.0-alpha
      apiVersion: rukpak.io/v1
      conditions:
      - type: Unpacked
        status: true  
    # older bundles visible during pivot
    - kind: Bundle
      name: plumbus.1.9.0
      apiVersion: rukpak.io/v1
      conditions:
      - type: Unpacked
        status: true
      - type: Replaced
        status: true
    - kind: ClusterServiceVersion
      namespace: operators
      name: plumbus.v2.0.0-alpha
      apiVersion: rukpak.io/v1alpha1
      conditions:
      - type: Installing
        status: True
        reason: AllPreconditionsMet
        message: deployment rolling out
        lastTransitionTime: "2019-09-16T22:26:29Z"
    - kind: CustomResourceDefinition
      name: plumbai.how.dotheydoit.com
      apiVersion: apiextensions.k8s.io/v1beta1
    - kind: ClusterRoleBinding
      namespace: operators
      name: rb-9oacj
      apiVersion: rbac.authorization.k8s.io/v1

When an Operator is created, an appropriate Input will be created. The Operator controller makes use of the Resolver API to determine a set of potential updates to highlight in the status (i.e. by querying in other channels / relaxing constraints).

Most users will not need to look at other APIs.

Future Work: Constraints, MetaConstraints, and Preference

This topic has not been nailed down to a degree that a strawman made sense. Instead, the scope of the problem is outlined below.

There is a big limitation in the current model (in deppy, specifically), which limits the constraints and meta-constraints that can be specified as input.

For example, this constraint:

type: olm.catalog
value: 
  name: community  
  namespace: my-ns
  package: plumbus
  channel: stable

Gets translated to a boolean formula for the sat-solver that looks like:

[A or B or C or D]

Where A, B, C, and D are all of the operators in the stable channel for the plumbus package. This constrains all solutions to only those that can be found in the specific package/channel, encoding it in a way that the solver understands.

Today OLM does the translation of constraints to these boolean formulas (we call a single formula an "installable" in the OLM codebase).

The resolver also has a set of constraints that enforce properties and invariants about the solution. Today these are things like "only one subscription to a package can exist in a channel" or "only one operator can provide a gvk in a namespace".

These "meta-constraints" are also translated into boolean formulas and fed to the solver.

In the model above, each new constraint type or constraint needs to be encoded in deppy, because deppy does the translation of constraints to solver inputs (boolean formulas).

We know that, when building a package ecosystem, it is useful to be able to define this type of constraint. We would like to be resilient to future extension needs (i.e. it should be straightforward to add helm-specific constraint types, as the need arises).

The rego constraint type gives us some flexiblity, but limits us to filtering that can be expressed as a rego query.

Some options we have discussed:

  • Constraints should all be specified as rego queries
  • Define plugin system that can be called directly from deppy (wasm?) for filtering
  • Define an InputClass "provisioner" that provides an API that converts from a constraint to a boolean formula
  • A Solve API that takes as input boolean constraints. Plugins translate constraints on Input into boolean constraints on Solve, the resolver outputs its results into the status of Solve.

Preference is also not currently pluggable.

Preference is the property in the resolver that tests solution in a particular order - it's the reason that an operator from the head of a channel is picked instead of an older release, if it can be. It's the reason operators from the redhat-catalog are preferred to operators in the community-catalog, and why dependencies are preferred from the same catalog from which they came.

As we expand to other artifact types, it's clear that we would like to be able to define preference order for other constraints.

Changelog

  • 0.6.2

    • migrated doc ownership
  • 0.6.1

    • move satisfies/dependency info to Inputs
    • add back in user stories
    • renamed depster -> deppy
  • 0.6

    • cleanup
    • clarify pluggability questions / future work
  • 0.5

    • One project for instance / bundle (rukpak)
    • One project for resolver + resolver apis (depster)
    • Plugins for those
  • 0.4

    • Separate operator-specific things from resolver+bundle things
  • 0.3

    • Added an arch diagram
    • Named resolver+bundle subproject RukPak with new group rukpak.io
    • Operator API puppets Input API for now
    • Error messages on Input API
    • Adds option for tracing resolver output, added to resolveset
    • Add sketch of Resolver API to answer questions about available updates / dry-run
    • Added a note about future extension via InputMetaClass
  • 0.2

    • add an FAQ
    • add a new API (Input) for constraints, separate from Operator
      • re-usable across other artifact types
      • factor out the top-level fields in Operator
      • entrypoint when installing somethign that is not an Operator
    • make the Approved condition match other condition apis
  • 0.1

    • initial e2e: do we get everything we need with just Operator + Bundle + Instance?



User Stories

Evaluate the design against target user stories.

Force install a single operator from a catalog: don't subscribe it to updates, don't resolve dependencies

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: force
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  version: "2.0.0"
  • Creates an Input with matching constraints
  • Creates an olm.resolveset Instance that contains every bundle that matches the constraints
    • This may pick multiple bundles
    • This does not fulfil dependencies - the resolveset contains only the bundles that match the query

Install an operator and its dependencies from a catalog but don't subscribe it to updates.

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
  version: "2.0.0"  
  • Creates an Input with matching constraints
  • Creates a resolveset Instance
    • Creates a set of Instances for all bundles
  • Constraints lock this to a particular version, no updates occur

Install a single operator from a catalog and subscribe it to updates, but don't subscribe its dependencies to updates.

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
  • Creates an Input with matching constraints
  • Creates a resolveset Instance
    • creates a set of Instances for all bundles
  • Constraints allow for updates

Install an operator and its dependencies and subscribe all of the operator and all of its dependencies to updates.

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: subscription
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
  • Creates an Input with matching constraints
  • Creates a resolveset Instance
    • creates a set of Instances for all bundles
  • For each Instance in the resolveset, creates an Input with appropriate constraints

Install a single operator bundle directly (no catalog)

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:  
  bundle:
    name: etcd-operator.v0.9.3
    spec:
      class: olm.bundle
      refs: 
      - docker://quay.io/etcd/bundle@sha256

which results in:

kind: Instance
spec: 
  # selector is used to find previous/current Bundles
  selector: 
    matchLabels:
      subscription: etcd-operator
  # a reference to the Bundle object that should be considered "Active".
  bundle: 
     name: etcd-operator.v0.9.3
     # optional, for direct application of a bundle definition
     spec:
       class: olm.bundle
       refs: 
       - docker://quay.io/etcd/bundle@sha256
       # or: volume-backed
status:
    # progress, failure, etc
    # 

Update the dependencies of an operator even if there is no "subscription" for the dependencies.

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable

  • Creates an Input with matching constraints
  • minimal-version means that the dependencies will only update when this one has requirements on newer versions.

Force update a single operator directly to a specific version, with no catalog available in the cluster.

Assuming there is an existing Instance on the cluster for etcd-operator:

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:  
  bundle:
    name: etcd-operator
    spec:
      class: olm.bundle
      refs: 
      - docker://quay.io/etcd/bundle@sha256

will result in:

kind: Instance
spec: 
  # selector that matches the existing Bundle we're updating
  selector: 
    matchLabels:
      subscription: etcd-operator
  # a reference to the Bundle object that should now be considered "Active".
  bundle: 
     name: etcd-operator
     spec:
       class: olm.bundle
       # ref to the bundle we want to update to
       refs: 
       - docker://quay.io/etcd/bundle@sha256
status:
    # progress, failure, etc
    # 
  • This attempts to directly update an installed bundle to another bundle, pivoting ownerrefs, etc

Force update a single operator directly to a specific version from a catalog

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: force
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  version: "2.0.0"
  • Creates an Input with matching constraints
  • Creates an olm.resolveset Instance for every bundle that matches the constraints, with appropriate "pivot" data added.

Able to change the properties or constraints of any installed bundle.

Properties and constraints, as determined by the resolver, are persisted as annotations on the Bundle. They can be edited by a user to change the way resolver sees them (this is generally inadvisable and should only be done when absolutely necessary).

kind: Bundle
apiVersion: rukpak.io/v1
metadata:
  name: plumbus-kjf122
  annotations:
    olm.properties: -|
     {"olm.gvk": {"group": "how.theydoit.com", "version": "v2alpha1", "kind": "Plumbus"}}
     {"olm.constraint": {"olm.gvk": {"group": "manufacturing.how.theydoit.com", "version": "v1alpha1", "kind": "Grumbo"}}
spec:
 class: registry.v1
 refs: 
 - docker://quay.io/plumbus/bundle@sha256:abcdef12345
status:
    unpacked: Done
    objects:
    - /objects/csv
    - /objects/role

Communicate to the user when the things that are already installed have unsatisfied dependencies and communicate to a user that dependencies are satisfied by specific other bundles.

Status is provided on Input. This is aggregated to the Operator status via existing mechanisms.

apiVersion: depster.io/v1
kind: Input
metadata:
  name: plumbus
spec:
  inputClass: subscription
  constraints:
  - type: olm.catalog
    value: 
      name: community
      namespace: my-ns
      package: plumbus
      channel: stable
status: 
  # instances with properties that match the constraints
  instances:
  - name: plumbus.v2.0.0-alpha
    # other instances satisfied by this instance
    satisfies: 
    - kind: Instance
      name: other-operator
      apiVersion: rukpak.io/v1
      meets:
      - {"olm.constraint": {"olm.name": "etcd-operator.v0.9.3"}}
    # dependencies introduced by this instance
    dependencies:
    - kind: Instance
      name: prometheus-operator-abc
      apiVersion: rukpak.io/v1
      meets:
      - {"olm.constraint": {"olm.gvk": {"group": "manufacturing.how.theydoit.com", "version": "v1alpha1", "kind": "Grumbo"}}
      - {"olm.constraint": {"olm.label": "LTS"}}
    # isntance-specific conditions
    conditions:
    - type: DependenciesMissing
      status: True
      reason: Only 2/3 dependency constraints are met.
      message: "etcd-operator has the following constraints: X,Y,Z. X and Z are satisfied by prometheus-operator-abc, but no instance satisfies Y"
      lastTransitionTime: "2019-09-16T22:26:29Z"

Communicate to the users when they are about to install operators, which dependencies would be resolved

  • Installing new content by default will result in an Instance waiting to be approved (due to android permissions)
  • User reviews the content of the instance before kubectl instance approve
  • Users that want more control can use a provisionerClass with DenyAll so that every single change requires review.

Don't block updates to operators when other unrelated operators are blocked.

The resolver will generate disjoint resolveset Instances.

Understand why an update is blocked

The resolver will emit events in the namespace.

Most other states should be readily visible in current Input (and Operator) status

If an operator is not subscribed to updates, be presented with options for update as they become available (OCP-style).

  • If using the Operator API, the status will include information about potential updates
  • The resolver API can be queried with constraints in the general case.

Operator updates available in other catalogs or channels are surfaced to users.

  • If using the Operator API, the status will include information about potential updates
  • The resolver API can be queried with constraints in the general case.

Ability to write policy for updates. Andriod-style permission policy comes by default.

Instances may have an Approved condition:

kind: Instance
metadata:
  name: resolved-654adh
spec:
  selector: 
    matchLabels:
      subscription: etcd-operator
  bundle:
    name: resolved-654adh
    spec:
      class: olm.resolveset
      refs:
      - file://content
      volumeMounts:
      - mountPath: /content
        configMap:
          name: resolved-654adh-content
          namespace: olm
status: 
  conditions:
  - lastUpdateTime: "2020-02-08T11:37:35Z"
    lastTransitionTime: "2020-02-08T11:37:35Z"
    message: Approved by my custom approver controller
    reason: ApprovedByMyPolicy
    status: True
    type: Approved

Depster ships with approval plugins as part of the olm.resolveset provisioner, the operator provisioner ships with approval plugins for olm.bundle. Additional plugins can be written, configured, and installed as needed.

Non-administrators should be able to install non-operator content, as long as they have sufficient permission for the install.

Strawman: As a non-administrator user, I would like to trigger a namespace-local installation of a helm chart via a bundle.

A helm provisioner is installed that watches Instance, and Bundle. It may also choose to include porcelein APIs that translate to Inputs

When I create an Instance that contains references a provisionerclass that points to the helm provisioner, the helm provisioner is used to install.

The helm provision may choose to provide additional apis for visibility or control (just as olm provides the Operator API).

The helm provisioner may provide a HelmChart API that is namespaced-scoped and performs access checks for a user before creating an Input on a user's behalf.

Strawman 2: An additional namespace-scoped API can be added, OperatorRequest, that either results in the creation of an unapproved Instance

As a non-administrator user, I would like to request an Operator install in the style of Kubernetes certificate signing requests getting approved by admins

This should more or less be covered by the approval workflow + a way to request installation with namespace scoped APIs.

Determine what would be installed under specific conditions - i.e. if the cluster were updated to a new version, if a specific operator is selected

apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: dry-run
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  version: "2.0.0"

results in

apiVersion: rukpak.io/v1 kind: Input metadata: name: plumbus spec: inputClass: dry-run constraints: - type: olm.catalog value: name: community namespace: my-ns package: plumbus channel: stable - type: olm.packageVersion value: packageName: plumbus version: "2.0.0"

The dry-run inputClass sets dryRun: true. When dry-run is true, the resolver will output a resolveset Bundle but not a resolveset Instance, so no changes occur to running bundles.

Alternatively, the resolver API can be queried interactively for this information.

If a dependency is in a degraded state, I can tell this from the top-level operator.

The Operator aggregates status of its components, which includes Input objects, which report dependency (satisfaction) problems.

Actual dependency health: TODO

Install at a particular version and then enable updates.

  1. Create an Operator to install at a specific version
apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version 
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  version: "2.0.0"
  1. Edit the Operator to enable updates
apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version 
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable

Install at a particular version, and walk through a set of updates, stopping before hitting latest.

  1. Create an Input to install at a specific version
apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version 
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  version: "2.0.0"
  1. Edit the Input to enable updates up to a specific release
apiVersion: operators.coreos.com/v1
kind: Operator
metadata:
  name: plumbus
spec:
  inputClass: minimal-version 
  catalog:
    name: community
    namespace: my-ns
    package: plumbus
    channel: stable
    
  version: "<= 2.4.5"

Install two operators that own the same APIs

Adjacent stories

Stories that are best addressed with features outside the scope of this enhancement.

View the update graph for a particular operator in the console.

The package server can be queried for this data, the console can display it.

Present a user with release notes for available updates, and with release notes for currently installed operators.

Given any bundle, the package-server can be queried for release notes.

Detailed release notes or docs may be part of an operator's bundle, visible after install.

Non-administrators can get info about operator-provided APIs based on their access to them.

Options:

  • Hack into discovery / openapi
  • Have a cluster-scoped api just for this info
  • Add extended property support to kube
  • "man pages" for kube
Select a repo