owned this note
owned this note
Published
Linked with GitHub
# WIP: How to OLM
**WARNING: this is just a strawman to explore a possible UX.**
## Bundle Your Operator
The first step towards life cycle management is packaging your operator for OLM.
We olms call our packaging format [the bundle spec](). It gives us a way to describe the k8s resources, configuration, and a host of other metadata that make up an operator.
Let's examine the anatomy of a faux bundle to get a feel for how bundles are constructed.
Consider a `plumbus-operator` that consists of the following k8s resource manifests (given as a YAML stream):
```yaml=
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: fleems.howtheydoit.io
spec:
group: howtheydoit.io
versions:
- name: v1
served: true
storage: false
schema:
openAPIV3Schema:
type: object
properties:
# ...
- name: v2alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
# omitted for brevity
scope: Namespaced
names:
plural: fleems
singular: fleem
kind: Fleem
shortNames:
- fl
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: plumbae.howtheydoit.io
spec:
group: howtheydoit.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
# omitted for brevity
scope: Namespaced
names:
plural: plumbae
singular: plumbus
kind: Plumbus
shortNames:
- pl
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller
labels:
app: plumbus-operator
spec:
replicas: 1
selector:
matchLabels:
app: plumbus-operator
template:
metadata:
labels:
app: plumbus-operator
spec:
serviceAccountName: operator
containers:
- name: controller
image: quay.io/howtheydoit/plumbus-controller@sha256:abc123...
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Namespace
metadata:
name: plumbus-operator
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: operator
namespace: plumbus-operator
----
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: fleem-controller
namespace: plumbus-operator
rules:
- apiGroups: ["howtheydoit.io"]
resources: ["fleems"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: plumbus-controller
aggregationRule:
clusterRoleSelectors:
- matchLabels:
howtheydoit.io/aggregate-to-plumbus-controller: "true"
rules:
- apiGroups: ["howtheydoit.io"]
resources: ["plumbae/status"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: plumbus-user
annotations:
howtheydoit.io/aggregate-to-plumbus-controller: "true"
rules:
- apiGroups: ["howtheydoit.io"]
resources: ["plumbae"]
verbs: ["*"]
- apiGroups: ["howtheydoit.io"]
resources: ["plumbae/status"]
verbs: ["read"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: operator
namespace: plumbus-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fleem-controller
subjects:
- kind: ServiceAccount
name: operator
namespace: plumbus-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: plumbus-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: plumbus-controller
subjects:
- kind: ServiceAccount
name: operator
namespace: plumbus-operator
```
Now that's a lot of YAML, let's summarize what we have:
- plumbus-operator `Namespace`: where the operator controller will live
- controller`Deployment`: deploys the operator controller to the plumbus-operator `Namespace`
- operator`ServiceAccount`: identity used by the operator controller
- fleem `CustomResourceDefinition`: defines the namespaced `Fleem` custom resource, used only by the controller in the plumbus-operator namespace
- plumbus `CustomResourceDefinition`: defines the namespace scoped `Plumbus` custom resource, consumed by plumbus-operator clients; everyone knows what it does, so there is no reason to explain it.
- fleem-controller`Role`: grants `*` on `resources: ["fleem"]` in the plumbus-operator `Namespace`
- plumbus-controller `ClusterRole`: grants `*` on `resources: ["plumbus/status"]`
- plumbus-user`ClusterRole`: grants`*` on `resources: ["plumbae"]`, `read` on `resources: ["plumbae/status"]`, and [aggregates](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) to the plumbus-controller `ClusterRole`
- operator `RoleBinding`: binds the fleem-controller `Role` to the operator `ServiceAccount` in the plumbus-operator namespace
- plumbus-operator `ClusterRoleBinding`: binds the plumbus-controller `ClusterRole` to the operator `ServiceAccount`
Assuming we split this YAML stream across several files in a directory:
```shell
$ tree .
.
├── crds.yaml
├── deployments.yaml
└── rbac.yaml
0 directories, 3 files
```
We can quickly initialize a bundle using the `opm init` command:
```shell
$ opm init plumbus-v1.0.0 | tar -C bundle -xvf -
x ./
x ./manifests/
x ./metadata/
x ./metadata/bundle.yaml
x ./manifests/crds.yaml
x ./manifests/deployments.yaml
x ./manifests/rbac.yaml
```
Tada! The `bundle` directory now contains an initialized bundle. Let's change directories and disect its content.
```shell
$ cd bundle
$ tree .
.
├── manifests
│ ├── crds.yaml
│ ├── deployments.yaml
│ └── rbac.yaml
└── metadata
└── bundle.yaml
2 directories, 4 files
```
Here we notice two directories, `manifests` and `metadata`.
Inspecting the `manifests` directory reveals that it contains our input manifests:
```shell
$ head -n5 manifests/*
==> manifests/crds.yaml <==
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: fleems.howtheydoit.io
==> manifests/deployments.yaml <==
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller
==> manifests/rbac.yaml <==
---
apiVersion: v1
kind: Namespace
metadata:
name: plumbus-operator
```
Turning our attention to the `metadata` directory, we find the file responsible for describing the management of our operator to OLM:`bundles.yaml`
> Note: Other files can be added to the `metadata` directory (e.g. `annotations.yaml`) to provide additional data to bundle clients. For more info, please see the [bundle spec]().
```yaml
schema: olm.bundle
name: angry-williams
properties:
- type: olm.gvk.provided
value:
group: howtheydoit.io
kind: Fleem
version: v2alpha1
- type: olm.gvk.provided
value:
group: howtheydoit.io
kind: Plumbus
version: v1
```
`opm init` generated this file based on the given manifests. For instance, the `olm.gvk.provided` property was inferred from the CRDs. Notice that it also generated a random name for our operator. Let's change the name to `plumbus-operator` before we confuse users.
Now that our bundle is set up, we can change the definition of our operator by directly updating these files. Jump into your favorite editor and update the value of the `name` field:
```yaml
schema: olm.bundle
name: plumbus-operator
properties:
# ...
```
> Note: we could have avoided this step by using the `--name` option to explicitly provide the name we wanted during initialization; e.g. `opm init plumbus-v1.0.0 --name plumbus-operator`
## Distribute Your Bundle
Before we can install a bundle, we need to package it into a distributable format that OLM understands.
It just so happens that `opm` provides a command that performs this packaging;`opm pack` converts a bundle directory into an equivalent docker-archive tarball, which can then be loaded into a container image and pushed to a compatible registry with the tool du jour.
Let's pack the plumbus-operator bundle and use `docker` to load and push the result.
```shell
$ tree .
.
├── manifests
│ ├── crds.yaml
│ ├── deployments.yaml
│ └── rbac.yaml
└── metadata
└── bundle.yaml
2 directories, 4 files
$ opm pack quay.io/howtheydoit/plumbus-operator:v1.0.0 . | docker load -q
Loaded image: quay.io/howtheydoit/plumbus-operator:v1.0.0
$ docker push quay.io/howtheydoit/plumbus-operator:v1.0.0
```
The `opm unpack` command performs the inverse operation. It converts a bundle docker-archive tarball into a bundle directory tarball.
Let's verify the content of the plumbus-operator bundle image we just pushed:
```shell
$ cd ..
$ tree .
.
├── bundle
│ ├── manifests
│ │ ├── crds.yaml
│ │ ├── deployments.yaml
│ │ └── rbac.yaml
│ └── metadata
│ └── bundle.yaml
└── deploy
├── crds.yaml
├── deployments.yaml
└── rbac.yaml
4 directories, 7 files
$ docker save quay.io/howtheydoit/plumbus-operator:v1.0.0 | opm unpack - | tar -C unpacked -xvf -
x ./
x ./manifests/
x ./metadata/
x ./metadata/bundle.yaml
x ./manifests/crds.yaml
x ./manifests/deployments.yaml
x ./manifests/rbac.yaml
$ diff -rq bundle unpacked
```
## Install Your Bundle
In order for OLM to install a bundle, it needs to be told which bundle to install. This is done using the `Instance` API.
Let's create an `Instance` for the `plumbus-operator` from its bundle image:
```shell
$ cat <<EOF | kubectl create -f -
apiVersion: rukpak.io/v1
kind: Instance
spec:
# selector is used to find previous/current Bundles
selector:
matchLabels:
subscription: plumbus-operator
# a reference to the Bundle object that should be considered "Active".
bundle:
name: plumbus-operator.v1.0.0
spec:
class: olm.bundle
refs:
- docker://quay.io/howtheydoit/plumbus-operator@sha256::e1488cb900233d035575f0a7787448cb1fa93bed0ccc0d4efc1963d7d72a8f17
EOF
```
> Note: OLM makes use of a generic packaging/distribution tool called [rukpak.io](). It implements a type of controller, called a provisioner, which is responsible for unpacking and applying bundle content to the cluster. `spec.class: olm.bundle` in the example above identifies OLM as the provisioner of choice. See the [ProvisionerClasses docs]() for more details.
Behind the scenes, a `Bundle` resource is generated based on the `spec.bundle` field of the `Instance` we just created:
```yaml=
apiVersion: rukpak.io/v1
kind: Bundle
metadata:
labels:
subscription: plumbus-operator
name: plumbus-operator.v1.0.0
spec:
class: olm.bundle
refs:
- docker://quay.io/howtheydoit/plumbus-operator@sha256::e1488cb900233d035575f0a7787448cb1fa93bed0ccc0d4efc1963d7d72a8f17
status:
unpacked: InProgress
# ...
```
This tracks the process of sourcing bundle content to the cluster. If this process is successful, the `status.unpacked` field will be set to `Done`:
```yaml
# ...
status:
unpacked: Done
# ...
```
> Note: `Bundles` can be created independently of `Instances`. This allows for several additional use cases. See the [bundle docs]() for more info.
Next, before any bundle content is applied to the cluster, the `Instance` is checked for approval. Approval is denoted by a status condition of type `Approved`. By default, OLM requires approval by setting the `Approved` status to `False` if the `Instance` creator isn't privileged to create every resource provided by the bundle:
```yaml=
apiVersion: rukpak.io/v1
kind: Instance
metadata:
name: plumbus-operator
spec:
# ...
status:
conditions:
- lastUpdateTime: "2020-02-08T11:37:35Z"
lastTransitionTime: "2020-02-08T11:37:35Z"
message: User plumbus-lover has insufficient privileges to apply target bundle content.
reason: Unauthorized
status: False
type: Approved
```
Once denied, a user with sufficient privileges _or_ that has the `approve` verb on the `Instance` can approve by setting the `status` field to `True`. This update can be achieved by issuing a json patch with a standard utility like `curl`, but is also codified concisely by the [RukPack kubectl plugin]() as `kubectl instance approve <instance-name>`:
```shell
$ kubectl instance approve plumbus-operator
plumbus-operator successfully approved
$ kubectl get instance plumbus-operator -o yaml
apiVersion: rukpak.io/v1
kind: Instance
metadata:
name: plumbus-operator
spec:
# ...
status:
conditions:
- lastUpdateTime: "2020-02-08T11:38:22Z"
lastTransitionTime: "2020-02-08T11:38:22Z"
message: User plumbus-admin manually approved this instance.
reason: ApprovedManually
status: True
type: Approved
```
> Note: OLM's approval behavior can be changed by updating the approval parameters in its [ProvisionerClass]().
After approval, all the resources in our plumbus-operator bundle are applied to the cluster. They should be easy enough to find since the `spec.selector` field of the `Instance` is projected onto every resource it created:
```shell
$ kubectl ns,crd,clusterrole,clusterrolebinding -l subscription=plumbus-operator
$ kubectl -n plumbus-operator get sa,role,rolebinding,deployment -l subscription=plumbus-operator
...
```
Additionally, OLM will ensure that these resources exist as long as the `Instance` does. If any of them are deleted or modified, they will be eventually restored:
```shell
$ kubectl -n plumbus-operator delete deployment controller
$ kubectl -n plumbus-operator get deployment controller -w
...
```
### Configuring an Operator Post-Install
Now that we've installed plumbus-operator, we need to configure the cluster so that it can be used by its clients. To do this, we need to ensure two properties:
1. A client is privileged enough to interact with the operator
2. The operator is privileged enough to interact with the client
> **Note:** Privileges can be asymmetric; i.e. a client may need a different set of privileges than an operator does to interact.
#### Au Naturel
These properties would apply to the operator even if its resources were directly applied to the cluster with `kubectl`. The fact that it was installed by OLM is of no consequence. Moreover, **notice that in our bundle resources, we've already ensured the operator is privileged enough to interact with any client by including the plumbus-operator `ClusterRoleBinding`**; grants `*` on `plumbus` and `plumbus-status`.
Now we only need to ensure a given client can interact with the operator. Luckily, we've included the plumbus-user `ClusterRole` in our bundle with just such an intent. Let's see how it can be used.
Say we wanted to grant user Suzan the ability to interact with plumbus-operator in the `Namespace` arrakis. To do this, we can bind Suzan to the plumbus-user `ClusterRole` with a `RoleBinding` in that `Namespace`:
```yaml=
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: plumbus-suzan
namespace: arrakis
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: plumbus-user
subjects:
- kind: User
name: Suzan
```
What if we wanted to grant the group plumbateers access in a set of namespaces? Just create `RoleBindings` in each namespace, binding plumbateers to the plumbus-user `ClusterRole`:
```yaml=
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: plumbus-plumbateers
namespace: arrakis
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: plumbus-user
subjects:
- kind: Group
name: plumbateers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: plumbus-plumbateers
namespace: caladan
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: plumbus-user
subjects:
- kind: Group
name: plumbateers
```
and if we wanted to grant the plumbus-lover `ServiceAccount` in the caladan `Namespace` access in all namespaces? Simply bind the `ServiceAccount` with a `ClusterRoleBinding`:
```yaml=
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: plumbus-plumbus-lover
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: plumbus-user
subjects:
- kind: ServiceAccount
name: plumbus-lover
namespace: caladan
```
#### Config Manager
What about more complex use cases?
- install the operator with a minimal set of privileges and increase them as we add clients
- the operator requires additional non-RBAC resources per client
- grant access to swaths of users over several namespaces at once
Complex use cases, like those listed above, are unwieldy using RBAC alone.
Enter [combo](https://github.com/operator-framework/combo); a tool that allows an author to templatize valid configurations for their operator.
Let's augment our bundle manifests with some `combo` resources so that we can minimize the operator's initial privileges.
To do this, we'll write the enable-plumbus `Template`:
```yaml=
apiVersion: combo.io/v1alpha1
kind: Template
metadata:
name: enable-plumbus
spec:
template: |
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: SUBJECT-enable-plumbus
namespace: NAMESPACE
subjects:
- kind: User
name: SUBJECT
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: plumbus-user
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: control-plumbus
namespace: NAMESPACE
subjects:
- kind: ServiceAccount
name: operator
namespace: plumbus-operator
roleRef:
kind: ClusterRole
name: plumbus-controller
apiGroup: rbac.authorization.k8s.io
parameters:
- key: SUBJECT
- key: NAMESPACE
```
enable-plumbus describes the manifests to add to a cluster that will enable a set of clients _and_ the operator itself to interact.
After adding the enable-plumbus `Template` to the `manifests` directory of our bundle, we need to replace the plumbus-operator `ClusterRoleBinding` with a `RoleBinding` to restrict the operator's initial set of permissions to the plumbus-operator namespace:
```yaml
apiVersion: rbac.authorization.k8s.io
kind: RoleBinding
metadata:
name: plumbus-operator
namespace: plumbus-operator
subjects:
- kind: ServiceAccount
name: operator
namespace: plumbus-operator
roleRef:
kind: ClusterRole
name: plumbus-controller
apiGroup: rbac.authorization.k8s.io
```
> **Note:** We could have also included a default `Combination` instead of the plumbus-operator `Role`. The effect on privileges would have been identical.
Before packaging the bundle for distribution, let's go over its new content:
- plumbus-operator `Namespace`: where the operator controller will live
- controller`Deployment`: deploys the operator controller to the plumbus-operator `Namespace`
- operator`ServiceAccount`: identity used by the operator controller
- fleem `CustomResourceDefinition`: defines the namespaced `Fleem` custom resource, used only by the controller in the plumbus-operator namespace
- plumbus `CustomResourceDefinition`: defines the namespace scoped `Plumbus` custom resource, consumed by plumbus-operator clients; everyone knows what it does, so there is no reason to explain it.
- fleem-controller`Role`: grants `*` on `resources: ["fleem"]` in the plumbus-operator `Namespace`
- plumbus-controller `ClusterRole`: grants `*` on `resources: ["plumbus/status"]`
- plumbus-user`ClusterRole`: grants`*` on `resources: ["plumbus"]`, `read` on `resources: ["plumbus/status"]`, and [aggregates]() to the plumbus-controller `ClusterRole`
- operator `RoleBinding`: binds the fleem-controller `Role` to the operator `ServiceAccount` in the plumbus-operator namespace
- plumbus-operator `RoleBinding`: binds the plumbus-controller `ClusterRole` to the operator `ServiceAccount` in the plumbus-operator namespace
- enable-plumbus `Pattern`: provides a template for stamping out the resources required for client and operator to interact post-install.
Once that checks out, go ahead and repackage/install the bundle.
After installation, we should be able to find the `Template` on cluster:
```sh
$ kubectl get template enable-plumbus
TODO
```
We can now can create a `Combination` to enable the operator for any combination of user/namespace we desire.
For example, if we wanted to enable the plumbus-operator for users Paul, Gurney, and Gaius the namespaces arrakis and caladan, we'd create the following `Combination`:
```yaml=
apiVersion: combo.io/v1alpha1
kind: Combination
metadata:
name: enable-plumbus-house-atreides
spec:
pattern:
name: enable-plumbus
arguments:
- key: SUBJECT
values:
- paul
- gurney
- gaius
- key: NAMESPACE
values:
- arrakis
- caladan
```
Under the hood, this will create the following `RoleBindings`:
| Namespace | RoleBinding |
| --------- | ----------- |
| arrakis | control-plumbus |
| arrakis | paul-enable-plumbus |
| arrakis | gurney-enable-plumbus |
| arrakis | gaius-enable-plumbus |
| caladan | control-plumbus |
| caladan | paul-enable-plumbus |
| caladan | gurney-enable-plumbus |
| caladan | gaius-enable-plumbus |
> **Note:** Instead of referencing users in `spec.subjects` directly, we could have referenced a `Group` containing them. This would have resulted in a smaller number of RoleBindings. For more info, see the [combo docs](https://github.com/operator-framework/combo).
Google Drive
Gist
Clipboard
Sign in with Wallet
