# Controller-Runtime Dynamic Cache Proposal Design Doc

## Motivation
Currently, Operators/Controllers require a certain set of permissions defined by the Operator/Controller author that are essentially static. If the Operator/Controller does not have these defined permissions, it will crash, infinitely loop, or enter some other unrecoverable state. Whenever an Operator/Controller enters one of these states, the only way to resolve it is to grant the RBAC and then restart it.

The key problem with this pattern is that cluster admins have to decide between ensuring the security of their cluster or installing an operator/controller. This design doc will show how a dynamic cache may be used by an operator/controller author to ensure:
- Their operator/controller is resilient to RBAC changes (i.e doesn't crash, block, or enter some other unrecoverable state)
- Their operator/controller uses a minimal permissions approach
- Cluster admins can install their operator/controller and still secure their cluster by limiting the operator/controller permissions via RBAC
- Their operator/controller is more approachable by industries where cluster security is extremely important

## Requirements

### User Stories

- As an operator author I want to develop an operator that can handle changes in permissions so that cluster admins can use Role Based Access Control (RBAC) to scope the permissions given to my operator.
- As a cluster admin I want to use RBAC to scope operator permissions so that I can ensure my cluster is as secure as possible while also gaining the benefits of using operators.

### Goals
- Operator/Controller authors can to add/remove watches
- Operator/Controller authors can specify how their watches should react when an error in the underlying informer occurs (i.e set the `WatchErrorHandler`)
- Operator/Controller authors can create minimal watches (i.e only create watches as needed and remove them when no longer needed)
- Cluster admins can change the RBAC applied to an Operator/Controller's ServiceAccount (using this cache implementation) without the operator entering a state that requires manual intervention

### Non-Goals
- The new cache implementation should not handle reconfiguration of watches. The cache should just be a way to keep track of watches, and the operator/controller author is responsible of handling any logic that would add, remove, or reconfigure watches.

## Proposal

Since there has already been some work done to create watches that can be added/removed/started/stopped in [controller-runtime#2159](https://github.com/kubernetes-sigs/controller-runtime/pull/2159) this proposal will focus on ways this can be built upon to meet the goals defined in the Requirements section above.

### Expose options for setting `WatchErrorHandler` when creating a watch
Building upon [controller-runtime#2159](https://github.com/kubernetes-sigs/controller-runtime/pull/2159) an operator/controller author should be able to set a [`WatchErrorHandler`](https://pkg.go.dev/k8s.io/client-go/tools/cache#WatchErrorHandler) to run error handling logic if the watch encounters some kind of error. The below example shows how an operator/controller author might create a watch that can be automatically stopped if it encounters a permissions error:

// For example purposes we are setting this to the source.Source interface.
// In reality, this can be replaced by any source.Source type
src := &source.Source{}

// For example purposes we are setting this to the handler.EventHandler
// interface. In reality, this can be replaced by any 
// handler.EventHandler type
eventHandler := &handler.EventHandler{}

watchErrHandler := func(r *Reflector, err error) {
    if apierrors.IsForbidden(err) {
        log.Info("Stopping watch due to permissions error")
        src.Stop()
    }
}

// This example is showing setting the `WatchErrorHandler` via a new
// parameter on the `Controller.Watch` function. There may be a more
// elegant approach that can be taken if this is not an ideal approach.
// The new Watch function definiton might look like:
// Watch(src source.Source, eventhandler handler.EventHandler, cgocache.WatchErrorHandler, predicates ...predicate.Predicate) error
ctrl.Watch(src, eventHandler, watchErrHandler)

:::warning
**Note**: I suspect this can already be accomplished by using the source.Informer source type. It *may* be worth making this possible for other sources as well in case an operator author doesn't want to manually create the informer and use a more simple approach like the source.Kind source type.
:::

### Create a new `cache.Cache` implementation

The next step in helping to achieve the goals in this proposal is to create a new `cache.Cache` implementation to help achieve the goal of establishing minimal watches.

:::info
A **minimal watch** is a watch that is filtered down to watch as few resources as possible. An example of a minimal watch would be establishing a watch that only populates a singular child resource created by the reconciliation of a CR.

<details>
<summary>Code Example of Creating a Minimal Watch</summary>

// For example purposes the client.Object interface is going to be used 
// to represent a custom resource. In reality this can be replaced with any object
// that satisfies the client.Object interface
cr := &client.Object{}

// For example purposes the kubernetes.Interface interface is going to be used 
// to represent a kubernetes client. In reality this can be replaced with any client
// that satisfies the kubernetes.Interface interface
client := &kubernetes.Interface{}

// Create the list options for filtering the resources. In this example,
// we are assuming that the CR name is "foo" and that the child resource
// created is a Deployment that should have a label `cr-ref=foo`
tweakListOpt := func(lo *v1.ListOptions) {
    lo.LabelSelector = labels.Set(map[string]string{"cr-ref": "foo"}).String()
}

// Create a SharedInformerFactory to ensure we can 
// filter the informer options as much as we want.
// For this example, we are assuming the CR was created in Namespace
// "bar" and that the child Deployment resource will be created in the same Namespace
sharedInfFact := informers.NewSharedInformerFactoryWithOptions(client, 10*time.Minute, informers.WithNamespace(cr.Namespace), informers.WithTweakListOptions(listOpt))

// Get the Deployment informer that should only be watching Deployments
// in the "bar" namespace with the label `cr-ref=foo`
deployInf := sharedInfFact.Apps().V1().Deployments().Informer()

// Now we can create the source to use with the `Watch()` function
src := &source.Informer{Informer: deployInf}

...
    

</details>
:::

The new `cache.Cache` implementation is necessary because when following the minimal watch pattern, you may need to create multiple informers for the same GVK. This may require configuring some way to set a mapping of an informer key (uid) --> informer to ensure the same exact informer isn't created more than once, but can instead have a reference count updated. This could look something like:

type ScopeInformer struct {
	informer cache.Informer
	dependents map[types.UID]metav1.Object
}

// Informers is a mapping of a string
// (meant to be a unique string) to a ScopeInformer
type Informers map[string]*ScopeInformer

// GvkToInformers is a mapping of GVK to Informers
type GvkToInformers map[schema.GroupVersionKind]Informers

In this implementation, the "dependents" would be any CRs that are reconciled that need the information provided by this informer. Using a map with the resource UID as the key, we ensure that we only track each CR using this informer once.

Examples in my current PoC:
- Example of `cache.Cache.GetInformer()` implementation: https://github.com/everettraven/telescopia/blob/main/pkg/cache/cache.go#L275-L362
- Example of creating an informer like this: https://github.com/everettraven/scoped-operator-poc/blob/poc/telescopia/controllers/memcached_controller.go#L298-L327

For more information on how this new `cache.Cache` implementation and usage could look, I recommend taking a look at the following resources:
- everettraven/telescopia (my `cache.Cache` implementation PoC):
- [Cache Design Doc](https://hackmd.io/@TJSe6k8SSICpsJVsxuD73Q/rJoS27yVo)
- [Cache Implementation](https://github.com/everettraven/telescopia/tree/main/pkg/cache)
- everettraven/scoped-operator-poc (my usage PoC):
- [Controller Using everettraven/telescopia](https://github.com/everettraven/scoped-operator-poc/blob/poc/telescopia/controllers/memcached_controller.go)
- [Diff between a regular controller and one using the PoC](https://github.com/everettraven/scoped-operator-poc/compare/main...poc/telescopia) (**Note**: These changes could probably be further optimized but this should help showcase the general idea.)