# Embedded Bundle Support

## Pre-Requisite Work

- [Bundle and ConfigMap Immutability](https://github.com/operator-framework/rukpak/pull/157/)

## Summary

Introduce support for embedding Bundle's within the BundleInstance (BI) API so users and higher level controllers can interact with a single API in the rukpak stack.

The BI API will be updated to support the dynamic creation of Bundle resources through this embedding mechanism.

> Supporting the static and dynamic creation of Bundle resources is considered out-of-scope for this design due to current UX concerns around this bi-lateral behavior.

When a user creates a BI resource, and that resource contains a reference to an embedded Bundle, all provisioners that implement the BI API are responsible for creating the Bundle according to the desired specification. The Bundle controller is still responsible for sourcing and unpacking Bundle resources.

In the case the Bundle has already been generated, the BI controller is responsible for ensuring that the current state of the Bundle resource eventually matches the desired state of the embedded BundleSpec.

The BI controller is also responsible for injecting owner references for generated, dependent Bundle resources.

See the upstream issue [^1] for more details and discussion.

## Goals

- Enable users and higher level controllers to interact with a single API
- Embedded Bundles are garbage collected when a BundleInstance resource has been deleted
- Embedded Bundles are garbage collected when a BundleInstance resource successfully pivots to a new Bundle resource

## Stetch-Goals

- Support Bundle and BundleInstances referencing different provisioner unique IDs
- Support more advanced rollout strategies when pivoting between Bundle resources

## Non-Goals

- Support the static and dynamic creation of Bundle resources in the BI API
- Support removing existing BundleInstance resources and adopting all their resources with new BundleInstances

## Proposal

Enable the dynamic creation and management of Bundle resources by updating the BI API to mirror the core Deployment Kubernetes resource. This can be done through the introduction label selection and templating of embedded Bundles to the BI API. See [Examples](#Examples) for a concrete embedded Bundle reference.

### Go API Type Changes

Update the BundleInstance API, and replace the `spec.BundleName` field with a custom `spec.Bundle` struct that closely resembles the core Deployment resource type definitions:

// BundleInstanceSpec defines the desired state of BundleInstance
type BundleInstanceSpec struct {
    // Label selector for Bundles. Existing Bundles that match this label
    // selector must match the label selector specified in the template field.
    Selector *metav1.LabelSelector `json:"selector"`
    // ProvisionerClassName sets the name of the provisioner that should reconcile this BundleInstance.
    ProvisionerClassName string `json:"provisionerClassName"`
    // Template describes the generated Bundle that this instance will manage.
    // +kubebuilder:pruning:PreserveUnknownFields
    Template *BundleTemplate `json:"template"`
}

// BundleTemplate defines the desired state of a Bundle resource
type BundleTemplate struct {
    // Standard object's metadata.
    // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
    // +optional
    metav1.ObjectMeta `json:"metadata,omitempty"`
    // Specification of the desired behavior of the Bundle.
    // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
    // +optional
    Spec BundleSpec `json:"spec,omitempty"`
}

### Reconciliation Logic

> Note: the following reconciliation logic is centered around rukpak's plain provisioner implementation.

<!-- Validation of the label selector specified to avoid metav1.All scenarios? -->

Update the plain provisioner's BI controller logic to get a list of Bundles that exist on cluster that match the BI's spec.Selector label selector value. For each of the Bundles returned, check whether any Bundle matches the desired spec.Template Bundle template value.

In the case that multiple Bundles match the desired template, sort the list of Bundles by metadata.CreationTimestamp, and chose the Bundle that has the newest value.

> Note: we may need to filer out spec.Template.Metadata values to properly compare the current and desired state

When the controller is unable to find a Bundle that matches the desired specification, it's responsible for generating a new Bundle resource that matches the spec.Template values. When generating Bundle resources, use the BI's metadata.Name value and append a hash to avoid collision with existing Bundle names.

Due to Bundle immutability, any Bundle that differs from the desired specification cannot be updated to match the Bundle template present in the BI resource. Instead, the BI controller is responsible for creating a new Bundle resource using the desired Bundle template, and waiting for that newly created Bundle resource to report an unpacked state before deleting the previous, out-of-date Bundle resource(s).

In the future, the BundleInstance API may support more advanced pivoting or rolling mechanisms.

## Example(s)

### Embedded Git-based BundleInstance

apiVersion: core.rukpak.io/v1alpha1
kind: BundleInstance
metadata:
  name: combo
spec:
  provisionerClassName: core.rukpak.io/plain
  selector:
    matchLabels:
      app.kubernetes.io/name: combo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: combo
    spec:
      provisionerClassName: core.rukpak.io/plain
      source:
        git:
          ref:
            branch: main
          repository: https://github.com/operator-framework/combo
        type: git

## Alternative Implementations

### BundleInstance supports embedding Bundles and existing Bundles

This alternative design is centered around the BundleInstance API supporting the static and dynamic creation of Bundle resources. This approach would mirror behavior that's present in core Kubernetes with the PersistentVolume and PersistentVolumeClaim resources [^3] where users can provisioner volumes statically, or dynamically when referencing StorageClass resources that support dynamic volume provisioning.

The BundleInstance API can be updated to use the same `spec.Bundle` custom structure, where only the `spec.Bundle.Name` is a required field. The BundleInstance controller would then be responsible for checking whether the nested `spec.Bundle.Spec` field has been specified by a user, and treat that resource as an embedded Bundle and dynamically manage that Bundle resource. Else, the controller would continue with the existing reconciliation behavior we've seen in the rukpak releases.

This approach was ultimately rejected due to initial UX concerns.

### Bundles are mutable

This alternative design takes into the same proposal detailed above, but instead of relying on Bundle/ConfigMap immutability as pre-requisite work, it treats Bundle resources as mutable entities. When the BI controller sees an existing, embedded Bundle resource, and the current state of the Bundle resource present on cluster doesn't match the desired spec outlined in the BI's spec.Bundle.Spec field, the controller attempts to update that Bundle resource with the updated spec. The Bundle controller would then be responsible for the unpacking and sourcing process.

### Introduce new API(s)

Introduce a new API that combines both of these two rukpak API primitives [^2].

And this new API could aim to emulate the ReplicaSet or Deployment resource(s) seen in core Kubernetes:

> Note: click the "details" tab to toggle the hidden YAML resources

<details>

apiVersion: core.rukpak.io/v1alpha1
kind: BundleInstanceSet ## Note: naming TDB
metadata:
  name: combo
spec:
  selector:
    matchLabels:
      app: combo
  bundle:
    template:
      metadata:
        name: combo
    spec:
      provisionerClassName: core.rukpak.io/plain
      source:
        type: git
        git:
          repository: https://github.com/operator-framework/combo
          ref:
            branch: main

</details>

And that higher level API would get translated into the following, lower level rukpak resources:

<details>

apiVersion: core.rukpak.io/v1alpha1
kind: Bundle
metadata:
  name: combo
spec:
  provisionerClassName: core.rukpak.io/plain
  source:
    type: git
    git:
      repository: https://github.com/operator-framework/combo
      ref:
        branch: main
---
apiVersion: core.rukpak.io/v1alpha1
kind: BundleInstance
metadata:
  name: combo
spec:
  provisionerClassName: core.rukpak.io/plain
  bundleName: combo

</details>

This approach was ultimately rejected due the current intention that the BundleInstance API more resembles the responsibilities of the Deployment resource, vs. the BundleInstance API resembles a lower level core Kubernetes API like a Pod.

[^1]: <https://github.com/operator-framework/rukpak/issues/73>
[^2]: <https://github.com/operator-framework/rukpak/issues/73#issuecomment-1084728436>
[^3]: <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#provisioning>