Helm Improvement Proposal: Pluggable Templating

--- hip: ??? title: "Helm Improvement Proposal: Pluggable Templating" authors: [ "John Ryan <ryanjo@vmware.com>" ] created: "2020-12-10" type: "feature" status: "draft" --- ## Abstract _(a short (~200 word) description of the technical issue being addressed.)_ ## Summary Allow Chart Authors to the ability to opt-in to an alternative rendering engine. This allows shops that use a variety of tools to wrangle manifests the ability to aggregate them under the Helm workflow. ### Single Chart Happy Path Briefly, 1. Engine author publishes a "Helm Engine" (i.e. a WASM executable) on Artifact HUB. 2. Chart author specifies an "Helm Engine" in optional Chart metadata 3. Helm user installs the chart 1. Helm calculates the Engine source URL from the Chart metadata and downloads it to a local cache. 2. Helm's Engine delegates to WASM executable for rendering. `Chart.yaml` ```yaml ... Engine: group: name: version: ... ``` - (e.g. `https://artifacthub.io/packages/helm-engine/carvel/ytt` to `$HELM_ENGINE_CACHE`). Concerns: - Questions: - ## Motivation _(The motivation is critical for proposals that want to change Helm in a significant way. It should clearly explain why existing features are inadequate to address the problem that the proposal solves, as well as who and what profiles/roles are impacted by this proposal. Proposal submissions without sufficient motivation may be rejected outright. )_ Native Helm Templating (gotpl+Sprig) provides a rich set of features aimed at externalizing commonly varying bits of configuration. Templating as a technique has its limits. Externalize enough configuration and the input values begin to mirror the underlying API. To mitigate this effect, authors can capture less-commonly-used configurations and aspect-oriented configuration in patches. Where a patching operation is more appropriate, Helm 3.1's post-render hook provides the access to the Helm workflow (TODO: term?) to apply such changes. However, as Kubernetes continues to see uptake in contexts with more complex deployments, Admins need more sophisticated ways of managing the growing piles of manifests. (TODO: cluster-api example of webhook deployment change). A new role emerges: the Enterprise Kubernetes Admin. _(TODO: elaborate on the specific needs of this proto persona?)_ Attempting to address this rising complexity for this specific cohort within native Helm templating would unnecessarily complicate the user experience for the rest of the community. Instead, provide the ability for this subset to opt-in to a sophisticated Kubernetes manifest management on a per-case basis. _(... or User Story format? ...)_ **As an Enterprise Kubernetes Admin** managing an ever-growing complex set of deployments, \ **I want to** build my resources leveraging some features of programming languages like libraries (DRY principles), values validation (better UX), interacting dynamically with (e.g. reading values from my environment) 1. Kustomize patches become unmanageable - include evidence: kind and kubeadm dropping kustomize dependency - ## Rationale _(The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other projects. )_ ### Design Concerns - Installation - In a happy path execution, the Helm User should be blissfully unaware which templating engine was used to produce their manifests. - - Safe Execution - no side-effects - no network access - no access to time or explicit source of randomness - exceptions/permissions granted by user? - if so, then provenance is required - Unified User Experience - When rendering fails, Helm core ought to have enough information to guide the Helm User: - where the inputs malformed? or invalid? - if so, which ones? - - SPI - define inputs and outputs - deterministic — identical inputs must result in identical outputs - Versioning - Helm User must be able to upgrade to any version of Helm using the same version of Charts. - - ## Specification _(The technical specification should describe the syntax and semantics of any new feature. If the proposal describes a standard, the specification should be detailed enough to describe how other community members may re-implement the standard in their product. )_ ### Installation Options: - embedded: in the chart (or as a common sub-chart?) - explicit: as a plugin, by the end-user - implicit: as a separate component, referenced version of engine is downloaded, verified, and installed on demand Consider "batteries included" plugin to leverage existing plugin "infrastructure". Questions: - **Q:** Would we allow for externalized configuration for the engine (e.g. flow environment vars, config files, etc)? Or require configuration reside in Charts (and passed in through SPI)? ### Helm Chart Lifecycle TODO: - [ ] review `EngineYard` interface (and associated) - [ ] survey popular templating tools for any other kinds of ins/outs. Details: - ins: - (ordered) list of named input streams (templates, data files, config files) - input Values — arguments for templates - outs: - rendered manifests - interruptible / timeout ## Backwards Compatibility _(The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other projects.)_ ## Reference Implementation _(The reference implementation must be completed before any proposal is given status "final", but it need not be completed before the proposal is accepted. While there is merit to the approach of reaching consensus on the specification and rationale before writing code, the principle of "rough consensus and running code" is still useful when it comes to resolving many discussions of API details.)_ ### Observations Through POC - Values handling is (slightly?) disjoint. Today, "coalescing" values is done separately from template rendering. - This allows Helm to inject values relating to workflow handling ( ) - These two operations would need to be combined behind a higher-level API call -