--- title: "Packaging API dependency management" authors: [ "Joao Pereira <joao.pereira@broadcom.com>" ] status: "draft" approvers: [ "Dmitriy Kalinin <dkalinin@vmware.com>" ] --- # Packaging API dependency management ## Problem Statement kapp-controller is being used in many contexts to deploy applications that users want to have in their clusters. One main problem that was never addressed was how can a user install multiple packages or even how can a user say that a particular package needs other packages or resources to be present in the cluster in order for it to work. The primary way this was being accomplished today was to create a Package that was able to create PackageInstalls for these other Packages, since kapp-controller does not do any validation of the resources that are being installed in the cluster. We believe that this method is effective when the package author is responsible for packaging all these other package, but in a world where we can have multiple package authors and each one of them only manages a subset of these packages we need a better solution. In this proposal we are going to propose some changes to the Packaging API that will allow the Package Authors to reference other Packages that they depend on as well as expand the API in a way that would allow them to be even more generic, by creating dependencies on the APIs present in the cluster. ## Terminology / Concepts *GVK* - Group Version Kind, it is a way in Kubernetes to define a particular API ## Proposal In this proposal we will describe the changes that are necessary to enable the Packaging API to support dependency management. With these changes we ### Goals and Non-goals #### Goals - Evolve the current Packaging API to support dependency management #### Non-goals - Define implementation details ### Specification / Use Cases Updated Package API ```yaml= apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: internalpackages.internal.packaging.carvel.dev spec: group: internal.packaging.carvel.dev names: kind: InternalPackage listKind: InternalPackageList plural: internalpackages singular: internalpackage scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: ... spec: properties: ... installationType: description: Different types of installation available. type: string enum: - onePerCluster - onePerNamespace - noRestrictions default: noRestrictions dependencies: description: Dependencies list of all the dependencies of the current package items: &dependencies description: Dependency contains the possible dependencies restrictions properties: gvk: description: GVK contains the version of an API that the Package depends on properties: apiVersion: description: APIVersion contains group and version of the API type: string kind: description: Kind contains the kind of the API type: string required: - apiVersion - kind type: object or: description: OrDependency contains a list of multiple dependencies that only 1 needs to be present type: array items: *dependencies package: description: Package contains the reference to a Package and version properties: refName: description: RefName Reference of the Package type: string version: description: Version Version of the Package type: string type: object type: object type: array provides: description: List of GVKs that this Package provides items: description: GVK API provided properties: apiVersion: description: APIVersion contains group and version of the API type: string kind: description: Kind contains the kind of the API type: string required: - apiVersion - kind type: object type: array ``` Updated PackageInstall API ```yaml= apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: packageinstalls.packaging.carvel.dev spec: group: packaging.carvel.dev names: categories: - carvel kind: PackageInstall listKind: PackageInstallList plural: packageinstalls shortNames: - pkgi singular: packageinstall scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: A Package Install is an actual installation of a package and its underlying resources on a Kubernetes cluster. It is represented in kapp-controller by a PackageInstall CR. A PackageInstall CR must reference a Package CR. properties: ... spec: properties: ... dependencies: description: Dependencies that need overrides properties: install: default: true description: When install is set to false it will only validate the present of the dependencies, when set to true it will try to install the Packages type: boolean override: description: Override the versions of packages to be installed items: properties: refName: type: string versionSelection: properties: constraints: type: string prereleases: properties: identifiers: items: type: string type: array type: object type: object type: object type: array values: description: List of secrets that contain values that will be used for package to be installed, only needed if Packages should be installed items: properties: alias: description: When Alias is set at the Package level this field should be used type: string refName: description: Reference to the Package the secret should be used on type: string secretRef: description: Reference to the secret properties: key: type: string name: type: string type: object required: - secretRef type: object type: array type: object status: properties: ... dependencies: description: Array will all the fullfilled dependencies information items: properties: created: description: Was this dependency created by the current PackageInstall type: boolean package: properties: packageInstallRef: description: PackageInstall reference type: string refName: description: Name of the Package type: string version: description: Exact version that was installed or is present in the cluster type string type: object gvk: description: Some of the information in this section might not be surfaced if the GVK is native to kubernetes or installed using a different method properties: providedBy: description: Package name that provides this GVK type: string version: description: Version of the Package type: string type: array ``` Package Example: ```yaml= apiVersion: data.packaging.carvel.dev/v1alpha1 kind: Package metadata: name: pkg.test.carvel.dev.1.0.0 spec: refName: pkg.test.carvel.dev version: 1.0.0 installationType: onePerCluster dependencies: packages: - or: - gvk: apiVersion: some.api.org/v1 kind: MainAPI - package: refName: pkg2.test.carvel.dev version: ">=1.0.0" - package: refName: pkg3.test.carvel.dev version: 0.1.0 - package: refName: pkg4.test.carvel.dev version: 0.2.0 optional: true ``` PackageInstall Example: ```yaml= apiVersion: packaging.carvel.dev/v1alpha1 kind: PackageInstall metadata: name: pkg-demo spec: serviceAccountName: default-ns-sa packageRef: refName: pkg.test.carvel.dev versionSelection: constraints: 1.0.0 dependencies: override: - refName: pkg1.test.carvel.dev versionSelection: contraints: 1.0.1 - refName: pkg4.test.carvel.dev install: false # Will not install this Package values: - refName: pkg1.test.carvel.dev secretRef: name: pkg1.dev-values - refName: pkg3.test.carvel.dev secretRef: name: pkg3.dev-values status: dependencies: - packageInstallRef: pkg1.test.carvel.dev.1.0.0 - packageInstallRef: pkg3.test.carvel.dev.1.0.0 ``` Example of a PackageInstall that is automatically created ```yaml= apiVersion: packaging.carvel.dev/v1alpha1 kind: PackageInstall metadata: name: pkg3.test.carvel.dev.1.0.0 annotations: kapp-controller.carvel.dev/owner: "PackageInstall/pkg-demo" spec: serviceAccountName: default-ns-sa packageRef: refName: pkg3.test.carvel.dev versionSelection: constraints: 1.0.0 values: secretRef: name: pkg3.dev-values ``` #### Dependency resolution The dependency resolution will be done one PackageInstall at a time, basically each PackageInstall will only create the PackageInstalls that it directly depends on. ex: ``` When pkg depends on pkg1 and pkg3, while pkg1 depends on pkg2 kapp-controller when it is processing pkg it will only create PackageInstalls for pkg1 and pkg3. pkg -> pkg1 -> pkg2 -> pkg3 ``` For future versions we can contenplate the possibility of each PackageInstall checking if the dependencies it has can be fulfilled with the Packages present in the Cluster. ##### Alternative dependency resolution Each package would be responsible for creating all the PackageInstall that it depends on as well as all the PackageInstalls that the dependencies might depend on ex: ``` When pkg depends on pkg1 and pkg3, while pkg1 depends on pkg2 kapp-controller when it is processing pkg it will create PackageInstalls for pkg1, pkg2 and, pkg3. pkg -> pkg1 -> pkg2 -> pkg3 ``` #### Use Cases ##### Package depends on another Package - When the Package that match the dependency is already installed, no new installation is needed - When the dependent Pacakage is not installed, it will try to install it - When a dependent Package is already installed but the version does not fullfil the requirements of the current PackageInstall, kapp-controller will try to install the version that matches - When there is no Package that fullfills the dependency of a PackageInstall the installation will fail - ##### Two Packages depend on the same Package In this scenario the first PackageInstall that creates the dependent PackageInstall will own it. This means that the second PackageInstall will not need to create the dependent PackageInstall ##### A Package uses "or" dependency In this case kapp-controller will go in order to try to fulfil this dependency. This scenario will be more clear when there is a GVK dependency or if there are multiple dependencies and the user decided not to install dependencies #### Future explorations ##### Automatic search and installation of packages In case of gvk dependencies or other dependencies that we can create in the future it is possible for a package to automatically find a Package that could provide the needed functionality. ```yaml= apiVersion: data.packaging.carvel.dev/v1alpha1 kind: Package metadata: name: some-pkg.test.carvel.dev.1.0.0 spec: refName: some-pkg.test.carvel.dev version: 1.0.0 dependencies: - gvk: group: api.carvel.dev kind: Resource version: v1 --- apiVersion: data.packaging.carvel.dev/v1alpha1 kind: Package metadata: name: provider.carvel.dev.1.0.0 spec: refName: provider.carvel.dev version: 1.0.0 provides: - gvk: apiVersion: api.carvel.dev/v1 kind: Resource ``` In the example above the author of the `some-pkg.test.carvel.dev` does not have to specify a particular Package to install because kapp-controller would be able to automatically find and install the needed package ##### Force deletion of dependent packages When creating a PackageInstall the user can define if they want to delete the dependent packages when the current PackageInstall is removed. ##### Enforce dependent PackageInstall cannot be modified The current proposal assume that by convension the PackageInstalls created by kapp-controller should not be updated. Nevertheless we can explore the possibility of creating a webhook that will enforce this behavior. ### Other Approaches Considered _Mention of other reasonable ways that the problem(s) could be addressed with rationale for why they were less desirable than the proposed approach._ ## Open Questions - What happens if the Resource did not change version but new fields where added and the package requires them but that version of the Resource is not present in the cluster? ## Answered Questions _A list of questions that have been answered._
Last changed by  
 
Carvel
A light-weight test of hackmd for Carvel's purposes.
59

Read more

Emitting comments from ytt templates

Summary ytt should have the facility to safely emit YAML comments from YAML templates. Today, the people who need to read and understand the output of ytt templates are having a bad time. Without the ability to weave in context for the humans (in the form of YAML comments), they must burn hours sleuthing back through whatever process(es) executed ytt, to find their inputs. Better would be the ability to include this additional information in the form of comments in the output. They could not only leave more explicit breadcrumbs, but name the source of a change or rationale for a given value. This not reduces the guesswork, but eliminates much of the toil in gathering the facts.

12/20/2023
OLM integration proposal

_This is a short summary of the problem that exists, why it needs to besolved: what specific needs are being met. Compelling problem statementsinclude concrete examples (even if only by reference). How exactly the proposalwould meet those needs should be located in the “Proposal” section, not this one.The goal of this section is to help readers quickly empathize with the target users’current experience to motivate the proposed change.

10/24/2023
Untitled

Title: Emitting comments in ytt Status: Scoping (Scoping | Pre-Alpha | In Alpha | In Beta | GA | Rejected) Originating Issue: ytt#63 Problem StatementCustomer Impact Proposal Emitting YAML Comments Emitting ytt templating

6/15/2023
Untitled

Title: "Emitting comments in ytt" UStatus: Scoping (Scoping | Pre-Alpha | In Alpha | In Beta | GA | Rejected) Originating Issue: ytt#63 Problem StatementCustomer Impact Proposal Emitting YAML Comments Emitting ytt templating

6/13/2023
Read more from Carvel
Published on HackMD