--- tags: recursive bundles, imgpkg --- # imgpkg Recursive Bundles Proposal - Proposal Status: **Draft**, In Review, Accepted, Rejected # Table of Contents [TOC] --- # Recursive Bundles Proposal ## Use Case **As** A Software Packager **I want to** Create a bundle to distribute the multiple applications that are part of a deployment **So that** Application consumers can retrieve all needed software from a single place ### Current Pain Points With today's approach, how might an Application consumer get multiple applications that are part of a single deployment? - Single Bundle with all the applications 1. The Packager creates an `images.yml` with all the individual OCI Images for each application 2. The Packager collects from the Application Teams all the needed configuration for each application 3. The Packager is responsible for organizing the configuration for all the Applications 4. The Packager creates a single bundle. This approach is complicated to maintain in the long run for the Packager. In case an Application Team creates a new update on the application, this process would have to be repeated by the Packager. This process can become a bottleneck because the Packager would need to have a deep understanding of each application and the configuration needs. - Multiple bundles: 1. Each Application Team would generate their Bundle with their application 2. The Packager collects all the Bundles SHA 3. The Packager creates a document with all the Bundle SHA 4. The Packager creates overlay's and adds some utilities to use during installation 5. The Packager generates a Bundle with these artifacts 6. The Consumer retrieves the Bundle generated by the Packager 7. The Consumer retrieves every other Bundle that is required This approach has challenges for both Packager and Consumer The Consumer would have to download multiple separate bundles and follow the instructions from the Packager to install all the applications The Packager would have to collect the information about the applications and generate an installation guide for the Consumer. ### Mitigate Pain Points In this proposal, we adapt `imgpkg` to try to mitigate some of the current pains. Given that `imgpkg` already supports [bundles](https://carvel.dev/imgpkg/docs/latest/resources/#bundle) with a simple extension that allows bundles to contain other bundles can be created to help with the problem. The workflow that this change enables is the following: 1. Each Application Team generates a bundle with all the needed images and the configuration 2. The Packager would collect the applications Image Bundle SHA's for all applications 3. The Packager can provide some overlay's for the configuration and add some utilities that are used during installation 4. Create a Bundle that contains all application Image Bundles artifacts generated in the previous point 5. The Application Consumer would be given a single Bundle Image that can be downloaded and consumed The benefits of this approach versus the current state are - Each Application Team manages their application's Bundle. The team that builds the application will manage all the requirements - The Packager just needs to collect the Bundle SHA for each application - The Consumer can get all the needed resources from a single Bundle to install all the applications needed # Proposed changes In this section, the proposal will only talk about parts that will be changed based on the Recursive Bundle solution ## Pushing a bundle The major proposed change in this section is to remove the validation done today that checks if the bundle being pushed contains bundles in the `.imgpkg/images.yml` Lock file. ### Required changes #### Validation removal Today we validate to ensure that when we are pushing a bundle, the images associated with it are not bundles. The error message that is shown read `Error: Expected image lock to not contain bundle reference:` As part of this proposal, this check no longer will be done. #### Ignore bundle specific folder When a bundle is pulled, and contain recursive bundles, the folder `.bundles` is created in the root of the bundle. As part of this proposal, we should ensure that if a user tries to create a bundle and it contains a folder called `.bundles` `imgpkg` should ignore the folder and provide the following message ``` Warning: .bundles folder could not be added to the bundle. This folder is used to store configurations of recursive bundles and cannot be checked into the bundle ``` ### Example Assuming the provided example in [here](https://github.com/k14s/design-docs/tree/002-recursive-bundles/imgpkg/002-recursive-bundles/examples) ```= $ imgpkg push -b ghcr.io/k14s/design-docs/simple-app-bundle -f examples/bundle-1 dir: . dir: .imgpkg file: .imgpkg/bundle.yml file: .imgpkg/images.yml file: Readme.md file: config.yml Pushed 'ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4' Succeeded ``` ## Copy a bundle ### From Repository to Tar #### Required changes ##### Retrieve all images for all bundles The most significant change in this operation is that when we copy the images `imgpkg` will have to traverse all the bundles to collect all the images to copy. ##### Layer deduped on disk When creating the tar file in disk `imgpkg` need to ensure that we do not store duplicated layers in disk #### Example Given that we create a bundle using the command `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` ```= imgpkg copy -b ghcr.io/k14s/design-docs/simple-app-install-package --to-tar recursive-bundle.tar copy | exporting 4 images... copy | will export ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 copy | will export ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 copy | will export ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 copy | will export ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f copy | exported 4 images copy | writing layers... copy | done: file 'manifest.json' (35.107µs) copy | done: file 'sha256-81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d.tar.gz' (46.817µs) copy | done: file 'sha256-9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1.tar.gz' (99.194µs) copy | done: file 'sha256-87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917.tar.gz' (119.416µs) copy | done: file 'sha256-8ece9ac45f2b7228b2ed95e9f407b4f0dc2ac74f93c62ff1156f24c53042ba54.tar.gz' (370.872834ms) Succeeded ``` ### From Tar to Repository #### Required changes ##### Retrieve all images for all bundles The most significant change in this operation is that when we copy the images `imgpkg` will have to traverse all the bundles to collect all the images to copy. #### Example Given that we create a bundle using the command `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` Followed by `imgpkg copy -b ghcr.io/k14s/design-docs/simple-app-install-package --to-tar recursive-bundle.tar` ```= imgpkg copy --tar recursive-bundle.tar --to-repo some-other.registry.io/recursive-bundle copy | importing 4 images... copy | importing ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> some-other.registry.io/recursive-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0... copy | importing ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f -> some-other.registry.io/recursive-bundle@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f... copy | importing ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 -> some-other.registry.io/recursive-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4... copy | importing ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 -> some-other.registry.io/recursive-bundle@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035... copy | imported 4 images Succeeded ``` ### From Repository to Repository #### Required changes ##### Retrieve all images for all bundles The most significant change in this operation is that when we copy the images `imgpkg` will have to traverse all the bundles to collect all the images to copy. ##### Bundles/Images deduped When copying the images/bundles ensure that `imgpkg` does not try to copy the same image multiple times #### Example Given that we create a bundle using the command `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` ```= imgpkg copy -b ghcr.io/k14s/design-docs/simple-app-install-package --to-repo some-other.registry.io/recursive-bundle copy | exporting 4 images... copy | will export ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 copy | will export ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f copy | will export ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 copy | will export ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 copy | exported 4 images copy | importing 4 images... copy | importing ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> some-other.registry.io/recursive-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0... copy | importing ghcr.io/k14s/design-docs/utilities@sha256:ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f -> some-other.registry.io/recursive-bundle@sha256:ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f... copy | importing ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 -> some-other.registry.io/recursive-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4... copy | importing ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 -> some-other.registry.io/recursive-bundle@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035... copy | imported 4 images Succeeded ``` ## Pull a bundle ### Required changes #### Change folder structure When downloading a bundle that contains other bundles to disk using the `pull` command will work as currently, except for that it will download the nested bundles into a hidden folder called `.bundles`. ##### Folder structure The structure of the output folder will be ``` $ tree -a recursive-bundle . ├── .bundles │   ├── sha256-{SHA Of the First Nested Bundle} │   │   ├── .imgpkg │   │   │   ├── bundle.yml │   │   │   └── images.yml │   │   └── config2.yml │   └── sha256-{SHA Of the Second Nested Bundle} │   │   ├── .imgpkg │   │   │   ├── bundle.yml │   │   │   └── images.yml │   └── config1.yml ├── .imgpkg │   ├── bundle.yml │   └── images.yml └── config.yml 7 directories, 9 files ``` The folder name will be the sha256-{SHA} where SHA is the SHA256 of the bundle OCI Image. The mapping between the folder names and the origin images can be found in the `.imgpkg/images.yml` of the bundle that included this bundle ###### Cyclic nesting To ensure that there is not cyclic nesting in the disk `imgpkg` will flatten the bundle structure to 1 level. ![](https://i.imgur.com/zSlnzg7.png) In the image above Bundle 1, Bundle 2 and, Bundle 3 will be all in a single level inside the `.bundles` folder. ### Example Given that we create a bundle using the command `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` ```= $ imgpkg pull -b ghcr.io/k14s/design-docs/simple-app-install-package -o /tmp/recursive-bundle Pulling bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035' Bundle Layers Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1) Nested bundles Pulling Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4' (1/2) Extracting layer 'sha256:81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d' (1/1) Found 1 Bundle packaged Pulling Nested Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f' (1/1) Extracting layer 'sha256:9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1' (1/1) Pulling Nested Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f' (2/2) Skipped, already downloaded Locating image lock file images... The bundle repo (ghcr.io/k14s/design-docs/simple-app-install-package) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml) Updating all images in the ImagesLock file: pull-tmp/.imgpkg/images.yml + Changing all image registry/repository references in pull-tmp/.imgpkg/images.yml to ghcr.io/k14s/design-docs/simple-app-install-package ``` ## List Images in Bundle *This feature is a nice to have* Enable the users of `imgpkg` to understand, without pulling the bundle, what images are part of the bundle. ### Proposed change Create a new command that could provide the user with information about the contents of a bundle ```= $ imgpkg info -b ghcr.io/k14s/design-docs/simple-app-install-package Images: - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 (Bundle) Images: - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 Annotations: kbld.carvel.dev/id: my.registry.io/simple-application - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f ```