Prepare Operation

# Prepare Operation # Meta [meta]: #meta - Name: (fill in the feature name: My Feature) - Start Date: (fill in today's date: YYYY-MM-DD) - Author(s): (Github usernames) - Status: Draft  - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Changelog - 2022/02/02 - [Defined changes to `io.buildpacks` schema](#Changes-to-iobuildpacks-namespace) - Including the proposed criteria for determining whether certain properties belong in `io.buildpacks.defaults`. - [Moved `io.buildpacks` namespace schema to Platform API](#Add-Project-Descriptor-iobuildpacks-namespace) - [Change prepare spec to say SHOULD instead of MUST for warnings](#Prepare) - 2022/01/31 - [Add alternatives section](#Alternatives) # Summary [summary]: #summary # Definitions [definitions]: #definitions  # Motivation [motivation]: #motivation  ### Goal 1: Serializing CLI configuration As a user, I would like to be able to serialize and share the parameters I use with certain platforms such as `pack`. ### Goal 2: A recognizable file in repositories As a user, I would like to be able to recognize, based on the file system, if a project is using Cloud Native Buildpacks. ### Goal 3: Platform recognition of project.toml As a user, I would like to ensure that my configuration in `project.toml` is being used. # What it is [what-it-is]: #what-it-is The proposal is composed of following changes: 1. [Moving `io.buildpacks` properties to `io.buildpacks.defaults`.](#Namespace-iobuildpacksdefaults) 2. [A replaceable new phase `prepare` that applies platform configuration.](#Preparer) 3. [Supporting Cloud Native Buildpack utilities.](#Cloud-Native-Buildpacks-utilities) 4. [A custom preparer process for Pack.](#Pack-Custom-preparer) - Based on proposal [buildpacks/rfcs#189](https://github.com/buildpacks/rfcs/pull/189) (`pack.toml`) ## Changes to `io.buildpacks` namespace ### Namespace `io.buildpacks.defaults` This namespace declares default values that platforms should acknowledge. A majority of properties should be spec'd but we would allow for unspec'd additional properties. The criteria for spec'd properties in this namespace would be based on whether they affect inputs to the lifecycle phases. For example, the order and environment variables are inputs to the lifecycle. The `builder`, in contrast, is not an input to the lifecycle so it wouldn't meet the criteria. Therefore the following changes have been made from the [latest schema][pd-02]: - Remove `builder` [pd-02]: https://github.com/buildpacks/spec/blob/extensions/project-descriptor/0.2/extensions/project-descriptor.md#iobuildpacks-optional ### Version `io.buildpacks` namespace Given that the project descriptor's `io.buildpacks` namespace is directly tied to the platform by the fact that it's the platform that needs to consume and parse it, it makes sense to unify the version of the schema with that of the platform API. ### Schema See [Platform Spec changes](#Platform-Spec). #### Usages Examples ###### Multi-Platform (pack + kpack) In the following example we show how various platform namespaces can have their own configuration but properties can be promoted to `io.buildpacks.defaults` if the user's intent is to apply it to all platforms. ```toml [io.buildpacks] schema-version = "0.2" ### # common buildpacks config ## [io.buildpacks.defaults] exclude = ["some-files/**"] ## # pack config ## [io.buildpacks.pack] schema-version = "0.1" builder = "cnbs/sample-builder:bionic" image = "my-app" # (example only) run-image = "my-run-image:latest" # (example only) ## # kpack config ## [com.vmware.kpack] schema-version = "0.1" [[com.vmware.kpack.build.env]] name="CUSTOM_ENV" value="SOME_VALUE" ``` Notice: 1. Both `pack` and `kpack` are expected to apply `io.buildpacks.defaults.exclude`. 2. `io.buildpacks.pack` has additional properties that are specific to `pack` only. ## Prepare phase A `prepare` phase would be a new lifecycle phase that gets executed before `create` or `detect` phase. Similar to the `rebaser`, it is outside of the standard [Build operation][build-operation]. [build-operation]: https://github.com/buildpacks/spec/blob/main/platform.md#build #### Responsibilities At minimum, the expected reponsibility of the `preparer` would be to: - Apply the requested configuration. #### Inputs The `prepare` phase would take all the same inputs as `creator` plus the addition of a path to `project.toml`. #### Outputs The `prepare` phase may affect the file system, and mutation or create files that would be consumed be futher phases. ## Cloud Native Buildpacks utilities The Cloud Native Buildpacks project will provide the following functionality in the form of utilities: ###### `go` prepare function A function that can help developers apply configuration to the filesystem based on a provided configuration. > This would be useful for builder providers or platform implementers that choose to provide their own [`preparer` executable](#preparer-executable). ###### `preparer` executable The Cloud Native Buildpacks project will have a `preparer` it ships along with the existing `lifecyle` image. > This would be useful for builder providers or platform implementers to not have to develop standard functionality. The default implementation COULD take care of applying the following configuration: - `io.buildpacks.defaults.group` → - Download buildpacks - Update `order.toml` - `io.buildpacks.defaults.build.env` → - Set build env vars in `<platform>/env` - Notify users of any other properties in `io.buildpacks.defaults` --- ## Custom `preparer` (for Pack) > NOTE: This is mostly an implementation detail and/or platform sub-team specific RFC to display how the general RFC could apply in practice to Pack. Pack would have its own preparer implementation which would do the following: - [Overlay `pack.toml` onto `project.toml`](#Overlay-Pack-specific-config) - Notify the user of any `io.buildpacks.defaults` properties that go unused. - Apply the requested configuration. #### Overlay Pack-specific config Overlaying is the process of merging a `project.toml` with the Pack-specific configuration file. Overlaying is done using the strategy defined in [IETF rfc7396][rfc7396] where **target** is the `io.buildpacks.defaults` namespace and the **patch** is the platform-specific configuration. Notes: - Arrays are not appended but replaced as per the RFC. - Due to [limitations][toml-no-null] in the TOML specification, in order to _unset_ a property an empty table (`{}`) may be declared to simulate a `null` value. (ie. `builder = {}`) [rfc7396]: https://datatracker.ietf.org/doc/html/rfc7396 [toml-no-null]: https://github.com/toml-lang/toml/issues/30 ##### Usage Examples ###### `pack` in `project.toml` Given the following config file: ```toml [io.buildpacks.defaults] builder = "image-from-defaults" [[io.buildpacks.defaults.build.env]] name = "env_from_def_1" value = "val_1" [[io.buildpacks.defaults.build.env]] name = "env_from_def_2" value = "val_2" [io.buildpacks.pack] builder = "image-from-pack-config" [[io.buildpacks.pack.build.env]] name = "env_from_pack_3" value = "val_3" ``` ... and the Pack-specific namespace is: `io.buildpacks.pack` ... the resulting merged _(effective)_ configuration should be: ```toml builder = "image-from-pack-config" [[build.env]] name = "env_from_pack_3" value = "val_3" ``` ###### `project.toml` + `pack.toml` Given the following `project.toml` file: ```toml [io.buildpacks] schema-version = "0.2" [io.buildpacks.defaults] builder = "image-from-defaults" [io.buildpacks.defaults] exclude = ["some/**/files"] [[io.buildpacks.defaults.build.env]] name = "env_from_def_1" value = "val_1" [[io.buildpacks.defaults.build.env]] name = "env_from_def_2" value = "val_2" ``` ... and `pack.toml` file: ```toml [pack] schema-version = "0.1" builder = "image-from-pack-config" [[pack.build.env]] name = "env_from_pack_3" value = "val_3" ``` ... and declared platform-specific namespace: `io.buildpacks.pack` ... the resulting merged _(effective)_ configuration should be: ```toml exclude = ["some/**/files"] builder = "image-from-pack-config" [[build.env]] name = "env_from_pack_3" value = "val_3" ``` --- # Drawbacks [drawbacks]: #drawbacks 1. Parsing and applying the `io.buildpacks` namespace becomes responsibility of the platform. - This is mitigated by providing [utilities](#Cloud-Native-Buildpacks-utilities) and the fact that the prepare phase is an independant and swappable. 2. Executing the Prepare operation may require an additional container to be spun up in some platforms; this would effectively increase the overall build process. # Alternatives [alternatives]: #alternatives #### [Converter RFC][converter-rfc] > The idea is to ship a binary with the lifecycle that would be responsible for translating project.toml from the schema defined in the project descriptor extension spec into something that the lifecycle knows the platform can understand i.e., a schema defined in the platform spec. - Benefits - Converting various possible user provided schemas to a platform API schema makes it easier for platforms to consume the configuration. - Counter-point: The "ease" of consumption could be provided by other means such as language-specific libraries that ease parsing to general models. - Drawbacks - Having a converter instead of an entire swappable prepare phase means that the process of applying configuration becomes a 2-step process for platforms. [converter-rfc]: https://github.com/buildpacks/rfcs/pull/182 #### Project Descriptor Buildpack The idea of a buildpack that can apply configuration from `project.toml` has been kicked around and would work for _some_ properties such as environment variables. - Benefits - Buildpacks are more robust. Updated versions can be used to apply the later versions of project descriptor with no platform operator/implementer intervention. - Drawbacks - Not all operations may be applied at the buildpack level. For example, buildpack order. There would need to be a higher-order operation to apply other parts of the configuration. Given a higher-order operation, it doesn't make sense to split the application of the configuration. # Prior Art [prior-art]: #prior-art - [rebase][rebase] - Another "out-of-build" operation. [rebase]: https://github.com/buildpacks/spec/blob/main/platform.md#rebase # Unresolved Questions [unresolved-questions]: #unresolved-questions - What determines whether a property is added to `io.buildpacks.defaults`? - Option 1: It must be an input to a lifecycle phase. - Should arbitrary properties be allowed in `io.buildpacks.defaults`?  # Spec. Changes [spec-changes]: #spec-changes ## Distribution Spec  ## Platform Spec --- ### Add Project Descriptor `io.buildpacks` namespace --- #### `project.toml` (TOML) The format of `project.toml` MUST adhere to version `0.2` of the [project descriptor specification][project-descriptor-spec]. Within the `project.toml` file the `io.buildpacks` namespace MAY be defined. [project-descriptor-spec]: https://github.com/buildpacks/spec/blob/main/extensions/project-descriptor.md ##### `io.buildpacks` namespace ```toml [io.buildpacks] schema-version = "<platform API version>" [io.buildpacks.defaults] include = ["<.gitignore pattern>"] exclude = ["<.gitignore pattern>"] [[io.buildpacks.defaults.pre.group]] id = "<buildpack ID>" version = "<buildpack version>" uri = "<url or path to the buildpack" [io.buildpacks.defaults.pre.group.script] api = "<buildpack API version>" shell = "<string>" inline = "<script contents>" [[io.buildpacks.defaults.group]] id = "<buildpack ID>" version = "<buildpack version>" uri = "<url or path to the buildpack" [io.buildpacks.defaults.group.script] api = "<buildpack API version>" shell = "<string>" inline = "<script contents>" [[io.buildpacks.defaults.post.group]] id = "<buildpack ID>" version = "<buildpack version>" uri = "<url or path to the buildpack" [io.buildpacks.defaults.post.group.script] api = "<buildpack API version>" shell = "<string>" inline = "<script contents>" [[io.buildpacks.defaults.build.env]] name = "<name>" value = "<value>" ``` Where: - `schema-version` (required): is the version of the schema which correlates with the platform API. - `defaults` (optional): is a table of default properties that all platforms should apply. _`include` and `exclude` are mutually exclusive. If both are present the build process MUST result in an error._ - `include` (optional): is an array of `.gitignore` pattern-based paths to include during the build operation. - `exclude` (optional): is an array of `.gitignore` pattern-based paths to exclude from the build operation and thereby produced image. - `group` (optional): is an array of buildpacks. _Either a `version`, `uri`, or `script` table MUST be included, but MUST NOT include any combination of these elements._ - `id` (optional): is the ID of the buildpack. - `version` (optional, default=`latest`): is the version of the buildpack. - `uri` (optional, default=`urn:buildpack:<id>`): is the URI to the buildpack. - `script` (optional): defines an inline buildpack. - `api` (required): is the api key defines its Buildpack API compatibility. - `shell` (optional, default=`/bin/sh`): defines the shell used to execute the inline script. - `inline` (required): is the build script for the inline buildpack. - `build` (optional): - `env` (optional): an array table that defines environment variables to be applied during the `build` phase. - `name` (required): is the name of the environment variable. - `value` (required): is the value of the environment variable. --- ### Add Prepare operation --- #### Prepare Before the [Build](#Build) operation is executed, a platform MUST prepare the build environment. During, the Prepare phase, the platform: - SHOULD apply provided [Project Descriptor][project-descriptor] configuration is present. - SHOULD generate a warning for any `io.buildpacks.defaults` property not applied. - MAY make changes to any path in the filesystem. [project-descriptor]: https://github.com/buildpacks/spec/blob/main/extensions/project-descriptor.md --- ### Add `prepare` section --- #### `preparer` Usage: ``` /cnb/lifecycle/preparer \ [-app <app>] \ [-buildpacks <buildpacks>] \ [-cache-dir <cache-dir>] \ [-cache-image <cache-image>] \ [-daemon] \ [-gid <gid>] \ [-launch-cache <launch-cache> ] \ [-launcher <launcher> ] \ [-layers <layers>] \ [-log-level <log-level>] \ [-order <order>] \ [-platform <platform>] \ [-previous-image <previous-image> ] \ [-process-type <process-type> ] \ [-project-descriptor <project-descriptor> ] \ [-project-metadata <project-metadata> ] \ [-report <report> ] \ [-run-image <run-image>] \ [-skip-restore <skip-restore>] \ [-stack <stack>] \ [-tag <tag>...] \ [-uid <uid> ] \ <image> ``` The `preparer` SHOULD accept the same inputs as the `creator` with the addition of the following: | Input | Environment Variable| Default Value| Description |-------------------|---------------------|--------------|---------------------- | `<project-descriptor>`| `CNB_PROJECT_DESCRIPTOR_PATH`| `<app>/project.toml` | Path to a [Project Descriptor](#projecttoml-TOML) ##### Outputs A `preparer` may make general changes to the file system, modify input files, or create input files. | Exit Code | Result |--- |--- | `0` | Success | `1-10`, `13-19` | Generic lifecycle errors  --- ## Project Descriptor Spec --- #### Remove `io.buildpacks` namespace ---

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.