---
tags: releases, software release
---
# Carvel Tools Release Process

- Proposal Status: Draft, In Review, **Accepted**, Rejected

# Table of Contents

[TOC]

---

# Release Process Proposal

## Goals

- Generate a checklist with the process we want to follow
- Document the reasons to create a new release
- Possibly add suggestions to make it better or point to places that could be improved
- This process document focus on the following tools release process: `ytt`, `kapp`, `kbld`, `kapp-controller`, `kwt`, `imgpkg` and `vendir`

## Anti Goals

- Develop tooling to help with the process

- Solution on said tooling

## Release decision
:construction:
:::spoiler see if it is enough to explain the why we should create a release _(click to reveal)_
We assume that there will be 4 major categories of reasons to release:
1. There is a critical bug fixed
1. The product achieved a milestone

A Milestone can be a Big Feature of a Chunk of that Big Feature. An example of this would be the Schema Feature for `ytt`, another example can be the MVP release of `imgpkg`
3. There is a set of features/bugs that warrant a new version

These set of features can be random issues or bugs that make sense to group in a new version.
5. A user asked for a new official release

For each one of the above categories, there are considerations that we should make before we release:
- Is there work in-flight on `develop` that would impact GA functionality? If so, consider waiting until "feature complete" or patching on the previous release.

- Does it make sense to release a new version today?

If we release today and an urgent issue is discovered tomorrow can we solve it?

Do we have enough time today to do the full release process?

Not releasing today is going to cause major problems for the tool? (Community or user wise)

What steps to take to decide:
1. Given any of the above prompt
2. Talk with product to align if it make sense right now
3. Run CVE scans (trivy) to ensure that we don't have any new CVE coming and we have the exceptions for the CVE's for which fix is not available.
4. Release
:::

## How to release

### Release Checklist

:::info
:information_source: **Track a Release in GitHub**

Grab a copy of the Markdown format of this checklist and create a GitHub issue to track progress of a release.
:::

- [ ] Create release in Github
- [ ] [Bump Version](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Bump-version)
- [ ] [Create Draft Github Release](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Create-Draft-Github-Release)
- [ ] [Create Release Notes](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Create-the-Release-Notes)
- [ ] [Generate Binaries and Checksum](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Generate-Binaries-and-Checksum)
- [ ] [Upload Binaries and Checksum](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Upload-Binaries-and-Checksum)
- [ ] [Publish the Draft Release](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Publish-the-draft-release)
- [ ] Close any GitHub issues that have been delivered.
- [ ] Add a link to the release on the issue.
- [ ] Update the packaging repositories
- [ ] [Update Homebrew](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Update-Homebrew)
- [ ] [Update Website Installation Script](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Update-Website-Installation-Script)
- [ ] [Update Chocolatey](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Update-Chocolatey)
- [ ] [Update Github Action](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Update-Github-Action)
- [ ] Update Documentation by [generating a new docs version](#Generate-new-docs-version)
- [ ] [Push any artifacts to a registry](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Push-OCI-Images-to-Registry)
- [ ] [Communicate in Slack](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Communicate-in-Slack)
- [ ] [Add to "Announcements" in Next Community Meeting Agenda](https://hackmd.io/uVpvITUuR4Cbwzkzb7MEpQ?view#Announce-in-community-meeting)

---

#### Bump version

##### Automatically Created Draft Release

`imgpkg`, `kapp`, `kbld`, `vendir`, and `ytt` all use a GitHub Action to create the Draft Release. They trigger when a tag in the shape of a version number is pushed to the repo.

_(for all other tools, see [Manually Created Draft Release](#Manually-Created-Draft-Release))_

1. Ensure you're on the `develop` branch (automation watches this branch, exclusively):

	$ git checkout develop

:::info
:information_source: **Tagging a Patch Release**

:::spoiler When tagging a patch release _after_ another
has already been released...

GitHub determines the order of releases based on the timestamp of the tags that mark each release.
So, the most recently _tagged_ release becomes the "latest" release (i.e. is the version users are redirected to from http://github.com/user/repo/releases/latest).

In git, a tag is a light-weight reference to a commit. So, the timestamp of the tag _is_ the timestamp of the commit it points to.

"Latest" is as expected if a patch release immediately follows its minor (e.g. from v0.37.0 to v0.37.1). However, if a patch release is preceeded by some _other_ minor release (e.g. v0.38.0), then "latest" becomes v0.37.1, when semver latest (and what users would expect) is v0.38.0.

To work around this, one needs to create an "annotated tag" (which has among other data a creation date) and set the date to position the tag in the timeline:

1. Checkout the minor version branch used to create the backport/patch.

	$ git co v0.37.x

2. Get metadata about the previous commits:

	$ git show v0.37.0 
	$ git show v0.38.0

3. Tag that commit with a timestamp that positions the patch release _after_ the previous patch version but _before_ the subsequent minor version (e.g. after v0.37.0, but before v0.38.0)

    $ GIT_COMMITTER_DATE="2013-12-31 00:00" git tag --annotate v0.37.x b217332279 -m "Publish v0.37.x"

where:
- replace `2013-12-31 00:00` with the time you've calculated (note: [this field supports multiple formats](https://git-scm.com/docs/git-tag#_date_formats))
- replace `v0.37.x` with your version
- replace `b217332279` with the commit you want to release at
4. Push the new tag to the GitHub clone:

    $ git push --tags origin v0.37.x

where:
- replace `v0.37.x` with your minor version branch
---
:warning: Some downstream tooling may not detect older commits/annotated tags. The next time we encounter this scenario, please document this experience.
:::

2. Tag that commit with the version:

	$ git tag vX.X.X

3. Publish the tag:

	$ git push origin vX.X.X

##### Manually Created Draft Release

`kapp-controller` and `kwt` have no automation for creating the Draft Release. For these tools, the version constant is updated manually.

Steps:
1. Checkout the `develop` branch
2. Update the major, minor, or patch version in the version file. For more information on versioning please [see this section](#Versioning)
- kapp-controller: [cmd/main.go](https://github.com/vmware-tanzu/carvel-kapp-controller/blob/develop/cmd/main.go)
- kwt: [pkg/kwt/cmd/version.go](https://github.com/vmware-tanzu/carvel-kwt/blob/develop/pkg/kwt/cmd/version.go)
3. Commit that change
4. Tag that commit with the version

	$ git tag vX.X.X

5. Push the commit to `develop` branch

    $ git push && git push --tags

#### Create Draft Github Release

For tools that use github actions to create the draft release (imgpkg, ytt, kbld, kapp, vendir), check to see if the new draft release has been added to GH by the goreleaser. Then continue to the [next section](#Create-the-Release-Notes) for more information on how to tailor the text to add in the `Describe Release` field.

For other tools (kapp, kwt, vendir):
:::spoiler Steps:
Using the Github UI, create a new release and tag the 'version bump' commit with the tag `vX.X.X`
1. Access the Github Repository
2. Click in `Releases`
3. Click `Draft a New Release` or access https://github.com/vmware-tanzu/REPOSITORY/releases/new
4. As a tag name use the tag created during the 'Bump Version' step above.
5. For the title use the same text as the tag.
6. Look at the [next section](#Create-the-Release-Notes) for more information on how to tailor the text to add in the `Describe Release` field
:::

#### Create the Release Notes
1. For imgpkg, ytt, kapp, vendir and kbld, the release notes should come with a list of changes made since the last release, this is not the case for kapp, kwt, and vendir. Read the commit messages and issues that have been closed since the last release.
2. Choose and include the most relevant feature changes or bugs
3. Follow the below proposed template for our release notes.

Title: `v1.2.3`

    # :sparkles: What's new
    - :exclamation: **Breaking Change** Something will no longer exist, #15
    - :scream: **Deprecation** Deprecated this particular feature because ..., #10
    - Small description of first issue #12,#14
    - Another feature #15, 16
    
    # :bug: Bug Fixes
    - Small bug, #9
    - This issue impacted the users in this particular way, #16

    # :speaker: Callouts
    Thanks to
    - User 1 @bananas
    - User 2 @pears

    For helping out with this release
    
    # :open_file_folder: Files Checksum
    
    \```
    d52b477612fbfd1651f55190d17fe8ef1a1df990e988cd9f49cf568d9018fcc9 ./app-darwin
    \```

To note:
- Add links to the issues that we tried to address, if possible
- There is no need to add a huge explanation on each point since an issue with context will be added
- Callouts maybe only make sense if they are non-core team and should be optional
- Do not forget to tag Deprecations, Breaking Changes and Major Bugs
- File checksum calculation information can be found in [here](#Generate-Binaries-and-Checksum)
- Not all sections need to be present
- Remove the `\` on the shasum section

:::spoiler **Image after rendering the template** _(click to reveal)_
:::

:::warning
:::spoiler Steps for not automated tools on generate Binaries and upload them to the release
#### [Not required] Generate Binaries and Checksum

Note: We have github actions running which takes care of it.

For tools that use github actions to create the draft release (imgpkg, ytt, kbld, kapp, vendir):
- the GitHub actions create the binaries and checksums, automatically.
- next step: [Upload Binaries and Checksum](#Upload-Binaries-and-Checksum)

For other tools (kapp, kbld, kwt, vendir):
On each repository there is a `hack/build-binaries.sh` file that we can use to generate the binaries and the checksum

Running `hack/build-binaries.sh` will generate a binary per platform and in the Standard Output prints the checksum for each file.

**kapp-controller**: This is a little bit of a different process because what we want to distribute is the configuration file and a OCI Image. To archive this run `hack/build-release.sh`.

This script will generate 2 artifacts, the OCI Image and the configuration YAML.
The OCI Image is pushed to Github Package Registry automatically by `kbld`, but the configuration YAML will have to be attached to the Github Release as per the next step

#### [Not Required] Upload Binaries and Checksum

Note: We have a Github action which takes care of it.

For tools that use github actions to create the release (imgpkg, ytt, kbld, kapp, vendir):
1. view the draft release, and find and download the `checksums.txt` file.
2. edit so that the filenames for the binaries start with a `./` (if you don't subsequent steps will fail)
3. copy + paste the contents to the bottom of the release notes.

For other tools (kapp, kbld, kwt, vendir):

Add the generated binaries to the Github Release, and edit the Release Notes to add the Checksums of the binaries

**kapp-controller**: The artifact that will be uploaded is the configuration file and the Checksum of the configuration file
:::

#### Publish the draft release

Click on the publish button... profit!

Wait for the Github Action `Checks after any release is published` to run successfully.

#### Update Homebrew

In the [Homebrew Tap repository](https://github.com/vmware-tanzu/homebrew-carvel) each tool has a file that contains the version and the checksum for Linux and MacOS binaries. These values need to be updated per release.

For tools that dispatch the GH action to update homebrew upon a release (`ytt`, `kbld`, `kapp`, `vendir`, and `imgpkg`):
- Confirm changes and sha256s in the the commit created by carvel-release-scripts in https://github.com/vmware-tanzu/homebrew-carvel

:::spoiler To update homebrew manually _(click to reveal)_:
1. Checkout https://github.com/vmware-tanzu/homebrew-carvel
2. Update the version on the file for the correct tool
3. Update the checksum on the file for the correct tool
4. Commit
5. Push
:::

#### Update Website Installation Script

This file is located in: https://github.com/vmware-tanzu/carvel/blob/develop/site/static/install.sh

For tools that dispatch the GH action to update the installation script upon a release (`ytt`, `kbld`, and `imgpkg`):
- Confirm changes and sha256s in the the commit created by carvel-release-scripts in https://github.com/vmware-tanzu/carvel

:::spoiler To update installation script manually _(click to reveal)_:
1. Checkout https://github.com/vmware-tanzu/carvel
2. Update the tool version and checksum for linux and MacOS on the file `site/static/install.sh`
3. Commit
4. Push
:::

#### Update Chocolatey

https://community.chocolatey.org/packages/ytt

:::danger
:rotating_light: **the ytt Chocolatey package is currently not maintained by Carvel.**

This means we currently do not maintain the package ourselves.

The "instructions" herein are a sketch of what we _would_ do, should we obtain ownership.
:::

:::info
:information_source: **Request the donation?**

At some point, perhaps we should consider requesting the owner to donate the package to this project.

It would be housed at: https://github.com/vmware-tanzu/carvel-chocolatey
:::

:::spoiler Possible Process _(click to reveal)_

:construction: Assumption: we can keep all tools in the same Chocolatey Github Repository

1. Checkout https://github.com/vmware-tanzu/carvel-chocolatey
2. Update the version on the file `tool-name.nuspec` for the correct tool. Update the release notes url to the correct version.
3. Update the version and checksum on the file `tools/VERIFICATION.txt` for the correct tool
4. Commit
5. Push

:construction: Unsure if other steps are needed
:::

#### Update Github Action

There is no need to do any update on the action. Nevertheless there is a Github Action called `Checks after release is published` that will run when a new Release is published. When releasing we should ensure that this check passes.

This Github Action is present in the repositories of each tool and will run automatically.

e.g. https://github.com/vmware-tanzu/carvel-ytt/actions/workflows/release-published.yml

#### Generate new docs version
This is a semi automated process that contains a script that will allow us to generate new documentation based on a version that already exist.

1. Clone the Carvel repository

Clone the repository `https://github.com/vmware-tanzu/carvel` using the command:

    git clone [e-mail]:vmware-tanzu/carvel.git

1. Enter the site folder

    cd carvel/site

1. Generate the documentation

- Generate documentation for the latest version

    # Execute the following command
    # replace `imgpkg` with the tool name
    # replace v0.24.0 with the new version just created
    ./hack/release-docs.sh imgpkg v0.24.0

- Generating the documentation for a patch version

    # Execute the following command
    # replace `imgpkg` with the tool name
    # replace v0.24.1 with the new version just created
    # replace v0.24.0 with the version that is the base of the patch
    ./hack/release-docs.sh imgpkg v0.24.1 v0.24.0

1. Verify and commit the changes

Check that the files added to git are the ones intended to be changed (ex: config.yaml will only be change when updating latest or doing a patch to the latest version)

1. Push changes and create a Pull Request

1. When the PR is merge, contact Nanci or Jonas or Joao so that they can rerun the crawler in the website to ensure the search will be updated

#### Push OCI Images to Registry

The only(?) tool that require an OCI Image is `kapp-controller` and the scripts that build the Image and generate the Configuration already pushes the image to the Github Package Registry.
The other tools are packages in a docker image for consumption and the repository is in https://github.com/vmware-tanzu/carvel-docker-image
We should create a new version of this image at least once every quarter.
1. ./hack/build.sh
1. ./hack/build-and-push.sh

#### Communicate in Slack

Please coordinate with Nanci ([@nancil]() in Slack) for communicating releases. She'll announce them on twitter and then link to them in slack.

For larger releases, we'll try to anticipate these releases so that we can prepare a blog post.

The message template we should use is:

:mega: Tool-Name v1.2.3

:sparkles: **Major Highlights**
- :exclamation: **Breaking Change** Something will no longer exist,
- :scream:  **Deprecation Deprecated** this particular feature
- :bug: Major issue This problem was huge

For more information visit github.com/vmware-tanzu/carvel-tool/releases/tag/v1.2.3

Image after rendering:

#### Announce in community meeting

Update the next community meeting Announcements section in https://hackmd.io/F7g3RT2hR3OcIh-Iznk2hw.

Example:

- ytt v0.31.0 ([release notes](https://github.com/vmware-tanzu/carvel-ytt/releases/tag/v0.31.0))

## Versioning

:construction: In here we should explain when we should bump each number

Heuristics to bump versions:
- Only a bug is present in this release: **Patch** bump
- No breaking changes: **Minor** bump
- Breaking changes: **Major** bump
- A feature deprecated: **Major** bump
- A big impact feature for the product that would change the way the product works: **Major** bump

These Guidelines are flexible and we should discuss before each bump to ensure that we are bumping the correct number.

## Deprecation of functionality

When deprecating functionality we should provide the users with enough time to prepare for this change. The recommendation for this is:
- When release cadence is higher than one release per month
- Ensure at least 3 minor releases for the consumers to adapt. Create a retro compatible version and 2 more versions with warning of deprecation.
- When release cadence is lower than one release per month
- Ensure at least 1 minor release exist with a retro compatible code exists that warns about the deprecation

## What is not true today in the process above

- Carvel team does not own the chocolatey release channel
- Github action is still in k14s organization
- Docker images only exist in Docker Hub and not Github Package Registry

## Possible improvements on the process

### Easy wins
- The version is read from the same file in all the repositories
- All PR's or Commits contain the Github formula that automatically closes Issues that they solve

### Automation of parts of the release process
1. Slack notifications

It would be interesting to provide a way to send automatic messages to both Slack channels when a release is published
1. Generation and inclusion of binaries in release notes

Automate the generation of the binaries and attachment to the release notes

1. Generate binaries
2. Generate the Checksums
3. Add the binaries to the draft release
4. Add the checksums to the text on the release
1. Simple way to gather all changes since last release

It would be helpfull to create a way to gather all the issues/stories that where finished since the last release. Even if this does not create the release note it would help us not forget any issue.

1. Update all the places an artifact needs to be updated

Having to update 4 or 5 different places with the same information is tedius and error prune. If we could automate the commit process in all the needed repositories would be awesome.

For some of these maybe we can leverage things like dependabot.

1. Packaging tool files automation

If there was a file in all the packaging repositories that contained the version and checksums. It would make the update easier, we could generate the final files using `ytt`.