Carvel Tools Release Process

Proposal Status: Draft, In Review, Accepted, Rejected

Carvel Tools Release Process
Table of Contents
Release Process Proposal

Release Process Proposal

Goals

Generate a checklist with the process we want to follow
Document the reasons to create a new release
Possibly add suggestions to make it better or point to places that could be improved
This process document focus on the following tools release process: ytt, kapp, kbld, kapp-controller, kwt, imgpkg and vendir

Anti Goals

Develop tooling to help with the process
Solution on said tooling

Release decision

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

see if it is enough to explain the why we should create a release (click to reveal)

We assume that there will be 4 major categories of reasons to release:

There is a critical bug fixed
The product achieved a milestone

A Milestone can be a Big Feature of a Chunk of that Big Feature. An example of this would be the Schema Feature for ytt, another example can be the MVP release of imgpkg
There is a set of features/bugs that warrant a new version

These set of features can be random issues or bugs that make sense to group in a new version.
A user asked for a new official release

For each one of the above categories, there are considerations that we should make before we release:

Is there work in-flight on develop that would impact GA functionality? If so, consider waiting until "feature complete" or patching on the previous release.
Does it make sense to release a new version today?

If we release today and an urgent issue is discovered tomorrow can we solve it?

Do we have enough time today to do the full release process?

Not releasing today is going to cause major problems for the tool? (Community or user wise)

What steps to take to decide:

Given any of the above prompt
Talk with product to align if it make sense right now
Run CVE scans (trivy) to ensure that we don't have any new CVE coming and we have the exceptions for the CVE's for which fix is not available.
Release

How to release

Release Checklist

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Track a Release in GitHub

Grab a copy of the Markdown format of this checklist and create a GitHub issue to track progress of a release.

Bump version

Automatically Created Draft Release

imgpkg, kapp, kbld, vendir, and ytt all use a GitHub Action to create the Draft Release. They trigger when a tag in the shape of a version number is pushed to the repo.

(for all other tools, see Manually Created Draft Release)

Ensure you're on the develop branch (automation watches this branch, exclusively):
```
$ git checkout develop
```

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Tagging a Patch Release

When tagging a patch release after another

has already been released…

GitHub determines the order of releases based on the timestamp of the tags that mark each release. So, the most recently tagged release becomes the "latest" release (i.e. is the version users are redirected to from http://github.com/user/repo/releases/latest).

In git, a tag is a light-weight reference to a commit. So, the timestamp of the tag is the timestamp of the commit it points to.

"Latest" is as expected if a patch release immediately follows its minor (e.g. from v0.37.0 to v0.37.1). However, if a patch release is preceeded by some other minor release (e.g. v0.38.0), then "latest" becomes v0.37.1, when semver latest (and what users would expect) is v0.38.0.

To work around this, one needs to create an "annotated tag" (which has among other data a creation date) and set the date to position the tag in the timeline:

Checkout the minor version branch used to create the backport/patch.
```
$ git co v0.37.x
```

Get metadata about the previous commits:

$ git show v0.37.0 
$ git show v0.38.0

Tag that commit with a timestamp that positions the patch release after the previous patch version but before the subsequent minor version (e.g. after v0.37.0, but before v0.38.0)
```
$ GIT_COMMITTER_DATE="2013-12-31 00:00" git tag --annotate v0.37.x b217332279 -m "Publish v0.37.x"
```
where:
- replace 2013-12-31 00:00 with the time you've calculated (note: this field supports multiple formats)
- replace v0.37.x with your version
- replace b217332279 with the commit you want to release at
Push the new tag to the GitHub clone:
```
$ git push --tags origin v0.37.x
```
where:
- replace v0.37.x with your minor version branch

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Some downstream tooling may not detect older commits/annotated tags. The next time we encounter this scenario, please document this experience.

Tag that commit with the version:
```
$ git tag vX.X.X
```
Publish the tag:
```
$ git push origin vX.X.X
```

Manually Created Draft Release

kapp-controller and kwt have no automation for creating the Draft Release. For these tools, the version constant is updated manually.

Steps:

Checkout the develop branch
Update the major, minor, or patch version in the version file. For more information on versioning please see this section
- kapp-controller: cmd/main.go
- kwt: pkg/kwt/cmd/version.go
Commit that change
Tag that commit with the version
```
$ git tag vX.X.X
```

Push the commit to develop branch

$ git push && git push --tags

Create Draft Github Release

For tools that use github actions to create the draft release (imgpkg, ytt, kbld, kapp, vendir), check to see if the new draft release has been added to GH by the goreleaser. Then continue to the next section for more information on how to tailor the text to add in the Describe Release field.

For other tools (kapp, kwt, vendir):

Steps:

Using the Github UI, create a new release and tag the 'version bump' commit with the tag vX.X.X

Access the Github Repository
Click in Releases
Click Draft a New Release or access https://github.com/vmware-tanzu/REPOSITORY/releases/new
As a tag name use the tag created during the 'Bump Version' step above.
For the title use the same text as the tag.
Look at the next section for more information on how to tailor the text to add in the Describe Release field

Create the Release Notes

For imgpkg, ytt, kapp, vendir and kbld, the release notes should come with a list of changes made since the last release, this is not the case for kapp, kwt, and vendir. Read the commit messages and issues that have been closed since the last release.
Choose and include the most relevant feature changes or bugs

Follow the below proposed template for our release notes.

Title: v1.2.3

# :sparkles: What's new
- :exclamation: **Breaking Change** Something will no longer exist, #15
- :scream: **Deprecation** Deprecated this particular feature because ..., #10
- Small description of first issue #12,#14
- Another feature #15, 16

# :bug: Bug Fixes
- Small bug, #9
- This issue impacted the users in this particular way, #16

# :speaker: Callouts
Thanks to
- User 1 @bananas
- User 2 @pears

For helping out with this release

# :open_file_folder: Files Checksum

\```
d52b477612fbfd1651f55190d17fe8ef1a1df990e988cd9f49cf568d9018fcc9 ./app-darwin
\```

To note:

Add links to the issues that we tried to address, if possible
There is no need to add a huge explanation on each point since an issue with context will be added
Callouts maybe only make sense if they are non-core team and should be optional
Do not forget to tag Deprecations, Breaking Changes and Major Bugs
File checksum calculation information can be found in here
Not all sections need to be present
Remove the \ on the shasum section

Image after rendering the template (click to reveal)

Steps for not automated tools on generate Binaries and upload them to the release

[Not required] Generate Binaries and Checksum

Note: We have github actions running which takes care of it.

For tools that use github actions to create the draft release (imgpkg, ytt, kbld, kapp, vendir):

the GitHub actions create the binaries and checksums, automatically.
next step: Upload Binaries and Checksum

For other tools (kapp, kbld, kwt, vendir): On each repository there is a hack/build-binaries.sh file that we can use to generate the binaries and the checksum

Running hack/build-binaries.sh will generate a binary per platform and in the Standard Output prints the checksum for each file.

kapp-controller: This is a little bit of a different process because what we want to distribute is the configuration file and a OCI Image. To archive this run hack/build-release.sh.

This script will generate 2 artifacts, the OCI Image and the configuration YAML. The OCI Image is pushed to Github Package Registry automatically by kbld, but the configuration YAML will have to be attached to the Github Release as per the next step

[Not Required] Upload Binaries and Checksum

Note: We have a Github action which takes care of it.

For tools that use github actions to create the release (imgpkg, ytt, kbld, kapp, vendir):

view the draft release, and find and download the checksums.txt file.
edit so that the filenames for the binaries start with a ./ (if you don't subsequent steps will fail)
copy + paste the contents to the bottom of the release notes.

For other tools (kapp, kbld, kwt, vendir):

Add the generated binaries to the Github Release, and edit the Release Notes to add the Checksums of the binaries

kapp-controller: The artifact that will be uploaded is the configuration file and the Checksum of the configuration file

Publish the draft release

Click on the publish button… profit!

Wait for the Github Action Checks after any release is published to run successfully.

Update Homebrew

In the Homebrew Tap repository each tool has a file that contains the version and the checksum for Linux and MacOS binaries. These values need to be updated per release.

For tools that dispatch the GH action to update homebrew upon a release (ytt, kbld, kapp, vendir, and imgpkg):

Confirm changes and sha256s in the the commit created by carvel-release-scripts in https://github.com/vmware-tanzu/homebrew-carvel

To update homebrew manually (click to reveal):

Checkout https://github.com/vmware-tanzu/homebrew-carvel
Update the version on the file for the correct tool
Update the checksum on the file for the correct tool
Commit
Push

Update Website Installation Script

This file is located in: https://github.com/vmware-tanzu/carvel/blob/develop/site/static/install.sh

For tools that dispatch the GH action to update the installation script upon a release (ytt, kbld, and imgpkg):

Confirm changes and sha256s in the the commit created by carvel-release-scripts in https://github.com/vmware-tanzu/carvel

To update installation script manually (click to reveal):

Checkout https://github.com/vmware-tanzu/carvel
Update the tool version and checksum for linux and MacOS on the file site/static/install.sh
Commit
Push

Update Chocolatey

https://community.chocolatey.org/packages/ytt

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

the ytt Chocolatey package is currently not maintained by Carvel.

This means we currently do not maintain the package ourselves.

The "instructions" herein are a sketch of what we would do, should we obtain ownership.

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Request the donation?

At some point, perhaps we should consider requesting the owner to donate the package to this project.

It would be housed at: https://github.com/vmware-tanzu/carvel-chocolatey

Possible Process (click to reveal)

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Assumption: we can keep all tools in the same Chocolatey Github Repository

Checkout https://github.com/vmware-tanzu/carvel-chocolatey
Update the version on the file tool-name.nuspec for the correct tool. Update the release notes url to the correct version.
Update the version and checksum on the file tools/VERIFICATION.txt for the correct tool
Commit
Push

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

Unsure if other steps are needed

Update Github Action

There is no need to do any update on the action. Nevertheless there is a Github Action called Checks after release is published that will run when a new Release is published. When releasing we should ensure that this check passes.

This Github Action is present in the repositories of each tool and will run automatically.

e.g. https://github.com/vmware-tanzu/carvel-ytt/actions/workflows/release-published.yml

Generate new docs version

This is a semi automated process that contains a script that will allow us to generate new documentation based on a version that already exist.

Clone the Carvel repository

Clone the repository https://github.com/vmware-tanzu/carvel using the command:
```
git clone git@github.com:vmware-tanzu/carvel.git
```
Enter the site folder
```
cd carvel/site
```

Generate the documentation

Generate documentation for the latest version

# Execute the following command
# replace `imgpkg` with the tool name
# replace v0.24.0 with the new version just created
./hack/release-docs.sh imgpkg v0.24.0

Generating the documentation for a patch version

# Execute the following command
# replace `imgpkg` with the tool name
# replace v0.24.1 with the new version just created
# replace v0.24.0 with the version that is the base of the patch
./hack/release-docs.sh imgpkg v0.24.1 v0.24.0

Verify and commit the changes

Check that the files added to git are the ones intended to be changed (ex: config.yaml will only be change when updating latest or doing a patch to the latest version)
Push changes and create a Pull Request
When the PR is merge, contact Nanci or Jonas or Joao so that they can rerun the crawler in the website to ensure the search will be updated

Push OCI Images to Registry

The only(?) tool that require an OCI Image is kapp-controller and the scripts that build the Image and generate the Configuration already pushes the image to the Github Package Registry. The other tools are packages in a docker image for consumption and the repository is in https://github.com/vmware-tanzu/carvel-docker-image We should create a new version of this image at least once every quarter.

./hack/build.sh
./hack/build-and-push.sh

Communicate in Slack

Please coordinate with Nanci (@nancil in Slack) for communicating releases. She'll announce them on twitter and then link to them in slack.

For larger releases, we'll try to anticipate these releases so that we can prepare a blog post.

The message template we should use is:

:mega: Tool-Name v1.2.3

:sparkles: **Major Highlights**
- :exclamation: **Breaking Change** Something will no longer exist,
- :scream:  **Deprecation Deprecated** this particular feature
- :bug: Major issue This problem was huge

For more information visit github.com/vmware-tanzu/carvel-tool/releases/tag/v1.2.3

Image after rendering:

Announce in community meeting

Update the next community meeting Announcements section in https://hackmd.io/F7g3RT2hR3OcIh-Iznk2hw.

Example:

- ytt v0.31.0 ([release notes](https://github.com/vmware-tanzu/carvel-ytt/releases/tag/v0.31.0))

Versioning

Image Not Showing Possible Reasons

The image file may be corrupted
The server hosting the image is unavailable
The image path is incorrect
The image format is not supported

Learn More →

In here we should explain when we should bump each number

Heuristics to bump versions:

Only a bug is present in this release: Patch bump
No breaking changes: Minor bump
Breaking changes: Major bump
A feature deprecated: Major bump
A big impact feature for the product that would change the way the product works: Major bump

These Guidelines are flexible and we should discuss before each bump to ensure that we are bumping the correct number.

Deprecation of functionality

When deprecating functionality we should provide the users with enough time to prepare for this change. The recommendation for this is:

When release cadence is higher than one release per month
- Ensure at least 3 minor releases for the consumers to adapt. Create a retro compatible version and 2 more versions with warning of deprecation.
When release cadence is lower than one release per month
- Ensure at least 1 minor release exist with a retro compatible code exists that warns about the deprecation

What is not true today in the process above

Carvel team does not own the chocolatey release channel
Github action is still in k14s organization
Docker images only exist in Docker Hub and not Github Package Registry

Possible improvements on the process

Easy wins

The version is read from the same file in all the repositories
All PR's or Commits contain the Github formula that automatically closes Issues that they solve

Automation of parts of the release process

Slack notifications

It would be interesting to provide a way to send automatic messages to both Slack channels when a release is published
Generation and inclusion of binaries in release notes

Automate the generation of the binaries and attachment to the release notes
1. Generate binaries
2. Generate the Checksums
3. Add the binaries to the draft release
4. Add the checksums to the text on the release
Simple way to gather all changes since last release

It would be helpfull to create a way to gather all the issues/stories that where finished since the last release. Even if this does not create the release note it would help us not forget any issue.
Update all the places an artifact needs to be updated

Having to update 4 or 5 different places with the same information is tedius and error prune. If we could automate the commit process in all the needed repositories would be awesome.

For some of these maybe we can leverage things like dependabot.
Packaging tool files automation

If there was a file in all the packaging repositories that contained the version and checksums. It would make the update easier, we could generate the final files using ytt.

ahurley

2020/12/16 22:37:49

# Table of Con

Is there supposed to be a link here? (Edited)

joaopereira

2020/12/17 20:54:22

I thought that maybe it would make sense to create a github issue to capture this work. But again this is left overs from Schemas. Do you think we should keep it or remove it?

2020/12/17 21:15:07

This feels like a part of our "proposal process" that is yet to be defined. I don't have a strong opinion on this right now. It would be nice to have this sort of thing be more visible in the future so that the community can contribute if they want.

2020/12/16 22:47:57

For each one of the above categories, there are consideratio

I'm not sure if I see the difference between these two items. Could you provide an example of a milestone that isn't a set of features/bugs? (Edited)

2020/12/16 22:49:22

If we release today and an urgent issue is discovered tomorrow can we solv

I'm not sure I'm following this guidance. Is this like the "don't deploy on Fridays" rule? (Edited)

cdean

2020/12/17 00:50:12

I see this as "a set of features or bugs" is a subset of a milestone. Maybe a milestone is a release in tracker for example, well this would be a not quite a milestone, but still a chunk of work. (Edited)

2020/12/17 14:52:57

Yes I was trying to differentiate between a full track of work, like Schemas and a group of solved github issues that are not part of any particular track of work (Edited)

Steven Locke

2020/12/18 17:51:48

Is there a github search we could use for this? (Edited)

2020/12/23 21:40:37

Unfortunately not that I am aware of. The best option would be to find the last tag and get a date. From that date try to get the commits

Guest Hardy2020/12/18 18:02:51

Breaking changes: **Major** bump

i think we can we a bit more "flexible" here given that tools havent reached 1.0. it's probably worth discussing case by case for now. (Edited)

Guest Hardy2020/12/18 18:04:06

enough time to prepare

currently proposed times to me sound very small for the type of foundational tools we are building. probably worth discussing further. (Edited)

2020/12/23 21:49:36

These are just numbers that I put together, we definitely should discuss to see if they are appropriate or not

Guest Hardy2020/12/18 18:06:05

The version is read from the same file in all the repositories

need to consider case when repo contains multiple binaries/controllers. (Edited)

2021/01/05 16:59:53

I agree with that, nevertheless maybe we can centralize in the root of the directory a file called `version.go` that would contain the version of all the binaries.

2020/12/18 18:10:35

I'm confused on these steps. I read this list as "Perform step 1. Steps 1a and 1b are the result". I think I'm mis-reading that :) Running the `hack/build-release.sh` script appears to generate a yml file. Is the script expected to push it to Docker? If not, how do we push it? (Edited)

2020/12/18 18:25:32

automatic messages

Should these be generated? Or would we manually create the message to send? (Edited)

2020/12/23 21:51:26

We do have a template for these messages. The only concern about auto generation is the fact that we cherry pick some messages from a place to another.

dhelfand

2020/12/21 15:51:18

The product achieved a milestone

What is the difference between these two? It would also be helpful to define what a milestone is. (Edited)

2020/12/23 21:29:56

Does it look better?

2020/12/21 15:59:47

Making note that there are tools like goreleaser for this that automatically generate templates when a release is pushed. The template is configurable through goreleaser and would help with having to have the person carrying out the release find all relevant bugs/features manually. (Edited)

2020/12/21 16:05:09

+1 I think some details are missing here on pushing the OCI image. Can confirm the script only generates the YAML as shown here: https://github.com/vmware-tanzu/carvel-kapp-controller/blob/5b4bb4308590142b83efe4be8a3ab28f52cc0d08/hack/build-release.sh. (Edited)

2020/12/21 16:08:36

I am still wondering if it makes sense to use Chocolatey versus a different package manager. Perhaps we should discuss before putting support behind this. (Edited)

2020/12/23 21:47:38

I agree was just assuming since we already have someone that did the work for chocolatey we could leverage it. But I am open to suggestions

2020/12/21 16:12:24

All PR's or Commits contain the Github formula that automatically closes Issues that they solve

There are ways to avoid having to make contributors go through this process of adding issues to commit messages, such as using tools like goreleaser. (Edited)

iamhsk

2021/01/04 22:42:11

product

I'm assuming both major & minor releases will have significant time lapsed in between so there would have been enough time to align between dev and product. For patch releases, i'm cool with the dev team deciding on when to do it while informing the rest of the team members including product. (Edited)

2021/03/10 18:25:01

A PR to update this repo as it currently exists (maintained by github user adriens) would include the tool binary. We would have better security control if we maintain this repo. I believe we haven't yet started submitting PRs to this current repo. Do we want to remove this step until we adopt this repository? (Edited)

Rohit Aggarwal

2021/12/07 05:48:29

#### Update Homebrew In the

What is the purpose of this? And how to verify as I get nothing in the web page. (Edited)

Carvel Authors

2022/07/27 00:17:37

If you're not seeing the Concourse pipeline running, you might need additional permissions. You may need to be a member of the GitHub organization named "pivotal-cf". It is through that membership that others in Carvel have access to those pipelines (and the source).

Guest Gibson2022/07/06 02:48:16

Update all the places an artifact needs to be updated

Jehdhhsjsjdhdjdkjdhxkxmxhdhdkld la hdkdhbjx🥰🥲🥰😄😚☺️🦮😚😄🥰😄😚☺️☺️😄🥰🦮🥰🦮 (Edited)

2020/12/16 22:56:40

I'm curious to hear Helen's thoughts on this as it would create a bottleneck. (Edited)

2020/12/17 14:54:57

Yes something of the sorts (Edited)

Guest Hardy2020/12/18 17:57:58

ive previously added "thanks to X" by the contibuted item to give more context (example: https://github.com/vmware-tanzu/carvel-kapp/releases/tag/v0.34.0) (Edited)

Guest Hardy2020/12/18 18:07:38

this one to me is most higher priority one as we add more distribution channels. (Edited)

2020/12/21 15:54:26

[

Would be helpful to more explicitly state to push to develop branch. (Edited)

2020/12/21 15:56:13

Would be helpful to include instructions on how to do this through GitHub UI. (Edited)

2020/12/21 16:15:03

If we are not using imgpkg here, we should consider it as a way of dogfooding are own tools. (Edited)

Guest Gibson2022/07/06 02:48:27

Hdjsoslahdhdklslshdkslsjdhdjdkslshdhdk (Edited)

Guest Gibson2022/07/06 02:48:37

Kkdujs😁🧘‍♂️💜🤣😅😉😅😅🧘‍♂️🤣💜🤣 (Edited)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Carvel Tools Release Process

Table of Contents

Release Process Proposal

Goals

Anti Goals

Release decision

How to release

Release Checklist

Bump version

Automatically Created Draft Release

Manually Created Draft Release

Create Draft Github Release

Create the Release Notes

[Not required] Generate Binaries and Checksum

[Not Required] Upload Binaries and Checksum

Publish the draft release

Update Homebrew

Update Website Installation Script

Update Chocolatey

Update Github Action

Generate new docs version

Push OCI Images to Registry

Communicate in Slack

Announce in community meeting

Versioning

Deprecation of functionality

What is not true today in the process above

Possible improvements on the process

Easy wins

Automation of parts of the release process