Tekton Catalog Reorganization

--- status: implementable title: Tekton Catalog Reorganization creation-date: '2022-07-12' last-updated: '2022-07-12' authors: - "@jerop" - "@vdemeester" - "@vinamra28" - "@QuanZhang-William" see-also: - TEP-0003 - TEP-0079 - TEP-0110 ---  - [Summary](#summary) - [Motivation](#motivation) - [Proposal](#proposal) - [Flat Structure](#flat-structure) - [Versioning](#versioning) - [Commits](#commits) - [Git Resolver](#git-resolver) - [Hub Resolver](#hub-resolver) - [Tags](#tags) - [Git Resolver](#git-resolver-1) - [Hub Resolver](#hub-resolver-1) - [HTTP Endpoint](#http-endpoint) - [Migration](#migration) - [References](#references)  # TEP-0115: Tekton Catalog Reorganization ## Summary Tekton Catalog organization is directory-based which presents challenges in versioning and scaling. In this TEP, we propose a git-based organization contract for Tekton Catalogs. This TEP builds on prior work in [TEP-0003][tep-0003] & [TEP-0110][tep-0110] and is a blocker for [TEP-0079][tep-0079]. ## Motivation As of today, the Tekton Catalog organization is directory-based: versions are expressed as directories and resource versions organized as directories. For further information, see [TEP-0003: Tekton Catalog Organization][tep-0003]. For example, the directory structure with two `Tasks` and one `Pipeline` is organized as such: ```shell ./task/ /argocd /0.1 /README.md /argocd.yaml /samples /tests /0.2 /README.md /argocd.yaml /samples /tests /OWNERS /golang-build /0.1 /README.md /golang-build.yaml /samples /tests /0.2 /README.md /golang-build.yaml /samples /tests /OWNERS ./pipeline/ /go-release /0.1 /README.md /go-release.yaml /samples /tests /0.2 /README.md /go-release.yaml /samples /tests /OWNERS ``` The directory structure presents challenges: - Bumping versions of resources is cumbersome which encourages minor making changes in existing versions. Some of these changes that seem minor lead to surprising breaking behavior for users, as described in [tektoncd/catalog#784][784]. - Resources that are frequently updated with new versions end up with a lot of duplications; because any change requires copying over everything into a new directory. For example, [git-clone][git-clone] `Task` has 7 versions currently. Previously, there was tight coupling between the organization and how users referenced resources which made it hard to change the organization of the Catalog without breaking users. In [TEP-0110][tep-0110], we proposed decoupling Catalog organization from resource reference; now that it's decoupled, we can reorganize the Catalog to address the challenges with the directory structure. The aim of this TEP is to create a new organization contract for the Tekton Catalog, building on the prior work and experiences from [TEP-0003][tep-0003]. In [TEP-0079: Tekton Catalog Support Tiers][tep-0079], we plan to add a Tekton Official Catalog which should use this new organization contract - this proposal is a blocker for [TEP-0079][tep-0079]. ## Proposal We propose supporting a git-based organization of Tekton Catalogs where versions are expressed using tags and commits. ### Flat Structure In this organization contract, we'll use a flat structure instead of the directory structure. The example shown above with the two `Tasks` and one `Pipeline` would be organized as such: ```shell ./task/ /argocd /README.md /argocd.yaml /samples /tests /OWNERS /golang-build /README.md /golang-build.yaml /samples /tests /OWNERS ./pipeline/ /go-release /README.md /go-release.yaml /samples /tests /OWNERS ``` ### Versioning #### Commits Users can reference specific versions of the resources using commits. ##### Git Resolver [Git Resolver][git-resolver] has a `"commit"` parameter that can be used to get a specific version of a resource, for example [this commit][bp-05] of buildpacks `Task`: ```yaml apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: buildpacks spec: taskRef: resolver: git resource: - name: url value: https://github.com/tekton/catalog.git - name: path value: task/buildpacks/buildpacks.yaml - name: commit value: d9183e36f00c712b5de59932650c071674ea89b8 ``` A user can specify either `"commit"` or `"branch"`, but not both. ##### Hub Resolver [Hub Resolver][hub-resolver] should take `"commit"` parameter that can be used to get a specific version of a resource, for example [this commit][bp-05] of buildpacks `Task`: ```yaml apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: buildpacks spec: taskRef: resolver: hub resource: - name: kind value: task - name: name value: buildpacks - name: commit value: d9183e36f00c712b5de59932650c071674ea89b8 ``` A user can specify either `"commit"` or `"version"`, but not both. #### Tags Tagged releases should be made so that they can be consumed by users who'd prefer to use tags instead of releases. The tags for versioning Catalogs should continue to use the simpler version of [semver][semver]: `{major}.{minor}`. TODO(jerop): add more details on tagging ##### Git Resolver [Git Resolver][git-resolver] has a `"branch"` parameter that can be used to get a tagged version of a resource, for example [this commit][bp-05] of buildpacks `Task` is tagged with "v0.5": ```yaml apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: buildpacks spec: taskRef: resolver: git resource: - name: url value: https://github.com/tekton/catalog.git - name: path value: task/buildpacks/buildpacks.yaml - name: branch value: v0.5 ``` A user can specify either `"commit"` or `"branch"`, but not both. ##### Hub Resolver [Hub Resolver][hub-resolver] has a `"version"` parameter that can be used to get a specific version of a resource, for example [this commit][bp-05] of buildpacks `Task` is tagged with "v0.5": ```yaml apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: name: buildpacks spec: taskRef: resolver: hub resource: - name: kind value: task - name: name value: buildpacks - name: version value: 0.5 ``` A user can specify either `"commit"` or `"version"`, but not both. ##### HTTP Endpoint In [TEP-0110][tep-0110], we added an HTTP endpoint that serves Tekton resources that users can apply directly to their clusters. These endpoints should continue working as: ```shell kubectl apply -f https://hub.tekton.dev/tekton/task/buildpacks/0.5/raw ``` ### Migration The Hub should support both the directory-based and git-based organization contracts to provide backwards compatibility during migration to git-based only. The Tekton Official Catalog - tektoncd/catalog-official - will be added using the git-based organization (no migration). The existing Tekton Catalog - tektoncd/catalog - should remain as-is using the directory-based organization, but is deprecated. We will create a new Tekton Community Catalog - tektoncd/catalog-community - that will use the git-based organization. The Hub will serve both Catalogs for 9 months during migration, after which tektoncd/catalog will be removed. This renaming from tektoncd/catalog to tekton/catalog-community also clarifies the naming of the two Catalogs that Tekton will provide for support tiers, as proposed in [TEP-0079][tep-0079]. ## References - Tekton Enhancement Proposals - [TEP-0003: Tekton Catalog Organization][tep-0003] - [TEP-0079: Tekton Catalog Support Tiers][tep-0079] - [TEP-0110: Decouple Catalog Organization and Resource Reference][tep-0110] - Issues: - [tektoncd/catalog#784: versioning resources whenever change is made][784] [tep-0003]: 0003-tekton-catalog-organization.md [tep-0079]: 0079-tekton-catalog-support-tiers.md [tep-0110]: 0110-decouple-catalog-organization-and-reference.md [784]: https://github.com/tektoncd/catalog/issues/784 [git-clone]: https://github.com/tektoncd/catalog/tree/main/task/git-clone/0.7 [git-resolver]: https://github.com/tektoncd/resolution/tree/7f92187843085874229aa4c43e5c6d7d392a26fa/gitresolver [hub-resolver]: https://github.com/tektoncd/resolution/tree/5d7918cb5b6f183d79cf0f91f4f08ecb204505a0/hubresolver [bp-05]: https://github.com/tektoncd/catalog/blob/d9183e36f00c712b5de59932650c071674ea89b8/task/buildpacks/0.5/buildpacks.yaml [semver]: https://semver.org/