Contributors:
- Jerop
- Vinamra
- Quan

# Automated Testing and Dogfooding

In this section, we design the infrastructure based on Tekton used to test
resources in the so that users can rely on them and maintainers can dogfood
Tekton.

#### Test Pipelines

To test a resource, the owners will to provide a `Pipeline` with three parts:
setup, execution and validation.

##### Setup

The `Pipeline` MAY have inline resources that do any required setup.

##### Execution

The `Pipeline` MUST have the resource under test.

The resource SHOULD be executed more than once. This will be made easier when
we support fanning out resources with combinations of `Parameters`, as proposed
in [TEP-0090: Matrix][tep-0090].

If the resource is a `Pipeline`, then we need `Pipelines` in `Pipelines` support
in Tekton Pipelines as proposed in [TEP-0056: Pipelines in Pipelines][tep-0056].

##### Verification

The `Pipeline` MUST have resources that assert that the resource under test has
worked by:

* Verifying all `Results`
* Verifying all `Workspace` mutations
* Verifying any expected “side effects” (e.g. a successful deployment)

##### Example

#### Test Infrastructure

Today, resource owners specify [hook scripts][hooks] that are run before applying
the resource files. We propose evolving the existing infrastructure by:

1. Implementing them with `Tasks` and `Pipelines` that glue them together
2. Providing pre-provisioned configuration via `Parameters` to the resource's
test `Pipeline`

We will iteratively provide more and more infrastructure for Tekton resources
to test against. At a minimum we will provide:

TODO

#### Versions of Tekton Pipelines

The testing infrastructure will run tests against:
1. The nightly release
2. The latest release
3. The minimum compatible release, as specified using `tekton.dev/minVersion` label

##### Nightly Release

The pro and con of this approach is the same: we may catch issues with Tekton Pipelines
itself. Since backwards incompatible changes require an `apiGroup` bump, this should never
fail, but probably sometimes will. Failures here are probably caused by bugs in Tekton
Pipelines. This means we may catch more issues before releases, but also means that Catalog
tests may fail for non-catalog related reasons.

Since these failures are more likely to be caused by Tekton Pipelines than by the resources,
we will only run these as periodic nightly jobs, and it will be up to build captains to track
down their failures.

##### Latest Release

##### Minimum Compatible Release

As described in [TEP-0003: Tekton Catalog Organization][tep-0003], resources in a Catalog
are required to declare the minimum Tekton Pipelines version they are compatible with, via
`tekton.dev/pipelines.minVersion: <version>` annotations on the resources.

---
(remember to remove links that do not apply)
[catalog-proposal]: https://docs.google.com/document/d/1O8VHZ-7tNuuRjPNjPfdo8bD--WDrkcz-lbtJ3P8Wugs/edit#heading=h.iyqzt1brkg3o
[catalog-hub-design]: https://docs.google.com/document/d/1pZY7ROLuW47ymZYqUgAbxskmirrmDg2dd8VPATYXrxI/edit#
[catalog-support-tiers]: https://docs.google.com/document/d/1BClb6cHQkbSpnHS_OZkmQyDMrB4QX4E5JXxQ_G2er7M/edit?usp=sharing
[tep-0003]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md
[tep-0003-org]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md#organization
[tep-0003-ownership]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md#ownership
[tep-0003-upstream]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md#upstream-catalogs
[tep-0003-hub]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md#the-hub-and-multiple-catalogs
[tep-0091]: https://github.com/tektoncd/community/pull/537
[tep-0060]: https://github.com/tektoncd/community/blob/main/teps/0060-remote-resource-resolution.md
[tep-0090]: https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md
[tep-0056]: https://github.com/tektoncd/community/blob/main/teps/0056-pipelines-in-pipelines.md
[tep-infra]: https://github.com/tektoncd/community/pull/170
[doc-infra]: https://docs.google.com/document/d/1-czjvjfpuIqYKsfkvZ5RxIbtoFNLTEtOxaZB71Aehdg
[github-rename]: https://docs.github.com/en/repositories/creating-and-managing-repositories/renaming-a-repository
[catalog-owners]: https://github.com/tektoncd/catalog/blob/main/OWNERS
[hub]: https://hub.tekton.dev/
[deprecation]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md#deprecation--removal-strategy
[task-authoring-recommendations]: https://github.com/tektoncd/catalog/blob/main/recommendations.md
[bundle]: https://tekton.dev/docs/pipelines/pipelines/#tekton-bundles
[hub-config]: https://github.com/tektoncd/hub/blob/68dfd7ed39ca9fc6ea8eb3c95a729110c6c7f81c/config.yaml#L37-L43
[catlin]: https://github.com/tektoncd/plumbing/tree/main/catlin
[rfc2119]: https://datatracker.ietf.org/doc/html/rfc2119
[bundle-resolver]: https://github.com/tektoncd/community/blob/main/teps/0060-remote-resource-resolution.md#bundle-resolver
[git-resolver]: https://github.com/tektoncd/community/blob/main/teps/0060-remote-resource-resolution.md#git-resolver
[hooks]: https://github.com/tektoncd/catalog/blob/43f52adb4dce0b7f7effb9e7f18e6c2d504c6e4b/DEVELOPMENT.md#end-to-end-testing
[catalog-structure]: https://github.com/tektoncd/catalog/blob/cbc2d9172cf23010c6249b10bf2b8e5adb0b1d76/README.md#catalog-structure