Helm: new `imageRegistry`, `imageNamespace`, and `imageSuffix` values

# Helm: new `imageRegistry`, `imageNamespace`, and `imageSuffix` values > [!NOTE] > > This document goes over the two main use-cases regarding the design proposal in [PR 8115](https://github.com/cert-manager/cert-manager/pull/8115). ## Problem The cert-manager's Helm chart is notoriously painful to use with OCI mirroring registries such as Artifactory, as shown by the multiple GitHub issues and PRs: - https://github.com/cert-manager/cert-manager/issues/6160, - https://github.com/cert-manager/cert-manager/issues/7101, - https://github.com/cert-manager/cert-manager/pull/7558, - https://github.com/cert-manager/cert-manager/pull/8115. In many other Helm charts, all you have to do is to set the `global.image.registry` to your Artifactory's mount point. But with cert-manager, you will have to rewrite the entire image names for all 4 images. ## Proposed solution We would like to add three additional Helm values to help with two use-cases. Here is what the default values will look like: ```yaml= imageRegistry: quay.io imageNamespace: jetstack imageSuffix: "" ``` Note that `imageRegistry` doesn't have to be an actual OCI registry (i.e., host:port). Like many other Helm charts do, we use a looser definition: "the thing that comes before the Docker repository". The Docker repository would be jetstack/cert-manager-webhook, in this case. You can read more about the term "repository" in the section [Have we been using the term repository wrong?](#Have-we-been-using-the-term-repository-wrong). Here is how images are constructed with these default values: ``` quay.io/jetstack/cert-manager-startupapicheck <-----> <------> <--------------------------><---> imageRegistry imageNamespace image.name imageSuffix (empty) ``` ## Use-case 1: Artifactory mirroring OSS images Users of the helm chart with Artifactory-mirrored images have to repeat the same thing over and over: ```yaml= image: registry: artifactory.domain.com/docker repository: jetstack/cert-manager-controller acmesolver: image: registry: artifactory.domain.com/docker repository: jetstack/cert-manager-acmesolver webhook: image: registry: artifactory.domain.com/docker repository: jetstack/cert-manager-webhook cainjector: image: registry: artifactory.domain.com/docker repository: jetstack/cert-manager-cainjector startupapicheck: image: registry: artifactory.domain.com/docker repository: jetstack/cert-manager-startupapicheck ``` This is because we chose, [in 2017](https://github.com/cert-manager/cert-manager/pull/145), to use the full image name as the default for `repository`, instead of just the part after the registry: ```yaml= # Current defaults: image: # registry: quay.io repository: quay.io/jetstack/cert-manager-controller ``` Once we add the three new values, Artifactory users will be able to do: ```yaml= imageRegistry: artifactory.domain.com/docker ``` Here is how the image is reconstructed: ``` artifactory.domain.com/docker/jetstack/cert-manager-startupapicheck <---------------------------> <------> <--------------------------><---> imageRegistry imageNamespace image.name imageSuffix ``` ## Use-case 2: CyberArk-specific use-cases Right now, CyberArk customers have to do: ```yaml image: repository: private-registry.venafi.cloud/cert-manager/cert-manager-controller acmesolver: image: repository: private-registry.venafi.cloud/cert-manager/cert-manager-acmesolver webhook: image: repository: private-registry.venafi.cloud/cert-manager/cert-manager-webhook cainjector: image: repository: private-registry.venafi.cloud/cert-manager/cert-manager-cainjector startupapicheck: image: repository: private-registry.venafi.cloud/cert-manager/cert-manager-startupapicheck ``` ### Use-case 2.1: non-FIPS + CyberArk registry When using non-FIPS images, they will be able to use: ```yaml imageRegistry: private-registry.venafi.cloud imageNamespace: cert-manager ``` Here is how the image is reconstructed: ``` private-registry.venafi.cloud/cert-manager/cert-manager-startupapicheck <---------------------------> <----------> <--------------------------><---> imageRegistry imageNamespace image.name imageSuffix ``` ### Use-case 2.2: FIPS + CyberArk registry If they are using our STS FIPS images, they will be able to use: ```yaml imageRegistry: private-registry.venafi.cloud imageNamespace: cert-manager imageSuffix: -fips ``` Here is how the image is reconstructed: ``` private-registry.venafi.cloud/cert-manager/cert-manager-startupapicheck-fips <---------------------------> <----------> <--------------------------><---> imageRegistry imageNamespace image.name imageSuffix ``` ### Use-case 2.3: non-FIPS + Artifactory mirroring CyberArk images And if they are using Artifactory as a mirror, imagining that their images are: ``` artifactory.domain.com/venafi/cert-manager/cert-manager-controller ``` Then they can do: ```yaml imageRegistry: artifactory.domain.com/venafi imageNamespace: cert-manager ``` ### Use-case 2.4: FIPS + Artifactory mirroring CyberArk images ```yaml imageRegistry: artifactory.domain.com/venafi imageNamespace: cert-manager imageSuffix: -fips ``` ## Coherence with the other cert-manager Helm charts Here is a comparison of how other cert-manager-related projects structure their Helm values for images: | Project | Where are images configured? | Existing default for `image.registry` | Existing `image.repository` | Notable differences | |-------------------------|-----------------------------------------------|---------------------------------------|-------------------------------------------------------------|-----------------------------------------------------------------------------------| | csi-driver | `image.*` + per-component images | no default set | `quay.io/jetstack/cert-manager-csi-driver` | Per-component sidecars (node registrar, liveness) with their own image blocks | | istio-csr | `image.*` | no default set | `quay.io/jetstack/cert-manager-istio-csr` | Standard single `image` block | | google-cas-issuer | `image.*` | "" | `quay.io/jetstack/cert-manager-google-cas-issuer` | Registry field exists but default is empty | | aws-privateca-issuer | `image.*` | field doesn't exist | `public.ecr.aws/k1n1h4h4/cert-manager-aws-privateca-issuer` | No registry override field at all | | approver-policy | `image.*` | no default set | `quay.io/jetstack/cert-manager-approver-policy` | Standard single `image` block | | trust-manager | `image.*` + `defaultPackageImage.*` | no default set | `quay.io/jetstack/trust-manager` | Has `global.rbac` only; no global image settings | | csi-driver-spiffe | `image.*` with repo map (`driver`,`approver`) | no default set | See *(1)* | `image.repository` is a map, not a string | | openshift-routes | `image.*` | no default set | `ghcr.io/cert-manager/cert-manager-openshift-routes` | Standard single `image` block | | cert-manager (proposal) | `image.*` + per-component images | no default set | `quay.io/jetstack/cert-manager-csi-driver-spiffe` | each componant has its own `<component>.image.*` block, controller uses `image.*` | *(1)* For csi-driver-spiffe, the `image.repository` is the only one that uses a map with keys `driver` and `approver`, each containing full image names, i.e., ```yaml= image: repository: driver: quay.io/jetstack/cert-manager-csi-driver-spiffe approver: quay.io/jetstack/cert-manager-csi-driver-spiffe-approver ```  ## Have we been using the term repository wrong? Short answer: yes, we use it the wrong way. "Docker repository" dates back to the beginning of Docker. Originally, this term belonged to the "Docker hosted" images. That's why a "Docker repository" doesn't have a domain: docker.io was assumed. Examples: - `ubuntu` (which is a shorthand for`library/ubuntu`) - `nicolaka/netshoot` Over time, other registries were created, and things became fuzzy. The term "image" also shifted from being just a repository and a tag (`ubuntu:latest`) to being a registry host, repository, and tag (`docker.io/library/ubuntu:latest`). The [OCI distribution spec](https://specs.opencontainers.org/distribution-spec/) clarified the meaning of "repository": > Repository: a scope for API calls on a registry for a collection of content (including manifests, blobs, and tags). > > To pull a manifest, perform a GET request to a URL in the following form: > > ``` > /v2/<name>/manifests/<reference> > ``` > > `<name>` refers to the **namespace of the repository**. <reference> MUST be either (a) the digest of the manifest or (b) a tag. The <reference> MUST NOT be in any other format. An "OCI repository" isn't `ubuntu`, nor `docker.io/library/ubuntu`... it's just scope for API calls. The identifiers `ubuntu` and such are "OCI repository namespaces" (shortened as "repository names" in the rest of the spec). Thus, an OCI repository doesn't contain the registry host. Many projects who started with docker.io images migrated away. In their Helm charts, they used the `repository` value to mean the "Docker-hosted repository". For example, that's what the ArgoCD Helm chart looked like a long time ago: ```yaml image: repository: argoproj/argocd ``` When ArgoCD moved away from docker.io [in 2021](https://github.com/argoproj/argo-helm/commit/3ba4cdb1ca9bf1c7bdc561b2255886ff4cb17949#diff-9eb59d56febda5e405748483497dfdc0c2dfbe9b4f386a12f0180c98d99051e0L366), the project kept using the term "repository": ```diff global: image: - repository: argoproj/argocd + repository: quay.io/argoproj/argocd ``` There is still no `global.image.registry` in ArgoCD's [values.yaml](https://github.com/argoproj/argo-helm/blob/6780539878c268460e3af9b3aefd54b15993a5d1/charts/argo-cd/values.yaml#L60) as of today. The cert-manager also started with docker.io. In 2017, its [values.yaml](https://github.com/cert-manager/cert-manager/commit/7194492447d5ea7bf5e0075cf87f02c3a3c2fb18) looked like this: ```yaml image: repository: jetstackexperimental/cert-manager-controller ``` KEDA did things somewhat differently... In 2020, they used to use the term "image" (which corresponds to the first in the Pod spec). When they [changed](https://github.com/kedacore/charts/commit/bfabf0fe055ae0af76ead237da57555d8e2d60a8#diff-3b4e4c10436059483f20fa5b4b2e507a0bcead63f71af2c02ca41dbff3c8b3adR7) it to "repository", they kept the same semantics: ```diff image: - keda: docker.io/kedacore/keda:2.0.0-rc - metricsApiServer: docker.io/kedacore/keda-metrics-apiserver:2.0.0-rc + keda: + repository: docker.io/kedacore/keda + metricsApiServer: + repository: docker.io/kedacore/keda-metrics-apiserver ``` When KEDA migrated away from docker.io [in 2021](https://github.com/kedacore/charts/commit/d5e7ec2e29cbb17a992dd5b9c352773c30a3abfd), it became: ```diff image: keda: - repository: docker.io/kedacore/keda + repository: ghcr.io/kedacore/keda metricsApiServer: - repository: docker.io/kedacore/keda-metrics-apiserver + repository: ghcr.io/kedacore/keda-metrics-apiserver ``` They changed to using a `registry` and `repository` later on: ```yaml image: keda: registry: ghcr.io repository: kedacore/keda metricsApiServer: registry: ghcr.io repository: kedacore/keda-metrics-apiserver ``` ## Erik's point: the problem with `imagePrefix` ```yaml= imagePrefix: artifactory.domain.com/docker/jetstack image: name: cert-manager-controller acmesolver: image: name: cert-manager-acmesolver webhook: image: name: cert-manager-webhook cainjector: image: name: cert-manager-cainjector startupapicheck: image: name: cert-manager-startupapicheck ``` ## Notes For consistency with other charts, let's go with the name "registry" even though it isn't the right term. We should not use `global.image.registry` because... that's why we went with `imageRegistry`. The `imageSuffix` isn't really as important, let's not have it as part of this design. ## Next things to be done next - [ ] Felix: Create a new design (PR) since Tim's design is somewhat different. Adjust the hackmd document to the format - [ ] Felix: Present to Atanas the baking + `imageRegistry` solution. - [x] Mael: investigate artifactory (not required but I want to) - [ ] Felix: Prototype it to make sure we can implement it in a non-breaking way