Possible implementations for registry.k8s.io

# Possible implementations for registry.k8s.io ## Anycast DNS An IP and a LoadBalancer in each CloudProvider pointed to by the same DNS record, spliting by proximity (preferable down to the ASNs of each CloudProvider). ## 302 redirect Perform a 302 redirect to the cloud provider based on the ASN. https://en.wikipedia.org/wiki/HTTP_302 https://github.com/goharbor/harbor/issues/14411 https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-snippet ## Config layer URLs Set up the config.json [layers[1]](https://github.com/opencontainers/image-spec/blob/master/config.md#layer) / [layers[2]](https://github.com/opencontainers/image-spec/blob/master/descriptor.md#properties) urls to point to multiple HTTPs URLs which are stored in different CloudProviders. Implement dynamic config.json generation or mutation based on the CloudProvider ASN in the URL. Use something like EnvoyProxy to rewrite the values on request depending on source IP (?). Will require extra layers of automation to function. Due to the nature of container registries, it appears not so possible to use different direct blob URLs in the URLs field of a layer in it's config - this is due to the requirement of a pull token per each container image pull. ## Multicluster Ingress Might require a service mesh, like Linkerd or Istio, but will provide distribution of traffic out to multiple clusters. Will require a few cronjobs to sync artifacts out. This would likely require more maintainance. # Harbor review for this project Harbor would be a solution at the provider level. ## URLs Currently some images are at the namespace level of the path in k8s.gcr.io, such as k8s.gcr.io/pause. Other images are at a project subpath in the URL, such as k8s.gcr.io/e2e-test-images/agnhost. In Harbor, it isn't possible to push an image to the namespace level of the URL path. Is it somehow possible to push to a URL like myregistry.com/pause? This might be possible to rewrite the request to point back to _/(something)_ with a custom [nginx-ingress server-snippet conditional](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-snippet) as a 302 redirect (?) ## Configuration For this project all configuration must be declaritive. As far as I'm aware it's not possible to configure things like projects, registries, and accounts through the Helm Chart. Is this possible or planned? https://github.com/goharbor/harbor-helm/issues/884 GCS storage backend can be configured with _persistence.imageChartStorage.gcs_ in the Helm Chart, provided with same options as: https://github.com/distribution/distribution/blob/main/docs/configuration.md#storage ## Artifacts It appears that Harbor can only store container images. Unsure about binary blob storage. ## Setting up k8s.gcr.io as a Harbor registry endpoint k8s.gcr.io can be cached with Harbor. Unique to Harbor, this requires a GCP service account. When setting it up, the url k8s.gcr.io cannot be used as a proxy-cache ... unless you first decare it as gcr.io before creating the project configured as a proxy-cache. # [Distribution](https://github.com/distribution/distribution) review for project ## Declaritive configuration Distribution supports declarative configuration, through a YAML file that is referred to when the registry is brought up. https://github.com/distribution/distribution/blob/main/docs/configuration.md ## Able to act as proxy-cache In the _proxy_ field of the distribution configuration allows for set up of this feature. Setting _proxy.remoteurl_ to gcr.k8s.io works as expected. https://github.com/docker/docker.github.io/blob/master/registry/recipes/mirror.md#run-a-registry-as-a-pull-through-cache ## Support for GCS backend and many others GCS can be configured as a storage backend in the _storage.gcs_ field. All blobs are configs will be stored in the bucket backend. ## CDN support through Middleware A CDN such as AWS Cloudfront or Google Cloud CDN. This would provide a layer on caching for all layers of the container images. https://github.com/distribution/distribution/blob/main/docs/configuration.md#example-middleware-configuration # [Container Image Promoter](https://github.com/kubernetes-sigs/k8s-container-image-promoter) review for project This would be responsible for reconciling registries and their images. This is already used in k8s infra production. Currently only supports GCP - this project requires extending it's registry support. Examples of usage: - Thin configuration: https://github.com/kubernetes/k8s.io/blob/main/k8s.gcr.io/manifests/k8s-image-staging-kind/promoter-manifest.yaml - ProwJob: https://github.com/kubernetes/test-infra/blob/821be15e2a39664243b363fe99607d70892881fd/config/jobs/kubernetes/sig-release/cip/container-image-promoter.yaml#L4