# Umbrella issue: k8s.gcr.io => registry.k8s.io solution #1834 This markdown is synced from https://hackmd.io/gN-1GeSpSgyNSvmjKSULbg?edit to https://github.com/kubernetes/k8s.io/issues/1834#issue-841372237 manually by @BobyMCBobs Scope: https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io) Design Doc: https://docs.google.com/document/d/1yNQ7DaDE5LbDJf9ku82YtlKZK0tcg5Wpk9L72-x2S2k/edit (shared w/ dev@kubernetes.io and SIG mailing list) Board: https://github.com/orgs/kubernetes/projects/77 DRAFT AIs that need filled turned into tickets: https://github.com/orgs/kubernetes/projects/77/views/2?filterQuery=is%3Adraft # What exactly are you doing? (and how?) - [x] We are setting up an AWS account with an IAM role and s3 buckets in AWS regions where we see a large percentage of source image pull traffic - [x] IAM Role + User for write - [k/k8s.io#3568](https://github.com/kubernetes/k8s.io/issues/3568) - [x] List of Regions with 80% traffic - [oci-proxy#39](https://github.com/kubernetes-sigs/oci-proxy/issues/39) - [x] Determine which regions should serve the image layers - [oci-proxy/issues#38](https://github.com/kubernetes-sigs/oci-proxy/issues/38) - kubernetes/k8s.io#3568 - kubernetes/k8s.io#3620 - kubernetes-sigs/promo-tools#533 - [x] Create Buckets per Region Names - [x] Add Terraform for registry-k8s-io s3 buckets [k/k8s.io#PR3605](https://github.com/kubernetes/k8s.io/pull/3605) - [x] Create s3 Buckets in AWS Regions [k/k8s.io#3595](https://github.com/kubernetes/k8s.io/issues/3595) - [x] We will iterate on a sandbox url (registry.sandbox.k8s.io) for our experiments and ONLY promote things to (registry.k8s.io) when we have complete confidence - [x] standup sandbox deployment [oci-proxy#13](https://github.com/kubernetes-sigs/oci-proxy/issues/13) - [x] Bulid sandbox infrastructure for oci-proxy [k8s.io#3317](https://github.com/kubernetes/k8s.io/issues/3317) - [x] Auto deploy for archeo in staging to registry-sandbox.k8s.io [k8s.io#3577](https://github.com/kubernetes/k8s.io/issues/3577) - [x] Full e2e testing [oci-proxy#24](https://github.com/kubernetes-sigs/oci-proxy/issues/24) - [x] both registry and registry.sandbox are serving traffic using oci-proxy on google cloud run - Is this true now? - [x] oci-proxy will be updated to identify incoming traffic from AWS regions based on IP ranges so we can route traffic to s3 buckets in that region. If a specific AWS region do not currently host s3 buckets, we will redirect to the nearest region which does have s3 buckets (tradeoff between storage and network costs) - [ ] [oci-proxy should detect per region (80% of total traffic) #39](https://github.com/kubernetes-sigs/oci-proxy/issues/39#issuecomment-1085230292) - We will bulk sync existing image layers to these s3 layers as a starting point (from GCS/GCR) - [x] [kubernetes/k8s.io#3666](https://github.com/kubernetes/k8s.io/issues/3666) - We will update image-promoter to push to these s3 buckets as well in addition to the current setup - [ ] Update image-promoter - [DRAFT](https://github.com/orgs/kubernetes/projects/77/views/2?filterQuery=Update+image-promoter+to+copy+to+multiple+regions) - We will set up monitoring/reporting to check on new costs we incur on the AWS infrastructure and update what we do in GCP infrastructure as well to include the new components - [ ] AWS Cost Monitoring - [DRAFT](https://github.com/orgs/kubernetes/projects/77/views/2?filterQuery=Research+Reports+for+AWS+Cost+Explorer) - [ ] Collection and loading of AWS cost explorer data [kubernetes/k8s.io/infra/aws/aws-costexplorer-export](https://github.com/kubernetes/k8s.io/tree/main/infra/aws/aws-costexplorer-export) - We will have a plan in place on how we could add additional AWS regions in the future - [x] Document adding additional AWS Regions - [kubernetes/k8s.io/infra/aws/terraform/registry.k8s.io/README.md#extending-regions](https://github.com/kubernetes/k8s.io/tree/main/infra/aws/terraform/registry.k8s.io#extending-regions) - We will have CI jobs that will run against registry.sandbox as well to monitor stability before we promote code to registry - [x] Full e2e testing using one parallel Conformance CI job - [k-sigs/oci-proxy#24](https://github.com/kubernetes-sigs/oci-proxy/issues/24) - [x] Add e2e tests - [oci-proxy/pr#32](https://github.com/kubernetes-sigs/oci-proxy/pull/32) - [x] Choose an e2e job to use registry-sandbox - [x] Configure chosen e2e sandbox with a testgrid dashboard - We will automate the deployment/monitoring and testing of code landing in the oci-proxy repository - [x] [Auto deploy for archeo in staging to registry-sandbox.k8s.io ยท Issue #3577](https://github.com/kubernetes/k8s.io/issues/3577)