# Status of usage of AWS Credits for Kubernetes

Google has been paying the bills for community infrastructure for a long time since the inception of the Kubernetes project. A few years ago Google switched to the mode where they wanted CNCF and Kubernetes community to own the infrastructure and ensure it is run by the community (and take Google folks as the only ones having admin access). Towards this effort Google granted credits of $3 million dollars per year. There was a Special Interest Group in Kubernetes community called SIG-K8s-infra to help move the infra out from google's hands BUT still run on GCP as there was no other alternative.

This SIG has been working on a lot of projects to help Kubernetes community have access to compute/network/storage resources across the board as needed by other SIG(s) in Kubernetes. Last few years a trend emerged where AWS customers, some of those who use EKS and many of those who run their clusters using own custom/proprietary k8s distros have been placing a lot of load on the community infrastructure. One example is container images, AWS generates a LOT of traffic across clouds as folks running things in AWS reach out across cloud boundaries (think egress costs) and pulling images from the google hosted registries (GCR). So the community reached out to AWS leadership and this year we are funding $3 million to match Google's commitment.

The SIG has been thinking about and building infrastructure across GCP and AWS now. For example the CI system called prow which runs on GCP has now been stitched together with a prow on EKS as well. So CI jobs in the community can run on either cloud. There are various classes of CI jobs, so some of them are easily portable and we are working on ensuring all classes of jobs can work on either cloud. Another example is scalability jobs for 100 node cluster and 500 node clusters etc which take a lot of $$'s to run as well. So community is coming up with ways and means to run these as well in AWS.

At the moment, one of the very first things that a community member or end of kubernetes will see is that there is a new container registry which is state of the art proxy that redirects traffic for large blobs to a location that is nearer to where the cluster is running. For example a cluster running on AWS will get redirected to a s3 bucket in the same region for large files. This helps in keeping egress costs down as well as speed up access and reduce latency as well. This is a net new win for AWS customers.

There are several such efforts in progress, the community has set itself a priority 0 and priority -1. priority zero is to ensure we use the credits from AWS responsibly and in a sustainable fashion to build a multi cloud infrastructure for the community. priority -1 is to reduce spend on GCP by any means necessary to create headroom there to ensure that we don't need to go ask google for extra funds in the last quarter of the year. This will also help with rolling out more things we can be doing in GCP that we have been holding off as we tended to use up all the credits there. There are things that will be happening for binary artifacts, signed release artifacts, CDN(s) etc over time.

What does AWS get out of this? (enumerating just a few here!):
- Community is in better shape
- More testing especially on AWS
- Better tools include AWS as first class citizen
- goodwill from the community
- Our customers have better access to community infrastructure
- Faster / lower latency
- EKS development team
- CI jobs are already running on AWS, easier to support versions of k8s that are not supported by the community

Most important of all, Community is more sustainable now which ensures that the revenue stream to AWS from EKS/Kubernetes will be more stable and growing going forward.

# What we can celebrate now?

Community started from a single registry in GCR, to a globally distributed solution in GCR, to now a globally distributed, multi-provider mirroring network across cloud providers. This registry running on GCP and AWS at first, with more expected to onboard. Everyone using the older registry (`k8s.gcr.io`) can and should move to the new registry at `registry.k8s.io`. Kubernetes clusters running on GCP will be served large files from within GCP and AWS customers who are running EKS or their own DIY Kubernetes clusters will get redirect to region specific S3 buckets for large/ frequently used files which will reduce egress traffic, latency and improve download speeds. All non-GCP users of `registry.k8s.io` are served from AWS S3 buckets as well.

More details at:

[1] https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/
[2] https://kubernetes.io/blog/2023/02/06/k8s-gcr-io-freeze-announcement/
[3] https://aws.amazon.com/blogs/containers/changes-to-the-kubernetes-container-image-registry/