In the default mode of the VPC CNI, without prefix delegation, singular IP addresses are assigned to ENIs. With prefix delegation, a range of IP addresses is assigned to an ENI. Traffic to an address within the assigned range is directed to the ENI. This greatly increases the number of IP addresses, and thus pods, that may be on a single node. More specifically, you can assign a private CIDR range (network prefix) to your network interfaces (ENIs) on EC2 instances. IPv4 and IPv6 ranges may be assigned to an ENI. The prefix assignments may be manually or automatically managed. Amazon VPC CNI assigns network prefixes to Amazon EC2 network interfaces to increase the number of IP addresses available to nodes and increase pod density per node. You can configure version 1.9.0 or later of the Amazon VPC CNI add-on to assign IPv4 and IPv6 CIDR instead of assigning individual IP addresses to network interfaces. [[GDC: JC do you prefer "ENABLE_PREFIX_DELEGATION is set to true" as the first line (techincal accuracy) or "When prefix delgation is enabled" (more general, users should follow the link to learn how to enable)]]

Operating an EKS cluster requires knowledge of VPC networking, in addition to Kubernetes networking. We recommend you understand the EKS control plane communication mechanisms before you start to plan your VPC or use existing VPCs to create Kubernetes clusters. Refer to Cluster VPC considerations and Amazon EKS security group considerations when architecting a VPC and subnets to be used with EKS. [[GDC: SJ & JC - I'm skeptical that the below section is relevant to the BPG. It's more than a general introduction to the reccomendations. The customer and AWS VPCs are a key part of EKS.]] EKS Control Plane Communication An EKS cluster consists of two VPCs:

It is critical to understand Kubernetes networking to operate your applications efficiently. This section of the EKS Best Practices advises on different cluster networking modes. How to use this guide? This guide introduces the Amazon VPC Container Network Interface (CNI) in the context of Kubernetes networking. The VPC CNI is highly configuable to support different use cases. This guide further includes dedicated sections on different VPC CNI use cases, operating modes, and sub-components. Generally, the VPC CNI plugin coordinates VPC resources with a Kubernetes Cluster. THE VPC CNI implements Kubernetes networking concepts using native AWS resources. Kubernetes Networking

A CNI is the container network interface that provides an application programming interface to configure network interfaces in containers. Multus CNI is a container network interface plugin for Kubernetes that enables attaching multiple network interfaces to Pods. In Kubernetes, each Pods has only one network interface by default, other than local loopback. With Multus, you can create multi-homed Pods that have multiple interfaces. Multus acts as ‘meta’ plugin that can call other CNI plugins to configure additional interfaces. Multiple network interfaces for Pods are useful in various use cases; examples include: Traffic splitting: Running network functions (NF) that require separation of control/management, and data/user plane network traffic to meet low latency Quality of Service (QoS) requirements. Performance: Additional interfaces often leverage specialized hardware specifications such as Single Root I/O Virtualization (SR-IOV) and Data Plane Development Kit (DPDK), which bypass the operating system kernel for increased bandwidth and network performance. Security: Supporting multi-tenant networks with strict traffic isolation requirements. Connecting multiple subnets to Pods to meet compliance requirements. What is a Multi-Homed Pods? The Multus CNI plugin allows Pods to have multiple interfaces in Kubernetes. The current version of EKS support for Multus bundles Amazon VPC CNI as the default delegate plugin (which is the only supported and validated default delegate plugin). The default delegate plugin configures the primary network interface (eth0) for Pods to enable Kubernetes control plane traffic, including the IP Address Management (IPAM) for the primary network interface for the Pods.

With the launch of IPv6 support in EKS, you can now create IPv6 Kubernetes clusters. In an IPv6 EKS cluster, pods and services will receive IPv6 addresses while maintaining compatibility with legacy IPv4 Endpoints. This includes the ability for external IPv4 endpoints to access services, and pods to access external IPv4 addresses. All the pod-to-pod communication within a cluster is always IPV6. Within a VPC (/56), the IPv6 CIDR block size for IPv6 subnets is fixed at /64. This provides approximately 18 quintillion IPv6 addresses, allowing you to scale your deployments on EKS. EKS’s support for IPv6 is focused on resolving the IP exhaustion problem, which is constrained by the limited size of the IPv4 address space. This is a significant concern raised by a number of our customers and is distinct from Kubernetes’ "IPv4/IPv6 dual-stack" feature. At cluster creation, you will have the option to specify IPv6 as the IP address family for the cluster. When you set up your cluster to run in IPv6 mode, pods and services in Kubernetes get IPv6 addresses. Amazon EKS IPv6 support leverages native VPC IPv6 capabilities. IPv6 support works for new and existing VPCs; you can opt in on a VPC-by-VPC basis. Each VPC is given an IPv4 address prefix (CIDR block size can be from /16 to /28) and a unique /56 IPv6 address prefix (fixed) from within Amazon’s GUA (Global Unicast Address); you can assign a /64 address prefix to each subnet in your VPC. IPv4 features, like Route Tables, Network Access Control Lists, Peering, and DNS resolution, work the same way in a VPC. Note that security groups for pods is not available during the present phase of IPv6 support. In the IPv6 world, every address is internet routable. The IPv6 addresses associated with the nodes and pods are public. Private subnets are supported by implementing an egress-only internet gateway (EIGW) in a VPC, allowing outbound traffic while blocking all incoming traffic. Best practices for implementing IPv6 subnets can be found in the VPC user guide.

A security group acts as a virtual firewall for instances to control inbound and outbound traffic. Just like applications running on EC2 instances, containerized applications (Pods) running on Amazon EKS frequently require access to other services running within the cluster as well as external AWS services, such as Amazon Relational Database Service (Amazon RDS) or Amazon ElastiCache. On AWS, controlling network level access between services is often accomplished via EC2 security groups. Security groups are automatically created when you provision an EKS cluster. By default, the Amazon VPC CNI will use security groups at the node level, and every pod on a node shares the same security groups. However, security groups at the node level can be customized by customers. As seen in the image below, all application pods operating on worker nodes will have access to the RDS database service. Although you may workaround this limitation by setting up a new node group for each application and defining taint and affinity rules to assign pods to the appropriate nodes, this appears to be a laborious process. This inefficient process is difficult to manage at scale and can result in underutilized nodes. You can use IAM roles for service accounts to address pod-level security challenges at the authentication layer. Using security groups for pods, on the other hand, may be a good idea if your organization's compliance rules also call for network segmentation as an extra defense in depth strategy. Kubernetes [network policies] provide a mechanism for controlling ingress and egress traffic both within and outside the cluster. Network policies are not, however, natively integrated with AWS security groups. This makes it difficult to restrict network access to VPC resources, such as RDS databases or EC2 instances, solely using network policies. With security groups for pods, you can reuse your operational knowledge, tooling, and experience around existing policies rather than reimplementing network policies at the Kubernetes layer.

By default, the Amazon VPC CNI will assign Pods a secondary IP address from the primary subnet. The primary subnet is the subnet CIDR that the primary ENI is attached to, usually the subnet of the node/host. If the primary subnet is too small, the CNI cannot acquire enough secondary IP addresses to assign to your Pods. Custom networking is one solution to this problem. Custom networking offers an alternative to using a single (primary) subnet. Instead of assigning an IP from the primary subnet CIDR, the CNI acquires IP addresses from a CIDR range defined in a ENIConfig CRD. The ENIConfig includes an alternate subnet CIDR range, along with the security group(s) that the Pods will belong to. Unless Pods are configured with hostNetwork, your Pods will not use the host’s primary ENI. Pods are bound to secondary ENIs that get attached to the EC2 instance by the VPC CNI. If the primary ENI is not used, the maximum number of Pods on the node is reduced. The primary reason to use custom networking is to reduce the risk of IP exhaustion -- when there are insufficient IP addresses available to allocate to all your Pods. This can occur when EKS clusters are deployed into VPCs that, at the time of creation, didn’t account for Pods getting IP address from the VPC’s primary CIDR range.

Kubernetes was designed to run distributed applications on a cluster of connected machines. Networking plays an integral part in orchestrating and running distributed applications in any Kubernetes environment. It is critical to understand the Kubernetes networking paradigm in order to build, run, and operate your applications efficiently. This section of the best practices advises on different cluster networking modes possible with EKS. How to use this guide? This guide is meant for EKS cluster administrators, operators, and developers. Although the main aim of the guide is to provide guidance and recommendations about cluster networking operations supported by EKS. The guide covers fundamentals of Kubernetes pods networking. Through the Amazon VPC Container Network Interface (CNI) plugin, Amazon EKS handles cluster networking. Amazon EKS supports native VPC networking via the VPC CNI plugin for Kubernetes. Amazon VPC CNI supports different networking modes. The guide is organized into different networking modes for easier consumption. Each topic begins with a brief overview, followed by use cases supported by each of the available modes, and a list of recommendations and best practices for ensuring the stability of your EKS clusters. We strongly recommend you learn about proposal and design of Amazon VPC CNI before considering the EKS cluster networking modes. Amazon VPC CNI and Configuration Modes The Amazon VPC CNI plugin offers networking for pods and enables Kubernetes Pods to have the same IP address on the VPC network and inside the pod. You may leverage Amazon VPC CNI's diverse modes and plugin configurations to address a variety of use cases, including:

Networking is a crucial component in Kubernetes. Pod-to-Pod networking also known as cluster networking is central part of Kubernetes and is focus of this guide. Kubernetes dictates the following requirements on cluster networking. Pods scheduled on the same node must be able to communicate with other Pods without using NAT (Network Address Translation). All system daemons (background processes, for example, kubelet) running on a particular node can communicate with the Pods running on the same node. Pods that use the host network must be able to contact all other Pods on all other nodes without using NAT. The Kubernetes network model defines a “flat” network and uses an IP per Pod model to meet the above requirements. Every Pod gets its own IP address and communicates with other Pods using IP addresses, reducing the complexity of mapping container ports to host ports. A Pod is modeled as a group of containers that share a network namespace, including their IP address and MAC address. Containers within a Pod all have the same IP address and port space assigned through the network namespace assigned to the Pod, and can find each other via localhost since they reside in the same network namespace. This creates a clean, backwards-compatible model where pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application, and configuration.

Amazon EKS implements cluster networking through the Amazon VPC Container Network Interface (CNI) plugin. The CNI plugin allows Kubernetes Pods to have the same IP address inside the Pod as they do on the VPC network. The CNI plugin uses Elastic Network Interface (ENI) for Pod networking. The maximum number of network interfaces, and the maximum number of private IPv4 addresses that you can use varies by the type of EC2 Instance. Since each Pod uses an IP address, the number of Pods you can run on a particular EC2 Instance depends on how many ENIs can be attached to it and how many IP addresses it supports. This file contains the maximum number of Pods you can run on an EC2 Instance. The limits in the file are invalid if you use CNI custom networking. Amazon VPC CNI has two components: CNI plugin, which will wire up host’s and Pod’s network stack when called. L-IPAMD (aws-node DaemonSet) runs on every node is a long-running node-Local IP Address Management (IPAM) daemon and is responsible for:

As of August 2021, you can assign a private IPv4 or IPv6 CIDR range, either automatically or manually, to your network interfaces on EC2 instances. Amazon VPC CNI leverages assigning prefixes to Amazon EC2 network interfaces to increase the number of IP addresses available to worker nodes. You can configure version 1.9.0 or later of the Amazon VPC CNI add-on to assign/28 (16 IP addresses) IPv4 address prefixes, instead of assigning individual IP addresses to network interfaces. This mode is used to maximize node use and improve pod density per node. When ENABLE_PREFIX_DELEGATION is set to true, Amazon VPC CNI will allocate /28 IPv4 address prefixes to the ENIs. With prefix assignment mode, the maximum number of elastic network interfaces per instance type remains the same, but you can now configure Amazon VPC CNI to assign /28 (16 IP addresses) IPv4 address prefixes, instead of assigning individual IPv4 addresses to network interfaces. The pods are assigned an IPv4 address from the prefix assigned to the ENI. Please follow the instructions mentioned in the EKS user guide to enable Prefix IP mode. Under the hood, CNI on worker node initialization, will assign a CIDR block prefix to the primary ENI. The CNI pre-allocates Prefix’s and IP addresses for faster pod startup by maintaining a warm pool. As IP needs increase (as more pods scheduled) additional prefixes will be requested for the existing ENI. When the number of prefixes exceeds the number of number can be assigned to a single network interface, the plugin begins allocating a new network interface, as long as the maximum number of network interfaces for the instance aren't already attached. Recommendations Calculate Maximum Pods You can use the following formula to determine the maximum number of pods you can deploy on a node when Prefix IP mode is enabled.

With the launch of IPv6 support in EKS, you can now create IPv6 Kubernetes clusters. In an IPv6 EKS cluster, pods and services will receive IPv6 addresses while maintaining the ability for legacy IPv4 endpoints to connect to services running on IPv6 clusters, as well as pods connecting to legacy IPv4 endpoints outside the cluster. All the pod-to-pod communication within a cluster is always IPV6. Within a VPC (/56), the IPv6 CIDR block size for IPv6 subnets is fixed at /64. This provides 2^64 (approximately 18 quintillion) IPv6 addresses allowing to scale your deployments on EKS. EKS’s support for IPv6 is focused on resolving the IP exhaustion problem, which is constrained by the limited size of the IPv4 address space, a significant concern raised by a number of our customers and is distinct from Kubernetes’ “IPv4/IPv6 dual-stack” feature. At cluster creation, you will have the option to specify IPv6 as the IP address family for the cluster. When you configure your cluster to run in IPv6 mode, Kubernetes pods and services receive IPv6 addresses. Amazon EKS IPv6 support leverages native VPC IPv6 capabilities. IPv6 support works for new and existing VPCs; you can opt in on a VPC-by-VPC basis. Each VPC is given an IPv4 address prefix (CIDR block size can be from /16 to /28) a unique /56 IPv6 address prefix (fixed) from within Amazon’s GUA (Global Unicast Address); you can assign a /64 address prefix to each subnet in your VPC. All the VPC features such as Security Groups, Route Tables, Network ACLs, Peering, and DNS resolution within a VPC all operate in the same way as IPv4. In the IPv6 world, every address is internet routable. The IPv6 addresses associated with the nodes and pods are public. Private subnets are supported by implementing an egress-only internet gateways (EIGW) in a VPC, allowing outbound traffic while blocking all incoming traffic. Best practices for implementing IPv6 subnets can be found in the VPC user guide. How it works IPv6 is supported in prefix assignment mode. Amazon VPC Container Network Interface (CNI) plugin is configured to assign an address from the prefix attached to the primary ENI. IPv6 prefix assignment now occurs only at the node startup and hence increasing the performance significantly by removing AWS api throttle for large clusters. You will not be required to update any of the WARM* settings as a single IPv6 Prefix-Delegation prefix has many addresses (/80 => ~10^14 addresses per ENI) and is big enough to support large clusters with millions of pods. The VPC CNI currently supports only prefix assignment mode for IPv6 clusters and only works with AWS Nitro-based EC2 instances.

By default, the Amazon VPC CNI will assign pods a secondary IP address from the host’s subnet CIDR. However, if the host’s subnet CIDR is too small, the CNI may not be able to acquire enough secondary IP addresses to assign to your pods. Custom networking offers an alternative to using the host’s subnet CIDR range, i.e. the subnet attached to eth0. Instead of assigning an IP from the host’s subnet CIDR, the CNI acquires IP addresses from a CIDR range defined in the ENIConfig, a Custom Resource Definition (CRD) that includes an alternate subnet CIDR range, along with the security group(s) that the pods will belong to. Unless pods are configured with hostNetwork, you forfeit the use of host’s primary ENI. All other pods are bound to secondary ENIs that get attached to the EC2 instance by the VPC CNI. The result is lower pod density per worker node. The primary reason to use custom networking is to reduce the risk of IP exhaustion where there are insufficient IP addresses available to allocate to all your pods. This can occur when EKS clusters are deployed into VPCs that, at the time of creation, didn’t account for pods getting IP address from the VPC’s primary CIDR range. While custom networking will accept all valid VPC CIDRs for secondary VPC CIDR range, we generally recommend that you use CIDRs from the CG-NAT space, i.e. 100.64.0.0/10 or 198.19.0.0/16 as those are less likely to be used in a corporate setting than other RFC1918 ranges. For additional information about the permitted and restricted CIDR block associations you can use with you VPC, see IPv4 CIDR block association restrictions in the VPC and subnet sizing ) section of the VPC documentation. As shown in the diagram below, the primary Elastic Network Interface (ENI) of the worker node still uses the primary VPC CIDR range (in this case 10.0.0.0/20) but the secondary ENIs for pods use the secondary VPC CIDR Range (in this case 100.64.0.0/10). Now, in order to have the pods using the 100.64.0.0/8 CIDR range, you will have to configure the CNI plugin to use custom networking. You can follow through the steps as documented here. If you want the CNI to assign IP addresses for Pods from a different subnet, you can set AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG environment variable to true.

A CNI is the container network interface that provides an application programming interface to configure network interfaces in containers. Multus CNI is a container network interface plugin for Kubernetes that enables attaching multiple network interfaces to pods. In Kubernetes, each pod has only one network interface by default, other than local loopback. With Multus, you can create multi-homed pods that have multiple interfaces. Multus acts as ‘meta’ plugin that can call other CNI plugins to configure additional interfaces. Multiple network interfaces for pods are useful in various use cases; examples include: Traffic splitting: Running network functions (NF) that require separation of control/management, and data/user plane network traffic to meet low latency Quality of Service (QoS) requirements. Performance: Additional interfaces often leverage specialized hardware specifications such as Single Root I/O Virtualization (SR-IOV) and Data Plane Development Kit (DPDK), which bypass the operating system kernel for increased bandwidth and network performance. Security: Supporting multi-tenant networks with strict traffic isolation requirements. Connecting multiple subnets to pods to meet compliance requirements. What is a multi-homed pod? The Multus CNI plugin allows pods to have multiple interfaces in Kubernetes. The current version of EKS support for Multus bundles Amazon VPC CNI as the default delegate plugin (which is the only supported and validated default delegate plugin). The default delegate plugin configures the primary network interface (eth0) for pods to enable Kubernetes control plane traffic, including the IP Address Management (IPAM) for the primary network interface for the pods.

The secondary IP mode was the original networking mode for Amazon VPC CNI. VPC CNI (secondary IP mode) was built from the ground up to allow EKS to fulfill all Kubernetes networking needs. As of today, EKS supports Secondary IP as the default mode for VPC CNI. In secondary IP mode, the CNI plugin uses Elastic Network Interface (ENI) for Pod networking. The CNI allocates ENIs to each worker node and uses the secondary IP range from each ENI for pods. The CNI pre-allocates ENIs and IP addresses for faster pod startup by maintaining a warm pool. When the number of pods running on the node exceeds the number of addresses that can be assigned to a single network interface, the plugin starts allocating a new network interface, as long as the maximum number of network interfaces for the instance aren't already attached. In this mode, each pod you deploy is assigned one secondary private IP address from one of the network interfaces attached to the instance. Since each Pod uses an IP address, the number of Pods you can run on a particular EC2 Instance depends on how many ENIs can be attached to it and how many IP addresses it supports. You can use following formula to determine maximum no of pods you can deploy on a node. (Number of network interfaces for the instance type × (the number of IP addressess per network interface - 1)) + 2 For a list of the maximum number of pods supported by each instance type, see eni-max-pods.txt on GitHub. Please be aware system pods also count towards the maximum pods.