###### tags: ` EKS` ` Immersion Day` ` Containers`

# Amazon Elastic Kubernetes Service Workshop - Tips and Tricks

This is my list of hints and tips for this course. It's markdown so you can save it, access it or store it anywhere. I'll add specific answers to questions I get during the course. I'll share it with everyone.

# Administrivia, Schedule and Planning
This immersion day will be delivered in multiple half hour blocks.

The EKS Workshop Labs https://www.eksworkshop.com/docs/introduction/ contains more than 26 labs. This is a significant body of work and not practical to squeeze into a single day.

The events will be delivered as follows:

| Session | Topics | Time | When |
|---------|-----------------------------|----------|--------|
| One | Intro to EKS | 30 mins | 9:30am |
| Two | Fundamentals | 60 | 10:00am |
| Three | Autoscaling | 60 | 11:00am |
| Four | Observability | 30 | 12:00pm |
| Five | Lunch | 30 | 12:30pm |
| Six | Security | 45 | 1:15pm |
| Seven | Networking | 45 | 2:00pm |

**JOIN EVENT:** https://catalog.workshops.aws/join

1. Complete OTP to an email address (Any that you can access over public internet)
2. **Code will be on the TV** screen for you to type.
3. Read "Terms and Conditions", select "**I agree with the Terms and Conditions**" and click "**Join event**".
4. Go straight to the first lab
5. In the [Event dashboard](https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US) in section `Event Outputs`, open your `IdeUrl` and use the `IdePassword`.
6. Click on [AWS Elastic Kubernetes Service Workshop](https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop), Open the link for the [eksworkshop](https://snapshots.eksworkshop.com/eae63d4d/docs/introduction/navigating-labs)

You will need minimum two tabs open in your browser:
1. [EKS Workskhop](AWS Elastic Kubernetes Service Workshop) for your guide to the labs.
2. The VS Code server to run your commands.

# EKS Immersion Day Labs
I've created new accounts and clusters with a clean data. The workshop which is built using terraform offers the ability to reset the environment at the beginning of each LAB. This process only takes a few minutes. If you ever have a probem choose to `reset-environment`.

### Kubectl Cheat Sheet
Before we get started, I always like to have two things.

1. **Kubectl References** (open in the browser ready)
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#run

::: info
The below commands have already been run in the VS Code Server for you, keep this as reference for your own IDE.
:::

2. **Kubectl autocomplete setup for Bash**

source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.

I always set the shorthand alias for `kubectl` that also works with completion. Speed up your CLI skills!

alias k=kubectl
complete -o default -F __start_kubectl k

## Workshop Intro

[**Getting Started**](https://snapshots.eksworkshop.com/eae63d4d/docs/introduction/getting-started/) 15 minutes
Clone the EKS Workshop Git repository in to the IDE environment. Make sure you run this in your CLI at a minimum.

`prepare-environment introduction/getting-started`

## Fundamentals
> We need to understand getting traffic into our cluster, storage and cluster upgrade process.
>
[**Exposing Applications**](https://snapshots.eksworkshop.com/eae63d4d/docs/fundamentals/exposing/) 25 minutes
The controller can provision the following resources:

* An AWS Application Load Balancer when you create a Kubernetes **Ingress**.
* An AWS Network Load Balancer when you create a Kubernetes **Service** of type LoadBalancer.

**LoadBalancer Service** -> Network Load Balancer (NLB) -> Layer 4 (TCP/UDP)
**Ingress** -> Application Load Balancer (ALB) -> Layer 7 (HTTP/HTTPS)

:::spoiler
AWS Load Balancer Controller supports:

* **Instance mode** - AWS NLB/ALB sends traffic to the instances and the kube-proxy on the individual worker nodes forward it to the pods through one or more worker nodes in the Kubernetes cluster
* **IP mode** - AWS NLB/ALB sends traffic directly to the Kubernetes pods behind the service, eliminating the need for an extra network hop through the worker nodes in the Kubernetes cluster. IP target mode supports pods running on both AWS EC2 instances and AWS Fargate

Limitations:

* Instance mode requires nodes in public subnets for public NLBs
* IP mode requires pods to have IPs from the VPC

:::

**Cluster upgrades** (See below)

##### EKS In-Place Upgrade Summary:

An In-Place upgrade updates an existing EKS cluster to a newer Kubernetes version without creating a new cluster.
:::spoiler
#### Process
1. Upgrading the control plane
2. Updating worker node AMIs
3. Replacing nodes gradually
4. Updating Kubernetes configurations
5. Testing and validation

#### Key Benefits
* Maintains existing cluster resources
* Keeps same API endpoint
* Less infrastructure overhead
* Easier management of stateful applications

#### Main Challenges
* Requires careful planning to minimize downtime
* Multiple upgrades may be needed for version jumps
* Difficult rollback process
* Extensive testing required
:::

##### EKS Blue-Green Upgrade Summary:

A Blue-Green upgrade creates a new EKS cluster (green) alongside the existing one (blue), gradually shifting traffic between them.
:::spoiler
##### Process
1. Create new cluster with target version
2. Deploy applications and add-ons
3. Test and validate new cluster
4. Gradually shift traffic from old to new
5. Monitor performance
6. Decommission old cluster

#### Key Benefits
* Controlled and safer upgrade process
* Can skip multiple K8s versions
* Easy rollback mechanism
* Minimal downtime

#### Main Challenges
* Additional infrastructure costs
* Complex traffic management
* Requires updating external integrations
* Challenging for stateful applications

#### Stateful Workload Considerations
* Plan data migration carefully
* Use tools like [Velero](https://velero.io/)
* Configure matching storage provisioning
* Consult application owners
* Keep data synchronized between clusters
:::

## Managing Clusters
> There is always cluster management to be done. Learn some of the key tasks associated with having a Kubernetes cluster.

[**Managed Node Group basics**](https://snapshots.eksworkshop.com/eae63d4d/docs/fundamentals/managed-node-groups/basics/) 35 minutes
An EKS cluster contains one or more EC2 nodes that Pods are scheduled on. EKS nodes run in your AWS account and connect to the control plane of your cluster through the cluster API server endpoint. You deploy one or more nodes into a node group. A node group is one or more EC2 instances that are deployed in an EC2 Auto Scaling group.

**Auto Mode**
::: info
Currently there is no Auto Mode section in the workshop. Lets chat about Auto-mode and what benefits it brings over managed node groups.
:::

:movie_camera: [Amazon EKS Auto Mode | Youtube](https://www.youtube.com/watch?v=IQjsFlkqWQY)

## Autoscaling

>There will be times where we need to scale Pods and Nodes. What should we use and how do they work?

[**Cluster Autoscaler**](https://snapshots.eksworkshop.com/eae63d4d/docs/autoscaling/compute/cluster-autoscaler/) 20 minutes
The Kubernetes Cluster Autoscaler, a component that automatically adjusts the size of a Kubernetes Cluster so that all pods have a place to run without unneeded nodes. The Cluster Autoscaler is a great tool to ensure that the underlying cluster infrastructure is elastic, scalable, and can meet the changing demands of workloads.

[**Karpenter**](https://snapshots.eksworkshop.com/eae63d4d/docs/autoscaling/compute/karpenter/) 25 minutes

Karpenter is designed to provide the right compute resources to match your application’s needs in seconds, instead of minutes by observing the aggregate resource requests of unschedulable pods and makes decisions to launch and terminate nodes to minimize scheduling latencies.

:movie_camera: [Karpenter vs Cluster Autoscaler | Youtube](https://www.youtube.com/watch?v=FIBc8GkjFU0)

[**Horizontal Pod Scaler**](https://snapshots.eksworkshop.com/eae63d4d/docs/autoscaling/workloads/horizontal-pod-autoscaler/) 10 minutes

Horizontal Pod Autoscaler (HPA) to scale pods in a deployment or replica set. It's implemented as a K8s API resource and a controller. The resource determines the behavior of the controller.

## Observability
> We will touch on logs from a system level in Kubernetes and then touch on the both AWS-Native toolsets and from the open source world.

[**Control Plane**](https://snapshots.eksworkshop.com/eae63d4d/docs/observability/logging/cluster-logging/) 10 minutes
The Kubernetes control plane is a set of components that manage Kubernetes clusters and produce logs used for auditing and diagnostic purposes. With Amazon EKS, you can turn on logs for different control plane components and send them to Amazon CloudWatch

[**Pod Logging**](https://snapshots.eksworkshop.com/eae63d4d/docs/observability/logging/pod-logging/) 15 minutes
Containerized applications should output their logs to stdout and stderr. This is also considered best practice in Kubernetes and cluster level log collection systems are built on this premise

[**AWS Native**](https://snapshots.eksworkshop.com/eae63d4d/docs/observability/container-insights/) 20 minutes
Use CloudWatch Container Insights to collect, aggregate, and summarize metrics and logs from your containerized applications and microservices. Container Insights is available for Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and Kubernetes platforms on Amazon EC2. Amazon ECS support includes support for Fargate.

[**Open Source** ](https://snapshots.eksworkshop.com/eae63d4d/docs/observability/open-source-metrics/) 25 minutes
Amazon Managed Service for Prometheus is a monitoring service for metrics compatible with the open source Prometheus project, making it easier for you to securely monitor container environments

## Security
> How do we give Pods access to other AWS Services and make EKS aware of IAM permissions? Operationally how do we keep track of best practive and alert on misconfiguration?

[**IAM Roles**](https://snapshots.eksworkshop.com/eae63d4d/docs/security/iam-roles-for-service-accounts/) 20 minutes
Applications in a Pod’s containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. For example, applications may need to upload files to an S3 bucket or query a DynamoDB table. To do so applications must sign their AWS API requests with AWS credentials. IAM Roles for Service Accounts (IRSA) provide the ability to manage credentials for your applications, similar to the way that IAM Instance Profiles provide credentials to Amazon EC2 instances.

[**Pod Identity**](https://snapshots.eksworkshop.com/eae63d4d/docs/security/amazon-eks-pod-identity/) 20 minutes
Applications in a Pod’s containers can use a supported AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions.

:movie_camera: [EKS Pod Idenitty vs IRSA | Youtube](https://www.youtube.com/watch?v=aUjJSorBE70)

:book: [Simplifying IAM Permissions for Amazon EKS Addons with EKS Pod Identity | AWS Blog](https://aws.amazon.com/blogs/containers/simplifying-iam-permissions-for-amazon-eks-addons-with-eks-pod-identity/)

[**GuardDuty Findings**](https://snapshots.eksworkshop.com/eae63d4d/docs/security/guardduty/) 40 minutes
Amazon GuardDuty for EKS Protection monitors control plane activity by analyzing Kubernetes audit logs from existing and new Amazon EKS clusters in your accounts. GuardDuty is integrated with Amazon EKS, giving it direct access to the Kubernetes audit logs without requiring you to turn on or store these logs

## Networking
> It's essential to understand Kubernetes networking to operate your cluster and applications efficiently. Pod networking, also called cluster networking, is the center of Kubernetes networking.

[**Network Policies**](https://snapshots.eksworkshop.com/eae63d4d/docs/networking/vpc-cni/network-policies/) 25 minutes
By default, Kubernetes allows all pods to freely communicate with each other with no restrictions. Kubernetes Network Policies enable you to define and enforce rules on the flow of traffic between pods, namespaces, and IP blocks (CIDR ranges).

[**Security Groups**](https://snapshots.eksworkshop.com/eae63d4d/docs/networking/security-groups-for-pods/) 15 minutes
Security groups, acting as instance level network firewalls, are among the most important and commonly used building blocks in any AWS cloud deployment. Containerized applications frequently require access to other services running within the cluster as well as external AWS services, such as Amazon Relational Database Service (Amazon RDS) or Amazon ElastiCache.

## Automation
> With ACK, you can leverage AWS services such as databases (RDS or others) and queues (SQS etc.) for your Kubernetes applications without manually defining resources outside of the cluster. This reduces the overall complexity of managing your application's dependencies.

[**AWS Controller for Kubernetes**](https://snapshots.eksworkshop.com/eae63d4d/docs/automation/controlplanes/ack/how-it-works)

You can find all available AWS Controllers for Kubernetes (ACK) on the official ECR Public Gallery:
https://gallery.ecr.aws/aws-controllers-k8s.
:::spoiler

#### Example

Deploy using Helm from the latest reference helm chart, for example [S3-Chart](https://gallery.ecr.aws/aws-controllers-k8s/s3-chart)

`public.ecr.aws/aws-controllers-k8s/s3-chart:1.0.32`

helm install ack-s3-controller \
  oci://public.ecr.aws/aws-controllers-k8s/s3-chart \
  --version 1.0.32 \
  --namespace ack-system \
  --create-namespace

Key components of this command:

ack-s3-controller
: Name of your helm release
--version 1.0.32
: Specifies the version you want to install
--namespace ack-system
: The namespace where the controller will be installed
--create-namespace
: Creates the namespace if it doesn't exist
:::

# Self Paced Learning
Best Practices for EKS
https://aws.github.io/aws-eks-best-practices
Containers from the Couch
:movie_camera: https://www.youtube.com/@ContainersfromtheCouch
Amazon EKS - Knowledge Badge Readiness Path
https://explore.skillbuilder.aws/learn/public/learning_plan/view/1931/amazon-eks-knowledge-badge-readiness-path

# Useful Links
#### Management
K9s
https://k9scli.io/
EKS Node Viewer
https://github.com/awslabs/eks-node-viewer

#### Observability
X-Ray Samples
https://github.com/aws-samples/aws-xray-kubernetes/
Set of Grafana dashboards for Kubernetes
https://github.com/dotdc/grafana-dashboards-kubernetes

#### Security
What is BottleRocket?
:movie_camera: https://www.youtube.com/watch?v=L33l7Yd8oZM&t=392s
Use Amazon Inspector to manage your build and deploy pipelines for containerized applications
https://aws.amazon.com/blogs/security/use-amazon-inspector-to-manage-your-build-and-deploy-pipelines-for-containerized-applications/

#### Deployment
[EKS Blueprints for Terraform](https://aws-ia.github.io/terraform-aws-eks-blueprints/)
[EKS Blueprints for CDK](https://aws-quickstart.github.io/cdk-eks-blueprints/)

#### General
The Twelve factor app https://12factor.net/

# FAQ
### EKS
* **What OS does the EKS Node use?**
Amazon Linux 2, BottleRocket, Ubuntu

### EKS Fargate
* **Can I use Fargate and Managed Node Groups in a single Cluster?**
Yes, you create a Fargate profile in your existing Cluster.

### EKS Auto-mode

Simplifies Kubernetes management by automatically provisioning infrastructure, selecting optimal compute instances, dynamically scaling resources, continuously optimizing costs, managing core add-ons, patching operating systems, and integrating with AWS security services

:movie_camera: [Amazon EKS Auto Mode | Youtube](https://www.youtube.com/watch?v=IQjsFlkqWQY)

### Key Differences: EKS Auto Mode vs. Fargate
* **EKS Auto Mode** is for more control over infrastructure, support for DaemonSets, stateful applications, and advanced networking.
* **Fargate** is used for its simplicity, serverless operation, and suitability for stateless workloads.
:::spoiler
### Worker Nodes

EKS Auto Mode: Uses managed EC2 instances.
Fargate: Serverless; no worker nodes required.

### Workload Scaling

EKS Auto Mode: Automatically right-sizes EC2 instances.
Fargate: Scales pods directly via Horizontal Pod Autoscaler (HPA).

### DaemonSets

EKS Auto Mode: Supported.
Fargate: Unsupported; requires sidecar containers instead.

### Stateful Apps

EKS Auto Mode: Supported (e.g., databases, persistent storage).
Fargate: Not recommended for stateful workloads.

### Networking

EKS Auto Mode: Supports public and private subnets.
Fargate: Limited to private subnets.

### Load Balancers

EKS Auto Mode: Compatible with Classic (CLB), Network (NLB), and Ingress.
Fargate: Only Ingress-supported.

### Custom Networking

EKS Auto Mode: Supports Container Networking Interface (CNI).
Fargate: CNI not supported.

### Cost

EKS Auto Mode: Pay for control plane + EC2 instance costs.
Fargate: Pay for control plane + per-pod runtime costs.

### Management Overhead

EKS Auto Mode: Moderate (AWS manages add-ons/updates; you manage nodes).
Fargate: Low (serverless; no node management).

### Use Cases

EKS Auto Mode: Flexible for mixed workloads (stateful, batch, custom apps).
Fargate: Ideal for stateless, event-driven, or serverless applications.
:::

[Migrate from EKS Fargate to EKS Auto Mode](https://docs.aws.amazon.com/eks/latest/userguide/auto-migrate-fargate.html)

### EKS Anywhere
* **What platforms can I deploy EKS-A onto?**
Bare-metal, VMWare & AWS Snowball Edge.
* **That's the cost?**
There are no costs to run EKS Anywhere. AWS Enterprise Support is a pre-requisite for purchasing an Amazon EKS Anywhere Enterprise Subscription. Basically if you want support on your EKS-A production cluster.