---
# System prepended metadata

title: Trustsoft K8s 1-2.12.23

---

Trustsoft K8s 1-2.12.23
===

# Local setup

 * https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
 * https://github.com/bitnami/charts/tree/main/bitnami/wordpress/templates
 * https://bitnami.com/stack/wordpress/helm

## Minikube

Start with `minikube`:

 * https://docs.google.com/presentation/d/1bX3UGMQRuBjkZW89urH-xvz0rOBXbD8q3rmwYW8JQZw/edit?usp=sharing
 
Now deploy wordpress
 
```
helm install wp oci://registry-1.docker.io/bitnamicharts/wordpress
helm list
```

Check documentation of values at [chart](https://github.com/bitnami/charts/tree/main/bitnami/wordpress/#installing-the-chart)

Check ingress setup.

```
minikube addons enable ingress
```

Update with values:

```
helm upgrade --install wp oci://registry-1.docker.io/bitnamicharts/wordpress -f wp/values.yml
```

Check wp:
```
kubectl get secret --namespace default wp-wordpress -o jsonpath="{.data.wordpress-password}" | base64 -d
```

Monitoring *:

```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install -f prometheus/only-prometheus.yml prometheus prometheus-community/prometheus
```

* we'll probably use the prometheus from the last session.

Check prometheus server:

```
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9090
```

pull/push metrics model:

 * traces: push/pull?
 * metrics: push/pull?
 * IoT: push/pull?

```
minikube addons configure registry-creds
minikube service wp-wordpress --url
```

Benchmark:

```
export WP_HOSTNAME=`minikube service wp-wordpress --url | head -n 1`
k6 run k6/index.js
```

Disable caching at values and check again.

Inconclusive, we'll see in the cloud.

Volumes!

 * Add picture
 * Kill the Pod
 * Check the wp again
 * Resize the claim

```
only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize
```

check `kubectl edit storageclass standard`

```
63s         Warning   ExternalExpanding              persistentvolumeclaim/wp-wordpress                       waiting for an external controller to expand this PVC
```

Could not be done as simple as needed.

# Deploy to EKS

```
terraform apply -target=module.vpc
terraform apply -target=module.vpc_cni_irsa
terraform apply -target=module.eks
```

`-target` is not a good practise:

```
Warning: Applied changes may be incomplete

The plan was created with the -target option in effect, so some changes
requested in the configuration may have been ignored and the output values may
not be fully updated. Run the following command to verify that no other
changes are pending:
    terraform plan
```

checkout cluster

```
aws eks update-kubeconfig --region eu-west-1 --name beranm-testing-01 --profile trustsoft
```

deploy wp

```
helm upgrade --install wp oci://registry-1.docker.io/bitnamicharts/wordpress -f wp/values.yml
```

check pending pods

```
kubectl get pods --field-selector status.phase=Pending
```

see

```
running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
```

add

```
module "ebs_csi_irsa_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name             = "${local.name}-ebs-csi"
  attach_ebs_csi_policy = true

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
    }
  }
}
...
    aws-ebs-csi-driver = {
      service_account_role_arn = module.ebs_csi_irsa_role.iam_role_arn
      most_recent              = true
    }
```

run

```
terraform apply -target=module.vpc -target=module.vpc_cni_irsa -target=module.eks -target=module.ebs_csi_irsa_role
```

Check https://eu-west-1.console.aws.amazon.com/ec2/home?region=eu-west-1#Volumes:

And now resize the claim

Allow resize at storage class:

```
allowVolumeExpansion: true
```

How about scaling the deployment? Will it work?

Check error on HPA:

```
the HPA was unable to compute the replica count: failed to get memory
utilization: unable to get metrics for resource memory: unable to fetch
metrics from resource metrics API: the server could not find the
requested resource (get pods.metrics.k8s.io)
```

Try to upload the picture again?

Let's scale for multiple nodes and change scaling to prefer different nodes

```
0/2 nodes are available: 2 node(s) had volume node affinity conflict. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
```

See https://stackoverflow.com/questions/51946393/kubernetes-pod-warning-1-nodes-had-volume-node-affinity-conflict

Check https://eu-west-1.console.aws.amazon.com/ec2/home?region=eu-west-1#Volumes:

Optional storage class:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: kubernetes.io/aws-ebs
parameters:
  fsType: ext4
  type: gp3
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - eu-west-1a
    - eu-west-1b
    - eu-west-1c
```

Works on instance termination, not for drains and cordons. Let's try. 10 minutes not a problem.

Check https://pet2cattle.com/2022/04/volume-affinity-conflict PVC has zones.

Change az setup of default node group:

```
  eks_managed_node_groups = {
    default_node_group = {
      use_custom_launch_template = false
      disk_size                  = 50
      desired_size               = 2
      min_size                   = 2
      subnet_ids                 = [module.vpc.private_subnets[0]]
    }
  }
```

Limit to single az and kill node again.

How about multi-az scaling?

## Enable EFS and try again

```
module "efs_csi_irsa_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name             = "${local.name}-efs-csi"
  attach_efs_csi_policy = true

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:efs-csi-controller-sa"]
    }
  }
}
```

and add to EKS module:

```
...
  eks_managed_node_group_defaults = {
    ami_type       = "AL2_x86_64"
    instance_types = ["t3.medium"]
    capacity_type  = "SPOT"

    iam_role_attach_cni_policy = true

    vpc_security_group_ids = [aws_security_group.eks.id]
  }
  ...
    aws-efs-csi-driver = {
      service_account_role_arn = module.efs_csi_irsa_role.iam_role_arn
      most_recent              = true
    }
```

and enable `efs.tf`.

Deploy new storage class

```
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-0bc88e6ad43d533f1
  directoryPerms: "700"
```

Change class in values.

Let's

 * scale
 * benchmark
 * enable caching, ...

## QoS classes

Pod priorities for scheduling

 * check [slides](https://docs.google.com/presentation/d/1T3c2_C5H6hSR6k4CPN-X67Nx_T6hKhYQh-QKLB1_ntY/edit#slide=id.g1c8c3191096_0_440)

```
helm upgrade --create-namespace  -n wp-guaranteed --install wp-guaranteed oci://registry-1.docker.io/bitnamicharts/wordpress -f wp/guaranteed-values.yml
```

```
helm upgrade --create-namespace  -n wp-best-efford --install wp-best-efford oci://registry-1.docker.io/bitnamicharts/wordpress -f wp/best-efford-values.yml
```

And now K6 both of them.

Do not use priority class unless it's enritely needed.

## Karpenter

[Example](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/karpenter/main.tf)

## KEDA

[Scalers](https://keda.sh/docs/2.12/scalers/)
