Full Kubernetes Setup

# Full Kubernetes Setup :::danger To be considered production ready, the following must also be done. 1. Setup CNI, ingress, and service mesh. - calico, flannel, cilium, ipv6 - countour, traefik - linkerd, istio - Their relations are actually more complicated. 2. Setup observability. - prometheus 3. Chaos engineering. - Litmus ::: [TOC] # Plan ## Environment Layout Deploy a kubernetes VM cluster consisting of 1 controller and 3 workers with [typhoon](https://typhoon.psdn.io/). The cluster is behind a self-managed gateway which acts as an external loadbalancer for the cluster. The cluster must be modified to run ELK. ### Environment to begin with - A safe LAN with no dhcp server. - Several subnets are broadcasting, among which is the internet gateway `172.16.1.57/24`. - We have superuser access to `172.16.1.99` which is a very powerful machine to host our VMs but only boasts a single NIC. ### Outline Same terms are used to refer to different things, which creates confusion. Conventions will be introduced in the next subsection to sort this out. - Run 5 guests (1 gateway, 1 controller, 3 workers) on the host which are all bridged to LAN. *Gateway* in the previous sentence refers to the gateway VM, not the internet facing gateway. - The gateway VM has WAN address `172.16.1.97`, LAN address `192.168.88.1`, and broadcasts the subnet `192.168.88.0/24` with a dhcp server leasing the range `192.168.88.200 - 192.168.88.254`. *WAN* and *LAN* in the previous sentence are relative to the gateway VM. - The host will obtain 2 ip addresses: - `172.16.1.199` which is bound to the original gateway with internet connection, and - `192.168.88.2` which is bound to the gateway VM. - To simplify things, with the pivotal intent to sidestep firewall rules injected by libvirt, all VMs are directly connected to LAN via MACVTAP in bridge mode. Since bridge mode MACVTAP devices cannot communicate with the physical NIC underneath, we have to add another MACVTAP device for the host to bind the address `192.168.88.2`. - The controller and workers will each get a static ip and dns entry from the gateway VM. Although possible, they should not bind to any other addresses and should use the gateway VM as their only gateway. ### Convention - local: the machine infront of you (this could be host) - LAN: the subnet `192.168.88.0/24` with DHCP server - WAN: the subnet `172.16.1.0/24` without DHCP server - gateway: the outermost gateway with internet connection - host: the host machine - pfsense: the gateway VM - controller: the controller VM - worker1: the first worker VM - worker2: the second worker VM - worker3: the third worker VM - workers: refers to worker1, worker2, worker3 collectively - cluster: refers to the controller and workers collectively ### Machine Specification - host: CentOS 8 - cpus: >= 20 - ram: >= 48G - disk: >= 1T - `172.16.1.99` - `192.168.88.2 matchbox.k8s` - pfsense: pfSense 2.4.5 - cpus: 2 - ram: 4G - disk: 30G - `172.16.1.97` - `192.168.88.1` - controller: Fedora CoreOS - cpus: 2 - ram: 4G - disk: 30G - `192.168.88.80 a.elk.k8s` - worker1: Fedora CoreOS - cpus: 4 - ram: 8G - disk: 60G - `192.168.88.81 1.elk.k8s elk.k8s` - worker2: Fedora CoreOS - cpus: 4 - ram: 8G - disk: 60G - `192.168.88.81 2.elk.k8s elk.k8s` - worker3: Fedora CoreOS - cpus: 4 - ram: 8G - disk: 60G - `192.168.88.81 3.elk.k8s elk.k8s` ### Illustration :::spoiler ``` +----------------------+ | | | +-------+ | host |NIC ===> 172.16.1.99 | +---+---+ | | |MACVTAP|===> 192.168.88.2 | +---+---+ | | | | | +------------+ | | | pfsense | | | | +---+---+ | | | |MACVTAP|===> 172.16.1.97 | | +---+---+ | | | |MACVTAP|===> 192.168.88.1 | +--------+---+---+ | | | | | +------------+ | | | controller | | | | +---+---+ | | | |MACVTAP|===> 192.168.88.80 | +--------+---+---+ | | | | | +------------+ | | | worker1 | | | | +---+---+ | | | |MACVTAP|===> 192.168.88.81 | +--------+---+---+ | | | | | +------------+ | | | worker2 | | | | +---+---+ | | | |MACVTAP|===> 192.168.88.82 | +--------+---+---+ | | | | | +------------+ | | | worker3 | | | | +---+---+ | | | |MACVTAP|===> 192.168.88.83 | +--------+-------+---+ | | +----------------------+ ``` ::: ## Procedure Layout ### Outline 1. Create pfsense, and set up WAN/LAN. 2. Create the controller and workers, but don't boot them yet. 3. Add static IP addresses and DNS entries according to the MAC addresses of the controller and workers. 4. Add a static IP address and a DNS entry for host, exposing the matchbox service. 5. Start matchbox. 6. Configure terraform. 7. Start terraform. Wait till introspection is done. 8. Stop matchbox. 9. Install kubectl, helm, and deploy ELK. 10. Expose ELK via ingress. ### Dependency of the process ```graphviz digraph hierarchy { nodesep=1.0 node [color=Red,fontname="Sans Serif"] edge [color=Blue,style=dashed] MAC -> {"IP/DNS" terraform} {"IP/DNS" TLS} -> {matchbox terraform} matchbox -> terraform -> kubectl -> ELK helm -> ELK } ``` ### Actual Approach 1. Preparation (superuser) - add user, setup ssh - install libvirt, matchbox, terraform, kubectl, helm - firewall, MACVTAP, partition for LVM pool 2. Setup libvirt - default connection, LVM pool - pfsense, controller, workers 3. Configure pfsense - WAN, LAN, IP, DNS, load balancing 4. Configure and start matchbox - TLS, assets 5. Configure and start terraform - `main.tf`, `providers.tf`, `out.tf`, ignition - `init`, `plan`, `apply` 6. Finalize - stop matchbox - deploy ELK # Hands-On Only *Preparation* requires superuser permission. We will create a non-priviledged user, add it to the libvirt group, and use the non-priviledged user only for beginning from step 2. To be honest, the non-priviledged user is quite priviledged, as the user is in the libvirt group. At first, we tried working with user session libvirt. This not only makes the user actually non-priviledged, it also isolates the libvirt session of the user with that of others, including the system session. Due to complexities, we adopted the following approach. - For performance, we use an LVM pool. - For connectivity, we use MACVTAPs. They mandate the user to connect to the system libvirtd. In our case, which is also very likely your case, we manage `host` from `local` remotely. Thus, ssh operations are also included. We assume the network layout as described previously. Since subnets `172.16.0.0/12` and `192.168.0.0/16` are reserved for private networks, we don't find it necessary to masquerade them. However, we hypothetically assume that the superuser account on *host* is exactly `superuser`, and that you can directly connect to *host* via `172.16.1.99`. We also suppose that the hostname for *local* (resp. *host*) is precisely `local` (resp. `host`). ## 1. Preparation (superuser) :::info Only this section requires superuser permission. ::: Suppose that `ssh superuser@172.16.199` directly logs us into the superuser account `superuser` on *host* and that we will be creating the non-privileged user `k8s`. It is recommended to add them to your ssh config file. It is also suggested to use ssh keys. ### Set up ssh on *local* :::info Read this section given that you don't have a similar setting. ::: Create the following files with the correct permission according to [this](https://hackmd.io/qRJdBJmqSkulCcS_cH4ygA#Detail). ``` ~/.ssh/config ~/.ssh/id_rsa ~/.ssh/id_rsa.pub ``` Append the following content to `~/.ssh/config`: ``` Host k8s User k8s HostName 172.16.1.99 IdentitiesOnly yes IdentityFile ~/.ssh/id_rsa Host superuser User superuser HostName 172.16.1.99 IdentitiesOnly yes IdentityFile ~/.ssh/id_rsa ``` From now on, the following to lines are equivalent. ``` ssh superuser@172.16.1.99 ssh superuser ``` Register the public key to `superuser@172.16.1.99`. ``` ssh-copy-id superuser ``` ### Create non-priviledged user on *host* Log in to *host* as superuser. ``` ssh superuser ``` Create user `k8s` with group `libvirt` on *host*. ``` sudo useradd -m -G libvirt k8s ``` Then, add the same public key to `/home/k8s/.ssh/authorized_keys` which can be found in `/home/superuser/.ssh/authorized_keys`. First, create directory with permission set. ``` sudo mkdir -m 700 /home/k8s/.ssh ``` Copy the whole authorized key file and set permission. ``` sudo cp /home/superuser/.ssh/authorized_keys /home/k8s/.ssh/authorized_keys sudo chmod 600 /home/k8s/.ssh/authorized_keys ``` Edit the destinated file so that it contains only the public created previously. `$EDITOR` could be `nano`, `vi`, `vim`, `nvim`, `emacs`, ... ``` sudo $EDITOR /home/k8s/.ssh/authorized_keys ``` Change the owner of everything in (and including) `/home/k8s/.ssh`. ``` sudo chown k8s:k8s -R /home/k8s/.ssh ``` ### Install tools As long as `/tmp` still has space, do the following operations in a directory in `/tmp`, then delete it afterwards. Create a temporary directory and change into it. ``` mkdir /tmp/download cd /tmp/download ``` Install [libvirt 6.6.0](https://libvirt.org/downloads.html) :::spoiler ``` curl -LO https://libvirt.org/sources/libvirt-6.6.0.tar.xz unxz libvirt-6.6.0.tar.xz tar xvf libvirt-6.6.0.tar ``` ::: Install libvirt 4.5.0 (for CentOS only). ``` sudo dnf install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install ``` Install [terraform 0.13.0](https://www.terraform.io/downloads.html). ``` curl -LO https://releases.hashicorp.com/terraform/0.13.0/terraform_0.13.0_linux_amd64.zip unzip terraform_0.13.0_linux_amd64.zip sudo install terraform /usr/local/bin ``` Install [Matchbox 0.8.3](https://github.com/poseidon/matchbox/releases). ``` curl -LO https://github.com/poseidon/matchbox/releases/download/v0.8.3/matchbox-v0.8.3-linux-amd64.tar.gz tar zxvf matchbox-v0.8.3-linux-amd64.tar.gz sudo install matchbox-v0.8.3-linux-amd64/matchbox /usr/local/bin ``` Install [kubectl 1.18.8](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-with-curl-on-linux). Latest version number can be found [here](https://storage.googleapis.com/kubernetes-release/release/stable.txt). ``` curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.8/bin/linux/amd64/kubectl sudo install kubectl /usr/local/bin ``` Install [Helm 3.3.0](https://github.com/helm/helm/releases). ``` curl -LO https://get.helm.sh/helm-v3.3.0-linux-amd64.tar.gz tar zxvf helm-v3.3.0-linux-amd64.tar.gz sudi install linux-amd64/helm /usr/local/bin ``` Get out of directory and dispose of the liquid assets. ``` cd rm -fr /tmp/download ``` ### Matchbox endpoint Since bridge mode MACVTAP devices cannot communicate with the physical NIC underneath, we have to add another MACVTAP device for the host to bind the address `192.168.88.2`. This process fits the following template. ``` ip link add link <beneath> <name> address <mac> type macvtap mode bridge ip addr add <ip/mask> brd + dev <name> ip link set <name> up ``` In our case, we want to create the MACVTAP named `matchbox` on top of the physical NIC `eno1np0` with random mac, set the ip to `192.168.88.2` with mask `255.255.255.0`, and create the route on *host* by default. ``` ip link add \ link eno1np0 matchbox \ address $(hexdump -n6 -e '/1 "%02x" 5/1 ":%02x"' /dev/urandom) \ type macvtap mode bridge ip addr add 192.168.88.2/24 brd + dev matchbox ip link set matchbox up ``` This address is for special purpose, i.e., to expose the matchbox readonly http endpoint. It would be most convenient to create a firewalld zone for port management.  Add the MACVTAP interface `matchbox@eno1np0` to the `public` zone and expose port `8080`. Port `8081` doesn't have to be exposed, interestingly. ``` sudo firewall-cmd --permanent --zone=matchbox --add-interface=matchbox@eno1np0 sudo firewall-cmd --permanent --zone=matchbox --add-port=8080 sudo firewall-cmd --reload ``` ### Partition for libvirt LVM pool Create a partition to boast the libvirt LVM pool. Depending on whether the disk uses `MBR` or `GPT`, use `fdisk` or `gdisk` for the task. In our case, the 600GB `/dev/sdb3` is created with `gdisk`. This would be enough. Don't create a physical volume or volume group from it. ## 2. Setup libvirt :::info Starting from this section, user log in as `k8s`, i.e., `ssh k8s`. ::: ### Configuration We will be working with libvirt through `virsh` solely. Libvirt is a client-server architecture; the command line client `virsh` communicates with the server-side daemon `libvirtd` through a UNIX socket. By default, virsh connects to session mode libvirtd which runs with user permission. The following two lines are identical. ``` virsh virsh -c qemu:///session ``` However, the operations we're about to perform requires elevated priviledges, which only system mode libvirtd can carry out. Thus, we have to do the following. ``` virsh -c qemu:///system ``` This requires read/write access to the system mode libvirtd socket. The easiest way to achieve this is by adding a user to the `libvirt` group, which is what we did previously. However, typing the connection each time we use virsh is too verbose. Fortunately, libvirt provides a simple way to change the default connection. Add the following to the file `~/.config/libvirt/libvirt.conf`. ``` uri_default = "qemu:///system" ``` Now we can add `/dev/sdb3` as our LVM pool `k8s`. ``` pool-define-as k8s logical - - /dev/sdb3 k8s /dev/k8s ``` The line above simply creates a libvirt entry (a XML file) for the pool `k8s` to use the partition `/dev/sdb3`. The next line will automatically create a LVM physical volume and a LVM volume group both named `k8s`. ``` pool-build k8s ``` Mark the LVM pool to start on boot. Then, activate the pool. ``` pool-autostart k8s pool-start k8s ``` See [this](http://linux.dell.com/files/whitepapers/KVM_Virtualization_in_RHEL_6_Made_Easy_Part2.pdf) for more information. ### Create VMs The latest Fedora CoreOS is a variant of Fedora 32. Due to the dated version of our libvirt installation, the closest we could get is Fedora 31. pfsense: `pfsense.k8s` ```bash virt-install \ --name pfsense.k8s \ --vcpus 2 \ --memory 4096 \ --boot hd,cdrom,useserial=yes,menu=on \ --disk pool=k8s,size=30,sparse=false,cache=none,io=native,bus=virtio \ --disk vol=default/pfsense.iso,device=cdrom \ --graphics none \ --os-variant freebsd12.0 ``` controller: `a.elk.k8s` ``` virt-install \ --name a.elk.k8s \ --vcpus 2 \ --memory 8192 \ --boot hd,network,useserial=on,menu=on \ --noreboot \ --disk pool=k8s,size=30,sparse=false,cache=none,io=native,bus=virtio \ --network type=direct,source=eno1np0,source_mode=bridge,model=virtio \ --graphics none \ --noautoconsole \ --os-variant fedora31 ``` worker: `1.elk.k8s`, `2.elk.k8s`, `3.elk.k8s` ``` virt-install \ --name 1.elk.k8s \ --vcpus 4 \ --memory 16384 \ --boot hd,network,useserial=on,menu=on \ --noreboot \ --disk pool=k8s,size=60,sparse=false,cache=none,io=native,bus=virtio \ --network type=direct,source=eno1np0,source_mode=bridge,model=virtio \ --graphics none \ --noautoconsole \ --os-variant fedora31 ``` ## 3. Configure pfsense Get the MAC address of each cluster VM. ``` for n in a 1 2 3 ; do [[ $(virsh domiflist ${n}.elk.k8s) =~ (([0-9a-f]{2}:){5}[0-9a-f]{2}) ]] && echo ${BASH_REMATCH[1]} done ``` :::danger TODO Assign MAC, IP, hostname, load-balancing. ::: Install the PfSense VM with VNC; SPICE is overkill, and I face problems with the serial install. #### System / General Setup - Hostname: pfSense - Domain: dev.k8s - DNS Servers: \<upstream dns ip\> - Timezone: Etc/GMT+8 - Timeservers: tw.pool.ntp.org #### Interfaces / WAN - Configuration Type: Static IPv4 - IPv4 Address: \<WAN ip\> / \<mask\> - IPv4 Upstream gateway: WANGW - \<WAN gateway ip\> - Block private networks and loopback addresses: *uncheck* #### Firewall / Rules / WAN - (Destination, Port) = (WAN address, HTTPS) - (Destination, Port) = (WAN address, SSH) #### Services / DHCP Server / LAN - Enable: *check* - Range: \<first ip\> - \<last ip\> - Network Booting - Enable: *check* - **set bootp?** - DHCP Static Mappings for this Interface - (MAC Address, IP Address, Hostname, Domain name) #### Services / DNS Resolver / General Settings - Enable: *check* - Network Interfaces: LAN, Localhost - Outgoing Network Interfaces: WAN - Static DHCP: *check* - Host Overrides - \<conform to kubeconfig\> ## 4. Configure and start matchbox :::info From now on, we will work in the dedicated directory `~/elk.k8s`. Do `mkdir -p ~/elk.k8s` to create it. Then, `cd ~/elk.k8s` to change into it. ::: Create the directories for matchbox. ``` mkdir -p matchbox/assets/fedora-coreos mkdir -p matchbox/etc/matchbox ``` Download the boot images for [Fedora CoreOS 32.20200715.3.0](https://getfedora.org/en/coreos/download?tab=metal_virtualized&stream=stable). ``` cd matchbox/assets/fedora-coreos # kernel curl -LO https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200715.3.0/x86_64/fedora-coreos-32.20200715.3.0-live-kernel-x86_64 # initramfs curl -LO https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200715.3.0/x86_64/fedora-coreos-32.20200715.3.0-live-initramfs.x86_64.img # raw image curl -LO https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200715.3.0/x86_64/fedora-coreos-32.20200715.3.0-metal.x86_64.raw.xz # signature for raw image curl -LO https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200715.3.0/x86_64/fedora-coreos-32.20200715.3.0-metal.x86_64.raw.xz.sig cd ~/elk.k8s ``` Create cerificates and keys, and copy them to the correct locations. ``` SAN=DNS.1:matchbox.k8s ./cert-gen cp tls/ca.crt tls/server.crt tls/server.key matchbox/etc/matchbox/ ``` ``` mkdir -p cluster/.matchbox cp tls/ca.crt tls/client.crt tls/client.key cluster/.matchbox ``` Start matchbox. ``` matchbox \ -log-level=debug \ -address=matchbox.k8s:8080 \ -rpc-address=matchbox.k8s:8081 \ -data-path=matchbox \ -assets-path=matchbox/assets \ -cert-file=matchbox/etc/matchbox/server.crt \ -key-file=matchbox/etc/matchbox/server.key \ -ca-file=matchbox/etc/matchbox/ca.crt ``` ## 5. Configure and start terraform `main.tf` ``` module "mercury" { source = "git::https://github.com/poseidon/typhoon//bare-metal/fedora-coreos/kubernetes?ref=v1.18.8" cluster_name = "elk.k8s" matchbox_http_endpoint = "http://matchbox.k8s:8080" os_stream = "stable" os_version = "32.20200809.3.0" install_disk = "vda" cached_install = true k8s_domain_name = "elk.k8s" ssh_authorized_key = "ssh-rsa AAA..." enable_aggregation = true controllers = [{ name = "a.dev.k8s" mac = "" domain = "a.dev.k8s" }] workers = [ { name = "1.dev.k8s", mac = "" domain = "1.dev.k8s" }, { name = "2.dev.k8s", mac = "" domain = "2.dev.k8s" }, { name = "3.dev.k8s", mac = "" domain = "3.dev.k8s" } ] } ``` `providers.tf` ``` provider "matchbox" { endpoint = "matchbox.k8s:8081" client_cert = file("~/elk.k8s/cluster/.matchbox/client.crt") client_key = file("~/elk.k8s/cluster/.matchbox/client.key") ca = file("~/elk.k8s/cluster/.matchbox/ca.crt") } provider "ct" {} terraform { required_providers { ct = { source = "poseidon/ct" version = "0.6.1" } matchbox = { source = "poseidon/matchbox" version = "0.4.1" } } } ``` `out.tf` ``` resource "local_file" "kubeconfig-mercury" { content = module.mercury.kubeconfig-admin filename = "~/dev.k8s/kube.conf" } ``` ignition ssh agent ``` eval `ssh-agent -s` ssh-add ~/.ssh/dev.k8s ``` Initialize terraform ``` terraform init teraform plan teraform apply ``` ``` journalctl -f -u bootstrap ``` ## 6. Finalize - stop matchbox - deploy ELK Create file `elk-limit.sh` with content: ``` echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf sudo sysctl -p sudo sed --in-place '/nofile/d' /etc/sysconfig/docker sudo systemctl restart docker ``` Then, run: ``` for n in a 1 2 3 ; do ssh core@${n}.elk.k8s 'bash -s' < elk-limit.sh done ``` This is due to [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-prod-prerequisites). See [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config.html) too. ``` export KUBECONFIG=$HOME/dev.k8s/kube.conf ``` ``` kubectl expose pod/elk-8546b7c8f4-hkqf2 \ -n elk \ --type=NodePort \ --port=22,5601,9200,5044 ``` ``` kubctl exec -it <pod> -n elk -- /bin/bash echo "root:mcnlab538" | chpasswd echo "Port 22" >> /etc/ssh/sshd_config echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config echo "PermitRootLogin yes" >> /etc/ssh/sshd_config echo "PasswordAuthentication yes" >> /etc/ssh/sshd_config ssh-keygen -t rsa -b 4096 -P '' -f /etc/ssh/ssh_host_rsa_key ssh-keygen -t ed25519 -b 4096 -P '' -f /etc/ssh/ssh_host_ed25519_key ssh-keygen -t ecdsa -b 521 -P '' -f /etc/ssh/ssh_host_ecdsa_key service ssh start ```