# Introduction

[TOC]

## TL;DR

We attempt to build **private kubernetes clusters** on a **private OpenStack cloud**.
Only very selective public facing services are exposed through a carefully managed gateway, whose management is outside the scope of this document.
We record useful information as we progress.

:::warning
This document contains much but not all information about our implementation.
:::

:::danger
Applicable only to the state of **summer 2020**.
:::

:::danger
We strive to follow best practices whenever possible, yet this doesn't make our setup production ready.
We still need to consult kubernetes/OpenStack/security experts to evaluate our actual implementation.
:::

## TL;DR longer

We originally aimed to build a [kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
cluster on baremetal, yet bootstrapping each node with
[kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
and
[kubelet](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/)
manually is clumsy.
After comparing some
[existing solutions](https://landscape.cncf.io/category=certified-kubernetes-distribution&format=card-mode&grouping=category),
we settled on the
[poseidon/typhoon-for-bare-metal](https://typhoon.psdn.io/fedora-coreos/bare-metal/)
approach, which is basically:
- configuring
[matchbox](https://matchbox.psdn.io/)
with
[terraform](https://www.terraform.io/intro/index.html)
to install
[Fedora CoreOS](https://docs.fedoraproject.org/en-US/fedora-coreos/)
on bare-metal machines that will permanently boot,
according to
[ignition files](https://docs.fedoraproject.org/en-US/fedora-coreos/fcct-config/),
as functional kubernetes
[nodes](https://kubernetes.io/docs/concepts/architecture/nodes/)
to form a
[cluster](https://kubernetes.io/docs/concepts/overview/components/).

After some time, due to reasons outlined below, we changed our approach to
- deploying several kubernetes clusters on VMs managed by
[OpenStack](https://www.openstack.org/software/)
which is, in turn, deployed on the bare-metal machines.

digraph structs {
    node [shape=record,fontname="Noto Sans"]
    struct3 [label="{
        {
            {custom | VM} |
            {custom | VM} |
            {custom | VM} |
            {ELK | {VM | VM | VM}} |
            {ELK | kubernetes | {VM | VM | VM}} |
            {CI/CD | kubernetes | {VM | VM | VM}} |
            {{ML | SPH} | HPC | {VM | VM | VM}}
        } |
        OpenStack |
        {
            baremetal |
            baremetal |
            baremetal
        }
    }"];
}

This solution proves to be more scalable and feature complete.
- As opposed to PXE boot, OpenStack can create VMs directly out of
[images](https://getfedora.org/en/coreos/download?tab=metal_virtualized&stream=stable#metal-virtualized:~:text=OpenStack).
- Build on top of the IaaS provided by OpenStack are
[LBaaS](https://docs.openstack.org/octavia/latest/reference/introduction.html#introducing-octavia:~:text=Octavia%20has%20become%20the%20reference%20implementation%20for%20Neutron%20LBaaS%20version%202),
[DNSaaS](https://docs.openstack.org/designate/latest/),
[FWaaS](https://docs.openstack.org/neutron/ussuri/admin/fwaas-v2-scenario.html),
[DBaaS](https://docs.openstack.org/trove/ussuri/index.html),
and
[FaaS](https://wiki.openstack.org/wiki/Faas),
to name a few.

### More than one way? How many exactly?

:::info
**Short answer:**

Two ways for Kubernetes, and two ways for OpenStack. For the final product, one from each will be selected.
:::

There are
[turnkey cloud solutions](https://kubernetes.io/docs/setup/production-environment/turnkey/)
provided by big-names like
[Google](https://cloud.google.com/kubernetes-engine),
[Amazon](https://aws.amazon.com/eks/),
[Microsoft](https://azure.microsoft.com/en-us/services/kubernetes-service/),
[IBM](https://www.ibm.com/cloud/container-service/),
and
[Alibaba](https://www.alibabacloud.com/product/kubernetes),
for which great tutorials exists, e.g.,
[kubernetes-the-hard-way](https://github.com/kelseyhightower/kubernetes-the-hard-way) for GKE.
Instead, we focus on bare-metal and OpenStack, mimicking the commercial cloud to some extent.

Before we deploy anything on bare-metal, we experiment with VMs first, and successful experiments can be directly translated to actual deployments.
Thus, we don't differentiate between them.

The exact kubernetes flavors are:
- Poseidon/typhoon on actual baremetal.
- [Fedora CoreOS OpenStack image](https://getfedora.org/en/coreos/download?tab=metal_virtualized&stream=stable#metal-virtualized:~:text=OpenStack) on OpenStack cloud.

The exact OpenStack flavors are:
- Manual installation.
- TripleO.

## Preface

With the addition of new machines to our lab racks, we took the chance to update our infrastructure.
This led to our expedition into the *cloud*.
Since cloud tech is complex, documentation is indispensible.
However, existing documents are at times oversimplified, lacking illustrations, or simply incorrect; being outdated is not particularly fatal, as long as accuracy is maintained in the supposed context.

This series is devoted to the status of Kubernetes and OpenStack in **summer 2020**.
The first part deals with deployment, which is aimed at bare-metal, KVM/QEMU/libvirt, and OpenStack.
This agrees with our orignal goal and excuses us from being overshadowed by great tutorials concerning deployment on commercial clouds, in particular, [kubernetes-the-hard-way](https://github.com/kelseyhightower/kubernetes-the-hard-way), by which our title is inspired.
The second part looks into the operation of the systems.

Initially, we wanted to deploy a bare-metal kubernetes cluster on the new machines, but the process is messy and sensitive.
As we gained experience with kubernetes operation, it becomes clear that kubernetes is no silver bullet; it seems that only workloads that are almost stateless and horizontally scalable are suitable to be deployed with kubernetes.
Certain networking aspects of kubernetes are inherently dependent on external environment, e.g., ingress and external load-balancing.
Moreover, our bare-metal machines are under-utilized and are clearly capable of hosting several clusters.
After some reflection, we find that building kubernetes on top of a cloud infrastructure is a more sensible approach.
See [Rationale](#Strategy-and-Rationale).

## Documentation Guideline

1. No bullsh*t.
2. Use rich text and diagrams.
3. Must be reproducible.
4. Keep track of reference.
5. Don't just paste code, state intention.
6. Record fail attempts and their explainations.

<!-- # Guidelines

## Principles

1. Choose tools with better **community support**.
+ fedora coreos vs ubuntu
3. Choose tools with better **reliability**.
4. Choose tools with better **security**.
5. Performance is only backed by **benchmarks**.
6. Infrastructure as code -- **IaC**.
7. Declarative

HA --[realization]-> redundancy --[complication]-> inconsistency & inefficiency -->

## Strategy and Rationale

Kubernetes is designed to scale containers and manage their connectivity.
With addons, kubernetes can achieve observability (telemetry), internal load-balancing (canary), internal network-policy, and other fancy stuff.
However, ingress, dynamic DNS, and external load-balancing rely on external services.
Moreover, bare-metal deployment of kubernetes is non-trivial.

On the other hand, OpenStack -- an open-sourced cloud solution -- is designed to introspect baremetal, provision VMs, and provide VM clusters with networking services -- firewall, routing, DNS, load-balancing, etc.
Although the installation of OpenStack is more demanding than that of kubernetes, one only has to do it once on a single machine.
Enrollment and introspection of other machines can be automated with ironic which takes advantage of PXE boot and out-of-band management, allowing the OpenStack cluster to grow at scale with little manual intervention.
Once the OpenStack services are in place, one is in the position to scale kubernetes cluster deployments with VMs.
High performance computation and other workloads unsuited for kubernetes can also be managed with OpenStack.

With OpenStack, one can easily put idle machines into work.
As VM provisioning becomes scalable, it makes sense to buy few powerful machines to host many VMs, as opposed to buying many weak machines.
VMs do incur performance hit, but they drive machine costs and management costs down.
Buying a machine with 72 cores, 288G ram, and 4.5T disk is cheaper than buying 8 machines each with 8 cores, 32G ram, and 500G disk; not to mention additional networking equipments for more machines.
Other cool things that VMs can do but bare-metal machines cannot do easily include live migration, taking snapshots, and emulating different computing architectures.

When it comes to machines, think in terms of VMs and not bare-metal.
We propose the following strategy.
- Buy machines with higher spec.
- Deploy OpenStack on them.
- Assign VMs to users requesting machines for personal workloads.
- Create VMs or reserve bare-metal machines for high performance computing.
- Deploy kubernetes clusters on VMs.

Before clusters, albeit kubernetes clusters or OpenStack clusters, are ready for production, one would like to drill through the deployment process with VMs.
This gets no easier with OpenStack.
For kubernetes clusters running on VMs, the process of deploying a cluster for test and production would be the same.