# Monitor Kubernetes using Prometheus

## Introduction
* In cloud environment, there is no fixed monitoring target, and nearly every monitored object in the cloud changes dynamically. Thus, Prometheus cannot statically monitor every device in the cloud.
* The solution is to introduce an intermediate agent. This agent has access to all current monitored targets. Prometheus only needs to ask the agent what monitoring targets there are. Such mechanism is called **service discovery**.

## Intorduction to Kubernetes
### Overview
* **Kubenetes** is an open source container orchestration tool developed by Google
* Kubernetes regards a series of hosts as a large amount of managed resources, which form a **cloud operating system** that can be easily expanded. Each container running in Kubernetes can be regarded as a process running in the cloud operating system.
* Kubernetes contains a central scheduler that is capable of scheduling, monitoring, and managing the entire system based.
### Structure

#### Pod
* Pod is the smallest scheduling resource in Kubernetes.
* Pod contains several containers, they can work together and provide a particular function.
* For the containers in a pod, they share the same network and storage resources, so they can be accessed directly through the local network.
#### Node
* A Node is a worker machine in Kubernetes. It may run on either a virtual or a physical machine.
* Pods always runs on a Node. A single Node may have multiple pods, and the Kubernetes Master can automatically handles scheduling the pods across the Nodes in the cluster
* Each Node cannot communicate directly with each other. They must be referred through the kube-apiserver

##### Kubelets
* The kubelet is the primary node agent that runs on each node. The kubelet takes a set of **PodSpecs** (a YAML or JSON object that describes a pod) that provided and ensures that the containers described in those PodSpecs are running healthy.
##### cAdvisor
* CAdvisor is Google's open source visualization tool for displaying and analyzing the running status of containers. It performs real-time monitoring and performance data collection on a Node machine, including CPU usage, memory usage, network throughput, file system usage, and so on.
##### Kube-Proxy
* Kube-Proxy serves as a network proxy run on each node. It is responsible for updating the iptables of a node, so that other objects in Kubernetes that are outside Node can know the status of all pods on the node.

#### Cluster
* A Cluster is a collection of multiple Nodes and Masters in Kubernetes. Basically it can be thought of as a unit where all Nodes are gathered together in the same environment

#### Kubernetes Master
* The Kubernetes Master is a center node responsible for managing all other Nodes. There are four components in a Master: kube-apiserver, etcd, kube-scheduler, and kube-controller-manager.
##### kube-apiserver
* The Kubernetes API server provides an endpoint for administrator to manage the entire Kubernetes. For example, the *kubectl* command communicates with the kube-apiserver and perform operations on the whole Kubernetes.
* The Kubernetes API server is also responsible for the authentication, authorization, and the access control of Kubernetes.
##### etcd
* The etcd is used to store Kubernetes data as backup. When the Master fails for some reason, the etcd can help us restore the Kubernetes state.
##### kube-controller-manager
* The Kubernetes controller manager is a series of processes responsible for monitoring the status of the cluster.
* For example, the Kubernetes controller manager will try to update the current state of a cluster when it does not match the desired state.
##### kube-scheduler
* The Kubernetes scheduler is a function that schedules availability, performance, and capacity thrpughput the Kubernetes clusters.
* For example, the Kubernetes scheduler can monitors the newly created Pods that have not yet been designated to run on which Node, and decide the most suitable Node to be placed according to the resource depending on regulations and hardware restrictions on each Node.

## Helm
* Helm is a tool that streamlines installing and managing Kubernetes applications. It is a packet manager for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes YAML files.
* Basic structure of a chart:
├── Chart.yaml
├── charts
├── templates
│ ├── deployment.yaml
│ ├── ingress.yaml
│ └── service.yaml
└── values.yaml

1. Chart.yaml: Metadata for the chart.
2. templates: A directory consists of configuration files for Kubernetes. The configuration is not hard coded in the YAML file. Instead, the parameters can be brought in via values.yaml, Chart.yaml, and so on.
3. values.yaml: Literally all the parameters of this Chart is defined in this YAML file. These parameters will be substituted into the components in templates.
4. charts: A directory containing any charts upon which this chart depends

## Ways to monitor Kubernetes

Prometheus provides 5 ways to automatically discover & monitor the Kubernetes cluster: node, pod, endpoint, service, and ingress monitoring.

### Example
#### Node Monitoring

* **Monitoring Kubelet**: We can extract information via **Kubelet** because Kubelet actually contains a built-in support for Promtheus directly. We can access **Kubelet** by visiting the Kube-Proxy.

* **Monitoring cAdvisor**: The **cAdvisor** also has built in support for Prometheus. We can also obtain the resource usage of all containers running on the current node via cAdvisor similarly by visiting the Kube-Proxy.

#### Endpoint Monitoring
* **Deploying node-exporters**: In order to collect the resource, we need to deploy a Node Exporter instance in each node. In this case, we need a Daemonset to achieve the desired result. (As the name suggests, Daemonset's management is similar to the daemon process in the operating system. Daemonset will ensure that a unique Pod is running on all nodes in the cluster.)

* **Kube-apiserver**: It manages the entire Kubernetes cluster, and it is responsible for exposing the Kubernetes API to external users. In fact, the actual back-end address of the kubernetes service is maintained through an endpoint. Thus, we can monitor kube-apiserver related information by quering the backend addresses through the endpoint.