# Kubernetes (K8S)

###### tags: `devops` `aws`

[TOC]

## Expect
familiar with the concept and critical components we use
Familiar with kubectl commands

**常常改pod的分布, deployment到哪, 擴增減少等等**

https://kubernetes.io/docs/concepts/
https://godleon.github.io/blog/Kubernetes/k8s-Deployment-Overview/

## Study Item

you use Kubernetes API objects to describe your cluster’s desired state: what applications or other workloads you want to run, what container images they use, the number of **replicas**, what network and disk resources you want to make available, and more.

### HandOn
#### Interactive Tutorial
https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive/

#### Minikube
https://github.com/kubernetes/minikube
minikube implements a local Kubernetes cluster on macOS, Linux, and Windows.
minikube's primary goals are to be the best tool for local Kubernetes application development and to support all Kubernetes features that fit.

## Basic Components

### Manage Structure

Manage component will talk later, we will focus on right part of this image.

### What's Pod

$ kubectl get pod -l "service in (http-server)"
NAME                     READY     STATUS    RESTARTS   AGE
nginx-3322722759-7vp34   1/1       Running   0          14m
nginx-3322722759-ip5w2   1/1       Running   0          14m
nginx-3322722759-q97b7   1/1       Running   0          14m

Kubernetes 運作的最小單位，通過該種抽象類別可以把更進階別的抽象內容增加到容器化組件。
* 一個 Pod 對應到一個應用服務（Application） ，舉例來說一個 Pod 可能會對應到一個 API Server。
* 每個 Pod 都有一個身分證，也就是屬於這個 Pod 的 yaml 檔
* 一個 Pod 裡面可以有一個或是多個 Container，這樣可以保證它們一直位於主機上，並且可以共享資源，但一般情況一個 Pod 最好只有一個 Container
* 同一個 Pod 中的 Containers 共享相同資源及網路，彼此透過 local port number 溝通
* Kubernetes中的每個pod都被分配一個唯一的（在叢集內的）IP位址這樣就可以允許應用程式使用同一埠，而避免了發生衝突的問題。
* pod可以通過Kubernetes API手動管理，也可以委託給控制器來實現自動管理。
* pod可以定義一個Volumes，例如本地磁碟目錄或網路磁碟，並將其暴露在pod中的一個容器之中。

#### What's Volumn
Pod可以定義一個Volumes，例如本地磁碟目錄或網路磁碟，並將其暴露在pod中的一個容器之中。

Solved Problem below
- file lost when a Container crashes. (when kubelet will restart it, but the files will be lost - the Container starts with a clean state.)
- Share files between Containers.

##### Comparing between Docker & K8S
**Docker**
In Docker, a volume is simply a directory on disk or in another Container. Lifetimes are not managed and until very recently there were only local-disk-backed volumes.

**k8s**
A Kubernetes volume, has an explicit lifetime - the same as the Pod that encloses it.

Consequently, a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts.

#### Kubernetes 是如何建立一個 Pod?
> 上圖為一個簡易的 Kubernetes Cluster，通常一個 Cluster 中其實會有多個 Master 作為備援，但為了簡化我們只顯示一個。
當使用者要部署一個新的 Pod 到 Kubernetes Cluster 時，使用者要先透過 User Command（kubectl）輸入建立 Pod 的對應指令（下面會在解說如何實際動手操作來建立一個 Pod）。此時指令會經過一層確認使用者身份的認證後，傳遞到 Master Node 中的 API Server，API Server 會把指令備份到 etcd 。
接下來 controller-manager 會從 API Server 收到需要創建一個新的 Pod 的訊息，並檢查如果資源許可，就會建立一個新的 Pod。最後 Scheduler 在定期訪問 API Server 時，會詢問 controller-manager 是否有建置新的 Pod，如果發現新建立的 Pod 時，Scheduler 就會負責把 Pod 配送到最適合的一個 Node 上面。
雖然上面的基本運作看似複雜，但實際上我們在操作時，只要輸入一行指令後 Kubernetes 就會自動幫我們完成後續的動作。
> https://medium.com/@C.W.Hu/kubernetes-basic-concept-tutorial-e033e3504ec0
> https://ithelp.ithome.com.tw/articles/10193248

#### kube-dns
DNS，全名為 Domain Name System。DNS 會有一張表格，紀錄每個 domain name 相對應的 IP 位址。如此，我們不再需要去紀錄該服務的 IP address，而是可以透過該服務的網域名稱連結到該服務。

Kubernetes 在每一個 Pod 創建時，都會在該 Pod 的 /etc/resolve.conf 檔案中，自動加入 kube-dns service 的 domain name 與相對應的 IP 位址。因此 其他 Pods 可以透過名稱為 kube-dns 的 Service 物件，找到正在運行的 kube-dns

### What's Node

The nodes in a cluster are the machines (VMs, physical servers, etc) that run your applications and cloud workflows. The Kubernetes master controls each node; you’ll rarely interact with nodes directly.


$ kubectl describe nodes minikub

**在 Worker Node 與 Master Node 的組件部分，因為 Kubernetes 本身其實都抽象的很好，所以在 Kubernetes 「基礎的」使用上如何不了解這些組建也不會有非常大的影響。**

#### Master Node
Kubernetes 運作的指揮中心，可以簡化看成一個特化的 Node 負責管理所有其他 Node。

> 一個 Master Node（簡稱 Master）中有四個組件：
> - kube-apiserver
>* etcd
> * kube-scheduler
> * kube-controller-manager
>
> **kube-apiserver**
> 管理整個 Kubernetes 所需 API 的接口（Endpoint），例如從 Command Line 下 kubectl 指令就會把指令送到這裏；
> 負責 Node 之間的溝通橋樑，每個 Node 彼此不能直接溝通，必須要透過 apiserver 轉介
> 負責 Kubernetes 中的請求的身份認證與授權
>
> **etcd**
> 整個 cluster 的配置與定義、以及各項工作的狀態資訊都是由 etc 負責統管的，用來存放 Kubernetes Cluster 的資料作為備份，當 Master 因為某些原因而故障時，我們可以透過 etcd 幫我們還原 Kubernetes 的狀態。
> 具有兩種部屬型態
> - 在 master 節點上以 pod 的形式部署
> 
> - 以獨立的 etcd cluster 方式部署
> 
>
> **kube-controller-manager**
> 負責管理並運行 Kubernetes controller 的組件，簡單來說 controller 就是 Kubernetes 裡一個個負責監視 Cluster 狀態的 Process，例如：Node Controller、Replication Controller 這些 Process 會在 Cluster 與預期狀態（desire state）不符時嘗試更新現有狀態（current state）。例如：現在要多開一台機器以應付突然增加的流量，那我的預期狀態就會更新成 N+1，現有狀態為 N，這時相對應的 controller 就會想辦法多開一台機器 controller-manager 的監視與嘗試更新也都需要透過訪問 kube-apiserver 達成
>
> **kube-scheduler**
> 整個 Kubernetes 的 Pods 調度員，scheduler 會監視新建立但還沒有被指定要跑在哪個 Node 上的 Pod，並根據每個 Node 上面資源規定、硬體限制等條件去協調出一個最適合放置的 Node 讓該 Pod 跑

#### Worker Node
一個 Worker Node（簡稱 Node）對應到一台機器, 虛擬機如 AWS 上的一台 EC2 或 GCP 上的一台 Computer Engine。

> 每個 Node 中都有三個組件：kubelet、kube-proxy、Container Runtime。
>
> **kubelet**
> 該 Node 的管理員，負責管理該 Node 上的所有 Pods 的狀態並負責與 Master 溝通
>
> **kube-proxy**
> 該 Node 的傳訊員，負責更新 Node 的 iptables，讓 Kubernetes 中不在該 Node 的其他物件可以得知該 Node 上所有 Pods 的最新狀態
>
> **Container Runtime**
> 該 Node 真正負責容器執行的程式，以 Docker 容器為例其對應的 Container Runtime 就是 Docker Engine

### What's Service

A Service in Kubernetes is an abstraction which defines a **logical set of Pods and a policy by which to access them**.


* The set of Pods targeted by a Service is usually determined by a *LabelSelector*.
> 

* Services enable a loose coupling between dependent Pods.
* A Service is defined using YAML (preferred) or JSON, like all Kubernetes objects.

* Although each Pod has a unique IP address, those IPs are not exposed outside the cluster without a Service. Services allow your applications to receive traffic.

Services can be exposed in different ways by specifying a type in the ServiceSpec:

1. ClusterIP (default) - Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.
1. NodePort - Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using <NodeIP>:<NodePort>. Superset of ClusterIP.
1. LoadBalancer - Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.
1. ExternalName - Exposes the Service using an arbitrary name (specified by externalName in the spec) by returning a CNAME record with the name. No proxy is used. This type requires v1.7 or higher of kube-dns.

#### [Headless Services](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services)

Sometimes you don’t need load-balancing and a single Service IP. In this case, you can create what are termed “headless” Services.

- specifying "None" for the cluster IP (.spec.clusterIP).
- You can use a headless Service to interface with other service discovery mechanisms, without being tied to Kubernetes’ implementation.
- a cluster IP is not allocated, kube-proxy does not handle these Services
- there is no load balancing or proxying done by the platform for them.

#### Service Topology
Service Topology enables a service to route traffic based upon the Node topology of the cluster. For example, a service can specify that traffic be preferentially routed to endpoints that are on the same Node as the client, or in the same availability zone.

### What's Cluster
Kubernetes 中多個 Node 與 Master 的集合。基本上可以想成在同一個環境裡所有 Node 集合在一起的單位。

## Controllers

在Kubernetes的世界中，**[ReplicaSet](/FlsxRUNHRNSJOpEhNmh-7A)和[Deployment](/pK2oPv8dQAy0weH2I1Z93Q)主要用於處理無狀態的服務**，無狀態服務的需求經常非常簡單並且輕量，每一個無狀態轉換器存儲的數據在重啟之後就會被刪除;
**[StatefulSet](/jkA5a4uRRHeuROMe5r5vQg)是用於管理有狀態應用的工作負載對象**，能管理Pod的對象，還它能夠保證這些Pod的順序性和唯一性。

### Stateless manage structure

從圖中可以看到一個 Deployment 掌管一或多個 ReplicaSet，而一個 ReplicaSet 掌管一或多個 Pod。
讓 Deployment 控管多個 ReplicaSet 的主要原因在於支援回滾機制 (Rollback)。每當部署新的 Deployment 設定時，Kubernetes 會依據新設定重新生成一個 ReplicaSet 並保留舊設定，未來有需要的話就能直接利用舊的 ReplicaSet 回滾至先前狀態。必須注意的是同時間只會有一個 ReplicaSet 生效 (DESIRED > 0)。

#### [Deployment](/pK2oPv8dQAy0weH2I1Z93Q)

可以使用 kubectl create 或 kubectl apply 去建立 nginx deployment

$ kubectl create -f nginx.yaml
deployment "nginx" created

$ kubectl get deployment
NAME                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx               3         3         3            3           8m

#### [Replica](/FlsxRUNHRNSJOpEhNmh-7A)

$ kubectl scale deployment/medium-api --replicas=2
$ kubectl get pods -o wide

ReplicaSet 是用來確保在 k8s 中，在資源允許的前提下，指定的 pod 的數量會跟使用者所期望的一致，也就是所謂的 desired status。

ReplicaSet 其實是 ReplicationController 的進化版，其中的差別僅在於 ReplicaSet 支援 set-based label selector，而 ReplicationController 僅支援 equality-based label selector。

$ kubectl get rs
NAME                DESIRED   CURRENT   AGE
nginx-3322722759    3         3         8m

### [StatefulSet](/jkA5a4uRRHeuROMe5r5vQg)

StatefulSet is the workload API object used to manage stateful applications.
eg. Zookeeper，[Kafka](/Mq2Kagu0TbSW6SrpC6fhfw)

Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods.

https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

#### Pros
1. Stable, unique network identifiers.
1. Stable, persistent storage.
1. Ordered, graceful deployment and scaling.
1. Ordered, automated rolling updates.

#### Limitation, Cons
- The **storage** for a given Pod must either be provisioned by a PersistentVolume Provisioner based on the requested storage class, or pre-provisioned by an admin.
- Deleting and/or scaling a StatefulSet down will not delete the volumes associated with the StatefulSet. This is done to ensure data safety, which is generally more valuable than an automatic purge of all related StatefulSet resources.
- StatefulSets currently **require a Headless Service to be responsible for the network identity of the Pods.** You are responsible for creating this Service.
- StatefulSets do not provide any guarantees on the termination of pods when a StatefulSet is deleted. To achieve ordered and graceful termination of the pods in the StatefulSet, it is possible to scale the StatefulSet down to 0 prior to deletion.
- When using Rolling Updates with the default Pod Management Policy (OrderedReady), it’s possible to get into a broken state that requires manual intervention to repair.

kubectl get pods -l app=nginx
NAME      READY     STATUS    RESTARTS   AGE
web-0     1/1       Running   0          1m
web-1     1/1       Running   0          1m

#### Scaling
Scaling a StatefulSet refers to increasing or decreasing the number of replicas. This is accomplished by updating the replicas field. You can use either ```kubectl scale``` or ```kubectl patch``` to scale a StatefulSet.

kubectl scale sts web --replicas=5

他會Sequencial increase, therefor you will see something like below

Watch Pods status

kubectl get pods -w -l app=nginx


Scale Down

kubectl patch sts web -p '{"spec":{"replicas":3}}'
statefulset.apps/web patched


[Rolling Update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#rolling-update)

#### Delete

### DaemonSet
A DaemonSet ensures that all (or some) Nodes run a copy of a Pod.
As nodes are added to the cluster, Pods are added to them.
As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

話說 ReplicaSet 是實現服務 AutoScale or Loadbalance 的方式，
但是如需在 K8s node 叢集環境下確保特定 pod 在每個 node 上皆能夠實現相同服務，
DaemonSet 反而是最佳選擇。

典型的應用服務包括：
1. 叢集儲存，比如 glusterd、ceph
1. 日誌收集，比如 fluentd、logstash
1. 系统監控，比如 Prometheus Node Exporter、collectd、New Relic agent、Ganglia gmond
1. 系统程序，比如 kube-proxy、kube-dns

DaemonSet 特性
* DaemonSet 會確保每個 node 上運行同一個 pod 服務～ 除非透過 nodeSelector。
* 例如系統常駐性監控服務，是非常適合透過 DaemonSet 實現副本需求。
* DaemonSet 如同 ReplicaSet 皆透過 Reconciliation Loops 可以確保目前 Status 是否符合預期 Spec。
* 當 K8s 新增 node 時，DaemonSet 會於 new node 上新增 pod 副本。

非 DaemonSet 特性
* 如果需要為了服務大量客戶流量，需要建立大量 pod 副本，
* 並且無需考量與 node 耦合性問題， 透過 ReplicaSet 進行 service scale 是正確的選擇。

## Storage
### PV - Persistent Volumes
k8s 利用 PV 將提供儲存區 (Storage) 與使用儲存區抽象化。PV 是一個在 k8s 中提供一串儲存區的虛擬化物件，是一個讓 Pod 可以保留儲存資料的元件，提供給 Container 將 Volume 中的資料永久地儲存下來，不會因為 Pod 被銷毀而不見。

PV對於 K8S 的管理者來說，就只是在 K8S Cluster 建立一個 Persistent Volume，像是裝上一顆硬碟的概念。目前 Container 真正的掛載使用需要透過 Persistent Volume Claim 機制的幫忙。

存取模式 (acessModes) 有三種
* ReadWriteOnce：被單一 Node 掛載為 read/write 模式
* ReadOnlyMany：被多個 Node 掛載為 read 模式
* ReadWriteMany：被多個 Node 掛載為 read/write 模式

在 K8S 架構中 Persistent Volume 算是底層的元件，目前已經有很多不同的實做，這些實做稱為 Provisioner.

有兩種提供模式
Static - admin建立好PVs
Dynamic- 如果PVC要的PV不存在, cluster會動態幫建立

可被掛載成Volumn使用

### PVC - Persistent Volume Claims
有了 PV 之後，我們可以透過 PVC 物件取得向 PV 發出要求儲存區的請求。它的概念就像是 Pod 會消耗 Node 的資源 (CPU, Memory) 一般，PVC 可以向 PV 要求存取 (大小、read/write 多次或一次等等) 儲存區。

## Configuration
### ConfigMap

A ConfigMap is an API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume.

A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.

目的:
為了讓應用程式可以方便轉移或避免重新編譯 image, 因此會將配置與應用程式解構, 以單獨的 ConfigMap 配置

### Secret
Kubernetes Secrets let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys.

To use a secret, a Pod needs to reference the secret.
A secret can be used with a Pod in two ways:
- As files in a volume mounted on one or more of its containers.
- By the kubelet when pulling images for the Pod.

### Kubeconfig file

Use kubeconfig files to organize information about clusters, users, namespaces, and authentication mechanisms. The kubectl command-line tool uses kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster.

* A running kubelet might authenticate using certificates.
* A user might authenticate using tokens.
* Administrators might have sets of certificates that they provide to individual users.

## Security

在整個環境上可以用4層架構來檢視資安結構

### Level1: Cloud
Cloud Security need to refer to AWS security.

### Level2: Cluster
* Securing the components that are configurable which make up the cluster
* Securing the components which run in the cluster

### NameSapce
用來進行資源的虛擬隔離之用，在 k8s 剛建立好的初期會有預設的 default & kube-system 兩個 namespace

### ServiceAccount

Default RBAC policies grant scoped permissions to **control-plane components, nodes, and controllers**, but grant no permissions to service accounts **outside the kube-system namespace**.

User Account & Service Account 是用來區別作用的範圍，但會有以下的差別：
- User Account 是用來辨識使用者，對應人的身份，因此是跨 namespace
- Service Account 則是用來辨識 k8s 中的各種服務, 對應的是 service 的身份，因此是與 namespace 有相關

### [Authorization module](https://godleon.github.io/blog/Kubernetes/k8s-API-Authorization/)

#### ABAC - Attribute base access control
API server 會檢查 API request attribute 來決定是否進行授權存取，在 k8s 有以下 API request attributes
* user, group, extra(額外的 key/value 資訊)
* API、Request Path(/api, /healthz), API request verb(get, list, create, update, patch, watch … etc)
* HTTP request verb(get, post, put, delete)
* Resource, Subresource (存取 resource 時需要的資訊)
* Namespace
* API group (存取 resource 時需要的資訊)

#### RBAC - Role-based access control
A method of regulating access to computer or network resources based on the roles of individual users within your organization.

* Role：定義在特定 namespace 下的 resource 的存取權限
* RoleBinding： 設定哪些使用者(or service account)與 role 綁定而擁有存取權限
* ClusterRole：定義在整個 k8s cluster 下的 resource 的存取權限
* ClusterRoleBinding：設定哪些使用者(or service account)與 role 綁定而擁有存取權限

### What's admission Webhook
* Admission webhooks are HTTP callbacks that receive admission requests and do something with them. 此模式是管理者在外部提供 HTTPS 授權服務，並設定 API server 透過與外部服務互動的方式進行授權。
* Admission webhooks are essentially part of the cluster control-plane.
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#what-are-admission-webhooks

### Level3: Container
OS based security

### Level4: Code
Code level security problem

## Resources

如果今天pod跑的Node資源足夠, container有機會 & 被允許比他request的用上更多資源
但container絕不能用超過limit

> For example, if you set a memory request of 256 MiB for a container, and that container is in a Pod scheduled to a Node with 8GiB of memory and no other Pods, then the container can try to use more RAM.
>
> If you set a memory ```limit``` of 4GiB for that Container, the kubelet (and container runtime) enforce the limit. The runtime prevents the container from using more than the configured resource limit. For example: when a process in the container tries to consume more than the allowed amount of memory, the system kernel terminates the process that attempted the allocation, with an out of memory (OOM) error.

#### request

#### limit
Limits can be implemented either reactively (the system intervenes once it sees a violation) or by enforcement (the system prevents the container from ever exceeding the limit). Different runtimes can have different ways to implement the same restrictions.

#### types
A resource type has a base unit.

**Huge pages** are a Linux-specific feature where the node kernel allocates blocks of memory that are much larger than the default page size.
> For example, on a system where the default page size is 4KiB, you could specify a limit, hugepages-2Mi: 80Mi. If the container tries allocating over 40 2MiB huge pages (a total of 80 MiB), that allocation fails.

**CPU unit**
unit: 1 vCPU/Core
A Container with ```spec.containers[].resources.requests.cpu``` of```0.5``` is guaranteed half as much CPU as one that asks for 1 CPU.

The expression ```0.1``` is equivalent to the expression ```100m```, which can be read as “one hundred millicpu”.

CPU is always requested as an absolute quantity, never as a relative quantity

**Mem unit**
plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K.
You can also use the **power-of-two equivalents***: Ei, Pi, Ti, Gi, Mi, Ki

> example. The following Pod has two Containers.
Each Container has a request of 0.25 cpu and 64MiB (226 bytes) of memory. Each Container has a limit of 0.5 cpu and 128MiB of memory.
Pod has a request of 0.5 cpu and 128 MiB of memory, and a limit of 1 cpu and 256MiB of memory.

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: wp
    image: wordpress
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

>

### ephemeral Storage
暫時的儲存空間, Limits and requests for ephemeral-storage are measured in bytes.

How Pods with ephemeral-storage requests are scheduled?
> When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods.
> 

### What's Pod Overhead
Pod Overhead is a feature for accounting for the resources consumed by the Pod infrastructure on top of the container requests & limits.

kubectl describe node | grep test-pod -B2

  Namespace                   Name                CPU Requests  CPU Limits   Memory Requests  Memory Limits  AGE
  ---------                   ----                ------------  ----------   ---------------  -------------  ---
  default                     test-pod            2250m (56%)   2250m (56%)  320Mi (1%)       320Mi (1%)     36m

### [Metrix API](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
1. **Resource metrics pipeline**: The resource metrics pipeline provides a limited set of metrics related to cluster components. Exposed via the ```metrics.k8s.io``` API.
2. **Full metrics pipeline**: respond to these metrics by automatically scaling or adapting the cluster based on its current state. implementing either the`custom.metrics.k8s.io` or `external.metrics.k8s.io API`.

### [Rolling update](/G714Qa_AQq-Omayv-naETw)
Kubernetes還提供了一個非常有用的特性——滾動更新（Rolling update），這個特性的好處就是我們不用停止服務就可以實現應用更新。默認更新的時候是一個Pod一個Pod更新的，所以整個過程服務不會中斷。當然你也可以設置一次更新的Pod的百分比。而且更新過程中，Service只會將流量轉發到可用的節點上面。更加重要的是，我們可以隨時回退到舊版本。

### What's Affinity

- Affinity, 當滿足了某些條件，就把Pod調度到集群上，
- Anti-Affinity則是當也就是當滿足了某些條件，就不要把Pod調度到集群上。

#### Node Affinity
可以透過設置nodeSelector來達成Node Affinity的需求
Node Affinity和Node Selector不同的是，它會有更多細緻的操作

node Affinity有兩種策略：
- preferredDuringSchedulingIgnoredDuringExecution 前面的就是軟策略（可以不滿足，不會怎樣）
- requiredDuringSchedulingIgnoredDuringExecution 後面的就是硬策略（一定要滿足，不然就翻臉）。
> 在matchExpressions底下所列的條件全都要滿足才會調度

#### Pod Affinity

- requiredDuringSchedulingIgnoredDuringExecution
- preferredDuringSchedulingIgnoredDuringExecution
唯一不同的是如果要使用互斥性，我們需要使用podAntiAffinity字段。


#### Taints and Tolerations

污點（Taints）與容忍（tolerations），對於nodeAffinity無論是硬策略還是軟策略方式，都是調度Pod到預期節點上，而Taints恰好與之相反，如果一個節點標記為Taints ，除非Pod也被標識為可以容忍污點節點，否則該Taints節點不會被調度Pod。

Taint的策略有三種：
* NoSchedule：POD 不會被調度到標記為taints 節點。
* PreferNoSchedule：NoSchedule 的軟策略版本。
* NoExecute：該選項意味著一旦Taint 生效，如該節點內正在運行的POD 沒有對應Tolerate 設置，會直接被逐出。

## Management Console

### k8s Control Panel

----

# Hands On

https://www.itread01.com/content/1507490413.html
https://kubernetes.io/docs/tasks/

1. Create a cluster

brew cask install minikube

$ minikube version
minikube version: v0.30.0

$ kubectl cluster-info

$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

$ minikube dashboard

2. Deploy an app

Let’s deploy our first app on Kubernetes with the kubectl create deployment command. We need to provide the deployment name and app image location (include the full repository url for images hosted outside Docker hub).

This performed a few things for you:
* searched for a suitable node where an instance of the application could be run (we have only 1 available node)
* scheduled the application to run on that Node
* configured the cluster to reschedule the instance on a new Node when needed

$ kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1
deployment.apps/kubernetes-bootcamp created

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1/1     1            1           4s

1 deployment running a single instance of your app. The instance is running inside a Docker container on your node.

The kubectl command can create a **proxy that will forward communications into the cluster-wide, private network**. The proxy can be terminated by pressing control-C and won't show any output while its running.

$ kubectl proxy

$ curl http://localhost:8001/version
{
  "major": "1",
  "minor": "17",
  "gitVersion": "v1.17.3",
  "gitCommit": "06ad960bfd03b39c8310aaf92d1e7c12ce618213",
  "gitTreeState": "clean",
  "buildDate": "2020-02-11T18:07:13Z",
  "goVersion": "go1.13.6",
  "compiler": "gc",
  "platform": "linux/amd64"
}

$ export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
echo Name of the Pod: $POD_NAME

Name of the Pod: kubernetes-bootcamp-69fbc6f4cf-csqw9

3. Expose your APP

所有的pod都需要被expose到service外界才能存取
如原本只有這樣

可以用這樣去expose他
Service called kubernetes-bootcamp.
Here we see that the Service received a unique cluster-IP, an internal port and an external-IP (the IP of the Node).

describe也可以用label

另外kubectl get pod/services的指令可帶 -L run=<label>

4. Update an App