# Assignment - Scheduling framework ###### tags: `Assignment`, `Scheduling framework` ## Requirement **Version： Kubernete <font color = red>1.17+</font>** 如果當前版本不符合要求，建議[安裝Kind](https://hackmd.io/@lsalab-k8s-2020/S1Gflygfv)作為練習 **Language： go 1.14** ## Problem Description 目前default的kube-scheduler只能做到FIFO(First-In-First-Out)的工作排程(scheduling)。在這次作業中，我們要實作一個自定義的[Scheduling framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/)，按以下規則排程工作(pod)： 1. 工作會依照使用者給定的label(podGroup)，分至不同的scheduling group 2. 每個scheduling group，依照使用者給定的label(groupPriority)，擁有不同的排程優先權(groupPriority值越高的優先順序越高)。 3. 同一個group中的工作，依照使用者給定的[QoS level](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)，決定排程的先後順序(`Guaranteed`>`Burstable`>`BestEffort`)。 4. 每個group只有當確認該Group的Pod數量>=最小副本數(miniAvailable)時，才會開始進行該group的工作排程動作。 Yaml 將會提供下列的Label，可透過Label來作為判斷的依據。(請注意在測試時，我們會假設擁有相同podGroup設定值的pods，他們的miniAvailable及groupPriority設定值是相同的。) + `podGroup`: 相同名稱視為同個Group + `minAvailable`: 該Group(服務、應用...)所需要的最小副本數 + `groupPriority`： 該Group的Pod優先權 ## Hint Implement plugin： + **QueueSort** + Compare the priority of pods first. + If the priority of pods are equal, then comapre the QOS + **PreFilter** + If the number of pods in same group < minAvailable, then the pod will reject and retry later. **Writing logs** is a good habit. You can download the file , and make appropriate **changes**： The files are about... + Test + Build ```bash= $ wget https://lsalab.cs.nthu.edu.tw/~riya/assignment/schedulingFramework.tar.gz $ tar zxvf schedulingFramework.tar.gz ``` ` ## Check ```bash= $ kubectl get pod -n kube-system ``` ![](https://i.imgur.com/2SdDCGG.png) --- ## Input Example 1 There is a job, `nginx` and its group is also called the same name. According the file of yaml, we can see there is a constraint that this service must has 3 pods, then the pods will start to be scheduled to the nodes. Actually, the operator made a mistake, he accidentally typed the wrong number. Thus, we can see ... The number of Pods in same group < minAvailable, so the pod status is `Pending`. When we check the log, we can find the situation that the number of pods in same group is less than minAvailable. ```bash= $ kubectl apply -f {filename} ``` :::spoiler ```yaml= apiVersion: batch/v1 kind: Job metadata: name: nginx spec: parallelism: 2 completions: 2 template: metadata: labels: app: nginx podGroup: nginx minAvailable: "3" spec: schedulerName: riya-scheduler containers: - name: nginx image: nginx resources: requests: cpu: 3000m memory: 500Mi limits: cpu: 3000m memory: 500Mi restartPolicy: Never ``` ::: ## Output Example 1 ```bash= $ kubectl get pod ``` ![](https://i.imgur.com/AuqzFXZ.png) ```bash= $ kubectl logs {your scheduler pod name} -n kube-system ``` ![](https://i.imgur.com/rq36Gje.png) ## Input Example 2 There are three jobs, <font color = red>red</font>, <font color = green>green</font> and <font color = #FFDC35>yellow</font> and the qos level is the same. Each job is a group consisting of 10 pods. Each job must have at least two pods to start scheduling. The order of scheduling pods is sorted by priority, and the priority is Red>Yellow>Green(1000>500>100). ```bash= $ kubectl apply -f {filename} ``` :::spoiler Green.yaml ```yaml= apiVersion: batch/v1 kind: Job metadata: name: green spec: parallelism: 10 completions: 10 template: metadata: labels: podGroup: "Green" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: green image: riyazhu/testprogram:cpu resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k restartPolicy: Never ``` ::: :::spoiler Yellow.yaml ```yaml= apiVersion: batch/v1 kind: Job metadata: name: yellow spec: parallelism: 10 completions: 10 template: metadata: labels: podGroup: "Yellow" groupPriority: "500" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: yellow image: riyazhu/testprogram:cpu resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k restartPolicy: Never ``` ::: :::spoiler Red.yaml ```yaml= apiVersion: batch/v1 kind: Job metadata: name: red spec: parallelism: 10 completions: 10 template: metadata: labels: podGroup: "Red" groupPriority: "1000" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: red image: riyazhu/testprogram:cpu resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k restartPolicy: Never ``` ::: ## Output Example 2 ```bash= $ kubectl logs {your scheduler pod name} -n kube-system ``` ![](https://i.imgur.com/yPqOvOJ.png) ```bash= $ kubectl get pod ``` ![](https://i.imgur.com/6m6SLkD.png) ## Input Example 3 This example is to check whether pods follow QOS level as the scheduling order when pods have the same priority. There are two jobs, hi and hello. The QOS level of hi is **Guaranteed**. The QOS level of hellow is **Best Effort**. ```bash= $ kubectl apply -f {filename} ``` :::spoiler Hi.yaml ```yaml= apiVersion: batch/v1 kind: Job metadata: name: hi spec: parallelism: 10 completions: 10 template: metadata: labels: podGroup: "Hi" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: hi image: riyazhu/testprogram:cpu # Guaranteed resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k restartPolicy: Never ``` ::: :::spoiler Hello.yaml ```yaml= apiVersion: batch/v1 kind: Job metadata: name: hello spec: parallelism: 10 completions: 10 template: metadata: labels: podGroup: "Hello" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: hello image: riyazhu/testprogram:cpu # Best-Effort # resources: # requests: # cpu: 1101m # memory: 262144k # limits: # cpu: 1101m # memory: 262144k restartPolicy: Never ``` ::: ## Output Example 3 We can expect that the jobs of hi was done first. ![](https://i.imgur.com/qFjSrEI.png) ![](https://i.imgur.com/yq6mibh.png) ## Input Example 4 The example you can test the pods in same group have the same priority, but have different qos level. The qos level of black-pusheen: **BestEffort**. The qos level of gray-pusheen: **Burstable**. The qos level of white-pusheen: **Guaranteed**. We can expected that the white-pusheen has the highest scheduling prioity and the black-pushenn has the lowest scheduling prioity in this case. ```bash= $ kubectl apply -f {filename} ``` :::spoiler pusheen.yaml ```yaml= apiVersion: v1 kind: Pod metadata: name: black-pusheen01 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: black-pusheen01 image: riyazhu/testprogram:cpu # BestEffort --- apiVersion: v1 kind: Pod metadata: name: gray-pusheen01 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: gray-pusheen01 image: riyazhu/testprogram:cpu # Burstable resources: requests: cpu: 500m memory: 10000k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: white-pusheen01 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: white-pusheen01 image: riyazhu/testprogram:cpu # Guaranteed resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: black-pusheen02 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: black-pusheen02 image: riyazhu/testprogram:cpu # BestEffort --- apiVersion: v1 kind: Pod metadata: name: white-pusheen02 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: white-pusheen02 image: riyazhu/testprogram:cpu # Guaranteed resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: gray-pusheen02 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: gray-pusheen02 image: riyazhu/testprogram:cpu # Burstable resources: requests: cpu: 500m memory: 10000k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: black-pusheen03 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: black-pusheen03 image: riyazhu/testprogram:cpu # BestEffort --- apiVersion: v1 kind: Pod metadata: name: white-pusheen03 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: white-pusheen03 image: riyazhu/testprogram:cpu # Guaranteed resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: gray-pusheen03 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: gray-pusheen03 image: riyazhu/testprogram:cpu # Burstable resources: requests: cpu: 500m memory: 10000k limits: cpu: 1101m memory: 262144k --- --- apiVersion: v1 kind: Pod metadata: name: black-pusheen04 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: black-pusheen04 image: riyazhu/testprogram:cpu # BestEffort --- apiVersion: v1 kind: Pod metadata: name: white-pusheen04 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: white-pusheen04 image: riyazhu/testprogram:cpu # Guaranteed resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: gray-pusheen04 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: gray-pusheen04 image: riyazhu/testprogram:cpu # Burstable resources: requests: cpu: 500m memory: 10000k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: black-pusheen05 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: black-pusheen05 image: riyazhu/testprogram:cpu # BestEffort --- apiVersion: v1 kind: Pod metadata: name: white-pusheen05 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: white-pusheen05 image: riyazhu/testprogram:cpu # Guaranteed resources: requests: cpu: 1101m memory: 262144k limits: cpu: 1101m memory: 262144k --- apiVersion: v1 kind: Pod metadata: name: gray-pusheen05 labels: podGroup: "pusheen" groupPriority: "100" minAvailable: "2" spec: schedulerName: riya-scheduler containers: - name: gray-pusheen05 image: riyazhu/testprogram:cpu # Burstable resources: requests: cpu: 500m memory: 10000k limits: cpu: 1101m memory: 262144k ``` ::: ## Output Example 4 This is show you that the qos level about `Burstable` > `BestEffort` ![](https://i.imgur.com/fHKd6M7.png) ![](https://i.imgur.com/tvNFIkR.png) ## Note Because the queue if first in first out, so maybe the creation of pod is too fast to observe hardly. ## Reference + Schduling framework： https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md + k8s testing： https://www.reddit.com/r/kubernetes/comments/be0415/k3s_minikube_or_microk8s/ + kubernetes in docker(kind)： https://kind.sigs.k8s.io/docs/user/quick-start/ + Go language： https://golang.org/ + Dockerfile： https://docs.docker.com/engine/reference/builder/ + client-go https://github.com/kubernetes/client-go + k8s qos https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/ + sample-scheduler-framework code https://github.com/cnych/sample-scheduler-framework