---
# System prepended metadata

title: Assignment - Scheduling framework
tags: [Assignment, Scheduling framework]

---

# Assignment - Scheduling framework
###### tags: `Assignment`, `Scheduling framework`


## Requirement


**Version： Kubernete <font color = red>1.17+</font>**

如果當前版本不符合要求，建議[安裝Kind](https://hackmd.io/@lsalab-k8s-2020/S1Gflygfv)作為練習

**Language： go 1.14** 


## Problem Description

目前default的kube-scheduler只能做到FIFO(First-In-First-Out)的工作排程(scheduling)。在這次作業中，我們要實作一個自定義的[Scheduling framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/)，按以下規則排程工作(pod)：
1. 工作會依照使用者給定的label(podGroup)，分至不同的scheduling group
2. 每個scheduling group，依照使用者給定的label(groupPriority)，擁有不同的排程優先權(groupPriority值越高的優先順序越高)。
3. 同一個group中的工作，依照使用者給定的[QoS level](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)，決定排程的先後順序(`Guaranteed`>`Burstable`>`BestEffort`)。
4. 每個group只有當確認該Group的Pod數量>=最小副本數(miniAvailable)時，才會開始進行該group的工作排程動作。


Yaml 將會提供下列的Label，可透過Label來作為判斷的依據。(請注意在測試時，我們會假設擁有相同podGroup設定值的pods，他們的miniAvailable及groupPriority設定值是相同的。)
+ `podGroup`: 相同名稱視為同個Group
+ `minAvailable`: 該Group(服務、應用...)所需要的最小副本數
+ `groupPriority`： 該Group的Pod優先權

   
## Hint

Implement plugin：

+ **QueueSort**
    +  Compare the priority of pods first.
    +  If the priority of pods are equal, then comapre the QOS
+ **PreFilter**
    + If the number of pods in same group < minAvailable,       then the pod will reject and retry later.

**Writing logs** is a good habit.

You can download the file , and make appropriate **changes**：

The files are about...
+ Test
+ Build
```bash=
$ wget https://lsalab.cs.nthu.edu.tw/~riya/assignment/schedulingFramework.tar.gz
$ tar zxvf schedulingFramework.tar.gz
```
`



## Check

```bash=
$ kubectl get pod -n kube-system
```

![](https://i.imgur.com/2SdDCGG.png)

---

## Input Example 1


There is a job, `nginx` and its group is also called the same name.
According the file of yaml, we can see there is a constraint that this service must has 3 pods, 
then the pods will start to be scheduled to the nodes. 

Actually, the operator made a mistake, he accidentally typed the wrong number.

Thus, we can see ...

The number of Pods in same group < minAvailable, so the pod status is `Pending`.  

When we check the log, we can find the situation that the number of pods in same group is less than minAvailable.

```bash=
$ kubectl apply -f {filename}
```


:::spoiler
```yaml=
apiVersion: batch/v1
kind: Job
metadata:
  name: nginx
spec:
  parallelism: 2
  completions: 2
  template:
    metadata:
      labels:
        app: nginx
        podGroup: nginx
        minAvailable: "3"
    spec:
      schedulerName: riya-scheduler
      containers:
        - name: nginx
          image:  nginx
          resources:
            requests:
              cpu: 3000m
              memory: 500Mi
            limits:
              cpu: 3000m
              memory: 500Mi
      restartPolicy: Never
```
:::



## Output Example 1

```bash=
$ kubectl get pod
```

![](https://i.imgur.com/AuqzFXZ.png)

```bash=
$ kubectl logs {your scheduler pod name} -n kube-system
```
![](https://i.imgur.com/rq36Gje.png)



## Input Example 2

There are three jobs, <font color = red>red</font>, <font color = green>green</font> and <font color = #FFDC35>yellow</font> and the qos level is the same.
Each job is a group consisting of 10 pods.
Each job must have at least two pods to start scheduling.
The order of scheduling pods is sorted by priority, and the priority is Red>Yellow>Green(1000>500>100).




```bash=
$ kubectl apply -f {filename}
```

:::spoiler Green.yaml
```yaml=
apiVersion: batch/v1
kind: Job
metadata:
  name: green
spec:
  parallelism: 10
  completions: 10
  template:
    metadata:
      labels:
        podGroup: "Green"
        groupPriority: "100"
        minAvailable: "2"
    spec:
      schedulerName: riya-scheduler
      containers:
        - name: green
          image:  riyazhu/testprogram:cpu
          resources:
            requests:
              cpu: 1101m
              memory: 262144k
            limits:
              cpu: 1101m
              memory: 262144k
      restartPolicy: Never
```
:::

:::spoiler Yellow.yaml
```yaml=
apiVersion: batch/v1
kind: Job
metadata:
  name: yellow
spec:
  parallelism: 10
  completions: 10
  template:
    metadata:
      labels:
        podGroup: "Yellow"
        groupPriority: "500"
        minAvailable: "2"
    spec:
      schedulerName: riya-scheduler
      containers:
        - name: yellow
          image:  riyazhu/testprogram:cpu
          resources:
            requests:
              cpu: 1101m
              memory: 262144k
            limits:
              cpu: 1101m
              memory: 262144k
      restartPolicy: Never
```
:::

:::spoiler Red.yaml
```yaml=
apiVersion: batch/v1
kind: Job
metadata:
  name: red
spec:
  parallelism: 10
  completions: 10
  template:
    metadata:
      labels:
        podGroup: "Red"
        groupPriority: "1000"
        minAvailable: "2"
    spec:
      schedulerName: riya-scheduler
      containers:
        - name: red
          image:  riyazhu/testprogram:cpu
          resources:
            requests:
              cpu: 1101m
              memory: 262144k
            limits:
              cpu: 1101m
              memory: 262144k
      restartPolicy: Never
```
:::
## Output Example 2

```bash=
$ kubectl logs {your scheduler pod name} -n kube-system
```
![](https://i.imgur.com/yPqOvOJ.png)


```bash=
$ kubectl get pod
```
![](https://i.imgur.com/6m6SLkD.png)



## Input Example 3

This example is to check whether pods follow QOS level as the scheduling order when pods have the same priority.

There are two jobs, hi and hello.

The QOS level of hi is **Guaranteed**.
The QOS level of hellow is **Best Effort**.

```bash=
$ kubectl apply -f {filename}
```

:::spoiler Hi.yaml
```yaml=
apiVersion: batch/v1
kind: Job
metadata:
  name: hi
spec:
  parallelism: 10
  completions: 10
  template:
    metadata:
      labels:
        podGroup: "Hi"
        groupPriority: "100"
        minAvailable: "2"
    spec:
      schedulerName: riya-scheduler
      containers:
        - name: hi
          image:  riyazhu/testprogram:cpu
          # Guaranteed
          resources:
            requests:
              cpu: 1101m
              memory: 262144k
            limits:
              cpu: 1101m
              memory: 262144k
      restartPolicy: Never
```
:::

:::spoiler Hello.yaml
```yaml=
apiVersion: batch/v1
kind: Job
metadata:
  name: hello
spec:
  parallelism: 10
  completions: 10
  template:
    metadata:
      labels:
        podGroup: "Hello"
        groupPriority: "100"
        minAvailable: "2"
    spec:
      schedulerName: riya-scheduler
      containers:
        - name: hello
          image:  riyazhu/testprogram:cpu
          # Best-Effort
          # resources:
          #   requests:
          #     cpu: 1101m
          #     memory: 262144k
          #   limits:
          #     cpu: 1101m
          #     memory: 262144k
      restartPolicy: Never
```
:::
## Output Example 3

We can expect that the jobs of hi was done first.

![](https://i.imgur.com/qFjSrEI.png)

![](https://i.imgur.com/yq6mibh.png)

## Input Example 4

The example you can test the pods in same group have the same priority, but have different qos level.

The qos level of black-pusheen: **BestEffort**.
The qos level of gray-pusheen: **Burstable**.
The qos level of white-pusheen: **Guaranteed**.

We can expected that the white-pusheen has the highest scheduling prioity and the black-pushenn has the lowest scheduling prioity in this case.

```bash=
$ kubectl apply -f {filename}
```

:::spoiler pusheen.yaml
```yaml=
apiVersion: v1
kind: Pod
metadata:
  name: black-pusheen01
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: black-pusheen01
    image: riyazhu/testprogram:cpu
    # BestEffort
---
apiVersion: v1
kind: Pod
metadata:
  name: gray-pusheen01
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: gray-pusheen01
    image: riyazhu/testprogram:cpu
    # Burstable
    resources:
      requests:
        cpu: 500m
        memory: 10000k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: white-pusheen01
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: white-pusheen01
    image: riyazhu/testprogram:cpu
    # Guaranteed
    resources:
      requests:
        cpu: 1101m
        memory: 262144k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: black-pusheen02
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: black-pusheen02
    image: riyazhu/testprogram:cpu
    # BestEffort
---
apiVersion: v1
kind: Pod
metadata:
  name: white-pusheen02
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: white-pusheen02
    image: riyazhu/testprogram:cpu
    # Guaranteed
    resources:
      requests:
        cpu: 1101m
        memory: 262144k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: gray-pusheen02
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: gray-pusheen02
    image: riyazhu/testprogram:cpu
    # Burstable
    resources:
      requests:
        cpu: 500m
        memory: 10000k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: black-pusheen03
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: black-pusheen03
    image: riyazhu/testprogram:cpu
    # BestEffort
---
apiVersion: v1
kind: Pod
metadata:
  name: white-pusheen03
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: white-pusheen03
    image: riyazhu/testprogram:cpu
    # Guaranteed
    resources:
      requests:
        cpu: 1101m
        memory: 262144k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: gray-pusheen03
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: gray-pusheen03
    image: riyazhu/testprogram:cpu
    # Burstable
    resources:
      requests:
        cpu: 500m
        memory: 10000k
      limits:
        cpu: 1101m
        memory: 262144k
---
---
apiVersion: v1
kind: Pod
metadata:
  name: black-pusheen04
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: black-pusheen04
    image: riyazhu/testprogram:cpu
    # BestEffort
---
apiVersion: v1
kind: Pod
metadata:
  name: white-pusheen04
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: white-pusheen04
    image: riyazhu/testprogram:cpu
    # Guaranteed
    resources:
      requests:
        cpu: 1101m
        memory: 262144k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: gray-pusheen04
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: gray-pusheen04
    image: riyazhu/testprogram:cpu
    # Burstable
    resources:
      requests:
        cpu: 500m
        memory: 10000k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: black-pusheen05
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: black-pusheen05
    image: riyazhu/testprogram:cpu
    # BestEffort
---
apiVersion: v1
kind: Pod
metadata:
  name: white-pusheen05
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: white-pusheen05
    image: riyazhu/testprogram:cpu
    # Guaranteed
    resources:
      requests:
        cpu: 1101m
        memory: 262144k
      limits:
        cpu: 1101m
        memory: 262144k
---
apiVersion: v1
kind: Pod
metadata:
  name: gray-pusheen05
  labels:
    podGroup: "pusheen"
    groupPriority: "100"
    minAvailable: "2"
spec:
  schedulerName: riya-scheduler
  containers:
  - name: gray-pusheen05
    image: riyazhu/testprogram:cpu
    # Burstable
    resources:
      requests:
        cpu: 500m
        memory: 10000k
      limits:
        cpu: 1101m
        memory: 262144k
```
:::
## Output Example 4

This is show you that the qos level about `Burstable` > `BestEffort`
![](https://i.imgur.com/fHKd6M7.png)

![](https://i.imgur.com/tvNFIkR.png)

## Note

Because the queue if first in first out,
so maybe the creation of pod is too fast to observe hardly.

## Reference
+ Schduling framework：
https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md
+ k8s testing：   https://www.reddit.com/r/kubernetes/comments/be0415/k3s_minikube_or_microk8s/
+ kubernetes in docker(kind)：
  https://kind.sigs.k8s.io/docs/user/quick-start/
+ Go language：
  https://golang.org/
+ Dockerfile：
  https://docs.docker.com/engine/reference/builder/
+ client-go
  https://github.com/kubernetes/client-go
+ k8s qos
  https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/
+ sample-scheduler-framework code
  https://github.com/cnych/sample-scheduler-framework

