# Deploying and testing the Descheduler in OpenShift 4.20
## Intro
Administrators who are managing virtual machines in OpenShift may want to configure the cluster to automatically rebalance the placement of VMs across cluster nodes.
The idea here is to enable a feature that users of VMware have become accustomed to: DRS, or Distributed Resource Scheduler's ability to reblance VMs across a VMware cluster.
The OpenShift scheduler makes placement decisions based on _requested_ resource usage, but does not automatically "reschedule" VMs if the node hosting the VMs becomes inhospitable due to a "noisy neighbor" workload which exceeds its requested resource usage and starts consuming more of the nodes compute resources.
The reason OpenShift/Kubernetes has traditionally not rebalanced workload comes down to one of the main differences between traditional, singleton workloads like VMs versus more modern, container workloads which often run as multiple "replicas" which are spread across the cluster. If one of the replicas of the container application on a particular node is struggling due to a noisy neighbor, the other replicas likely are not. If they are, the application could be set up to scale the number of replicas onto different nodes which may not be experiencing pressure.
Now that OpenShift has become a popular platform for running Virtual Machines, the Descheduler is offered as a way to provide this DRS-like functionality in Kubernetes.
## Install the Descheduler operator
Search for the Descheduler operator in the Ecosystem -> Software Catalog
## Configure your nodes for Load Aware Balancing
This rolls out a change to your nodes via a MachineConfig object.
This MachineConfig appends `psi=1` to the kernel command line parameters on master nodes to enable **Pressure Stall Information** tracking. This change will trigger a rolling reboot of the targeted nodes. This information is key to the Descheduler’s recent capabilities in Load Aware Balancing.
Choose this config if you are running the classic 3-node compact cluster in a test environment. It will target the "master" Machine Config Pool:
```yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-master-psi
labels:
machineconfiguration.openshift.io/role: master
spec:
kernelArguments:
- psi=1 #Pressure Stall Information
```
Wait for this to roll out to all 3 of your nodes.
Choose this config if you are running what we might call a more traditional cluster comprised of separate Worker and Control Plane nodes:
```yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-worker-psi
labels:
machineconfiguration.openshift.io/role: worker
spec:
kernelArguments:
- psi=1 #Pressure Stall Information
```
Wait for this to roll out to all your Worker nodes.
They will reboot and rejoin the cluster once they are done applying the change. Expect this step to take 15+ minutes depending on the size of the machine pool you are targeting and the age/size/personality of your hardware. You can watch its progress with:
```
# oc get mcp <mcp-name>
# Get the mcp you're interested in with
# oc get mcp
oc get master
# If you want to check other pools like 'worker'
# oc get mcp <poolname>
```
Example: for a 3-node cluster, look for UPDATING to be False (indicates it is complete), READYMACHINECOUNT to be 3 and UPDATEDMACHINECOUNT to be 3
You can see which node it is working on by looking for the one that is Scheduling Disabled and/or Not Ready.
```
oc get nodes
```
## Create an instance of the Descheduler
There are fields you can customize ([see the docs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html-single/virtualization/index#nodes-descheduler-profiles_virt-enabling-descheduler-evictions)) but the intention here is to get you a starting point for testing.
```yaml
---
apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
name: cluster
namespace: openshift-kube-descheduler-operator
spec:
managementState: Managed
deschedulingIntervalSeconds: 30 # Run every 30s - good for testing
# Probably need a more conservative value for prod.
profiles:
- KubeVirtRelieveAndMigrate # Use PSI metrics
mode: Automatic # vs Predictive (which is informative-only)
profileCustomizations:
devEnableSoftTainter: true
devDeviationThresholds: Low
devActualUtilizationProfile: PrometheusCPUCombined
```
## Set up conditions such that only one node runs a lot of VMs
If running a 3-node cluster, cordon 2 of your 3 nodes so that we can launch a storm of VMs on one node only, overloading it and creating conditions that are ripe for descheduling evictions.
(If running more than 3 nodes, cordon all but one of your worker nodes.)
```
oc get nodes
oc adm cordon <nodename>
# to Uncordon
# oc adm uncordon <nodename>
```
Check to see the new status of your nodes. You should see two of them be `Ready, SchedulingDisabled`. Here is a view of my 6-node cluster with two Workers cordoned:
## Launch a pool of VMs
Thanks to Mark DeNeve and his wonderful blog post "[OpenShift Virtualization and the Kubernetes Descheduler Revisited](https://xphyr.net/post/ocpv_descheduler_revisited/)" for the VirtualMachinePool manifest below. I modified it slightly to include the necessary descheduler annotation in a different location.
```yaml
apiVersion: pool.kubevirt.io/v1alpha1
kind: VirtualMachinePool
metadata:
name: vm-pool-fedora
spec:
replicas: 3
selector:
matchLabels:
kubevirt.io/vmpool: vm-pool-fedora
virtualMachineTemplate:
metadata:
creationTimestamp: null
labels:
kubevirt.io/vmpool: vm-pool-fedora
spec:
runStrategy: Always
template:
metadata:
creationTimestamp: null
labels:
kubevirt.io/vmpool: vm-pool-fedora
annotations:
descheduler.alpha.kubernetes.io/evict: "true"
spec:
domain:
cpu:
cores: 1
maxSockets: 4
model: host-model
sockets: 1
threads: 1
devices:
disks:
- disk:
bus: virtio
name: containerdisk
- disk:
bus: virtio
name: cloudinitdisk
memory:
guest: 2048Mi
terminationGracePeriodSeconds: 0
volumes:
- containerDisk:
image: quay.io/kubevirt/fedora-cloud-container-disk-demo
name: containerdisk
- cloudInitNoCloud:
userData: |
#cloud-config
user: fedora
password: fedora
chpasswd: { expire: False }
ssh_pwauth: True
disable_root: false
package_update: true
packages:
- stress-ng
runcmd:
- stress-ng --matrix 0
name: cloudinitdisk
```
Generally speaking, you opt-in the VMs you want to be relocatable via the Descheduler
This can be done in the UI or via YAML. Below is the key annotation which does this.
```yaml
kind: VirtualMachine
spec:
template:
metadata:
annotations:
descheduler.alpha.kubernetes.io/evict: "true"
```
## Create pressure by scaling up the number of VMs
The above will start with 3 but on your nodes that may not be enough to trigger pressure.
The magic number of VMs which will trigger the Descheduler to act depends on...
1) How much CPU your cluster nodes have
2) How much other workload is already running on them
3) Certain settings in your descheduler configuration
Perhaps start by scaling up to 25 and see what happens.
You can always scale down or up.
```
oc scale vmpool vm-pool-fedora --replicas 25
```
## Monitor what’s happening
I recommend having a few terminal windows or tmux panes, one for each of these ```watch``` commands.
```
watch oc get vmpool
```
For this one, pay attention to the NODENAME field. You want to see this field change for some of them once you uncordon a node.
```
watch oc get vmi
```
## Watch the descheduler logs
You’ll notice it ignores high priority, non-VM pods with “`Pod fails the following checks…pod has system critical priority…`”.
Also notice it is only evicting one at a time (one every 30-sec descheduler interval, as we configured above). That rate is configurable.
```
oc logs -n openshift-kube-descheduler-operator -l app=descheduler -f
I1212 22:30:36.344112 1 defaultevictor.go:268] "Pod fails the following checks" pod="openshift-cnv/hco-operator-7d65c9f6f9-gf6mg" checks="[pod has system critical priority, pod has higher priority than specified priority class threshold]"
I1212 22:30:36.344144 1 defaultevictor.go:268] "Pod fails the following checks" pod="openshift-workload-availability/self-node-remediation-controller-manager-c58d6cb8d-rtmnh" checks="[pod has system critical priority, pod has
higher priority than specified priority class threshold]"
I1212 22:30:36.344175 1 defaultevictor.go:268] "Pod fails the following checks" pod="openshift-ovn-kubernetes/ovnkube-node-nnsjn" checks="[pod has system critical priority, pod has higher priority than specified priority class threshold, pod is related to daemonset and descheduler is not configured with evictDaemonSetPods]"
I1212 22:30:36.344203 1 defaultevictor.go:268] "Pod fails the following checks" pod="openshift-workload-availability/self-node-remediation-ds-g4p9f" checks="[pod has system critical priority, pod has higher priority than specified priority class threshold, pod is related to daemonset and descheduler is not configured with evictDaemonSetPods]"
I1212 22:30:36.344229 1 nodeutilization.go:200] "Pods on node" node="ip-10-0-3-178.us-east-2.compute.internal" allPods=130 nonRemovablePods=46 removablePods=84
I1212 22:30:36.344253 1 nodeutilization.go:216] "Evicting pods based on priority, if they have same priority, they'll be evicted based on QoS tiers"
I1212 22:30:36.489479 1 evictions.go:601] "Eviction in background assumed" pod="vm-load-test/virt-launcher-vm-pool-fedora-33-rhn2n"
I1212 22:30:36.489514 1 nodeutilization.go:334] "Currently, only a single pod eviction is allowed"
I1212 22:30:36.489533 1 profile.go:376] "Total number of evictions/requests" extension point="Balance" evictedPods=0 evictionRequests=1
I1212 22:30:36.489550 1 descheduler.go:403] "Number of evictions/requests" totalEvicted=0 evictionRequests=1
```
Sign in with Wallet
Connect another wallet