Akri Bug Bash – February 2022

tags: Bug Bash

The goal of this bug bash is to discover any bugs before Akri creates a new release. This release contains the following major changes:

• Installs crictl directly in the Agent
• Enable applying multiple Configurations that use the same Discovery Handler
• Job Brokers (breaking)

Please leave a comment in the scenario outcomes section with the scenario's you tested and whether it was successful. If you find issues, please create an issue on Akri's GitHub and comment it in the discovered issues section.

As always, feel free to post any questions on Akri's Slack.

Background

Akri is an Open Source project that automates the discovery and usage of IoT devices around Kubernetes clusters on the Edge. Akri can automatically deploy user-provided workloads to the discovered devices. It handles device appearance and disappearance, allowing device deployments to expand and contract and enabling high resource utilization.

Setting Up an Environment

Akri is regularly tested on K3s, MicroK8s, and standard Kubernetes clusters versioned 1.16-1.21 (see previous release for list of exact versions tested) with Ubuntu 20.04 node. While we only test on these K8s distributions, feel free to try it out on the distribution and Linux OS of your choice.

• Skip set up and use the Kubernetes Playground
• Hyper-V Ubuntu 20.04 VM
• Set up Linux VM with cloud provider
• Try out Akri on a managed Kubernetes service
• etc

Scenarios

Choose any of the following scenarios (none are pre-requisite of the others). The first is the most structured scenario. Make sure to use the akri-dev chart (helm install akri akri-helm-charts/akri-dev) when installing Akri with Helm. If you have previously installed akri, be sure to run helm repo update.

Scenario A: Test crictl in the Agent (10 – 30 minutes)

Akri depends on crictl to track whether Pods deployed by the Akri controller are still running. This release adds new functionality such that crictl is pre-installed in the Agent container so that it does not need to be installed on each node. However, some user configuration is still needed, namely telling the agent where the container runtime socket lives so it can successfully execute crictl. This scenario should check that the embedded crictl is working as expected, given the user appropriately sets the runtime socket location as explained in the Akri configure crictl documentation. After setting up your cluster, set AKRI_HELM_CRICTL_CONFIGURATION env var as explained in the Akri configure crictl documentation. Install Akri with the debug echo discovery handler and configuration:

export AKRI_HELM_CRICTL_CONFIGURATION=<set based on distro>

helm repo add akri-helm-charts https://project-akri.github.io/akri/ 

helm install akri akri-helm-charts/akri-dev  \
    $AKRI_HELM_CRICTL_CONFIGURATION  \
    --set agent.allowDebugEcho=true  \
    --set debugEcho.discovery.enabled=true  \
    --set debugEcho.configuration.enabled=true  \
    --set debugEcho.configuration.brokerPod.image.repository=nginx  \
    --set debugEcho.configuration.brokerPod.image.tag=stable-alpine  \
    --set debugEcho.configuration.shared=false 

Assert that crictl calls are successful – be sure to wait 30 seconds after the Agent is running to make sure an initial
check has happened.

kubectl logs $(kubectl get pods | grep agent | awk '{ print $1 }' | head -1) | grep "get_node_slots - crictl called successfully"

If there is no output, check for a reported error in the logs and report it.

kubectl logs $(kubectl get pods | grep agent | awk '{ print $1 }' | head -1) | grep -i "get_node_slots - failed to call crictl"

Scenario B: Enable applying multiple Configurations that use the same Discovery Handler (10-30 minutes)

Multiple configurations should be able to be applied that use the same DiscoveryHandler in order to support deploying different types of brokers to devices and discovering different sets of devices. For example, you may want to deploy two Configurations that use the udev Discovery Handler as follows: one that discovers USB cameras and deploys a broker that grabs frames from the camera and another that discovers GPUs and deploys inferencing pods. This scenario will check that two DebugEcho configurations can be applied. To extend this scenario, see the Akri documentation on the debug echo testing Discovery Handler.

export AKRI_HELM_CRICTL_CONFIGURATION=<set based on distro>

#Get Akri's Chart and install Akri's Controller, Agent, and Debug Echo Discovery Handler 

helm install akri akri-helm-charts/akri-dev  $AKRI_HELM_CRICTL_CONFIGURATION  --set agent.allowDebugEcho=true  --set debugEcho.discovery.enabled=true 

# Assert that no Configurations have been applied nor debug echo Instances created 
kubectl get akric,akrii  


# Install a Configuration that discovers "fee" debug echo devices 
helm install fee-debug-echo-devices akri-helm-charts/akri-dev  \
    --set controller.enabled=false  \
    --set agent.enabled=false \
    --set rbac.enabled=false  \
    --set debugEcho.configuration.enabled=true  \
    --set debugEcho.configuration.name="fee-devices"  \
    --set debugEcho.configuration.discoveryDetails.descriptions[0]="fee-1"  \
    --set debugEcho.configuration.discoveryDetails.descriptions[1]="fee-2"  \
    --set debugEcho.configuration.shared=false 

# Assert that the fee Configuration exists and two fee debug echo Instances created 
watch kubectl get akric,akrii  

# Install a second Configuration that discovers "foo" debug echo devices 
helm install foo-debug-echo-devices akri-helm-charts/akri-dev  \
    --set controller.enabled=false  \
    --set agent.enabled=false  \
    --set rbac.enabled=false  \
    --set debugEcho.configuration.enabled=true  \
    --set debugEcho.configuration.name="foo-devices"  \
    --set debugEcho.configuration.discoveryDetails.descriptions[0]="foo-1"  \
    --set debugEcho.configuration.discoveryDetails.descriptions[1]="foo-2"  \
    --set debugEcho.configuration.shared=false 


# Assert that the foo Configuration ALSO exists and two foo debug echo Instances created 
watch kubectl get akric,akrii  

# Play around with deleting and re-add Configurations. IE: 

helm delete fee-debug-echo-devices 

watch kubectl get akric,akrii  

helm install fee-debug-echo-devices akri-helm-charts/akri-dev  \
    --set controller.enabled=false  \
    --set agent.enabled=false  \
    --set rbac.enabled=false  \
    --set debugEcho.configuration.enabled=true  \
    --set debugEcho.configuration.name="fee-devices"  \
    --set debugEcho.configuration.discoveryDetails.descriptions[0]="fee-1"  \
    --set debugEcho.configuration.discoveryDetails.descriptions[1]="fee-2"  \
    --set debugEcho.configuration.shared=false 

watch kubectl get akric,akrii  

Scenario C: Test Akri support for Job Brokers (20-45 minutes)

Now Akri has support for deploying Jobs to devices discovered by the Akri Agent. Previously, Akri only supported deploying Pods that were not intended to terminate (and would be restarted if they did). Adding Jobs will enable more scenarios with Akri such as device management scenarios. More background can be found in the Jobs proposal.

export AKRI_HELM_CRICTL_CONFIGURATION=<set based on distro>

# Install Akri with a debugEcho Configuration that will deploy a Job to each (foo0 and foo1) discovered device. The Job will echo "Hello World" and then sleep for 5 seconds before terminating. 

helm install akri akri-helm-charts/akri-dev  \
    $AKRI_HELM_CRICTL_CONFIGURATION \
    --set agent.allowDebugEcho=true \
    --set debugEcho.discovery.enabled=true  \
    --set debugEcho.configuration.brokerJob.image.repository=busybox  \
    --set debugEcho.configuration.brokerJob.command[0]="sh"  \
    --set debugEcho.configuration.brokerJob.command[1]="-c"  \
    --set debugEcho.configuration.brokerJob.command[2]="echo 'Hello World'"  \
    --set debugEcho.configuration.brokerJob.command[3]="sleep 5"  \
    --set debugEcho.configuration.enabled=true 

watch kubectl get akric,akrii,pod  



# Say you are feeling more exuberant and want the Job to echo "Hello Amazing World", you can change the brokerSpec. 

# Upgrade the installation to do so and watch Akri delete all the resources (instances, jobs) and recreate them, deploying the new job (Do kubectl logs on the Pods to verify). 



helm upgrade akri akri-helm-charts/akri-dev  \
    $AKRI_HELM_CRICTL_CONFIGURATION  \
    --set agent.allowDebugEcho=true  \
    --set debugEcho.discovery.enabled=true  \
    --set debugEcho.configuration.brokerJob.image.repository=busybox  \
    --set debugEcho.configuration.brokerJob.command[0]="sh"  \
    --set debugEcho.configuration.brokerJob.command[1]="-c"  \
    --set debugEcho.configuration.brokerJob.command[2]="echo 'Hello Amazing World'"  \
    --set debugEcho.configuration.brokerJob.command[3]="sleep 5"  \
    --set debugEcho.configuration.enabled=true 

watch kubectl get akric,akrii,pod  



# Test out some of the Job settings and modify the Job to run the Job twice per device (completions=2) in parallel 

# (parallelism=2). These Helm settings simply modify equivalently named parts of the Kubernetes JobSpec 

# (https://kubernetes.io/docs/concepts/workloads/controllers/job/) 

helm upgrade akri akri-helm-charts/akri-dev  \
    $AKRI_HELM_CRICTL_CONFIGURATION  \
    --set agent.allowDebugEcho=true  \
    --set debugEcho.discovery.enabled=true  \
    --set debugEcho.configuration.brokerJob.image.repository=busybox  \
    --set debugEcho.configuration.brokerJob.command[0]="sh"  \
    --set debugEcho.configuration.brokerJob.command[1]="-c"  \
    --set debugEcho.configuration.brokerJob.command[2]="echo 'Hello Amazing World'"  \
    --set debugEcho.configuration.brokerJob.command[3]="sleep 5"  \
    --set debugEcho.configuration.brokerJob.parallelism=2  \
    --set debugEcho.configuration.brokerJob.completions=2  \
    --set debugEcho.configuration.enabled=true 

watch kubectl get akric,akrii,pod  



# Continue to play around with setting `backoffLimit`, `parallelism`, and `completions` 

# (debugEcho.configuration.brokerJob.<backoffLimit|completions|parallelism>) confirming that the number and order of 

# brokers are produced as would be expected of a standard Kubernetes JobSpec 

# (https://kubernetes.io/docs/concepts/workloads/controllers/job/)

Discovered Issues or Enhancements

https://github.com/project-akri/akri/issues/447
https://github.com/project-akri/akri/issues/448
https://github.com/project-akri/akri/issues/450

Scenario Outcomes

Environment	Scenario	Success/Issue
KinD (0.11.1) with K8s v1.21.2	A, B, C	Success
MicroK8s on Windows K8s 1.18.20	A, B, C	Success
Kubernetes Playground (K8s version 1.18)	A, B, C	Success
K3s 1.22.6 Ubuntu 20.04 Hyper-V	A, B, C	Success
AKS 1.21.9	A	Success. Note, needed to configure crictl for containerd (export AKRI_HELM_CRICTL_CONFIGURATION="--set agent.host.containerRuntimeSocket=/run/containerd/containerd.sock)