# Conformance Tests using Sonbuoy
[Sonobuoy(so·no·buoy | \ ˈsä-nə-ˌbȯi , -ˌbü-ē \)](https://www.merriam-webster.com/dictionary/sonobuoy) is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of plugins (including Kubernetes conformance tests) in an accessible and non-destructive manner. It is a customizable, extendable, and cluster-agnostic way to generate clear, informative reports about your cluster.
Its selective data dumps of Kubernetes resource objects and cluster nodes allow for the following use cases:
- Integrated end-to-end (e2e) conformance-testing
- Workload debugging
- Custom data collection via extensible plugins.
To check confromance results on cluster created with `airshipctl`, go to section [Results](#Sonobuoy-Conformance-Results-with-airshipctl)
## Usage
This document is to demonstrate usage of Sonobuoy to run the following conformance tests, on a Kubernetes cluster:
- default conformance tests (uses plugins e2e, systemd-logs)
- cis conformance tests (uses kubebench)
For demo, kind will be used. Scripts used can be found in [WIP PS](https://review.opendev.org/#/c/756538/)
By default, sonobuoy runs plugins e2e and systemd-logs for conformance tests.
In addition, one can also run custom plugins such as `CIS Benchmarks` which utilizes the kube-bench implementation of the CIS security benchmarks. It is technically two plugins; one to run the checks on the master nodes and another to run the checks on the worker nodes.
The document also contains information on running conformance tests on target cluster created with airshipctl using capd provider(docker)
To check conformance results on cluster created with `airshipctl`, go to section [Results](#Sonobuoy-Conformance-Results-with-airshipctl)
[](https://asciinema.org/a/F3qBSflKSV0lUUi1FZdus2peX)
```
❯ sudo swapoff -a
```
```
❯ cat kind.yaml
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
- role: control-plane
- role: worker
```
```
❯ kind delete cluster --name capi-docker
Deleting cluster "capi-docker" ...
```
```
❯ kind create cluster --name capi-docker --config kind.yaml --wait 120s
Creating cluster "capi-docker" ...
✓ Ensuring node image (kindest/node:v1.18.2) 🖼
✓ Preparing nodes 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
✓ Waiting ≤ 2m0s for control-plane = Ready ⏳
• Ready after 25s 💚
Set kubectl context to "kind-capi-docker"
You can now use your cluster with:
kubectl cluster-info --context kind-capi-docker
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
```
```
❯ kubectl wait --for=condition=Ready nodes --all --timeout 50s
node/capi-docker-control-plane condition met
node/capi-docker-worker condition met
```
```
❯ kubectl get nodes
NAME STATUS ROLES AGE VERSION
capi-docker-control-plane Ready master 83s v1.18.2
capi-docker-worker Ready <none> 47s v1.18.2
```
```
❯ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-5b8t4 1/1 Running 0 74s
kube-system coredns-66bff467f8-6dlrz 1/1 Running 0 75s
kube-system etcd-capi-docker-control-plane 1/1 Running 0 82s
kube-system kindnet-4c6zk 1/1 Running 0 74s
kube-system kindnet-tjjq7 1/1 Running 1 57s
kube-system kube-apiserver-capi-docker-control-plane 1/1 Running 0 82s
kube-system kube-controller-manager-capi-docker-control-plane 1/1 Running 0 82s
kube-system kube-proxy-5gt9m 1/1 Running 0 57s
kube-system kube-proxy-pw7m5 1/1 Running 0 74s
kube-system kube-scheduler-capi-docker-control-plane 1/1 Running 0 82s
local-path-storage local-path-provisioner-bd4bb6b75-wcqjd 1/1 Running 0 75s
```
```
❯ KUBE_CONFIG=~/.kube/config ./tools/deployment/sonobuoy/01-install_sonobuoy.sh
+ : 0.18.2
+ : /home/rishabh/.kube/config
+ URL=https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.18.2/sonobuoy_0.18.2_linux_amd64.tar.gz
+ rm -rf /tmp/sonobuoy
+ mkdir /tmp/sonobuoy
+ sudo -E curl -sSLo /tmp/sonobuoy/sonobuoy_0.18.2_linux_amd64.tar.gz https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.18.2/sonobuoy_0.18.2_linux_amd64.tar.gz
+ tar xvf /tmp/sonobuoy/sonobuoy_0.18.2_linux_amd64.tar.gz -C /tmp/sonobuoy/
LICENSE
sonobuoy
+ sudo mv /tmp/sonobuoy/sonobuoy /usr/bin/sonobuoy
+ sudo chmod +x /usr/bin/sonobuoy
+ echo /home/rishabh/.kube/config
/home/rishabh/.kube/config
+ sonobuoy version --kubeconfig /home/rishabh/.kube/config
Sonobuoy Version: v0.18.2
MinimumKubeVersion: 1.16.0
MaximumKubeVersion: 1.18.99
GitSHA: f13b4cc6bbb9ed38a4bf593fd822886494cb6a92
API Version: v1.18.2
```
```
❯ KUBE_CONFIG=~/.kube/config ./tools/deployment/sonobuoy/02-run_default.sh
+ : /home/rishabh/.kube/config
+ : quick
+ : v1.18.2
+ : 300
+ rm -rf /tmp/sonobuoy_snapshots/e2e
+ mkdir -p /tmp/sonobuoy_snapshots/e2e
+ cd /tmp/sonobuoy_snapshots/e2e
+ sonobuoy delete --wait
INFO[0000] already deleted kind=namespace namespace=sonobuoy
INFO[0000] deleted kind=clusterrolebindings
INFO[0000] deleted kind=clusterroles
+ sonobuoy run --plugin e2e --plugin systemd-logs -m quick --kube-conformance-image gcr.io/google-containers/conformance:v1.18.2 --kubeconfig /home/rishabh/.kube/config --wait --timeout 300 --log_dir /tmp/sonobuoy_snapshots/e2e
INFO[0000] created object name=sonobuoy namespace= resource=namespaces
INFO[0000] created object name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] created object name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] created object name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] created object name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] created object name=sonobuoy-master namespace=sonobuoy resource=services
+ kubectl get all -n sonobuoy --kubeconfig /home/rishabh/.kube/config
NAME READY STATUS RESTARTS AGE
pod/sonobuoy 1/1 Running 0 2m
pod/sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-k7pwx 2/2 Terminating 0 105s
pod/sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-xntcz 2/2 Terminating 0 105s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/sonobuoy-master ClusterIP 10.101.224.88 <none> 8080/TCP 2m
+ sonobuoy status --kubeconfig /home/rishabh/.kube/config
PLUGIN STATUS RESULT COUNT
e2e complete passed 1
systemd-logs complete passed 2
Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
+ sonobuoy logs
namespace="sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-k7pwx" container="systemd-logs"
Sleeping for 1h to avoid daemonset restart
namespace="sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-k7pwx" container="sonobuoy-worker"
time="2020-10-09T04:40:36Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.1.5]:8080/api/v1/progress/by-node/capi-docker-control-plane/systemd-logs"
time="2020-10-09T04:40:36Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-10-09T04:40:37Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
time="2020-10-09T04:40:37Z" level=info msg="Results transmitted to aggregator. Sleeping for 3600 seconds"
time="2020-10-09T04:40:55Z" level=info msg="received a signal. Waiting then sending the real shutdown signal." signal=terminated
namespace="sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-xntcz" container="systemd-logs"
Sleeping for 1h to avoid daemonset restart
namespace="sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-xntcz" container="sonobuoy-worker"
time="2020-10-09T04:40:37Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-10-09T04:40:37Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.1.5]:8080/api/v1/progress/by-node/capi-docker-worker/systemd-logs"
time="2020-10-09T04:40:38Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
time="2020-10-09T04:40:38Z" level=info msg="Results transmitted to aggregator. Sleeping for 3600 seconds"
time="2020-10-09T04:40:55Z" level=info msg="received a signal. Waiting then sending the real shutdown signal." signal=terminated
namespace="sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-k7pwx" container="sonobuoy-worker"
time="2020-10-09T04:40:36Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.1.5]:8080/api/v1/progress/by-node/capi-docker-control-plane/systemd-logs"
time="2020-10-09T04:40:36Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-10-09T04:40:37Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
time="2020-10-09T04:40:37Z" level=info msg="Results transmitted to aggregator. Sleeping for 3600 seconds"
time="2020-10-09T04:40:55Z" level=info msg="received a signal. Waiting then sending the real shutdown signal." signal=terminated
namespace="sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-16892447dafb4f56-k7pwx" container="systemd-logs"
Sleeping for 1h to avoid daemonset restart
++ sonobuoy retrieve --kubeconfig /home/rishabh/.kube/config
+ results=202010090439_sonobuoy_6f0dead8-468b-4799-be0e-63ee3cca5d89.tar.gz
+ echo 'Results: 202010090439_sonobuoy_6f0dead8-468b-4799-be0e-63ee3cca5d89.tar.gz'
Results: 202010090439_sonobuoy_6f0dead8-468b-4799-be0e-63ee3cca5d89.tar.gz
+ sonobuoy results 202010090439_sonobuoy_6f0dead8-468b-4799-be0e-63ee3cca5d89.tar.gz
Plugin: e2e
Status: passed
Total: 4992
Passed: 1
Failed: 0
Skipped: 4991
Plugin: systemd-logs
Status: passed
Total: 2
Passed: 2
Failed: 0
Skipped: 0
+ ls -ltr /tmp/sonobuoy_snapshots/e2e
total 404
-rw-rw-r-- 1 rishabh rishabh 412570 Oct 8 21:41 202010090439_sonobuoy_6f0dead8-468b-4799-be0e-63ee3cca5d89.tar.gz
```
```
❯ KUBE_CONFIG=~/.kube/config ./tools/deployment/sonobuoy/03-kubebench.sh
+ : /home/rishabh/.kube/config
+ : https://raw.githubusercontent.com/vmware-tanzu/sonobuoy-plugins/master/cis-benchmarks/kube-bench-master-plugin.yaml
+ : https://raw.githubusercontent.com/vmware-tanzu/sonobuoy-plugins/master/cis-benchmarks/kube-bench-plugin.yaml
+ : 1.18
+ : quick
+ : 300
+ rm -rf /tmp/sonobuoy_snapshots/kubebench
+ mkdir /tmp/sonobuoy_snapshots/kubebench
+ cd /tmp/sonobuoy_snapshots/kubebench
+ sonobuoy delete --wait
INFO[0000] deleted kind=namespace namespace=sonobuoy
INFO[0000] deleted kind=clusterrolebindings
INFO[0000] deleted kind=clusterroles
+ sonobuoy run -m quick --kubeconfig /home/rishabh/.kube/config --plugin https://raw.githubusercontent.com/vmware-tanzu/sonobuoy-plugins/master/cis-benchmarks/kube-bench-master-plugin.yaml --plugin https://raw.githubusercontent.com/vmware-tanzu/sonobuoy-plugins/master/cis-benchmarks/kube-bench-plugin.yaml --plugin-env kube-bench-master.KUBERNETES_VERSION=1.18 --plugin-env kube-bench-master.KUBERNETES_VERSION=1.18 --wait --timeout 300 --log_dir /tmp/sonobuoy_snapshots/kubebench
INFO[0000] created object name=sonobuoy namespace= resource=namespaces
INFO[0000] created object name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] created object name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] created object name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] created object name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] created object name=sonobuoy-master namespace=sonobuoy resource=services
+ kubectl get all -n sonobuoy --kubeconfig /home/rishabh/.kube/config
NAME READY STATUS RESTARTS AGE
pod/sonobuoy 1/1 Running 0 20s
pod/sonobuoy-kube-bench-master-daemon-set-ee18fe65b9484c3b-85qnm 2/2 Terminating 0 18s
pod/sonobuoy-kube-bench-node-daemon-set-489f90a585f645d0-54wf2 2/2 Terminating 0 18s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/sonobuoy-master ClusterIP 10.97.30.103 <none> 8080/TCP 20s
+ sonobuoy status --kubeconfig /home/rishabh/.kube/config
PLUGIN STATUS RESULT COUNT
kube-bench-master complete failed 1
kube-bench-node complete failed 1
Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
+ sonobuoy logs
namespace="sonobuoy" pod="sonobuoy-kube-bench-master-daemon-set-ee18fe65b9484c3b-85qnm" container="plugin"
tar: removing leading '/' from member names
Sleeping for 1h to avoid daemonset restart
namespace="sonobuoy" pod="sonobuoy-kube-bench-master-daemon-set-ee18fe65b9484c3b-85qnm" container="sonobuoy-worker"
time="2020-10-09T04:42:01Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-10-09T04:42:01Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.1.8]:8080/api/v1/progress/by-node/capi-docker-control-plane/kube-bench-master"
time="2020-10-09T04:42:02Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/results.tar.gz
time="2020-10-09T04:42:02Z" level=info msg="Results transmitted to aggregator. Sleeping for 3600 seconds"
time="2020-10-09T04:42:12Z" level=info msg="received a signal. Waiting then sending the real shutdown signal." signal=terminated
namespace="sonobuoy" pod="sonobuoy-kube-bench-master-daemon-set-ee18fe65b9484c3b-85qnm" container="plugin"
tar: removing leading '/' from member names
Sleeping for 1h to avoid daemonset restart
namespace="sonobuoy" pod="sonobuoy-kube-bench-node-daemon-set-489f90a585f645d0-54wf2" container="plugin"
tar: removing leading '/' from member names
Sleeping for 1h to avoid daemonset restart
namespace="sonobuoy" pod="sonobuoy-kube-bench-node-daemon-set-489f90a585f645d0-54wf2" container="sonobuoy-worker"
time="2020-10-09T04:42:07Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-10-09T04:42:07Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.1.8]:8080/api/v1/progress/by-node/capi-docker-worker/kube-bench-node"
time="2020-10-09T04:42:08Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/results.tar.gz
time="2020-10-09T04:42:08Z" level=info msg="Results transmitted to aggregator. Sleeping for 3600 seconds"
time="2020-10-09T04:42:12Z" level=info msg="received a signal. Waiting then sending the real shutdown signal." signal=terminated
namespace="sonobuoy" pod="sonobuoy-kube-bench-master-daemon-set-ee18fe65b9484c3b-85qnm" container="sonobuoy-worker"
time="2020-10-09T04:42:01Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-10-09T04:42:01Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.1.8]:8080/api/v1/progress/by-node/capi-docker-control-plane/kube-bench-master"
time="2020-10-09T04:42:02Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/results.tar.gz
time="2020-10-09T04:42:02Z" level=info msg="Results transmitted to aggregator. Sleeping for 3600 seconds"
time="2020-10-09T04:42:12Z" level=info msg="received a signal. Waiting then sending the real shutdown signal." signal=terminated
++ sonobuoy retrieve --kubeconfig /home/rishabh/.kube/config
+ results=202010090441_sonobuoy_7da9bc70-f77a-4998-9ffd-631147d1a0e2.tar.gz
+ echo 'Results: 202010090441_sonobuoy_7da9bc70-f77a-4998-9ffd-631147d1a0e2.tar.gz'
Results: 202010090441_sonobuoy_7da9bc70-f77a-4998-9ffd-631147d1a0e2.tar.gz
+ sonobuoy results 202010090441_sonobuoy_7da9bc70-f77a-4998-9ffd-631147d1a0e2.tar.gz
Plugin: kube-bench-master
Status: failed
Total: 65
Passed: 41
Failed: 13
Skipped: 11
Failed tests:
1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)
1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)
1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Scored)
1.2.21 Ensure that the --profiling argument is set to false (Scored)
1.2.22 Ensure that the --audit-log-path argument is set (Scored)
1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Scored)
1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Scored)
1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Scored)
1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Scored)
1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Scored)
1.3.2 Ensure that the --profiling argument is set to false (Scored)
1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
1.4.1 Ensure that the --profiling argument is set to false (Scored)
Plugin: kube-bench-node
Status: failed
Total: 23
Passed: 14
Failed: 6
Skipped: 3
Failed tests:
4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
4.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
+ ls -ltr /tmp/sonobuoy_snapshots/kubebench
total 76
-rw-rw-r-- 1 rishabh rishabh 76268 Oct 8 21:42 202010090441_sonobuoy_7da9bc70-f77a-4998-9ffd-631147d1a0e2.tar.gz
```
## Conformance results with airshipctl
Currently, we have created an experimental pipeline on zuul, which consists of jobs
- airship-airshipctl-docker-kubebench-conformance
- airship-airshipctl-docker-cncf-conformance
that run conformance tests using sonobuoy on target cluster created by airshipctl using docker provider (CAPD)
The pipeline can be triggered by the comment "check experimental" on the ps [WIP PS](https://review.opendev.org/#/c/756538/)
To trigger the pipeline, visit [WIP PS](https://review.opendev.org/#/c/756538/) and comment "check experimental"
For checking results, go to https://zuul.opendev.org/t/openstack/status, and look for change "756538", and check for jobs
running under the section experimental pipeline.
The results will also be posted on the patchset when the jobs finish execution.
- CNCF Conformance Tests are passing on cluster created with `airshipctl` using capd
- Kubebench Tests (12 Tests Failed)
### airship-airshipctl-docker-cncf-conformance
```
Plugin: systemd-logs
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0
Plugin: e2e
Status: passed
Total: 4992
Passed: 277
Failed: 0
Skipped: 4715
```
### Kubebench Failiures
Failed tests:
4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
4.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
4.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
#### airship-airshipctl-docker-kubebench-conformance
```
Plugin: kube-bench-master
Status: failed
Total: 65
Passed: 41
Failed: 13
Skipped: 11
Failed tests:
1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)
1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)
1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Scored)
1.2.21 Ensure that the --profiling argument is set to false (Scored)
1.2.22 Ensure that the --audit-log-path argument is set (Scored)
1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Scored)
1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Scored)
1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Scored)
1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Scored)
1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Scored)
1.3.2 Ensure that the --profiling argument is set to false (Scored)
1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
1.4.1 Ensure that the --profiling argument is set to false (Scored)
Plugin: kube-bench-node
Status: failed
Total: 46
Passed: 28
Failed: 12
Skipped: 6
Failed tests:
4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
4.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)
4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)
4.2.4 Ensure that the --read-only-port argument is set to 0 (Scored)
4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored)
4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)
4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)
```
### Other Limitations
1. tenant max-job-timeout
10800 (3 hrs)
2. e2e conformance tests require atleast 3 nodes
tests are skipped or failed on anything less than 3 nodes
https://github.com/kubernetes/kubernetes/issues/69601
4. https://github.com/kubernetes/kubernetes/issues/92601
sonobuoy delete --wait doesnot delete resources properly, when used in cncf mode
## CNCF Conformance Process
[CNCF Conformance](https://www.cncf.io/certification/software-conformance/)
[Instructions on how to submit conformance results](https://github.com/cncf/k8s-conformance/blob/master/instructions.md)
## WIP
1. What action can further be taken with the reports and sanpshots generated
2. Alignment with CNCF conformance process to create PR, Submit Results
## Reference
### CIS and Kubernetes Version Mapping
kube-bench supports the tests for Kubernetes as defined in the [CIS Kubernetes Benchmarks](https://www.cisecurity.org/benchmark/kubernetes/).
The latest CIS version supported by kube-bench is cis-1.6.
| CIS Kubernetes Benchmark | kube-bench config | Kubernetes versions |
|---|---|---|
| [1.5.1](https://workbench.cisecurity.org/benchmarks/4892) | cis-1.5 | 1.15- |
| [1.6.0](https://workbench.cisecurity.org/benchmarks/4834) | cis-1.6 | 1.16- |
### Resources
| Topic | Reference |
|---|---|
| Sonobuoy Home Page | https://sonobuoy.io/ |
| Sonobuoy Getting Started | https://github.com/vmware-tanzu/sonobuoy#getting-started |
| Plugins | https://sonobuoy.io/plugins/ |
| E2E Testing Plugin| https://sonobuoy.io/docs/master/e2eplugin/ |
| Systemd-Logs Plugin | https://github.com/vmware-tanzu/sonobuoy-plugins/tree/master/systemd-logs |
| CIS Benchmarks Plugin | https://github.com/vmware-tanzu/sonobuoy-plugins/tree/master/cis-benchmarks |
| K8s Conformance Tests Repo | https://github.com/kubernetes/kubernetes/tree/master/cluster/images/conformance |
| CNCF Conformance Program | https://www.cncf.io/certification/software-conformance/ |
| High Level Overview | https://tanzu.vmware.com/content/blog/certifying-kubernetes-with-sonobuoy |
<style>.markdown-body { max-width: 1250px; }</style>
Sign in with Wallet
Connect another wallet