# :sparkles: Debugging kpng e2e failures :sparkles: ## Clone repos :cyclone: ``` $ git clone git@github.com:kubernetes-sigs/kpng.git $ git clone git@github.com:kubernetes/kubernetes.git ``` ## Create a kpng test cluster :desktop_computer: ``` $ cd kpng // -d flag will simply create a cluster but **not** run any tests // -b is the backend of your choice // -i is the ip family of your choice $ bash hack/test_e2e.sh -d -b iptables -i ipv4 // verify if cluster & pods are created and running $ kind get clusters kpng-e2e-ipv4-iptables $ k get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-78fcd69978-pw75r 1/1 Running 0 171m kube-system coredns-78fcd69978-q7vph 1/1 Running 0 171m kube-system etcd-kpng-e2e-ipv4-iptables-control-plane 1/1 Running 0 171m kube-system kindnet-9cl5s 1/1 Running 0 171m kube-system kindnet-fmsps 1/1 Running 0 171m kube-system kindnet-nkx8r 1/1 Running 0 171m kube-system kpng-6xx2n 2/2 Running 0 171m kube-system kpng-78gdc 2/2 Running 0 171m kube-system kpng-thrpl 2/2 Running 0 171m kube-system kube-apiserver-kpng-e2e-ipv4-iptables-control-plane 1/1 Running 0 171m kube-system kube-controller-manager-kpng-e2e-ipv4-iptables-control-plane 1/1 Running 0 171m kube-system kube-scheduler-kpng-e2e-ipv4-iptables-control-plane 1/1 Running 0 171m local-path-storage local-path-provisioner-85494db59d-r86qb 1/1 Running 0 171m ``` ## Generate the e2e.test file :computer: All e2e tests in kpng are from the k/k repo. The tests are designed to delete all resources after test run - no matter success or failure. This makes debugging hard when you want the environment to be around to inspect CRs and pods Look through [this directory](https://github.com/kubernetes/kubernetes/tree/master/test/e2e) and locate your test(s). Example, let's say you want to debug this test - [It should serve multiport endpoints from pods](https://github.com/kubernetes/kubernetes/blob/2e68fd2857cded9cb470e3bf22ba01bb4934c6f8/test/e2e/network/service.go#L843) 1. comment out all `defer functions` in the test 2. in the beginning of the test, add this line `framework.TestContext.DeleteNamespace = false` Next you need to generate the e2e.test file with the above changes ``` $ cd kubernetes $ make WHAT=test/e2e/e2e.test ``` Depending on what all tests you want to run, you can make use of the `--focus` flag in ginkgo, this will run only the tests you focus on instead of all tests. Continuing with the above example, ``` $ cd kpng $ ginkgo --focus="should serve multiport endpoints from pods" /home/kokoni/kubernetes/_output/bin/e2e.test -- --kubeconfig=/home/kokoni/kpng/temp/e2e/artifacts/kubeconfig_tests.conf --provider=local --dump-logs-on-failure=false --report-dir=reports --disable-log-dump=true ``` Now, everytime you run this test, all resources and state are preserved. ## Troubleshooting :hammer_and_pick: ### kpng cluster is not created / failed / stuck There could be 2 reasons - 1. docker build failed (the test script runs in quiet mode, so you won't see the error output) a. either comment out the `--quiet` flag in the test script b. run docker build yourself - from the repo root `docker build -t kpng:test .` 2. there is a runtime error - panic caused by nil assignments, segmentation fault etc. a. inspect the kpng container using `k logs <kpng-pod-name> -n kube-system kpng` b. this will usually show you on what line number is the error ### I exec'd into my pod, but can't see iptables / any rules This usually happens when there is some grave error in your backend. So best to inspect the backend container logs. Most likely to find a lead there. `k logs <kpng-pod-name> -n kube-system kpng-iptables -f` `-f` flag is used to keep watching aka following the logs # Happy Debugging :heart: