---
# System prepended metadata

title: ':sparkles: Debugging kpng e2e failures :sparkles:'

---

# :sparkles: Debugging kpng e2e failures :sparkles: 

## Clone repos :cyclone: 
```
$ git clone git@github.com:kubernetes-sigs/kpng.git
$ git clone git@github.com:kubernetes/kubernetes.git
```
## Create a kpng test cluster :desktop_computer: 
```
$ cd kpng

// -d flag will simply create a cluster but **not** run any tests
// -b is the backend of your choice
// -i is the ip family of your choice
$ bash hack/test_e2e.sh -d -b iptables -i ipv4

// verify if cluster & pods are created and running
$ kind get clusters
kpng-e2e-ipv4-iptables

$ k get po -A
NAMESPACE            NAME                                                           READY   STATUS    RESTARTS   AGE
kube-system          coredns-78fcd69978-pw75r                                       1/1     Running   0          171m
kube-system          coredns-78fcd69978-q7vph                                       1/1     Running   0          171m
kube-system          etcd-kpng-e2e-ipv4-iptables-control-plane                      1/1     Running   0          171m
kube-system          kindnet-9cl5s                                                  1/1     Running   0          171m
kube-system          kindnet-fmsps                                                  1/1     Running   0          171m
kube-system          kindnet-nkx8r                                                  1/1     Running   0          171m
kube-system          kpng-6xx2n                                                     2/2     Running   0          171m
kube-system          kpng-78gdc                                                     2/2     Running   0          171m
kube-system          kpng-thrpl                                                     2/2     Running   0          171m
kube-system          kube-apiserver-kpng-e2e-ipv4-iptables-control-plane            1/1     Running   0          171m
kube-system          kube-controller-manager-kpng-e2e-ipv4-iptables-control-plane   1/1     Running   0          171m
kube-system          kube-scheduler-kpng-e2e-ipv4-iptables-control-plane            1/1     Running   0          171m
local-path-storage   local-path-provisioner-85494db59d-r86qb                        1/1     Running   0          171m
```

## Generate the e2e.test file :computer: 
All e2e tests in kpng are from the k/k repo. The tests are designed to delete all resources after test run - no matter success or failure. This makes debugging hard when you want the environment to be around to inspect CRs and pods
Look through [this directory](https://github.com/kubernetes/kubernetes/tree/master/test/e2e) and locate your test(s).
Example, let's say you want to debug this test - [It should serve multiport endpoints from pods](https://github.com/kubernetes/kubernetes/blob/2e68fd2857cded9cb470e3bf22ba01bb4934c6f8/test/e2e/network/service.go#L843)
1. comment out all `defer functions` in the test
2. in the beginning of the test, add this line
`framework.TestContext.DeleteNamespace = false`


Next you need to generate the e2e.test file with the above changes
```
$ cd kubernetes
$ make WHAT=test/e2e/e2e.test
```

Depending on what all tests you want to run, you can make use of the `--focus` flag in ginkgo, this will run only the tests you focus on instead of all tests. Continuing with the above example,
```
$ cd kpng
$ ginkgo --focus="should serve multiport endpoints from pods" /home/kokoni/kubernetes/_output/bin/e2e.test -- --kubeconfig=/home/kokoni/kpng/temp/e2e/artifacts/kubeconfig_tests.conf --provider=local --dump-logs-on-failure=false --report-dir=reports --disable-log-dump=true
```

Now, everytime you run this test, all resources and state are preserved.

## Troubleshooting :hammer_and_pick: 
### kpng cluster is not created / failed / stuck
There could be 2 reasons -
1. docker build failed (the test script runs in quiet mode, so you won't see the error output)
    a. either comment out the `--quiet` flag in the test script
    b. run docker build yourself - from the repo root `docker build -t kpng:test .`
2. there is a runtime error - panic caused by nil assignments, segmentation fault etc.
    a. inspect the kpng container using `k logs <kpng-pod-name> -n kube-system kpng`
    b. this will usually show you on what line number is the error
    
### I exec'd into my pod, but can't see iptables / any rules
This usually happens when there is some grave error in your backend. So best to inspect the backend container logs. Most likely to find a lead there.
`k logs <kpng-pod-name> -n kube-system kpng-iptables -f`
`-f` flag is used to keep watching aka following the logs

# Happy Debugging :heart: 