# Kubernetes Pod Issues:

Currently we are facing two issues:
* Kube-system pods are in pending state
* Deployment pods are in terminating state

## Pods are in pending state:

When pods are in pending state, "kubectl descirbe pods" command should display the messages from the scheduler and it will explain why they can't be scheduled. However, "Pending" state can redirect to tow reasons.

First, it may bound to hostPort and second, we may have insufficient
resources such as memory or CPU.

* Diagonise the issue

`kubectl describe pods -n <NAMESPACE> <PODNAME>`

this will tell us the reason behind the pending state.

* Scale out, scale up of worker node

If we have reached the resouce limits, we can do scale out by adding one more worker node to the cluster. To scale up, we just need to increase the node memory or CPU on the existing worker node.

* Reduce resouce request for the pod
If we don't want to do scale up or scale out of worker node, need to reduce the existing resouce request by chaging the following fields for container running inside the pod:

spec.containers[].resources.requests.cpu
spec.containers[].resources.requests.memory

Link: [Manage Resources for containers](https://https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)

All the extra informtaion from these steps can be added to the opened
kubernetes issue

## Deployments pods are in terminating state

These can be result of two reasons, one is node issue where kubelet inhibitor is not working properly. In our case we can see that kubelet inhibitor is active. The second reason can be the configuration for shutdownGracePeriod and shutdownGracePeriodCriticalPods. But you have tested it with increased value for them but still having this issue. So we can try few workaround to see, if it help us. Each of the following steps are independent.

* Check the finalizer for terminating pod

If ther is a finalizer present for the deletig pod, it will stay in terminating state and deleting will be failed

`kubectl get pods -n <NAMESPACE> <PODNAME> -o yaml`

check under metadata and finalizers. If the finalizers is not empty, try to use the following command to convert the value of finalizers to null:

`kubectl patch pod -n <NAMESPACE> <PODNAME> -p '{"metadata":{"finalizers":null}}'`

* Force delete the pods

We can force delete those pods which are in terminating state by using the following command

`kubectl delete pod --grace-period=0 --force -n <NAMESPACE> <PODNAME>`

this should immediately remove the terminating pod.

* Restart the kubelet of the worker node

We can restart the kubelet of that node to see if the node recovers:
ssh worker node and run-> `swapoff -a && systemctl restart kubelet`

Finally, If we are not willing to do the force deletion or deleting the finalizer, then need to check the following logs:

- [x] Verify kubelet logs to see what is causing the issue "journalctl -u kubelet"
- [ ] Verify docker logs: journalctl -u docker.service
- [x] Check if pod's volume mount points still exist and if anyone holds lock on it.
- [ ] Verify if host is out of memory or disk

## Test results

After the test done on different steps described above, we have seen multiple entries of errors in kubelet logs leads to volume issue where kubelet is unable to delete the volume directory and that is may be the main issue behind unsuccessful graceful node shutdow. After looking into the community, we can see open issues in kubernetes which are still hanging to get approval from the community. [Orphaned pod issue](https://github.com/kubernetes/kubernetes/issues/105536).

Common Error: Unnecessary parts are trimmed

Sep 19 07:04:42 worker-pool1-58voifse-ejlnoox-ibd-stack kubelet[1862]: 1862 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"d01af12d-e6b1-4377-98d4-2e808919e2b5\" found, but error occurred when trying to remove the volumes dir: not a directory" numErrs=1
Sep 19 07:04:44 worker-pool1-58voifse-ejlnoox-ibd-stack kubelet[1862]: 1862 kubelet_volumes.go:245] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"d01af12d-e6b1-4377-98d4-2e808919e2b5\" found, but error occurred when trying to remove the volumes dir: not a directory" numErrs=1

This kinds of volume error occurs when kubelet doesn't have enough time to remove the volume or failed to remove the directory. If the orphaned pod directory exists and contains stale volume directory with a file, in that case we can face this kind of issue.

I was checking the kubelet code and it looks like kublet is able to delete a volume directory if it is empty. Here is the comment in the code base: [kubelet code](https://github.com/kubernetes/kubernetes/blob/e054181e517b48a3c862207537092c28604aaad9/pkg/kubelet/kubelet_volumes.go#L115)

# Possible solution

The current solution is to first enter the `/var/lib/kubelet/pods` directory on the node which is facing the issue, delete the pod’s folder or file corresponding to the pod `id` that reported in the error log.

ssh to the node and run the following command to see the list of files in the pod directory

`sudo ls -la /var/lib/kubelet/pods/<podID>/volumes/<directory>`

the output should show a json file similar to the following:
`-rw-r--r-- 1 root root vol_data.json`

Deleting this file should immediately resolve the kubelet error within few seconds.

Otherwise, you need to check how to fix the volume issue from your end and what is causing this problem, so that it doesn't start this kind of situation where kubelet is unable to delete the file or directory.