# Logs

could not run steps: step upgrade-tests-aws-ocp-46-continuous failed: "upgrade-tests-aws-ocp-46-continuous" test steps failed: "upgrade-tests-aws-ocp-46-continuous" pod "upgrade-tests-aws-ocp-46-continuous-serverless-e2e" failed: the pod ci-op-f4d867df/upgrade-tests-aws-ocp-46-continuous-serverless-e2e failed after 40m41s (failed containers: test): ContainerFailed one or more containers exited

## serverless-e2e
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-knative-serverless-operator-main-4.6-upgrade-tests-aws-ocp-46-continuous/1423071484703674368/artifacts/upgrade-tests-aws-ocp-46-continuous/serverless-e2e/build-log.txt

01:15:30.158 DEBUG:   [[ $(oc get pods -n knative-eventing --field-selector=status.phase!=Succeeded -o jsonpath='{.items}') != '[]' ]] : Waiting until non-zero (max 600 sec.)
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
01:29:37.074 ERROR:   Time out of 600 exceeded

## describe of kafka-ch-dispatcher-7b86fdf85f-zdlzh pod

seems it failed to be scheduled due to a node error. quick googling shows it might be due to the node having too many network adapters.

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-knative-serverless-operator-main-4.6-upgrade-tests-aws-ocp-46-continuous/1423071484703674368/artifacts/upgrade-tests-aws-ocp-46-continuous/serverless-e2e/artifacts/gather-knative/quay-io-openshift-knative-must-gather-sha256-51049ace7b41db7499a225b27bf11492e419245a3c49581020c184d6c816d410/knative-eventing/pods/describe.log

Events:
  Type     Reason                  Age        From                                                Message
  ----     ------                  ----       ----                                                -------
  Normal   Scheduled               <unknown>                                                      Successfully assigned knative-eventing/kafka-ch-dispatcher-7b86fdf85f-zdlzh to ip-10-0-251-72.us-west-2.compute.internal
  Normal   AddedInterface          23m        multus                                              Add eth0 [10.128.2.90/23]
  Warning  FailedCreatePodSandBox  23m        kubelet, ip-10-0-251-72.us-west-2.compute.internal  Failed to create pod sandbox: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF

and that's why it couldn't be deleted.

## pod stuck terminating
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-knative-serverless-operator-main-4.6-upgrade-tests-aws-ocp-46-continuous/1423071484703674368/artifacts/upgrade-tests-aws-ocp-46-continuous/gather-extra/artifacts/oc_cmds/pods

kafka                                              my-cluster-zookeeper-2                                                1/1     Running       0          43m     10.131.0.27    ip-10-0-159-90.us-west-2.compute.internal    <none>           <none>
kafka                                              strimzi-cluster-operator-c8ddd59d-nfdrq                               1/1     Running       0          44m     10.129.2.8     ip-10-0-171-142.us-west-2.compute.internal   <none>           <none>
knative-eventing                                   kafka-ch-dispatcher-7b86fdf85f-zdlzh                                  0/2     Terminating   0          28m     <none>         ip-10-0-251-72.us-west-2.compute.internal    <none>           <none>
knative-eventing                                   v0.23-kafka-storage-version-migration-948tq                           0/1     Completed     0          28m     10.128.2.78    ip-10-0-251-72.us-west-2.compute.internal    <none>           <none>
openshift-apiserver-operator                       openshift-apiserver-operator-6c46f9bd8d-p85fh                         1/1     Running       1          75m     10.128.0.15    ip-10-0-193-135.us-west-2.compute.internal   <none>           <none>

## kubelet on node

Aug 05 01:35:03.696529 ip-10-0-251-72 hyperkube[1500]: I0805 01:35:03.696502    1500 kubelet_pods.go:980] Pod "kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-5096-4356-9fff-49a5759c714b)" is termi
nated, but pod cgroup sandbox has not been cleaned up

Aug 05 01:06:44.808772 ip-10-0-251-72 hyperkube[1500]: I0805 01:06:44.808509    1500 kubelet.go:1866] SyncLoop (ADD, "api"): "kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-5096-4356-9fff-49a5759c714b)"

Aug 05 01:06:45.346111 ip-10-0-251-72 hyperkube[1500]: I0805 01:06:45.335410    1500 kubelet.go:1882] SyncLoop (DELETE, "api"): "kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-5096-4356-9fff-49a5759c714b)"

  52873 Aug 05 01:06:48.197129 ip-10-0-251-72 crio[1462]: time="2021-08-05 01:06:48.197101193Z" level=info msg="RunSandbox: removing pod sandbox from storage: 6e260bd1f9a50c9f18e9859b3d582204c42a48f07635a5307ec23
  52873 115f92266bd" id=8ef82c4d-45eb-4fce-bf08-c4773796e28f name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
  52874 Aug 05 01:06:48.208095 ip-10-0-251-72 crio[1462]: time="2021-08-05 01:06:48.208058224Z" level=info msg="RunSandbox: releasing container name: k8s_POD_kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing_
  52874 3b1d0b45-5096-4356-9fff-49a5759c714b_0" id=8ef82c4d-45eb-4fce-bf08-c4773796e28f name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
  52875 Aug 05 01:06:48.208095 ip-10-0-251-72 crio[1462]: time="2021-08-05 01:06:48.208100489Z" level=info msg="RunSandbox: releasing pod sandbox name: k8s_kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing_3b
  52875 1d0b45-5096-4356-9fff-49a5759c714b_0" id=8ef82c4d-45eb-4fce-bf08-c4773796e28f name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
  52876 Aug 05 01:06:48.208242 ip-10-0-251-72 hyperkube[1500]: E0805 01:06:48.208220    1500 remote_runtime.go:113] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = error reading contai
  52876 ner (probably exited) json message: EOF
  52877 Aug 05 01:06:48.208296 ip-10-0-251-72 hyperkube[1500]: E0805 01:06:48.208255    1500 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-509
  52877 6-4356-9fff-49a5759c714b)" failed: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF
  52878 Aug 05 01:06:48.208296 ip-10-0-251-72 hyperkube[1500]: E0805 01:06:48.208265    1500 kuberuntime_manager.go:741] createPodSandbox for pod "kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-50
  52878 96-4356-9fff-49a5759c714b)" failed: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF
  52879 Aug 05 01:06:48.208367 ip-10-0-251-72 hyperkube[1500]: E0805 01:06:48.208297    1500 pod_workers.go:191] Error syncing pod 3b1d0b45-5096-4356-9fff-49a5759c714b ("kafka-ch-dispatcher-7b86fdf85f-zdlzh_knati
  52879 ve-eventing(3b1d0b45-5096-4356-9fff-49a5759c714b)"), skipping: failed to "CreatePodSandbox" for "kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-5096-4356-9fff-49a5759c714b)" with CreatePod
  52879 SandboxError: "CreatePodSandbox for pod \"kafka-ch-dispatcher-7b86fdf85f-zdlzh_knative-eventing(3b1d0b45-5096-4356-9fff-49a5759c714b)\" failed: rpc error: code = Unknown desc = error reading container (pr
  52879 obably exited) json message: EOF"
  52880 Aug 05 01:06:48.208367 ip-10-0-251-72 hyperkube[1500]: I0805 01:06:48.208327    1500 event.go:291] "Event occurred" object="knative-eventing/kafka-ch-dispatcher-7b86fdf85f-zdlzh" kind="Pod" apiVersion="v1
  52880 " type="Warning" reason="FailedCreatePodSandBox" message="Failed to create pod sandbox: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF"