## Incident Roles

| | |
| ------------------------------ | ---------------------------------------- |
| :worker: Incident Responder(s) | The person who has the hands-on keyboard |
| :captain: Incident Commander | Communicating with stakeholders |
| :talk: Incident Communicator | Overseeing the incident |

The steps below are applicable for Sev 1 & 2 incidents – see the [Escalations](#Escalations) section for more detailed steps.

<span style="font-size: 28px; color: grey">:1:</span> The first responder automatically becomes the incident commander until it is transferred: <br/>

!!! warning
You are still the :captain: commander until someone else says :speak: *I have command*

<span style="font-size: 28px; color: grey; ">:2:</span> The incident commander is also automatically the incident communicator until it is transferred:

!!! warning
You are still :talk: communicator until someone else says :speak: *I have comms*

<span style="font-size: 28px; color: grey">:3:</span> Setup a communications channel on %%{support.channel}%%

<span style="font-size: 28px; color: grey">:4:</span> Communicate a new incident and channel details to %%{support.email}%%

## Severity Classification

<table>
<tr>
<td>
</td>
<td>Sev 1
</td>
<td>Sev 2
</td>
<td>Sev 3
</td>
</tr>
<tr>
<td>Kubernetes
</td>
<td>Cannot schedule pods
</td>
<td>
</td>
<td>Other performance issues
</td>
</tr>
<tr>
<td>Harbor
</td>
<td>Cannot Pull Images
</td>
<td>Cannot Push Images
</td>
<td>Performance issues / Cannot use UI
</td>
</tr>
<tr>
<td>Postgres
</td>
<td>
</td>
<td>
</td>
<td>No replicas / HA lost
</td>
</tr>
<tr>
<td>Vault
</td>
<td>
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<td>Elasticsearch
</td>
<td>
</td>
<td>
</td>
<td>
</td>
</tr>
</table>

## Escalations

After receiving a new alert or ticket, spend a few minutes (&lt; 5m) doing a preliminary investigation to identify the reproducibility and impact of the incident. Once confirmed as an issue being the formal incident response process:

## Hypothesis development

:1: Start with checking general cluster health

:2: Check recent changes that may have had an impact

:3: Check recent trends in consumption metrics:

* has a threshold been reached? A threshold can be both fixed (e.g. memory) and cumulative (disk space)

:4: Check performance metrics

* Are metrics within bounds?
* Are any metrics abnormal

:5:

## Hypothesis Testing

## Mitigation

See [Generic mitigations](https://www.oreilly.com/content/generic-mitigations/)

### :kubernetes: Kubernetes

### :heart: Health Checks

* :bash: `karina status` (lists control-plane / etcd versions / leaders / orphans )
* :bash: `karina status pods`
* :octicons-graph-16: control-plane logs [TODO - elastic query]
* :octicons-graph-16: karma, canary alerts
* :bash: [kubectl-popeye](https://github.com/derailed/popeye)

| | |
| ----------------------------- | ------------------------------------------------------------ |
| Deployments | See [Troubleshooting Deployments](https://learnk8s.io/troubleshooting-deployments) <br>:bash: <a href="https://github.com/aylei/kubectl-debug">kubectl-debug</a> |
| No Scheduling | :worker: Manual schedule by specifying node name in spec |
| Network Connectivity | See <a href="https://itnext.io/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727">Guide to K8S Networking</a> and :material-video: <a href="https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/">Networking Model</a> <a href="https://www.youtube.com/watch?v=RQNy1PHd5_A">Packet-level Debugging</a> <br>:bash: <a href="https://github.com/eldadru/ksniff">kubectl-sniff</a> - tcpdump specific pods<br/>:bash: <a href="https://soluble-ai.github.io/kubetap/">kubectl-tap</a> - expose services locally<br/>:bash: <a href="https://github.com/mehrdadrad/tcpprobe">tcpprobe</a> - measure 60+ metrics for socket connections <br/>:octicons-graph-16: check node to node connectivity using :material-docker: <a href="https://github.com/bloomberg/goldpinger">goldpinger</a> : <br/>:action: Restart CNI controllers / agents <br> |
| End User Access Denied | :action: Temporarily increase access level<br />Check access using :bash: <a href="https://github.com/corneliusweig/rakkess">rbac-matrix</a> |
| Disk / Volume Space | :action: Check PV usage using :bash: <a href="https://github.com/yashbhutwala/kubectl-df-pv">kubectl-df-pv</a><br/>:worker: Remove old filebeat/journal logs <br>:worker: Scale down replicated storage <br>:worker: Reduce replicas from 3 → 2 → 1 |
| DNS Latency | :octicons-graph-16: Check DNS request/failure count on grafana<br /> :octicons-graph-16: Check pod ndots configuration and reduce if possible<br />:octicons-graph-16: Check node-local cache hit rates<br /> :worker: Scale coredns replicas and/or increase cpu limits on coredns and node-local-dns |
| Failing Webhooks | :worker: Temporarily disable the webhooks by either deleting them or setting the **FailurePolicy** to *ignore* |
| Loss of Control Plane Access | :worker: Try gain access to master nodes and regen certs using `/etc/kubernetes/pki` <br />:worker: Downscale cluster to 1 master, and regen certs using kubeadm, scale masters back up <br /> |
| Failure during rolling update | Run :bash: `karina terminate-node` followed by `karina provision` |
| Worker Node failure | Run :bash: `kubectl terminate-node` followed by `karina provision` |
| Control Plane Node Failure | :action: Run :bash: `kubectl terminate-node` followed by `karina provision`<br />:worker: Remove any failed etcd members using :bash: `karina etcd remove-member` |
| Namespace Overutilization | |
| Load Balancer Failure | |
| Cluster Failure | :action: Cordon the cluster by removing GSLB entries<br />:action: Without PVCs Run :bash: `kubectl terminate` followed by `karina provision` to reprovision the cluster <br />:action: With PVCs: Try take a backup first (`karina backup or velero backup`), provision a new cluster with a new name, restore from backup `karina restore or velero restore` |
| Cluster over-utilized | :action: Increase capacity if possible (even temporarily)<br />:action: Shed load starting with <br />* moving workloads to other clusters <br />* reducing replica counts <br />* terminating non critical workloads (dynamic namespaces, etc) |
| Node Performance | :octicons-graph-16: Check container top using :bash: `crictl stats` and `crictl ps<br /`<br />:octicons-graph-16: Check VM host CPU/Memory/IO staturation<br />See <a href="https://publib.boulder.ibm.com/httpserv/cookbook/">Performance Cookbook</a> and <a href="http://www.brendangregg.com/USEmethod/use-linux.html">USE</a> |
| Unable to SSH | Try :bash: <a href="https://github.com/kvaps/kubectl-node-shell">kubectl-node-shell</a> |

### :etcd: Etcd

| | |
| ---------------- | ------------------------------------------------------------ |
| Slow Performance | :octicons-graph-16: Check disk I/O <br>:worker: Reduce size of etcd cluster |
| Loss of Quorum | See <a href="https://etcd.io/docs/v3.4.0/op-guide/recovery/">Disaster recovery</a> |
| Key Exposure | |
| DB Size Exceeded | :action: Run etcd compaction and/or reduce version history retention |
| | |
| | |

### :postgres: Postgresql

| | |
| --------------- | ------------------------------------------------------------ |
| Slow Performace | |
| Data Loss | Recover from backup |
| Key Exposure | |
| Failover | :bash: run `patronctl failover`<br />Delete config endpoint if failover is stuck without any master |

### :vault: Vault / :consul: Consul

| | |
| --------------- | ------------------- |
| Slow Performace | |
| Data Loss | Recover from backup |
| Key Exposure | |
| Failover | |

### :harbor: Harbor

| | |
| --------------- | ------------------------------------------------------------ |
| Inaccessible | :worker: compare accessibilty via UI / API / docker login<br /> :worker: Reset CSRF_TOKEN in harbor-core configmap<br /> :worker: Reset admin password in `harbor_users` postgres table |
| Slow Performace | :octicons-graph-16: Check performance of underlying registry storage (S3 / Disk etc)<br />:octicons-graph-16: Check CPU load/throttling on the postgres DB |
| Data Loss | :worker: Fail forward and ask dev-teams to rebuild and repush images <br />:worker: Recover images from running nodes |
| Key Exposure | |
| Failover | |

### :elastic: Elasticsearch

| | |
| --------------- | ------------------------------------------------------------ |
| Slow Performace | :octicons-graph-16: Check performance of underlying disks <br />:octicons-graph-16: Check CPU load/throttling on the elastic instance <br />:octicons-graph-16: Check memory saturation under elastic node health |
| Data Loss | |
| Key Exposure | |
| Failover | |

## Incident Resolution