---
# System prepended metadata

title: Untitled

---


## Incident Roles

|                                |                                          |
| ------------------------------ | ---------------------------------------- |
| :worker: Incident Responder(s) | The person who has the hands-on keyboard |
| :captain: Incident Commander   | Communicating with stakeholders          |
| :talk: Incident Communicator   | Overseeing the incident                  |

The steps below are applicable for Sev 1 & 2 incidents – see the [Escalations](#Escalations) section for more detailed steps.

<span style="font-size: 28px; color: grey">:1:</span> The first responder automatically becomes the incident commander until it is transferred:      <br/>

!!! warning
    You are still the :captain: commander until someone else says  :speak:  *I have command*

<span style="font-size: 28px; color: grey; ">:2:</span> The incident commander is also automatically the incident communicator until it is transferred:

!!! warning
    You are still :talk: communicator until someone else says  :speak:  *I have comms*

<span style="font-size: 28px; color: grey">:3:</span> Setup a communications channel on %%{support.channel}%%

<span style="font-size: 28px; color: grey">:4:</span> Communicate a new incident and channel details to %%{support.email}%%



## Severity Classification

<table>
  <tr>
   <td>
   </td>
   <td>Sev 1
   </td>
   <td>Sev 2
   </td>
   <td>Sev 3
   </td>
  </tr>
  <tr>
   <td>Kubernetes
   </td>
   <td>Cannot schedule pods
   </td>
   <td>
   </td>
   <td>Other performance issues
   </td>
  </tr>
  <tr>
   <td>Harbor
   </td>
   <td>Cannot Pull  Images
   </td>
   <td>Cannot Push Images
   </td>
   <td>Performance issues / Cannot use UI
   </td>
  </tr>
  <tr>
   <td>Postgres
   </td>
   <td>
   </td>
   <td>
   </td>
   <td>No replicas / HA lost
   </td>
  </tr>
  <tr>
   <td>Vault
   </td>
   <td>
   </td>
   <td>
   </td>
   <td>
   </td>
  </tr>
  <tr>
   <td>Elasticsearch
   </td>
   <td>
   </td>
   <td>
   </td>
   <td>
   </td>
  </tr>
</table>




## Escalations

After receiving a new alert or ticket, spend a few minutes (&lt; 5m) doing a preliminary investigation to identify the reproducibility and impact of the incident. Once confirmed as an issue being the formal incident response process:



## Hypothesis development

:1: Start with checking general cluster health

:2: Check recent changes that may have had an impact

:3: Check recent trends in consumption metrics:

*  has a threshold been reached? A threshold can be both fixed (e.g. memory) and cumulative (disk space)

:4: Check performance metrics

* Are metrics within bounds?
* Are any metrics abnormal

:5: 

## Hypothesis Testing


## Mitigation

See [Generic mitigations](https://www.oreilly.com/content/generic-mitigations/)



### :kubernetes: Kubernetes

### :heart: Health Checks

* :bash: `karina status` (lists control-plane / etcd versions / leaders / orphans )
* :bash: `karina status pods`
* :octicons-graph-16: control-plane logs [TODO - elastic query]
* :octicons-graph-16: karma, canary alerts
* :bash: [kubectl-popeye](https://github.com/derailed/popeye)



|                               |                                                              |
| ----------------------------- | ------------------------------------------------------------ |
| Deployments                   | See [Troubleshooting Deployments](https://learnk8s.io/troubleshooting-deployments) <br>:bash: <a href="https://github.com/aylei/kubectl-debug">kubectl-debug</a> |
| No Scheduling                 | :worker: Manual schedule by specifying node name in spec     |
| Network Connectivity          | See <a href="https://itnext.io/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727">Guide to K8S Networking</a> and :material-video: <a href="https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/">Networking Model</a>  <a href="https://www.youtube.com/watch?v=RQNy1PHd5_A">Packet-level Debugging</a> <br>:bash: <a href="https://github.com/eldadru/ksniff">kubectl-sniff</a> - tcpdump specific pods<br/>:bash: <a href="https://soluble-ai.github.io/kubetap/">kubectl-tap</a>   - expose services locally<br/>:bash: <a href="https://github.com/mehrdadrad/tcpprobe">tcpprobe</a> - measure 60+ metrics for socket connections <br/>:octicons-graph-16: check node to node connectivity using :material-docker: <a href="https://github.com/bloomberg/goldpinger">goldpinger</a> :  <br/>:action: Restart CNI controllers / agents <br> |
| End User Access Denied        | :action: Temporarily increase access level<br />Check access using :bash: <a href="https://github.com/corneliusweig/rakkess">rbac-matrix</a> |
| Disk / Volume Space           | :action: Check PV usage using  :bash: <a href="https://github.com/yashbhutwala/kubectl-df-pv">kubectl-df-pv</a><br/>:worker: Remove old filebeat/journal logs <br>:worker: Scale down replicated storage <br>:worker: Reduce replicas from 3 → 2 → 1 |
| DNS Latency                   | :octicons-graph-16: Check DNS request/failure count on grafana<br /> :octicons-graph-16: Check pod ndots configuration and reduce if possible<br />:octicons-graph-16: Check node-local cache hit rates<br /> :worker: Scale coredns replicas and/or increase cpu limits on coredns and node-local-dns |
| Failing Webhooks              | :worker: Temporarily disable the webhooks by either deleting them or setting the **FailurePolicy** to *ignore* |
| Loss of Control Plane Access  | :worker: Try gain access to master nodes and regen certs using `/etc/kubernetes/pki`  <br />:worker:  Downscale cluster to 1 master, and regen certs using kubeadm, scale masters back up <br /> |
| Failure during rolling update | Run :bash: `karina terminate-node` followed by `karina provision` |
| Worker Node failure           | Run :bash: `kubectl terminate-node` followed by `karina provision` |
| Control Plane Node Failure    | :action: Run :bash: `kubectl terminate-node` followed by `karina provision`<br />:worker:  Remove any failed etcd members using :bash: `karina etcd remove-member` |
| Namespace Overutilization     |                                                              |
| Load Balancer Failure         |                                                              |
| Cluster Failure               | :action: Cordon the cluster by removing GSLB entries<br />:action: Without PVCs Run :bash: `kubectl terminate` followed by `karina provision` to reprovision the cluster <br />:action: With PVCs:  Try take a backup first (`karina backup or velero backup`), provision a new cluster with a new name, restore from backup `karina restore or velero restore` |
| Cluster over-utilized         | :action: Increase capacity if possible (even temporarily)<br />:action: Shed load starting with <br />* moving workloads to other clusters <br />* reducing replica counts <br />* terminating non critical workloads (dynamic namespaces, etc) |
| Node Performance              | :octicons-graph-16: Check container top using :bash: `crictl stats` and `crictl ps<br /`<br />:octicons-graph-16: Check VM host CPU/Memory/IO staturation<br />See <a href="https://publib.boulder.ibm.com/httpserv/cookbook/">Performance Cookbook</a> and <a href="http://www.brendangregg.com/USEmethod/use-linux.html">USE</a> |
| Unable to SSH                 | Try :bash: <a href="https://github.com/kvaps/kubectl-node-shell">kubectl-node-shell</a> |

### :etcd: Etcd

|                  |                                                              |
| ---------------- | ------------------------------------------------------------ |
| Slow Performance | :octicons-graph-16: Check disk I/O <br>:worker: Reduce size of etcd cluster |
| Loss of Quorum   | See <a href="https://etcd.io/docs/v3.4.0/op-guide/recovery/">Disaster recovery</a> |
| Key Exposure     |                                                              |
| DB Size Exceeded | :action: Run etcd compaction and/or reduce version history retention |
|                  |                                                              |
|                  |                                                              |



### :postgres: Postgresql

|                 |                                                              |
| --------------- | ------------------------------------------------------------ |
| Slow Performace |                                                              |
| Data Loss       | Recover from backup                                          |
| Key Exposure    |                                                              |
| Failover        | :bash: run `patronctl failover`<br />Delete config endpoint if failover is stuck without any master |




### :vault: Vault / :consul: Consul


|                 |                     |
| --------------- | ------------------- |
| Slow Performace |                     |
| Data Loss       | Recover from backup |
| Key Exposure    |                     |
| Failover        |                     |



### :harbor: Harbor

|                 |                                                              |
| --------------- | ------------------------------------------------------------ |
| Inaccessible    | :worker: compare accessibilty via UI / API / docker login<br /> :worker: Reset CSRF_TOKEN in harbor-core configmap<br /> :worker: Reset admin password in `harbor_users` postgres table |
| Slow Performace | :octicons-graph-16: Check performance of underlying registry storage (S3 / Disk etc)<br />:octicons-graph-16: Check CPU load/throttling on the postgres DB |
| Data Loss       | :worker: Fail forward and ask dev-teams to rebuild and repush images <br />:worker: Recover images from running nodes |
| Key Exposure    |                                                              |
| Failover        |                                                              |



### :elastic: Elasticsearch

|                 |                                                              |
| --------------- | ------------------------------------------------------------ |
| Slow Performace | :octicons-graph-16: Check performance of underlying disks <br />:octicons-graph-16: Check CPU load/throttling on the elastic instance <br />:octicons-graph-16: Check memory saturation under elastic node health |
| Data Loss       |                                                              |
| Key Exposure    |                                                              |
| Failover        |                                                              |






## Incident Resolution
