# Operate First on-call log

New randomly generated on-call order via [shuffle tool]:
### On-Call Order
- [ ] @HumairAK (pto Sept 1st to 11th)
- [ ] @SamoKopecky
- [ ] @durandom
- [x] @Gregory-Pereira
- [ ] @tumido
- [ ] @schwesig
- [ ] @mimotej

### ToDo
* continue labeling issues/prs (apps and operations repo)
* on-call will remove stale labels, and prompt for responses from the issue stakeholder during this first round of issue pruning/cleanup

## 2022-09-02
* on-call @durandom

## 2022-09-01
* on-call @SamoKopecky
* Auth issues on smaug
* Storage seems to be the issue
* Unknown error at https://console-openshift-console.apps.smaug.na.operate-first.cloud/ocs-dashboards/block-file
* Logs and events aren't avaliable for any smaug pods

## 2022-08-31
* on-call @HumairAK

## 2022-08-30
* on-call @SamoKopecky
* Generated new on-call order
* Created PR to remove obsolete getting started page in our docs: [operate-first/apps#2333](https://github.com/operate-first/apps/pull/2333)
* Create docs for vault CLI login using k8s auth: [operate-first/apps#2334](https://github.com/operate-first/apps/pull/2334)
* Add some more missing labels/remove rotten labels

## 2022-08-29
* on-call @mimotej
* Looked into - https://github.com/operate-first/apps/issues/2318
* @HumairAK can we capture this in a new issue? --> https://github.com/operate-first/support/issues/568#issuecomment-1218144874
* Looked into problems with CI in [following PR](https://github.com/operate-first/apps/pull/2205), issue was [once noticed](https://github.com/operate-first/apps/pull/2296) and is captured in issue [here](https://github.com/operate-first/toolbox/issues/94). Currently quick solution is rebase.
* Labeled issues which were missing labels
* handover @mimotej -> @SamoKopecky

## 2022-08-26
* on-call @tumido
* handover @schwesig -> @tumido

## 2022-08-25 [@gregory-Pereira] @mimotej
* on-call @schwesig
* Should @Gregory-Pereira deal with this @mimotej https://github.com/operate-first/apps/issues/2288? The `grep` command in https://github.com/operate-first/toolbox/blob/master/scripts/test-kubeval-validation#L8-L12 is not working properly (can step through example), basically it should list every file that needs to be changed
* Until this gets resolved, temporarily doubling memory for kubeval check (https://github.com/operate-first/apps/pull/2289)
* cluster-resources-smaug app is reporting out of sync on a rolebinding, even though it's properly sync'd: https://argocd.operate-first.cloud/applications/cluster-resources-smaug?resource=sync%3AOutOfSync
* resolution: bump resource usage, we cant check against half-baked manifests

## 2022-08-24 @durandom
* merged https://github.com/operate-first/apps/pull/2281
* picked up https://github.com/operate-first/apps/issues/2278

## 2022-08-23 [humair]
* Infra cluster upgrade from 4.8-4.9 started
* Confirmed no resources at risk of being removed due to [deprecated resource removal](https://access.redhat.com/articles/6955985)
* Noticed cluster-resources-smaug app was out of sync, leading to 3 different pr changes not being applied live.
* attempting to sync failed, created issue here: https://github.com/operate-first/apps/issues/2278
* leaving for next on-call to attempt a fix
* Kubeval failing due to OOM kills
* Can we only run kubeval on changed root directories?
* Kubeval doc pr review https://github.com/operate-first/apps/pull/2273
* Leaving 4.9 upgrade going
* For next on call:
* Check status on 4.9 upgrade by monitoring here: https://console-openshift-console.apps.moc-infra.massopen.cloud/settings/cluster
* If 4.9 upgrade is complete, create a pr in the similar to: https://github.com/operate-first/apps/pull/2277 , next version will be 4.10.26
* If upgrade is failing, investigate and figure out why, escalate to Lars if he is needed and returned from PTO
* Investigate why cluster-resources-smaug app is failing to sync fybrik resources, see https://github.com/operate-first/apps/issues/2278 for details
* Confirm that apps repo is valid, by running [this script](https://github.com/operate-first/toolbox/blob/master/scripts/test-kubeval-validation) from the toolbox
* If repo is valid, then we should know why kubeval is not catching these api errors
* if these crds are invalid, let fybrik team (e.g. [roytman](https://github.com/roytman)) know.
* If repo is failing kubeval validation, we should fix the errors otherwise all ci will keep failing on this step

## 2022-08-19 [greg]
* Continued labeling issues and removing the stale, I focused on label `lifecycle-stale` rather than `lifecycle-rotten` in `org:operate-first` as a whole
* Created new image for operate-first/argocd using kustomize v4.4.0 binary (https://github.com/operate-first/operations/issues/421)

## 2022-08-18 [sam]
* Add myself to triagers (https://github.com/operate-first/common/pull/142)
* Issues/PRs in apps repo were checked for stale/rotten labels and give new ones if needed
* About half of the issues in the operations repo are checked
* Consider also checking Issues/PRs for common repo while on-call

## 2022-08-17 [mimotej]

* Went over issues/prs in support repository removed stale/rotten labels and begin investigation whether these issues are relevant
* Apps and operations repository still needs to be done
* Consider adding labels for size change requests

---
[Original order of on-call people]: https://github.com/operate-first/apps/blob/master/slack-first/overlays/moc/smaug/config.yaml
[shuffle tool]: https://onlinerandomtools.com/shuffle-lines

---
[This document]: https://hackmd.io/SHcaWaXLS56Fo2Wr9cRfxg
[Operate First Team Charter]: https://docs.google.com/document/d/15NroPyCRsoGS_j5ZIaXsCfrAbj1Go6MBQe3Efyk2vf4/edit
[Operate First Community Cloud support process]: https://github.com/operate-first/blueprint/blob/main/adr/0021-sre-cloud-support.md
[Contributing to GitOps Docs]: https://www.operate-first.cloud/apps/CONTRIBUTING.html
[On-call document]: https://hackmd.io/@durandom/HkzQaW5C5/edit