# CI flakes/failures notes

This document is meant to collect CI test flakes and failures tracked by CI team for release cycle v1.4.0

## 12.12.2022 - 16.12.2022, week2:
Responsible: Furkat Gofurov

* **12.12.2022:**
* 2 jobs [capi-e2e-release-1-2-1-23-1-24](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.2#capi-e2e-release-1-2-1-23-1-24) and [capi-e2e-release-1-2-1-22-1-23](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.2#capi-e2e-release-1-2-1-22-1-23) are failing constantly for 5days:
- [x] CAPI Issue: https://github.com/kubernetes-sigs/cluster-api/issues/7738
- [x] Test infra issue: https://github.com/kubernetes/test-infra/issues/28233
- [x] Debug PRs' in test-infra and fix in CAPI:
- [x] https://github.com/kubernetes/test-infra/pull/28238
- [x] https://github.com/kubernetes/test-infra/pull/28243
- [x] https://github.com/kubernetes/test-infra/pull/28241
- [x] https://github.com/kubernetes-sigs/cluster-api/pull/7505
* 1 job [capi-pr-verify-main](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-pr-verify-main) has failed couple of times, triaged as flake:
- [x] Issue: https://github.com/kubernetes-sigs/cluster-api/issues/7736

* **13.12.2022:**
* Monitoring the CI signal before patch releases (v1.2.X & v1.3.x) scheduled for tomorrow, so far we have a 🟢 signal

* **14.12.2022:**
* CI is 🟢, new patch releases are out ([v1.3.1](https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.3.1) and [v1.2.8](https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.2.8))

* **15.12.2022:**
* 1 job [capi-e2e-release-1-1-1-23-1-24](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.1#capi-e2e-release-1-1-1-23-1-24) failing since 9th December (runs every other day).
- [x] CAPI Issue: https://github.com/kubernetes-sigs/cluster-api/issues/7768

* **16.12.2022:**
* Working on the fix of #7768

----

## 19.12.2022 - 23.12.2022, week3:

Responsible: Alexander Demicev

* **20.12.2022:**
* [capi-e2e-release-1-1-1-23-1-24](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.1#capi-e2e-release-1-1-1-23-1-24) is still failing.

* **21.12.2022:**
* same as yesterday

* **22.12.2022:**
* all jobs were green except for the one from 20.12.2022

* **23.12.2022:**

----
## 26.12.2022 - 30.12.2022, week4:

* 2 jobs [capi-e2e-release-1-0-1-22-1-23](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-e2e-release-1-0-1-22-1-23) and [capi-e2e-release-0-4-1-22-1-23](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-e2e-release-0-4-1-22-1-23) are failing:
- [ ] CAPI Issue: https://github.com/kubernetes-sigs/cluster-api/issues/7811 ==> this can be ignored until the tests decided to be kept. Leaving it open, can be marked as done, once tests are dropped or fixed.

----
## January 2nd 2023 - January 6th 2023, week 5:

Watcher: Nawaz

* **January 2nd 2023**
* everything looks good so far.
* **January 3rd 2023**
* The following below test jobs seem to be failing.
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-mink8s-main
* Initial failure on Jan 3rd.
* **Resolution**: Was flake. Been running green EOD
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-e2e-release-0-4-1-22-1-23
* Failing consistently since Dec 20th
* **Resolution**: [We might not want to keep tests for unsupported version for long](https://github.com/kubernetes-sigs/cluster-api/issues/7811#issuecomment-1370573057)
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-e2e-release-1-0-1-22-1-23
* Failing consistently since Dec 20th
* **Resolution**: [We might not want to keep tests for unsupported version for long](https://github.com/kubernetes-sigs/cluster-api/issues/7811#issuecomment-1370573057)
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main
* Been flacky; failed 4 times consecutively but then went green again
* **Resolution**: Was flake. Been running green EOD.
* **January 4th 2023**
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-pr-e2e-main
* This is flacky but it could be flacky for multitude of reasons and need not be pursued as these are PR tests.
* **January 5th 2023**
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-e2e-release-0-4-1-22-1-23
* Still failing since Dec 20th.
* We have a PR addressing this from Stefan https://github.com/kubernetes-sigs/cluster-api/pull/7856
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-e2e-release-1-0-1-22-1-23
* Still failing since Dec 20th.
* We have a PR addressing this from Stefan https://github.com/kubernetes-sigs/cluster-api/pull/7856
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.2#capi-e2e-release-1-2
* Been flacky and parts of it has failed intermittently making the whole test fail four times consecutively.
* This test runs every 4 hours. Depending on the next run's output, will raise a new issue for it.
* https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3#capi-pr-e2e-informing-release-1-3
* Has failed thrice since Jan4th 23:20PST.
* Will wait for another run for it and raise alarm if it fails again.
* Raised https://github.com/kubernetes-sigs/cluster-api/issues/7858 to track this.

----
## 09.01.2023 - 13.01.2023, week6:
Responsible: Furkat Gofurov

* **09.01.2023:**
* 2 jobs are failing since last week:
* [capi-e2e-release-1-3](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3#capi-e2e-release-1-3)
* [capi-e2e-release-1-2](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.2#capi-e2e-release-1-2)

- [x] Tracking CAPI [Issue](https://github.com/kubernetes-sigs/cluster-api/issues/7858)
- [x] Expected fix in main branch for above issue: backport PRs of [7856](https://github.com/kubernetes-sigs/cluster-api/pull/7856) into release-1.2/1.3 branches
- [x] backport [PR](https://github.com/kubernetes-sigs/cluster-api/pull/7871) to release-1.3
- [x] backport [PR](https://github.com/kubernetes-sigs/cluster-api/pull/7870) to release-1.2

* **10.01.2023:**
- both backport PRs landed, watching the CI closely.

* **11.01.2023:**
- CI is green, patch releases are out.

* **12.01.2023:**
- All green.

* **13.01.2023:**
- [x] [capi-pr-e2e-full-main](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-pr-e2e-full-main) failed twice, but it is a PR job. Monitoring for now, no actions needed.

----
## 16.01.2023 - 20.01.2023, week7:
Responsible: Aniruddha Basak

* **16.01.2023:**
* All Green but majority of the [sig-cluster-lifecycle-cluster-api-1.3](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3) is flaky.

* **17.01.2023:**
- Same as yesterday.

* **18.01.2023:**
- Most of the [sig-cluster-lifecycle-cluster-api-1.3](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3) is flaky.
- Failing tests -
- [capi-e2e-release-1-0-1-22-1-23](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-e2e-release-1-0-1-22-1-23)
- [capi-e2e-release-0-4-1-22-1-23](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-e2e-release-0-4-1-22-1-23)

* **19.01.2023:**
- Same as yesterday.

* **20.01.2023:**
- All [sig-cluster-lifecycle-cluster-api-1.3](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3) flaky tests are green.

**Conclusion:** CI was mostly green, only open item to follow-up on is tracking open issue https://github.com/kubernetes-sigs/cluster-api/issues/7858 and close it depending on CI signal.

----
## 23.01.2023 - 27.01.2023, week8:
Responsible: Alex Demicev

* **23.01.2023:**
- miniK8S e2e is failing https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-mink8s-main, issue tracking thread https://kubernetes.slack.com/archives/C8TSNPY4T/p1674469385575859
- capi-e2e-release-1-0-1-22-1-23, capi-e2e-release-0-4-1-22-1-23: still failing from last week - https://kubernetes.slack.com/archives/C8TSNPY4T/p1674057996218879

* **24.01.2023:**
- miniK8S is ok now

* **25.01.2023:**
- Every job looks fine

* **26.01.2023:**
- All was fine, some failures that we decided to ignore

* **27.01.2023:**
- same as yesterday

ç

## January 30th 2023 - February 3rd 2023, week 9:

Watcher: Nawaz

* **January 30th 2023**
* [capi-test-release-0-3](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.3#capi-test-release-0-3) has been failing since Jan 25th with the below error:
- ```
*=== RUN TestKubeadmConfigReconciler_Reconcile_GenerateCloudConfigData
381
I0130 03:49:32.253250 8681 kubeadmconfig_controller.go:366] "msg"="Creating BootstrapData for the init control plane" "kind"="Machine" "kubeadmconfig"={"Namespace":"default","Name":"control-plane-init-cfg"} "name"="control-plane-init-machine" "version"="1"
382
I0130 03:49:32.254219 8681 kubeadmconfig_controller.go:813] "msg"="Altering ClusterConfiguration" "kubeadmconfig"="default/control-plane-init-cfg" "ClusterName"="cluster"
383
I0130 03:49:32.254256 8681 kubeadmconfig_controller.go:839] "msg"="Altering ClusterConfiguration" "kubeadmconfig"="default/control-plane-init-cfg" "KubernetesVersion"="v1.19.1"
384
--- FAIL: TestKubeadmConfigReconciler_Reconcile_GenerateCloudConfigData (1.43s)
385
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
386
panic: runtime error: invalid memory address or nil pointer dereference
387
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xaa1d4a]

  - [ periodic-cluster-api-e2e-workload-upgrade-1-22-1-23-release-0-4](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-e2e-release-0-4-1-22-1-23) failing since Jan 17th, need to follow up if this test is to be ignored?
  - [periodic-cluster-api-e2e-workload-upgrade-1-22-1-23-release-1-0](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-e2e-release-1-0-1-22-1-23) failing since Jan 17th, need to follow up if this test needs to be ignored.
  - [pull-cluster-api-e2e-informing-release-1-3](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3#capi-pr-e2e-informing-release-1-3) failing since Jan 18th, need to check if this is an important test signal.

- **January 31st 2023**
    - Everything looks normal, except 
        - https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.3#capi-test-release-0-3
        - https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-test-release-0-4
        - https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-test-release-1-0
- **February 1st 2023**
    - Everything looks normal, except 
        - https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.3#capi-test-release-0-3
        - https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-0.4#capi-test-release-0-4
        - https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.0#capi-test-release-1-0
- **February 2nd 2023**
    - Everything looks normal
- **February 3rd 2023**
    - Everything looks normal

----
## 06.02.2023 - 10.02.2023, week 10:
Responsible: Furkat Gofurov

* **06.02.2023:**
    * all green (disregarding failing jobs in the EOL branches which soon to be removed)
* **07.02.2023:**
    * Few jobs were failing due to image bump, fix in https://github.com/kubernetes/test-infra/pull/28654 landed
    
* **08.02.2023:**
    * All green, few flaky tests in some jobs (capi e2e, capi-e2e-mink8s)

* **09.02.2023:**
    * all green

* **10.02.2023:**
    * all green

----
## 20.02.2023 - 24.02.2023, week 12:
Responsible: Aniruddha Basak

* **20.02.2023:**
    * CI Failing -
        * [capi-e2e-main](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main)
        * [capi-e2e-mink8s-main](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-mink8s-main)
* **21.02.2023:**
    * All green
    
* **22.02.2023:**
    * All green

* **23.02.2023:**
    * All green

* **24.02.2023:**
    * All green

----
## February 27th 2023 - March 3rd 2023, week 13:

Watcher: Nawaz

* Februrary 27th
    * Everything looks green
* February 28th
    * Everything looks green
* March 1st
    * Everything looks green
* March 2nd
    * Everything looks green
* March 3rd
    * Everything looks green

----
## March 6th 2023 - March 10th 2023, week 14:

Responsible: Furkat

* March 6th
    * All 🟢
* March 7th
    * All 🟢
* March 8th
    * `capi-e2e-main` job is flaky, tracking
* March 9th
    * `capi-e2e-main` job is flaky
* March 10th
    * `capi-e2e-main` job is extremely flaky, but not red yet, might need an escalation (e.g. issue)

----
## March 20th 2023 - March 24th 2023, week 16:

Responsible: Furkat

* March 20
    * All 🟢
* March 21
    * All 🟢
* March 22
    * All 🟢
* March 23
    * All 🟢
* March 24
    * All 🟢

----

## March 27th - March 31st 2023, week 17:

Watcher: Nawaz K

* March 27th
    * Mostly green 🟢 and some flakes
* March 28th
    * Mostly green 🟢
* March 29th
    * Mostly green 🟢
* March 30th
    * Mostly green 🟢
* March 31st
    * Mostly green 🟢