SIG Testing Annual Report - 2021

# SIG Testing Annual Report - 2021 Authors: @spiffxp, Draft date: 2021-03-08  Based on template at: This report reflects back on CY 2020 and was written in Feb-Mar. 2021. ## Operational ### How is the group doing with operational tasks in [sig-governance.md]? #### Is your [README] accurate? Yes. However, our [charter.md] could benefit from a refresh. It hasn't been refreshed/reviewed in 2.5 years. e.g. - Missing from "in scope" is the fact that we've been actively involved in maintaining the build system for kubernetes. - The "deploying changes" section could also stand a refresh to reflect what is concretely done today. #### Have a [CONTRIBUTING.md] file? No. In general, our subprojects follow the CONTRIBUTING.md files for each repo they cover. #### All subprojects correctly mapped and listed in [sigs.yaml]? TODO: what does "correctly mapped" mean in this case? #### What’s your meeting culture? We regularly meet bi-weekly Tuesdays at 10am PT, to try and overlap with PT, ET, and CET timezones. Agenda is open to community suggestion, deadline to add is 1pm PT the day prior. If we have nothing on the agenda, we cancel. We haven't had much in the way of regular structure, e.g. no regular subproject checkins, no review of testgrid dashboards, no slot just for issue triage. We recently discussed our meeting culture at a meeting, and consensus amongst attendees was the following: - people valued the time and enjoyed hanging out with fellow contributors - no desire to further reduce meeting frequency, nor move back to 30min/weekly - the vast majority of speaker time goes to a few individuals, mostly the leads - leads are interested in how to encourage others to speak up or participate more - but attendees were generally happy with status quo, found leads informative - "this meeting could have been an email" or "this could have been prerecorded" - the spontenaity of Q&A often provided the most insight to attendees - we are considering trialing use of GitHub discussions, or more regular e-mails for info-dumps etc. - walking a board for issue triage eats entire meetings, more success when curating a few to discuss: - ... which is ultimately what lands on our agenda anyway: things contributors want to discuss/demo, and things leads want to discuss/demo - we are considering setting aside a block of time for help-wanted issues ##### Large/small, active/quiet, learnings? TOO, 10-20 attendees ##### Meeting notes up to date? Yes ##### Are you keeping recordings up to date/trends in community members watching recordings? Yes, although uploading recordings is a manual process for us; they currently live on @spiffxp's channel. Can lag behind by a week or two. 690 views over the last year - 290 came from a "Deflaking Kubernetes Tests" video clipped from SIG Testing meeting with @liggitt presenting. - 5-32 view range for SIG Testing recordings from 2020 - no other subprojects are holding regular meetings ### How does the group get updates, reports, or feedback from [subprojects][subproject-definition]? Nothing formal. Subproject owners will e-mail the list with announcements or design proposals, show up at our meetings for design review or discussion. But not much in the way of regular review of roadmaps/accomplishments etc. #### Are there any springing up or being retired? We moved the "testing-commons" subproject to "best effort", meaning it still exists as a conceptually separate area of focus, and is owned by SIG Testing, but nobody is actively leading the effort. We archived the #testing-commons slack channel and have directed folks to come to the general #sig-testing slack channel and meeting for future discussions on things in the testing-commons area. It is intended to represent ownership of everything under kubernetes/kubernetes/test, not in terms of "we'll write and fix all of these tests" (explicitly out of scope, per our charter) but instead in an advisory capacity that aligns with our bandwidth. #### Are OWNERS.md files up to date in these areas? TODO ### How does the group get updates, reports, or feedback from Working Groups? Same as for subprojects. @spiffxp happens to be a lead for WG K8s Infra so often passes info back and forth. We haven't yet formally shared info between WG Reliability and SIG Testing, though we've collaborated on some proposals. #### Are there any springing up or being retired? No #### Are OWNERS.md files up to date in these areas? TODO ### When was your last public community-wide update? TODO: (provide link to deck and/or recording) ## Membership ### Are all listed SIG leaders (chairs, tech leads, and subproject owners) active? TODO ### How do you measure membership? We don't. If we had to, it would depend on the reason. For example: - to gather relevant feedback, the mailing list, e.g. sig-testing design docs should be shared with the kubernetes-sig-testing mailing list (but it's always safe to include kuberntes-dev too) - to get a sense of staffing, examine regular meeting attendance, and participation in issues/PRs that aren't targeted job/testgrid config changes ### How does the group measure reviewer and approver bandwidth? TODO #### Do you need help in any area now? TODO #### What are you doing about it? TODO ### Is there a healthy onboarding and growth path for contributors in your SIG? Probably not. These are the sorts of things we lack: - dedicated onboarding docs - dedicated roles with playbooks and mentors - SIG chairs with enough spare bandwidth to handle all ad-hoc questions - measurement of anything related to contributors (growth, productivity, satisfaction, bounce rate, etc.) #### What are some activities that the group does to encourage this? These are the sorts of things we do have: - we are most responsive on #sig-testing in slack (and whatever slack threads some of us get pulled into) - we consider #release-ci-signal members to be an unofficial onboarding path; they are motivated to become educated power-users of our tooling, and to reduce the toil of using project test results - we welcome reviewers, and will promote to approver if they put in the required work (not automated, but human reviewed ~per release cycle) - we welcome all topics at our bi-weekly meeting (within first-come, first-serve timeboxed constraints) - we periodically groom `help wanted` issues about once per release cycle, and try to be responsive to contributors who pick those up ([sig-testing board: Help Wanted column](https://github.com/orgs/kubernetes/projects/52#column-12861951)) - subproject owners are more responsive to ad-hoc questions (e.g. boskos, kind, kubetest2, prow) - some members have occasionally paired or had 1:1 meetings with contributors #### What programs are you participating in to grow contributors throughout the contributor ladder? None ### What programs do you participate in for new contributors? None ### Does the group have contributors from multiple companies/affiliations? Yes #### Can end users/companies contribute in some way that they currently are not? Yes. Two examples come to mind. Knowledge sharing: - are there OSS things outside of the kubernetes bubble we should know about? - are there downstream problems that upstream could better solve? - are there questions about how to use the tooling we do have? Friction logs: - what makes it difficult to test kubernetes? - what makes it difficult to use (solely) tests to believe kubernetes (itself/features/post-upgrade) is stable/reliable? - what kinds of questions have you needed to answer for testing kubernetes? - when looking for answers, how and where did you expect to find them? how and where did you actually find them? ## Current initiatives and project health ### What are initiatives that should be highlighted, lauded, shout outs, that your group is proud of? TODO: I think most of this will be a restatement/summary of the answers below #### Currently underway? TODO #### What are some of the longer tail projects that your group is working on? TODO ### Year to date KEP work  TODO: what's relevant YTD - [714 - Breaking apart the Kubernetes test tarball][kep-714] - done but lacking stage - [917 - go modules][kep-917] - [960 - Behavior-driven Conformance Testing][kep-960] - [1143 - Appropriate use of node-role labels][kep-1143] - [1179 - Building Kubernetes Without In-Tree Cloud Providers][kep-1179] - [1209 - Metrics Stability Framework][kep-1209] - [1333 - Ensure Conformance Tests Do Not Require Beta APIs or Features][kep-1333] - [1498 - Kubernetes Yearly Support Period][kep-1498] - [1547 - Building a Dockerless Kubelet][kep-1547] - [1732 - Kubernetes Community Artifact Serving][kep-1732] - [2290 - New label for trusted PR identification][kep-2290] - [2291 - Presubmit config inside the tested repo][kep-2291] - [2390 - Reporting Conformance Test Results to Testgrid][kep-2390] - [2420 - Reducing Kubernetes Build Maintenance][kep-2420] - [2464 - Kubetest2 CI Migration][kep-2464] - [2539 - Continuously Deploy K8s Prow][kep-2539] - [2572 - Defining the Kubernetes Release Cadence][kep-2572] - [1933 - Defend against accidental credential logging via static analysis interation into Prow.][kep-1933] #### What's now stable? TODO #### Beta? [KEP 2420 - Reducing Kubernetes Build Maintenance][kep-2464] #### Alpha? [KEP 2464 - Kubetest2 CI Migration][kep-2464] [KEP 2539 - Continuously Deploy K8s Prow][kep-2539] #### Road to alpha? KEPs we're "participating" in (vs. owning): - [KEP 2572 - Defining the Kubernetes release cadence][kep-2572] - [KEP 1143 - Clarify use of node-role labels within Kubernetes and migrate old components][kep-1143] ### What initiatives are you working on that aren't being tracked in KEPs? Migration of project infrastructure from Google to community (a.k.a WG K8s Infra) ### What areas and/or subprojects does the group need the most help with? TODO ### What metrics/community health stats does your group care about and/or measure? TODO Examples?  [sigs.yaml]: https://git.k8s.io/community/sigs.yaml [sig-governance.md]: https://git.k8s.io/community/committee-steering/governance/sig-governance.md [sig-definition]: https://github.com/kubernetes/community/blob/master/governance.md#sigs [sig-roles]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance.md#roles [subproject-definition]: https://github.com/kubernetes/community/blob/master/governance.md#subprojects [wg-definition]: https://github.com/kubernetes/community/blob/master/governance.md#working-groups  [charter.md]: https://git.k8s.io/community/sig-testing/charter.md [CONTRIBUTING.md]: https://git.k8s.io/community/sig-testing/CONTRIBUTING.md [README]: https://git.k8s.io/community/sig-testing/README.md  [kep-116]: https://github.com/kubernetes/enhancements/issues/116 [kep-714]: https://github.com/kubernetes/enhancements/issues/714 [kep-917]: https://github.com/kubernetes/enhancements/issues/917 [kep-960]: https://github.com/kubernetes/enhancements/issues/960 [kep-1143]: https://github.com/kubernetes/enhancements/issues/1143 [kep-1179]: https://github.com/kubernetes/enhancements/issues/1179 [kep-1209]: https://github.com/kubernetes/enhancements/issues/1209 [kep-1333]: https://github.com/kubernetes/enhancements/issues/1333 [kep-1498]: https://github.com/kubernetes/enhancements/issues/1498 [kep-1547]: https://github.com/kubernetes/enhancements/issues/1547 [kep-1732]: https://github.com/kubernetes/enhancements/issues/1732 [kep-2290]: https://github.com/kubernetes/enhancements/issues/2290 [kep-2291]: https://github.com/kubernetes/enhancements/issues/2291 [kep-2390]: https://github.com/kubernetes/enhancements/issues/2390 [kep-2420]: https://github.com/kubernetes/enhancements/issues/2420 [kep-2464]: https://github.com/kubernetes/enhancements/issues/2464 [kep-2539]: https://github.com/kubernetes/enhancements/issues/2539 [kep-2572]: https://github.com/kubernetes/enhancements/issues/2572 [kep-1933]: https://github.com/kubernetes/enhancements/issues/1933