Release CI Improvements

# Release CI Improvements ## Goal: * gather issues noticing during releases * prioritize what to do first * differentiate between "fix/first/fast" [F] and "change-proposal/soon" [RFE] ## Meeting Template ``` ## MMM DD ### Previously (copy from prev-mtg-next-steps) ### Notes ### Next Steps * post minutes to [Discourse thread](https://discourse.pulpproject.org/t/discussion-pulp-release-process-improvements/860/2) ``` ## June 29 * Wrapping up working group * Last two action items * Automating update CI branches for Y releases * Automating Pulp-OCI-images release steps ## June 22 * Review PLEASE: * https://github.com/pulp/plugin_template/pull/751 [DONE] * Last two items removing manual steps: * Remove need to manually update CI branches for Y releases [AI: gerrod] * need to have a programatically-reliable location of “list of supported branches” * change “…and latest-Y to just-released Y” * Automating Pulp-oci-images release steps * What next? * working-group is set to end at end of June * we have only the auto-ci-branches Thing to do * looks like we can declare working-group Done at that point? * pulp-oci-images work should be done as well * discussion around pulp-oci-image status * work w/ decko on our needs here? * Some proposed tasks (in proposed order): * get p-o-i to always have a branch for each pulpcore-y-branch * https://github.com/pulp/pulp-oci-images/issues/503 [dkliban] * pull apart p-o-i action so that the individual Things are separate actions * https://github.com/pulp/pulp-oci-images/issues/504 * automate the p-o-i branch-creation in a way that makes it possible to invoke from pulpcore-release-workflow * https://github.com/pulp/pulp-oci-images/issues/505 * automate p-o-i release in a way that it can also be invoked from core-release * Action Items: * Remove need to manually update CI branches for Y releases [AI: gerrod] * https://github.com/pulp/pulp-oci-images/issues/503 [dkliban] ## Jun 15 * try to reserve brain-space to pick up items below as we can this week * [F] Fix conflicting auto-on-release PRs: Update Changelog vs Cherrypick Changelog * Possible fix - remove the cherrypick changelog, on bump minor version PR delete the changelog entries, solely have the update changelog job * https://github.com/pulp/plugin_template/issues/754 * [F] Remove Tests from Release process * https://github.com/pulp/plugin_template/pull/751 * [F] create new backport-label for Y-releases * https://github.com/pulp/plugin_template/pull/745 * Should be converted to a separate workflow that only runs when plugin_template is modified * Should update the backport labels to reflect ci_update_branches * [F] Check if this job is necessary: * https://github.com/pulp/pulp-ci/actions/workflows/supported.yml * It updates a file in pulp-ci that is used in the docs header to tell if a user is looking at the latest version * Should change the logic to have the header look at that plugin's repository and then remove this job * Also the supported versions docs header should probably list the ci_update_branches ## Jun 8 * Removed tests from release workflow * https://github.com/pulp/plugin_template/pull/751 * and there was Much Rejoicing! * eyes/review please * Simplify (separate out) OCI release automation * All of the CI seems to be in one workflow file... * https://github.com/pulp/pulp-oci-images/blob/latest/.github/workflows/pulp_images.yml * From what I can tell this workflow is called in three scenarios: * on pull-requests * daily cron-job * from api-request (usually called by the daily cron-job) * The logic of what the workflow does is dependent on how it is called (99% sure it is doing lots of redundant work, even within a single call) * Pulpcore <3.23 branches have their own special logic and use a different workflow file... * https://github.com/pulp/pulp-oci-images/blob/latest/.ci/scripts/update_ci_branches.py#L25-L28 * https://github.com/pulp/pulp-oci-images/blob/3.21/.github/workflows/publish_images.yaml * Interestingly this special workflow isn't on the 3.22 branch, so the special logic check is wrong... * There might be a bug somewhere for the logic to release & tag updated Z-streams to not work if the corresponding Y image was never released * I think this is what happened to 3.25 * Some proposed tasks: * pull apart p-o-i action so that the individual Things are separate actions * get p-o-i to always have a branch for each pulpcore-y-branch * automate the p-o-i branch-creation in a way that makes it possible to invoke from pulpcore-release-workflow * automate p-o-i release in a way that it can also be invoked from core-release * Is there a technical issue that gets in the way of separating our one-large-workflow into more tightly-focused files? * possibly? needs some investigation ## Jun 01 ### Previously (copy from prev-mtg-next-steps) * ~~focus on core/3.25-compat releases **first**~~ * when does the oci-image get published? * not out yet - AI: decko - hit up Mike to figure out what's up * try to reserve brain-space to pick up items below as we can this week * [F] Fix conflicting auto-on-release PRs: Update Changelog vs Cherrypick Changelog * [F] Remove Tests from Release process * [F] create new backport-label for Y-releases * https://github.com/pulp/plugin_template/pull/745 * [F] Check if this job is necessary: * https://github.com/pulp/pulp-ci/actions/workflows/supported.yml * [F] disable deprecations-test from release-branches "automagically" * yes please ### Notes * Check release script: * https://github.com/pulp/plugin_template/pull/750 * looks thru CHANGES directory, does Cool Stuff in response * Note: we don't appear to currently have git-python as a requirement anywhere * is there a "release_requirements.txt" somewhere? should there be? * release_requirements.txt is under a DEPRECATED_FILES in `plugin_template` * ci_requirements? (no - stuff that needs to be installed before tests, not release) * we should separate this-script, from "we should have a releasae_requirements.txt file" - that will be its own PR * gerrod is picking up "remove tests from release" ASAP * backport-releases took up A LOT of time * the one-off/transient test-failures are A Pain * high-level goal: document what the CI is "actually doing", so ppl have a place to go to try and figure out problems * a to-do for "after everything else is stable/quiescent" ### Next Steps * AI: decko - hit up Mike to figure out what's up w/ oci-images and 3.25 * try to reserve brain-space to pick up items below as we can this week * [F] Fix conflicting auto-on-release PRs: Update Changelog vs Cherrypick Changelog * [F] Remove Tests from Release process * [F] create new backport-label for Y-releases * https://github.com/pulp/plugin_template/pull/745 * [F] Check if this job is necessary: * https://github.com/pulp/pulp-ci/actions/workflows/supported.yml * [F] disable deprecations-test from release-branches "automagically" * yes please * post minutes to [Discourse thread](https://discourse.pulpproject.org/t/discussion-pulp-release-process-improvements/860/2) ## May 25 ### Previously * "just do" things labelled [F] below * [F] Fix conflicting auto-on-release PRs: Update Changelog vs Cherrypick Changelog * [F] Remove Tests from Release process * [F] create new backport-label for Y-releases * https://github.com/pulp/plugin_template/pull/745 * [F] Check if this job is necessary: * https://github.com/pulp/pulp-ci/actions/workflows/supported.yml * ~~bring list to larger team for review/thoughts/observations~~ * ~~https://discourse.pulpproject.org/t/discussion-pulp-release-process-improvements/860/3~~ * meeting weekly (Thurs 0900 GMT-5) ### Notes * Finish releasing plugin pulpcore>=3.25 releases * file/rpm/python/maven/deb/certguard are done! /cheers wildly * container, ostree, ansible (actually release) * talk w/ lubos * AI [gerrod] ansible * cookbook, gem, npm for nightly (main branch only) * AI [decko] cookbook * to be removed -> https://github.com/pulp/pulp-oci-images/pull/486 * AI [gerrod x9c4] gem, ansible * AI [x9c4] npm * potentially remove from images for now * [F] disable deprecations-test from release-branches "automagically" * yes please * Working on check-release script (gerrod) * Multiple branches are missing changelog updates * seems to have been failures in changlog/remove-changefiles/update-dev? * needs investigation * seems not to affect any branches "we care about" * discussion around "auto actions at create-release-branch action (Y branches)" * create new backport label (PR#745) * should prob take over from Michal since he's in finals for a few weeks * update ci-branches automatically * re-sync backport labels to to ci-branches ### Next Steps * focus on core/3.25-compat releases **first** * try to reserve brain-space to pick up items below as we can this week * [F] Fix conflicting auto-on-release PRs: Update Changelog vs Cherrypick Changelog * [F] Remove Tests from Release process * [F] create new backport-label for Y-releases * https://github.com/pulp/plugin_template/pull/745 * [F] Check if this job is necessary: * https://github.com/pulp/pulp-ci/actions/workflows/supported.yml * [F] disable deprecations-test from release-branches "automagically" * yes please * post minutes to [Discourse thread](https://discourse.pulpproject.org/t/discussion-pulp-release-process-improvements/860/2) ## May 18 ### Next Steps * "just do" things labelled [F] below * bring list to larger team for review/thoughts/observations * AI: [ggainey] open a discourse thread * put on agenda for Monday's team mtg * meeting weekly (Thurs 0900 GMT-5) ### Issues * [F] Fix conflicting auto-on-release PRs: Update Changelog vs Cherrypick Changelog * why are there two? * which one is doing The Right Thing? * remove at least one * maybe it's a collision between Y-release/Z-release processes? * [F] Remove Tests from Release process * yes please * Remove manual CI steps for OCI-images * https://github.com/pulp/pulp-oci-images/blob/latest/docs/developer-instructions.md * should be automatable in pulp-bot * can use args on the manual steps * or we can use args to programaticaly build the images * Remove need to manually update CI branches for Y releases * need to have a programatically-reliable location of "list of supported branches" * change "...and latest-Y to just-released Y" * [F] create new backport-label for Y-releases * [RFE] and remove unneeded ones? * see "reliable location of list of supported branches" * [F] Check if this job is necessary: * https://github.com/pulp/pulp-ci/actions/workflows/supported.yml * is this "latest docs don't warn, prev docs say 'not most recent version'"? * The magic is happening here? https://github.com/pulp/pulp-ci/actions/runs/5010048693/jobs/8979570319#step:6:1 * Create script to determine if releases are needed * Checks the release branches for presences of changelog additions * can determine Y- vs Z- based on CHANGES extension * x9c4 has a git-log cmd * "some day", we could get to a completely-automated process * Create script to auto-generate an announcement message * autopost to discourse? * autopost to twitter? * or mastodon? * y-release only? or all? * matrix channel $TOPICs? * slack too?