Compose-id pinning and promotion workflow

Brainstorming

1. Compose-id pinning:

   • 2 labels so far:
     • centos-ci-testing (periodic promotion of CentOS latest-compose)
     • current-centos (latest known good compose-id)

2. Compose-id promotion workflow:

   • latest-compose -> centos-ci-testing
     • Promoter code work already started in ci-config
     • Do we need more labels here? e.g: previous-centos-ci-testing
   • centos-ci-testing -> current-centos
     • needs a nodepool image pinned to centos-ci-testing compose-id
       • we can use an image pinned to an older compose, and then update repos + update packages
     • needs to define jobs and promotion criteria
     • will need to promote the nodepool image to current-centos
       • We can use an images pinned to an older compose-id and update repos with an specific compose-id. Only compose-ids will need to be promoted.

TripleO Repos and Centos Stream Compose Pinning

Date: Nov 11, 2021

Current Situation on CS9:

• CS9 nodepool image has centos.repo which pulls packages from centos stream mirrors.

  • Pre-installed packages on nodepool images are from centos stream mirrors.

• In tripleo CI jobs, we enable compose repos using tripleo-repos or using release file to lay down to compose repos.

  • Here we mix packages and metadata from centos mirror and compose
  • Which further leads to following issues
    • https://bugs.launchpad.net/tripleo/+bug/1950466 - CS9 nothing provides libgcc >= 11.2.1-6.el9
      • Enable compose repos in pre: https://review.opendev.org/c/openstack/tripleo-ci/+/817455
    • https://bugs.launchpad.net/tripleo/+bug/1949601 - package buildah filtered out by modular filtering
      • Use module_hotfix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/816648

Long term Plan:

• In TripleO CI integration and upstream check/gate jobs, we need to pull centos contents from centos mirrors.
  • Contents gets synced from centos mirror to opendev mirror (https://review.opendev.org/c/opendev/system-config/+/817136)
    • Once CentOS Stream 9 gets released.

Main Goal:

• Catch these issues early in the pipeline.

  • One Idea: Can we use CentOS Stream Pinning?

    • Life span of a compose content is approx 20 days
      • Latest compose will be generally pushed to mirrors (Need to clarify)?
    • Two composes (which one to use)
      • production: https://odcs.stream.centos.org/production/
      • Development: https://odcs.stream.centos.org/development/

  • Current Solution (We are driving)

    • Use tripleo-repos to enable compose in periodic line
    • Switch to mirror once contents gets synced

Notes:

• First: get system config patch in and use mirror content everywhere
  • switch to using the mirror content and no longer composes in release files
  • retest container and images everything
• Second: Get the dependency line
• Understand centos release pattern
• mirror content is n -1 on prod composes if true - forward looking n with latest-compose else latest-compose is n-1 and then test with development compose
• Store old compose ids
• Build nodepool images with a particular compose

Tasks

CentOS Composes - Dependency Pipeline

1. Add a new repo type, 'repo_composes', to 'repo-setup' role that configures repos based on a compose-id

   • we can easily set the compose-url in the release files and use yum-config
   • WIP: https://review.opendev.org/c/openstack/tripleo-quickstart/+/819019

2. Create a new dependency repo_config for compose-pinning

   • we may want to overwrite 'repo' and 'add_repos' vars to avoid compose-repos being overwritten
   • make sure that we have delorean repos and deps repos being set

3. Implement function to get compose-url from humam-readable label

   • today the content is a compose-id, but we can change content to compose-url (easier to handle between different distros - which may have different urls)
   • e.g. https://images.rdoproject.org/centos_compose/centos8/master/centos-ci-testing (compose-id)

4. Create a new or update an existing 'pre' to use tripleo-yum-config to generate compose repos

   • it shall configure repos and update packages
   • not all the jobs use release file content to setup repos
   • the 'pre' should validate if the current image is too new for the compose-if being tested
     • where this info leaves in image?

5. Create new jobs

   • list of jobs TBD
   • first, validate with a provider + scenario
   • define image to be used for compose-pinning, need to be a old compose-id

6. Create pipeline and add to rdo-jobs

   • run every day or two?

7. Generate diff repo list

   • Use/adjust existing playbooks to compile the diff repo list

CentOS Composes - Promotions

1. Create promotion jobs:
   • promote latest-compose-id to centos-ci-testing
     • periodic-tripleo-centos-8-promote-latest-compose-master
     • need to add to periodics
     • https://review.rdoproject.org/r/q/topic:"compose-promoter"+(status:open OR status:merged)
     • Update to include previous if needed (ci-config)
   • promote centos-ci-testing to current-centos
     • Improve promoter code to also promote based on criterias
     • see https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34612
     • define criteria
     • create promotion environment file

Questions

1. Which image should we use in pinning jobs?
2. How do we check which compose-id was the current in image creation?