# centos-stream dependency pipeline via dnf versionlock
###### tags: `Design`

## Problem

centos-stream-8 updates have now broken upstream and 3rd party for the second time in 3 weeks. We can expect more issues in the future. We can also expect A LOT MORE churn and bugs w/ centos-stream-9.

## History

We were executing a centos-stream-8 dependency pipeline to test the conversion of centos-8 to centos-stream-8. We were also investigating a pending or staging repo for centos-stream-8. After asking rhos-ops and directly to the centos maintainers no such pending/staging repo is available for centos-stream releases. This broke our pattern for the dependency pipeline and the dependency pipeline was abandoned for centos-stream.

### UPDATE: found https://composes.centos.org/ which may enable the original design of the dependency pipeline.
https://review.rdoproject.org/r/c/testproject/+/31586
https://review.opendev.org/c/openstack/tripleo-quickstart/+/796536 Update centos 8 dep pipeline to run with compose repos

## New thoughts

Sandeep employeed dnf's plugin version lock for a couple baseos / appstream bugs encountered in the pervious ruck/rover sprint. Now is a good time to explore again the centos-stream dependency pipeline with version lock to stabilize upstream and 3rd party while watching for the latest changes that break OpenStack.

## How dnf versionlock works:

Installed:
  python3-dnf-plugin-versionlock-4.0.21-1.fc33.noarch                                                                                                                                          

Complete!
[root@localhost ~]# dnf versionlock add kernel-5.12.8-200.fc33.x86_64

Adding versionlock on: kernel-0:5.12.8-200.fc33.*
[root@localhost ~]# cat /etc/dnf/
aliases.d/          dnf.conf            modules.d/          modules.defaults.d/ plugins/            protected.d/        vars/               
[root@localhost ~]# cat /etc/dnf/plugins/
copr.conf               copr.d/                 debuginfo-install.conf  versionlock.conf        versionlock.list        
[root@localhost ~]# cat /etc/dnf/plugins/versionlock.list 

# Added lock on Mon Jun  7 11:54:24 2021
kernel-0:5.12.8-200.fc33.*

### Proposal #1 DNF VersionLock

Build a versionlock.list (manifest) of known good centos packages from the logs of our integration jobs. Create a single file from all our jobs that contain packages from non delorean repos.

In each upstream and 3rd party and periodic job including integration and component.

1. install dnf versionlock
2. add the package manifest to /etc/dnf/pluguins/versionlock.list

The CentOS-Stream dependency pipeline would be NOT run with the version lock and would be responsible for updating the versionlock.list file if all the jobs pass.

The new manifest of packages would be added to each integration pipeline and promoted as part of each branches promotion.

This way as we promote a branch integration we would be promoting a set of known good openstack packages, AND centos packages along side the containers and overcloud images.

### Proposal #2 CentOS-Stream Compose build

Take advantage of the CentOS compose build ID and lock to a specific CentOS compose while always testing the latest compose in a dependency pipeline.

* https://composes.centos.org/
* upstream and 3rd party would be locked to CenOS-Stream-8-2021*.n.0
* CentOS dependency pipeline would constantly test "latest-CentOS-Stream-8" compose
* current-tripleo + latest CentOS-8 compose
* CentOS compose id becomes a promotable artifact by name.
* e.g. CentOS-Stream-8-20210608.n.0
* store / read artifact on http://images.rdoproject.org/centos8/ per branch?

Work required:
* ensure https://composes.centos.org/ is in the upstream allow list for the proxies
* e.g.
* https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/templates/mirror.vhost.j2#L200-L218
* https://review.opendev.org/c/opendev/system-config/+/683240
* ensure we have a tool that read/write compose id by date
* check date of latest -> get the pinned version
* release config reads centos pin and creates pinned centos repos
* works across quickstart, tripleo-repos etc.

digraph hierarchy {

nodesep=0.1
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "get latest pinned compose yyyymmmm"->{ "test in dep pipeline" }
    "test in dep pipeline"->{ "pass dep pipeline?" }
    "pass dep pipeline?"->{ "PASSED dep pipeline" }
    "pass dep pipeline?"->{ "FAILED dep pipeline" }
    "PASSED dep pipeline"->{ "set build to centos-ci-testing"}
    "set build to centos-ci-testing"->{ "Integration line picks up yyyymmmm" }
    "Integration line picks up yyyymmmm"->{ "pass integration?" }
    "pass integration?"->{ "PASSED integration" }
    "pass integration?"->{ "FAILED integration" }
    "PASSED integration"->{ "promote to current-centos-8"}
}

## thoughts w/ rlandy

Integration line:
* periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing
* create the centos rpm lock file at the same time the dlrn-hash is pinned. e.g. a job named
* periodic-tripleo-centos-8-master-promote-centos-ci-testing ( this will be the first job in the line. )
* This creates the rpmlock.list, runs md5sum, creates lock file on image server
* This then becomes a promotion artifact.

2 lists...
* rpmlock.list
* excludes config ( any rpm in this file will NOT get unlocked )

Ideally rpms are locked as early in a zuul job run as possible. e.g. the first tripleo owned zuul pre.yaml

* pre.yaml
* install dnf version
* run rpm -qa
* drop in rpmlock.list

The escape hatch to turn off..
* on the file server.. edit lines or nuke lock list entirely.. Must be outside the tripleo-ci check/gate.

## What about RHEL and OSP
* The same concepts apply to osp pipelines, however we maintain two seperate lists.

## How to road test.
* pick a unimported branch.. try it end to end.

## What work is required...
1. zuul changes
2. job config and pre.yaml that is branch aware
3. install rpmlock.list
2. tool to create the locklist, and md5
3. upload to image server
4. define how to turn off either via specific rpm or globally.
5. Need to design / understand when rpm include/exclude version. for example.. excludes config should override the rpmlock.list
6. promotions now include promotion of the rpmlock.list as an artifact and it's own module.

## Lock modules
* podman

## Alternative approaches

* Alternative approach to promote compose builds, Aim one moving target (compose build) instead of two moving target (openstack builds(tripleo-ci-testing) and compose )
* Primary approach proposed above https://hackmd.io/iYAL3ueUTzWCHE5qFA6jVQ#Proposal-2-CentOS-Stream-Compose-build
* Benefits: Segregation **we don't block current-centos-8 promotion if openstack build promotion is blocked due to Openstack issues**.

digraph hierarchy {

nodesep=0.2
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "Get the latest pinned compose yyyymmmm"->{ "For a branch - Test Compose yyyymmmm in deps line with current-tripleo (Include some jobs we run in check)" }->{ "pass dep pipeline?" }
    "pass dep pipeline?"->{ "PASSED dep pipeline" }
    "pass dep pipeline?"->{ "FAILED dep pipeline" }
    "PASSED dep pipeline"->{ "Promote to current-centos for particular branch (upstream/periodic/component everything pick newcompose)"}
}

* running two integration lines for a branch one centos locked, one unlocked
* locking in a dependency line, integration is unlocked.. and lock file is only used for rollback
* denpency pipeline w/ unlocked centos (w/ current-tripleo ) creates the lock file that feeds the integration line.
* inject lock file at zuul side
* [Sandeep]
1) We create a different module stream like module stream 3-tripleo-ci, and ci team decide which rpm moves from one stream to another stream (which our jobs consumes).
2) Content-provider as yum server

- pin job
- cj
- tj
- tj
- tj
- promote job

~~~
Sharing an alternative thought other than versionlock, as versionlock might be causing deps issue if multiple packages are available.(We will know better once we PIC)

We will have 2 lines :-

* We call our current "Periodic intergration lines" - new deps line
* 2nd line pull older rpm - the new stable "Periodic intergration lines"

Working of second line:-

* We create base like content-provider, which pull the OS rpm we have validated previously from Centos/openstack upstream mirror
* We create ftp/http yum Server on content provider job which serves the validated set of RPMs
* Container-build/image build and all the Consumer jobs pull repos from content-provider/base job

Check job:-

* We fetch list of older working OS rpm.
* On content provider job/ we create another base like content-provider, which pull the OS rpm we have validated previously from Centos/openstack upstream mirror
* We create ftp/http yum Server on content provider job which serves the validated set of RPMs
* Consumer jobs pull repos from content-provider

~~~

### Host our own centos stream mirror..
* configure upstream mirrors to accept our mirror.

### yum modules

### small side repo that contains just the required rpms

### name: upper-rpm-constraints

# Proposed design and next steps for CentOS-8

digraph hierarchy {

nodesep=0.2
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "Get the latest pinned compose yyyymmmm"->{ "For a branch - Test Compose yyyymmmm in deps line with current-tripleo (Include some jobs we run in check)" }->{ "pass dep pipeline?" }
    "pass dep pipeline?"->{ "PASSED dep pipeline" }
    "pass dep pipeline?"->{ "FAILED dep pipeline" }
    "PASSED dep pipeline"->{ "Promote to current-centos for particular branch (upstream/periodic/component everything pick newcompose)"}
}

## Tasks required in order
* Set up images server to handle compose ID
* http://images.rdoproject.org/centos8/centos_compose
* Create job to get latest compose ID and upload to images server (promote-to-centos-ci-testing)
* centos_compose dir
* CentOS-Stream-8-20210617.n.1
* CentOS-Stream-8-20210629.n.0
* current-centos-testing --> CentOS-Stream-8-20210629.n.0
* current-centos --> CentOS-Stream-8-20210617.n.1
* in each a file
* file w/ contain
* doesn't matter if we use a file w/ a link or just use the compose id a file.

          https://composes.centos.org/CentOS-Stream-8-20210706.n.0/

* Pre playbook to get repos setup with latest compose - should be done as soon as we have access ( will require dnf setup)
* use yum-config-module to edit mirror url inline or disable && create new centos-repos. For each repo in the compose... do the action
*

    [DIR]	AppStream/	2021-07-06 23:39	-	 
    [DIR]	BaseOS/	2021-07-06 23:38	-	 
    [DIR]	HighAvailability/	2021-07-06 23:42	-	 
    [DIR]	NFV/	2021-07-06 23:42	-	 
    [DIR]	PowerTools/	2021-07-06 23:42	-	 
    [DIR]	RT/	2021-07-06 23:42	-	 
    [DIR]	ResilientStorage/	2021-07-06 23:42	-	 
    [DIR]	metadata/	2021-07-06 23:51	-	 

* Create dependency jobs to run with ^^ pre playbook
* Add some diff table/tool??
* [link to diff](https://logserver.rdoproject.org/openstack-dependencies-containertools/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-container-tools-container-build-train/d8f485c/logs/undercloud/home/zuul/control_test_diff_table.log.txt.gz)
* link to the code that runs the diff
* Add jobs to line
* Create promote job to promote ci-centos-testing to current-centos? (naming?)
* Update one release to use current-centos in integration line
* Update the current-tripleo of the same release to use current-centos in check
* Update all releases to use current-centos