Designcentos-stream-8 updates have now broken upstream and 3rd party for the second time in 3 weeks. We can expect more issues in the future. We can also expect A LOT MORE churn and bugs w/ centos-stream-9.
We were executing a centos-stream-8 dependency pipeline to test the conversion of centos-8 to centos-stream-8. We were also investigating a pending or staging repo for centos-stream-8. After asking rhos-ops and directly to the centos maintainers no such pending/staging repo is available for centos-stream releases. This broke our pattern for the dependency pipeline and the dependency pipeline was abandoned for centos-stream.
https://review.rdoproject.org/r/c/testproject/+/31586 https://review.opendev.org/c/openstack/tripleo-quickstart/+/796536 Update centos 8 dep pipeline to run with compose repos
Sandeep employeed dnf's plugin version lock for a couple baseos / appstream bugs encountered in the pervious ruck/rover sprint. Now is a good time to explore again the centos-stream dependency pipeline with version lock to stabilize upstream and 3rd party while watching for the latest changes that break OpenStack.
Installed:
python3-dnf-plugin-versionlock-4.0.21-1.fc33.noarch
Complete!
[root@localhost ~]# dnf versionlock add kernel-5.12.8-200.fc33.x86_64
Adding versionlock on: kernel-0:5.12.8-200.fc33.*
[root@localhost ~]# cat /etc/dnf/
aliases.d/ dnf.conf modules.d/ modules.defaults.d/ plugins/ protected.d/ vars/
[root@localhost ~]# cat /etc/dnf/plugins/
copr.conf copr.d/ debuginfo-install.conf versionlock.conf versionlock.list
[root@localhost ~]# cat /etc/dnf/plugins/versionlock.list
# Added lock on Mon Jun 7 11:54:24 2021
kernel-0:5.12.8-200.fc33.*
Build a versionlock.list (manifest) of known good centos packages from the logs of our integration jobs. Create a single file from all our jobs that contain packages from non delorean repos.
In each upstream and 3rd party and periodic job including integration and component.
The CentOS-Stream dependency pipeline would be NOT run with the version lock and would be responsible for updating the versionlock.list file if all the jobs pass.
The new manifest of packages would be added to each integration pipeline and promoted as part of each branches promotion.
This way as we promote a branch integration we would be promoting a set of known good openstack packages, AND centos packages along side the containers and overcloud images.
Take advantage of the CentOS compose build ID and lock to a specific CentOS compose while always testing the latest compose in a dependency pipeline.
Work required:
digraph hierarchy {
nodesep=0.1
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
"get latest pinned compose yyyymmmm"->{ "test in dep pipeline" }
"test in dep pipeline"->{ "pass dep pipeline?" }
"pass dep pipeline?"->{ "PASSED dep pipeline" }
"pass dep pipeline?"->{ "FAILED dep pipeline" }
"PASSED dep pipeline"->{ "set build to centos-ci-testing"}
"set build to centos-ci-testing"->{ "Integration line picks up yyyymmmm" }
"Integration line picks up yyyymmmm"->{ "pass integration?" }
"pass integration?"->{ "PASSED integration" }
"pass integration?"->{ "FAILED integration" }
"PASSED integration"->{ "promote to current-centos-8"}
}
Integration line:
2 lists…
Ideally rpms are locked as early in a zuul job run as possible. e.g. the first tripleo owned zuul pre.yaml
The escape hatch to turn off.. * on the file server.. edit lines or nuke lock list entirely.. Must be outside the tripleo-ci check/gate.
digraph hierarchy {
nodesep=0.2
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
"Get the latest pinned compose yyyymmmm"->{ "For a branch - Test Compose yyyymmmm in deps line with current-tripleo (Include some jobs we run in check)" }->{ "pass dep pipeline?" }
"pass dep pipeline?"->{ "PASSED dep pipeline" }
"pass dep pipeline?"->{ "FAILED dep pipeline" }
"PASSED dep pipeline"->{ "Promote to current-centos for particular branch (upstream/periodic/component everything pick newcompose)"}
}
Sharing an alternative thought other than versionlock, as versionlock might be causing deps issue if multiple packages are available.(We will know better once we PIC)
We will have 2 lines :-
* We call our current "Periodic intergration lines" - new deps line
* 2nd line pull older rpm - the new stable "Periodic intergration lines"
Working of second line:-
* We create base like content-provider, which pull the OS rpm we have validated previously from Centos/openstack upstream mirror
* We create ftp/http yum Server on content provider job which serves the validated set of RPMs
* Container-build/image build and all the Consumer jobs pull repos from content-provider/base job
Check job:-
* We fetch list of older working OS rpm.
* On content provider job/ we create another base like content-provider, which pull the OS rpm we have validated previously from Centos/openstack upstream mirror
* We create ftp/http yum Server on content provider job which serves the validated set of RPMs
* Consumer jobs pull repos from content-provider
* configure upstream mirrors to accept our mirror.
digraph hierarchy {
nodesep=0.2
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
"Get the latest pinned compose yyyymmmm"->{ "For a branch - Test Compose yyyymmmm in deps line with current-tripleo (Include some jobs we run in check)" }->{ "pass dep pipeline?" }
"pass dep pipeline?"->{ "PASSED dep pipeline" }
"pass dep pipeline?"->{ "FAILED dep pipeline" }
"PASSED dep pipeline"->{ "Promote to current-centos for particular branch (upstream/periodic/component everything pick newcompose)"}
}
https://composes.centos.org/CentOS-Stream-8-20210706.n.0/
[DIR] AppStream/ 2021-07-06 23:39 -
[DIR] BaseOS/ 2021-07-06 23:38 -
[DIR] HighAvailability/ 2021-07-06 23:42 -
[DIR] NFV/ 2021-07-06 23:42 -
[DIR] PowerTools/ 2021-07-06 23:42 -
[DIR] RT/ 2021-07-06 23:42 -
[DIR] ResilientStorage/ 2021-07-06 23:42 -
[DIR] metadata/ 2021-07-06 23:51 -
-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
Subscribe