# RHOS 16.2 Component and Integration Pipelines
###### tags: `Design`

## Team sync meeting points
* timelines
* basic status
* share dependencies
* steps required before next sync call

## Design and steps

Order of operations 16.2:

### mini-integration
- [x] container builds ( jobs )
- [x] overcloud image builds ( jobs )
- [x] ONE standalone basic job
- [x] ATTEMPT passing not required fs001

### component pipeline
- [x] create standalone job to each component.
- [x] ensure this one job is in criteria

### add other jobs
1. Add the rest of the jobs as you can to both integration and component pipeline.
2. Use the same job set found in rhos-17 integration and component

## Open reviews

### Upstream
* ~~https://review.opendev.org/#/c/744786/ ~~
* ~~https://review.opendev.org/#/c/744166/~~ WIP: Add container excludes for rhos-16.2 builds
* ~~https://review.opendev.org/#/c/744562/ Separate required-projects and roles from base jobs~~
* ~~https://review.opendev.org/#/c/745346/~~ fix openstack collections in tq

* https://review.opendev.org/#/c/743432/ native container builds in standalone
* READY FOR REVIEW
* ~~https://review.rdoproject.org/r/#/c/28775/~~ test cr repos WIP: Add dependency job for centos 8 CR repo

* ~~Waiting on Train CentOS 8 container patches to merge and reach downstream repos (in upstream gates)~~
~~https://review.opendev.org/#/c/736752/
Implement a new Container Images layout
https://review.opendev.org/#/c/742208/
Revert "Disable ubi8 job on train"
https://review.opendev.org/#/c/740095/
Add openstack tripleo container image build~~
* ~~https://review.rdoproject.org/r/28730 Add rhos-16.2 defaults to get_hash role~~

### Downstream
**CONTAINERS**
* ~~https://code.engineering.redhat.com/gerrit/#/c/207802/ Make container jobs for 16.2 use overrides~~
* ~~https://code.engineering.redhat.com/gerrit/#/c/208964/ Add shadowing jobs in zuul.d/required-projects-overrides.yaml~~
* ~~https://code.engineering.redhat.com/gerrit/#/c/195884/ Test container push~~

**STANDALONE**
* ~~https://code.engineering.redhat.com/gerrit/#/c/208575/ Periodic 16.2 standalone job~~

* ~~https://code.engineering.redhat.com/gerrit/#/c/209162/ -
Add rhel-8.3 beta repo to 16.2 release files~~
* ~~https://code.engineering.redhat.com/gerrit/#/c/209530/~~ Add sc007 component and integration jobs in 16.2

**OVB**
* ~~https://code.engineering.redhat.com/gerrit/#/c/209571/~~ WIP: Add rhos-16.2 OVB job

**COMPONENT**

~~Update 16.2 component jobs and add them to template https://code.engineering.redhat.com/gerrit/209243~~

**PIPELINE**
~~* Add component-pipeline-rhos-16.2 templates https://code.engineering.redhat.com/gerrit/#/c/209387/~~

~~* https://code.engineering.redhat.com/gerrit/#/c/209208/ Add integration pipeline template for rhos 16.2~~
~~* https://code.engineering.redhat.com/gerrit/#/c/209209/ Add rhos-16.2 integration pipeline~~

~~* https://bugzilla.redhat.com/show_bug.cgi?id=1869379 bug on the 16.2 standalone - fixed by latest rpms~~

~~* https://review.opendev.org/#/c/745819/ Generalize vars in template file in build-test-packages role~~

* ~~https://code.engineering.redhat.com/gerrit/#/q/topic:prep-16-2+(status:open+OR+status:merged)
container build and push/upload
image build/push
RHOS-16.2 release files and criteria files~~
* ~~https://code.engineering.redhat.com/gerrit/207509 Add component jobs for rhos-16.2~~
* ~~https://code.engineering.redhat.com/gerrit/#/c/208642/ - Temp add missing packages - awaiting import and use virt:8.3 instead of virt:8.2~~
* ~~https://code.engineering.redhat.com/gerrit/#/c/208453/ Remove rhos-17 hard coded vlaues from build-image~~

## Open bugs:
* https://bugzilla.redhat.com/show_bug.cgi?id=1867132 Buildah package upgrade from 1.14.9 ->1.15.x for rhos-16.2-rhel-8-trunk-build
* https://bugzilla.redhat.com/show_bug.cgi?id=1867133 Downstream repo needed for ansible-collections-openstack
* https://github.com/containers/buildah/issues/2521
* https://bugzilla.redhat.com/show_bug.cgi?id=1784001 - We are hitting similiar issue on Rhel8.3
* https://bugzilla.redhat.com/show_bug.cgi?id=1828681
* https://bugzilla.redhat.com/show_bug.cgi?id=1869379 [OSP 16.2] Containers fail to start for step 2 - mysql_init_bundle error

* Latest podman(podman.x86_64-1.6.4-13) is causing deployment failure, Using older - podman.x86_64 1.6.4-12.module+el8.2.0+6670+014d0ff8 to get pass the issue - **Need to open bug**

## Required directories/namespaces for 16.2
* new namespace on Internal Upshift registry: https://url.corp.redhat.com/e32bba0
/tripleorhos-17 - RHOS 17 containers (in use now)
~~/tripleorhos-16-2 - RHOS 16.2 containers~~
* ~~images and dockerfile uploads on internal images server
ADDED~~

## Naming convention (16.2/16_2/16-2)
* https://url.corp.redhat.com/0db0776 DLRN rhel8-osp16-2/
* https://url.corp.redhat.com/e14f53e Trunk candidate rhos-16.2-rhel-8-trunk-candidate/

## Dependency issues
* jjoyce> rlandy: 'brew list-pkgs --tag rhos-16.2-rhel-8-trunk-build' would give you a list of what we have. Let me find the url
shreshtha> rlandy, you can also compare tag-inheritance for 16.2 and 17.0 to get answer of your initial question
shreshtha> brew list-tag-inheritance rhos-16.2-rhel-8-build has rhel-8.3.0-z-build-base whereas 17.0 has rhel-8-build-base
jjoyce> In theory 17.0 and 16.2 should be using the same rhel version

### Dependency versions per pipeline
* container tools 8.2 (until 8.3) in integration/component lines
* containers tools from rhel8 stream in dependency pipeline

## Other Issues

* RESOLVED: With rhel8.3 - image build job failing with below error -

failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory[3].(Same job is working with rhel8.2)

https://sf.hosted.upshift.rdu2.redhat.com/logs/10/207510/14/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-16.2/2e91c58/logs/build.log.txt.gz

We are hitting similiar issue on Rhel8.3

https://bugzilla.redhat.com/show_bug.cgi?id=1784001

## RHEL 8.3 nodepool node
apevec> 8.3 should be in snapshots phase
apevec> next one ES2 compose is Aug 25
apevec> it goes to CDN for HTB customers so should be reasonable stable
apevec> publish Sep 2
rlandy> so worth building a rhel 8.3 node?
apevec> I'd say not before ES2
apevec> let's start doing it Aug 26, that's the official "layered product start testing
* ~~https://code.engineering.redhat.com/gerrit/210130~~ Add rhel8.3-cloud.yaml for RHEL 8.3 nodepool node

## Working jobs:
* promotion jobs
* image builds
* container build/push/upload
* Standalone job
* fs001 running on OVB
* scenario job 007, 004

## Ensure imports from upstream -> 16.2
* should use https://trunk.rdoproject.org/centos8-train/promoted-components/

## Dependency Pipeline

### CENTOS use centos8-stream vs. cr
http://mirror.den01.meanservers.net/centos/8-stream/
http://cloud.centos.org/centos/8-stream/x86_64/images/

### RHEL
Isolate the base and various important dependencies.

* baseos 8.2 -> 8.3 -> 8.4
* ansible
* http://download-node-02.eng.bos.redhat.com/rhel-8/nightly/ANSIBLE/latest-ANSIBLE-2.9-RHEL-8/compose/Base/x86_64/os/
* libpod,podman,buildah
* latest 8.2. compose -> app-stream
* module:
* container-tools:8.2 --> 8.3
* both are contained in the repo below, need adjust the config.
* http://download-node-02.eng.bos.redhat.com/rhel-8/rel-eng/RHEL-8/latest-RHEL-8.3/compose/AppStream/x86_64/os/
* adv-virt
* http://download-node-02.eng.bos.redhat.com/rhel-8/nightly/ADVANCED-VIRT-8/latest-ADVANCED-VIRT-8.1.1-RHEL-8/compose/Advanced-virt/x86_64/os/
* pacemaker
* http://download-node-02.eng.bos.redhat.com/rhel-8/rel-eng/RHEL-8/latest-RHEL-8.2/compose/HighAvailability/x86_64/os/
* openvswitch / ovn
* http://download-node-02.eng.bos.redhat.com/rhel-8/nightly/FDP/latest-FDP-8-RHEL-8/compose/Server/x86_64/os/
* the dependency-repos
* puppet?

Stable RHEL 8.1 - 16.1

as 8.2 is approaching...

* base-os repo 8.1.0 stream, 8.1.1 stream ( looks like centos cr. )

These can be two different pipelines:
AppStreams:
* buildah/podman
* python-depencies
* http://download-node-02.eng.bos.redhat.com/rhel-8/rel-eng/RHEL-8/latest-RHEL-8.3/compose/AppStream/x86_64/os/
* https://docs.engineering.redhat.com/display/PRODCHAIN/OSP+16.1+Repo+and+module+configuration

#### DESIGN - repo setup for the dep-pipeline

-e @tripleo-env/config/release/tripleo-ci/rhel16.2
+
-e @tripleo-env/config/release/dep_pipeline/RHEL/8.2/container-tools/2.0.yaml
-e @tripleo-env/config/release/dep_pipeline/RHEL/8.2/container-tools/rhel8.yaml

#### TASKS - remove any 3rd party dependency jobs
* Once container-tools jobs are running, we no longer need buildah/podman 3rd party jobs.
* Remove these 3rd party jobs as the last stage of setting up the dependency lines.

#### Diff script design
* Where does the script run and where is it displayed
Suggestion - actually run the diff within the job and store the output in the logs but have a script that we can run independently
Output the diff file link in the cockpit
* What does the diff script display?
- The difference between the two lists ONLY
- Not a combined list of thw two outputs
- If a file exists in only one OR if it exists as a different version
* Keep the output file naming consistent
- control list
- dependency list
* Deciding if there is a difference in the versions:
- rpm out looks as follows: {name}-{number/letter sequence}
- pattern split:

import re
package_name = re.split('(-[0-9]+)', i)[0]

### CENTOS (usptream)