RDO Components and the Component Promotion Pipeline

# RDO Components and the Component Promotion Pipeline ###### tags: `Design` This document describes how the component promotion pipeline has been designed in a simple form. [TOC] ## Background Info :::spoiler Components and the Component Pipeline (click to expand/collapse) ### What are Components? OpenStack packages are built by DLRN. The RDO team has grouped OpenStack projects e.g. openstack/neutron or openstack/glance and [grouped](https://review.rdoproject.org/r/#/c/22394/) into logical buckets called Components using metadata in [rdoinfo](https://github.com/redhat-openstack/rdoinfo/). Currently there are 15 components: * baremetal * cinder * clients * cloudops * common * compute * glance * manila * network * octavia * security * swift * tempest * tripleo * ui * validations ( to be added soon ) ### What is the component pipeline? The **component promotion pipeline** is a means to consistently deliver validated working components of OpenStack without having to hold back all of OpenStack due to one failing piece. When a component passes it's validation it's added to a group of packages tagged with a label called "promoted-components". Essentially the **promoted-components** are a known good working set of the components in OpenStack. The **component promotion pipeline** is an additional layer to the traditional promotion pipeline that provides a more reliable set of components to the continuous build and delivery of TripleO. ### Motivation In older OpenStack releases a single build is created for all of OpenStack and TripleO. There are hundreds of changes in any given build that can cause a build to fail. This is a large problem to debug and solve, and takes a considerable amount of time to debug. By breaking OpenStack into several smaller components we narrow the reduce the change set from hundreds of changes to just a few and limited to the scope of the OpenStack projects in the component. Additionally one of the goals of the group has been to be able to release a version TripleO at any given time. In order to release at *any time* in a cycle you have to maintain a known good list and builds of OpenStack. The component pipeline achieves this goal by only promoting working builds of each component. Working components proceed through the pipelines while broken components are held back until fixed. ### Goals - [x] Release at any time - [x] Provide more reliable components that can be validated later in the workflow by more rigorous integration tests - [x] Reduce time to investigate issues and fix broken components ### Production View The upstream component pipelines can be found in the following links * [master](http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline) * [ussuri](http://dashboard-ci.tripleo.org/d/BA9imlRMz/component-pipeline-ussuri) ### Glossary | Term | Description | | ------ | -------- | | Single/common hash | A single DLRN build hash that containers all OpenStack and TripleO packages | | Component | A group of packages or services (e.g. compute, keystone, etc.) | Component Pipeline | A new pipeline that validates and promotes components individually | | Component promotion jobs | A set of CI jobs that validate and promote an individual component | | Integration Pipeline | A pipeline of jobs that validate all the individually promoted components together to ensure tha latest components work together | | Integration jobs | The set of jobs in the Integration Pipeline. Any voting upstream job is required to be an integration job | ::: ## Completed Work * A component pipeline for the master branch * A component pipeline for the ussuri branch * An internal master/osp-17 pipeline * Monitoring ## Further Development * add a validations component * add a pipeline for stable/train / OSP-16 * Update the DLRN dependency repositories into components * https://trunk.rdoproject.org/centos8-master/deps/latest/ * https://trunk.rdoproject.org/centos8-master/build-deps/latest/ * 3rd party dependency rpms into components * libpod / podman, buildah * openvswitch * etc ## Overview The new promotion pipeline is composed by component and integration promotions: ```graphviz digraph hierarchy { nodesep=0.1 node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box] edge [color=darkgreen] "compute/consistent"->{ "compute/component-ci-testing" } "security/consistent"->{ "security/component-ci-testing" } "cinder/consistent"->{ "cinder/component-ci-testing" } "compute/component-ci-testing"->{ "compute/promoted-components" } "cinder/component-ci-testing"->{ "cinder/promoted-components" } "security/component-ci-testing"->{ "security/promoted-components" } node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box] edge [color=darkgreen] "compute/promoted-components"->{ "promoted-components" } "cinder/promoted-components"->{ "promoted-components" } "security/promoted-components"->{ "promoted-components" } node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box] edge [color=darkgreen,style=normal] "tripleo-ci-testing" ->{ "current-tripleo"} "promoted-components"->{ "tripleo-ci-testing"} } ``` :red_circle: *<component>/consistent*, *<component>/component-ci-testing*, *<component>/promoted-components* labels are to the DLRN yum repos :large_blue_circle: *promoted-components* replaces *consistent* from the old promotion pipeline :black_circle: *tripleo-ci-testing* and *current-tripleo* remains the same from the existing promotion pipeline :::info A real time view into the above promotions can be found here: * [high level view of all promotions](http://dashboard-ci.tripleo.org/d/QrkFP-zGz/upstream-and-rdo-promotions?orgId=1) * [component level promotions](http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline) ::: ### Another example with failing components Example of a security failing, while the latest compute and network patches flow through. ```graphviz digraph hierarchy { nodesep=0.1 node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box] edge [color=darkgreen] "compute/consistent"->{ "compute/component-ci-testing" } "security/consistent"->{ "security/component-ci-testing" } "network/consistent"->{ "network/component-ci-testing" } "compute/component-ci-testing"->{ "compute/promoted-components" } "network/component-ci-testing"->{ "network/promoted-components" } "security/component-ci-testing"->{ "security/promoted-components" [color=darkred,fontcolor=darkred,shape=box3d] } node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box] edge [color=darkgreen] "compute/promoted-components"->{ "promoted-components" } "network/promoted-components"->{ "promoted-components" } node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box] edge [color=darkgreen,style=normal] "tripleo-ci-testing" ->{ "current-tripleo"} "promoted-components"->{ "tripleo-ci-testing"} } ``` ### Zuul Pipelines The component pipeline is broken down into separate Zuul pipelines per component: - openstack-component-compute - openstack-component-keystone - openstack-component-cinder - openstack-component-[...] https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-compute ![](https://i.imgur.com/uQtmox7.png) The job that promotes to *promoted-components* hash runs in a separate pipeline as well: - openstack-promote-component https://review.rdoproject.org/zuul/builds?pipeline=openstack-promote-component ![](https://i.imgur.com/ZxO0nCp.png) ## Design Component promotion pipeline consumes the individual built component repos containing unique hashes, run a set of validation tests, and promotes each component hash to feed the integration pipeline. ```graphviz digraph hierarchy { nodesep=1.5 node [color=Red,fontname=Courier,shape=box] edge [color=Blue, style=dashed] build_md5->{compute keystone cinder etc} compute->{ hash_x } keystone->{ hash_y } cinder->{ hash_w } etc->{ hash_z } } ``` :::info :information_source: **build_md5** :id: is a MD5 hash of [delorean.repo](https://trunk-staging.rdoproject.org/centos7/current-tripleo/delorean.repo) file containing all promoted components with their respective repository hashes. ::: ### Components build Components are individually built and placed (by the infra team) in a repository at https://trunk.rdoproject.org/centos8-master/component/ https://trunk.rdoproject.org/centos8-master/component/ \ compute keystone ... ### Promotion pipeline workflow :one: Get the latest build for each component from the repositories tagged as "consistent" :two: Pin the latest built by promoting its hash to **component-ci-testing** :three: Run component *validation jobs*[^component_jobs] and report result to delorean api [^component_jobs]: **validation jobs**: CI jobs in the Zuul pipeline that compose the criteria to promote a individual component, such as container build, standalone, etc. :four: Check promotion criteria :five: Promote component build repository to **promoted-components** hash if criteria is met. Discard build otherwise. ```flow start=>start: Start end=>end: End consistent=>operation: consistent component-ci-testing=>operation: component-ci-testing meet-criteria=>condition: meet promotion criteria ? promoted-components=>operation: promoted-components start->consistent->component-ci-testing->meet-criteria meet-criteria(yes)->promoted-components promoted-components->end meet-criteria(no)->end ``` ### Promotion hashes ```graphviz digraph hierarchy { nodesep=0.1 node [color=blue,shape=oval,style=dashed] edge [color=red] "consistent (compute)"->{ "component-ci-testing (compute)"} "consistent (keystone)"->{ "component-ci-testing (keystone)"} "consistent (common)"->{ "component-ci-testing (common)"} "component-ci-testing (compute)"->{ "promoted-components" } "component-ci-testing (keystone)"->{ "promoted-components" } "component-ci-testing (common)"->{ "promoted-components" } } ``` > Individual components are promoted from unique label *consistent* (pointing to different hashes) to a common *promoted-components* label (pointing to a common build id - a md5 hash generated for the repo file containing all component promoted hashes). :::info consistent >> component-ci-testing >> promoted-components #### consistent Each component built will get its own repository setup and an unique build hash on https://trunk.rdoproject.org/centos8-master/component/compute/consistent/ #### component-ci-testing Before any component promotion job runs, the consistent hash is pinned to *component-ci-testing* hash, so the build can be consumed even if a new build updates consistent hash. Component promotion jobs run and validate the built component with a stable version of all other components. #### promoted-components Components that successfully run the promotion jobs are promoted to *promoted-components* hash. For example, for *compute* component, delorean promotes by naming the repository location as follows: https://trunk.rdoproject.org/centos8-master/component/compute/promoted-components Delorean also creates the consolidated repo for all promoted components, to be placed at https://trunk.rdoproject.org/centos8-master/promoted-components/delorean.repo ``` [delorean-component-common] name=delorean-python-vmware-nsxlib-27d4662f90008790da824942a835c4f861973e61 baseurl=https://trunk-staging.rdoproject.org/centos7/component/common/27/d4/27d4662f90008790da824942a835c4f861973e61_462060d5 enabled=1 gpgcheck=0 priority=1 [delorean-component-compute] name=delorean-openstack-nova-ee16ae1b39962c8d07467326a0108606433f2280 baseurl=https://trunk-staging.rdoproject.org/centos7/component/compute/ee/16/ee16ae1b39962c8d07467326a0108606433f2280_a545aa4b enabled=1 gpgcheck=0 priority=1 ``` ::: ### Jobs #### Promote consistent to component-ci-testing The consistent hash of a component repository is promoted to *component-ci-testing* to pin the hash for running the promotion jobs with the same hash, as new hashes may be generated with new builds. #### Component (validation) jobs For now standalone is the only validation job in the component pipeline. The job overrides the stable containers w/ the new built ones to be tested for a single component. #### Promote component-ci-testing to *promoted-components* The component is promoted by a Zuul job in an independent pipeline. It runs the [promote-hash role](https://github.com/rdo-infra/review.rdoproject.org-config/tree/master/roles/promote-hash/), checks the promotion criteria with the delorean api[^dlrnapi], and promotes the hash to *promoted-components*. [^dlrnapi]: **delorean api**: https://trunk-staging.rdoproject.org/api-centos-master-uc Component promotion sequence: ```sequence promotion-->delorean api: get reported jobs for hash 'xyz' Note right of delorean api: query delorean db delorean api->promotion: json containing reported jobs Note left of promotion: check promotion criteria promotion->delorean api: promote hash to *promoted-component* ``` ------ ## Speculatively test two unpromoted components together There are times when one component will not promote until another component promotes due to cross rpm dependencies. One could force a promotion of one of the components to unlock the dependency. A better way is to test two components together. * In this example we're taking the tripleo component job and adding the clients component ( component-ci-testing ) to the tripleo component job. ``` - project: check: jobs: - periodic-tripleo-ci-centos-8-standalone-tripleo-secondary-clients-master - job: name: periodic-tripleo-ci-centos-8-standalone-tripleo-secondary-clients-master parent: periodic-tripleo-ci-centos-8-standalone-tripleo-master vars: add_repos: - type: generic reponame: "{{ component }}" filename: "{{ component }}-component.repo" priority: 1 baseurl: "https://{{ dlrn_server_url }}/centos8-{{ release }}/component/{{ component }}/component-ci-testing" update_container: true - type: generic reponame: "clients" filename: "clients-component.repo" priority: 1 baseurl: "https://{{ dlrn_server_url }}/centos8-{{ release }}/component/clients/component-ci-testing" update_container: true ``` ## Implications to the Promotion Pipeline The promotion pipeline is now broken down into 2 parts: - component pipeline (the new layer added to promote individual components) - integration pipeline (former promotion pipeline) > The component pipeline feeds the integration pipeline w/ all the components combined in a single build. Each individual component points to a separate repository containing an unique hash. ```flow component=>start: COMPONENT PIPELINE integration=>end: INTEGRATION PIPELINE component->integration ``` #### A higher level view of the internal workflow with component and integration can be found here [link](https://docs.google.com/drawings/d/1KcKh-WzRNrLAVF70w1aXtBDlaVp_TyVdJbxxd4sMSi4/edit) ### commit_hash vs build_id #### commit_hash The commit_hash uniquely identifies a single component repository. It preserves the same functionality from the former *promotion pipeline*, but only for a single component. This implies in modifications to how jobs parse commit.yaml to get the hash that identifies the repository to be used. https://trunk.rdoproject.org/centos8-master/component/compute/0e/d0/0ed00643bff0002f04288e4e208c88ed614f111f_c9bfa013/ :::warning :warning: **commit_hash** has a completely different meaning in the new promotion pipeline, and no longer represents the consolidated repository w/ all packages. ::: Jobs from integration pipeline won't get [commit.yaml](https://trunk.rdoproject.org/centos8-master/component/compute/0e/d0/0ed00643bff0002f04288e4e208c88ed614f111f_c9bfa013/commit.yaml) to setup repos or parse delorean.repo to upload images. Instead, they will use *build_id* to identify a group of components promoted and ready to consume. #### build_id build_id is a md5 hash generated to uniquely identify a build of components collection. It is generated everytime a component is promoted. The [delorean.repo](https://trunk-staging.rdoproject.org/centos7/current-tripleo/delorean.repo) file is updated with the hash of the promoted component and a new md5 build_id is created. In the integration pipeline, instead of consuming *commit_hash* from commit.yaml file, jobs need to parse a new **build.yaml** file containing the *build_id* with the consolidated repository with all promoted components. Assumptions: - infra needs to support delorean report jobs w/ --build-id option (that replaces commit_hash) - infra needs to generate build_id md5 - ci needs to update get-hash, upload-cloud-images, and other parts of the code that refer to commit.yaml and commit_hash. ### Spec for promoting promoted-components -> tripleo-ci-testing (spec by wes hayutin, owner rlandy) ::: info A job will handle the promotion of promoted-components -> tripleo-ci-testing. Promotions for current-tripleo and current-tripleo-rdo will be handled by the promotion-server ::: #### [When md5_url is defined](https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/get_hash/tasks) md5_url is the key that trigger promoting all the components indivdually at once to tripleo-ci-testing. * In a zuul unprotected repo create a task that creates the information required to promote all the components at once. As discussed a new task in [https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/get_hash/tasks](https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/get_hash/tasks) * Information required for the promotion. The information will be consumed by the [Promote hash label task](https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-primary-distro.yaml) and [promote_hash.sh](https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/promote-hash.sh) script. rfolco, rlandy, and wes discussed the following.. * Do as much work as possible in unprotected repos. * curl/fetch the [promoted-components/delorean.repo](https://trunk.rdoproject.org/centos8-master/promoted-components/delorean.repo) file * Extract the full dlrn_hash for each component * Create a file named $component for each component. Filename: baremetal ( match the [name](https://trunk.rdoproject.org/centos8-master/component/)) * In each component file * export COMPONENT_NAME ( for logging ) * export COMMIT_HASH * export DISTRO_HASH :::info A file per component will allow us to loop and source the component information easily. ::: #### Loop through components and promote :::info An example of how this works can be found in the [gist](https://gist.github.com/weshayutin/c77a2aa33c0d7a4c9b93481dc2f789f4) ::: :::info See: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-primary-distro.yaml for details ::: * Create a version of the task Promote hash label for two use cases. 1. When a single hash ( centos-7 ) * this should only need an additional when: md5_url not defined 2. When md5_url is defined ( centos-8 ) * loop through each component w/ a list of components in ansible * source the appropriate file * execute the tasks as currently written in [source](https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-primary-distro.yaml#L6-L16) #### MOLECULE ( test it), owner: rfolco ## Monitoring * master: http://dashboard-ci.tripleo.org/d/M1DYSidGz/component-pipeline?orgId=1 * ussuri: http://dashboard-ci.tripleo.org/d/MhDLSiOGz/component-pipeline-ussuri?orgId=1 * train: http://dashboard-ci.tripleo.org/d/mOvYIiOMk/component-pipeline-train?orgId=1

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.