RDO Components and the Component Promotion Pipeline

tags: `Design`

This document describes how the component promotion pipeline has been designed in a simple form.

RDO Components and the Component Promotion Pipeline

Background Info

Components and the Component Pipeline (click to expand/collapse)

What are Components?

OpenStack packages are built by DLRN. The RDO team has grouped OpenStack projects e.g. openstack/neutron or openstack/glance and grouped into logical buckets called Components using metadata in rdoinfo.

Currently there are 15 components:

baremetal
cinder
clients
cloudops
common
compute
glance
manila
network
octavia
security
swift
tempest
tripleo
ui
validations ( to be added soon )

What is the component pipeline?

The component promotion pipeline is a means to consistently deliver validated working components of OpenStack without having to hold back all of OpenStack due to one failing piece. When a component passes it's validation it's added to a group of packages tagged with a label called "promoted-components". Essentially the promoted-components are a known good working set of the components in OpenStack.

The component promotion pipeline is an additional layer to the traditional promotion pipeline that provides a more reliable set of components to the continuous build and delivery of TripleO.

Motivation

In older OpenStack releases a single build is created for all of OpenStack and TripleO. There are hundreds of changes in any given build that can cause a build to fail. This is a large problem to debug and solve, and takes a considerable amount of time to debug. By breaking OpenStack into several smaller components we narrow the reduce the change set from hundreds of changes to just a few and limited to the scope of the OpenStack projects in the component.

Additionally one of the goals of the group has been to be able to release a version TripleO at any given time. In order to release at any time in a cycle you have to maintain a known good list and builds of OpenStack. The component pipeline achieves this goal by only promoting working builds of each component. Working components proceed through the pipelines while broken components are held back until fixed.

Goals

Release at any time
Provide more reliable components that can be validated later in the workflow by more rigorous integration tests
Reduce time to investigate issues and fix broken components

Production View

The upstream component pipelines can be found in the following links

Glossary

Term	Description
Single/common hash	A single DLRN build hash that containers all OpenStack and TripleO packages
Component	A group of packages or services (e.g. compute, keystone, etc.)
Component Pipeline	A new pipeline that validates and promotes components individually
Component promotion jobs	A set of CI jobs that validate and promote an individual component
Integration Pipeline	A pipeline of jobs that validate all the individually promoted components together to ensure tha latest components work together
Integration jobs	The set of jobs in the Integration Pipeline. Any voting upstream job is required to be an integration job

Completed Work

A component pipeline for the master branch
A component pipeline for the ussuri branch
An internal master/osp-17 pipeline
Monitoring

Further Development

add a validations component
add a pipeline for stable/train / OSP-16
Update the DLRN dependency repositories into components
- https://trunk.rdoproject.org/centos8-master/deps/latest/
- https://trunk.rdoproject.org/centos8-master/build-deps/latest/
3rd party dependency rpms into components
- libpod / podman, buildah
- openvswitch
- etc

Overview

The new promotion pipeline is composed by component and integration promotions:

digraph hierarchy {

nodesep=0.1
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "compute/consistent"->{ "compute/component-ci-testing" }
    "security/consistent"->{ "security/component-ci-testing" }
    "cinder/consistent"->{ "cinder/component-ci-testing" }
    "compute/component-ci-testing"->{ "compute/promoted-components" }
    "cinder/component-ci-testing"->{ "cinder/promoted-components" }
    "security/component-ci-testing"->{ "security/promoted-components" }
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "compute/promoted-components"->{ "promoted-components" }
    "cinder/promoted-components"->{ "promoted-components" }
    "security/promoted-components"->{ "promoted-components" }
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen,style=normal]
    "tripleo-ci-testing" ->{ "current-tripleo"}
    "promoted-components"->{ "tripleo-ci-testing"}

}

<component>/consistent, <component>/component-ci-testing, <component>/promoted-components labels are to the DLRN yum repos
promoted-components replaces consistent from the old promotion pipeline
tripleo-ci-testing and current-tripleo remains the same from the existing promotion pipeline

A real time view into the above promotions can be found here:

Another example with failing components

Example of a security failing, while the latest compute and network patches flow through.

digraph hierarchy {

nodesep=0.1
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "compute/consistent"->{ "compute/component-ci-testing" }
    "security/consistent"->{ "security/component-ci-testing" }
    "network/consistent"->{ "network/component-ci-testing" }
    "compute/component-ci-testing"->{ "compute/promoted-components" }
    "network/component-ci-testing"->{ "network/promoted-components" }
    "security/component-ci-testing"->{ "security/promoted-components" [color=darkred,fontcolor=darkred,shape=box3d] }
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "compute/promoted-components"->{ "promoted-components" }
    "network/promoted-components"->{ "promoted-components" }

node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen,style=normal]
    "tripleo-ci-testing" ->{ "current-tripleo"}
    "promoted-components"->{ "tripleo-ci-testing"}

}

Zuul Pipelines

The component pipeline is broken down into separate Zuul pipelines per component:

openstack-component-compute
openstack-component-keystone
openstack-component-cinder
openstack-component-[…]

https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-compute

The job that promotes to promoted-components hash runs in a separate pipeline as well:

openstack-promote-component

https://review.rdoproject.org/zuul/builds?pipeline=openstack-promote-component

Design

Component promotion pipeline consumes the individual built component repos containing unique hashes, run a set of validation tests, and promotes each component hash to feed the integration pipeline.

digraph hierarchy {

nodesep=1.5
node [color=Red,fontname=Courier,shape=box]
edge [color=Blue, style=dashed]

    build_md5->{compute keystone cinder etc}
    compute->{ hash_x }
    keystone->{ hash_y }
    cinder->{ hash_w }
    etc->{ hash_z }
}

build_md5 is a MD5 hash of delorean.repo file containing all promoted components with their respective repository hashes.

Components build

Components are individually built and placed (by the infra team) in a repository at https://trunk.rdoproject.org/centos8-master/component/

https://trunk.rdoproject.org/centos8-master/component/
\
 compute
 keystone
 ...

Promotion pipeline workflow

Get the latest build for each component from the repositories tagged as "consistent"

Pin the latest built by promoting its hash to component-ci-testing

Run component validation jobs^[1] and report result to delorean api

Check promotion criteria

Promote component build repository to promoted-components hash if criteria is met. Discard build otherwise.

start=>start: Start
end=>end: End
consistent=>operation: consistent
component-ci-testing=>operation: component-ci-testing

meet-criteria=>condition: meet
promotion
criteria ?
promoted-components=>operation: promoted-components

start->consistent->component-ci-testing->meet-criteria
meet-criteria(yes)->promoted-components
promoted-components->end
meet-criteria(no)->end

Promotion hashes

digraph hierarchy {

nodesep=0.1
node [color=blue,shape=oval,style=dashed]
edge [color=red]

    "consistent (compute)"->{ "component-ci-testing (compute)"}
    "consistent (keystone)"->{ "component-ci-testing (keystone)"}
    "consistent (common)"->{ "component-ci-testing (common)"}
    "component-ci-testing (compute)"->{ "promoted-components" }
    "component-ci-testing (keystone)"->{ "promoted-components" }
    "component-ci-testing (common)"->{ "promoted-components" }
}

Individual components are promoted from unique label consistent (pointing to different hashes) to a common promoted-components label (pointing to a common build id - a md5 hash generated for the repo file containing all component promoted hashes).

consistent >> component-ci-testing >> promoted-components

consistent

Each component built will get its own repository setup and an unique build hash on
https://trunk.rdoproject.org/centos8-master/component/compute/consistent/

component-ci-testing

Before any component promotion job runs, the consistent hash is pinned to component-ci-testing hash, so the build can be consumed even if a new build updates consistent hash.

Component promotion jobs run and validate the built component with a stable version of all other components.

promoted-components

Components that successfully run the promotion jobs are promoted to promoted-components hash.

For example, for compute component, delorean promotes by naming the repository location as follows:
https://trunk.rdoproject.org/centos8-master/component/compute/promoted-components

Delorean also creates the consolidated repo for all promoted components, to be placed at https://trunk.rdoproject.org/centos8-master/promoted-components/delorean.repo

[delorean-component-common]
name=delorean-python-vmware-nsxlib-27d4662f90008790da824942a835c4f861973e61
baseurl=https://trunk-staging.rdoproject.org/centos7/component/common/27/d4/27d4662f90008790da824942a835c4f861973e61_462060d5
enabled=1
gpgcheck=0
priority=1

[delorean-component-compute]
name=delorean-openstack-nova-ee16ae1b39962c8d07467326a0108606433f2280
baseurl=https://trunk-staging.rdoproject.org/centos7/component/compute/ee/16/ee16ae1b39962c8d07467326a0108606433f2280_a545aa4b
enabled=1
gpgcheck=0
priority=1

Jobs

Promote consistent to component-ci-testing

The consistent hash of a component repository is promoted to component-ci-testing to pin the hash for running the promotion jobs with the same hash, as new hashes may be generated with new builds.

Component (validation) jobs

For now standalone is the only validation job in the component pipeline. The job overrides the stable containers w/ the new built ones to be tested for a single component.

Promote component-ci-testing to promoted-components

The component is promoted by a Zuul job in an independent pipeline. It runs the promote-hash role, checks the promotion criteria with the delorean api^[2], and promotes the hash to promoted-components.

Component promotion sequence:

promotion-->delorean api: get reported jobs for hash 'xyz'
Note right of delorean api: query delorean db
delorean api->promotion: json containing reported jobs
Note left of promotion: check promotion criteria
promotion->delorean api: promote hash to *promoted-component*

Speculatively test two unpromoted components together

There are times when one component will not promote until another component promotes due to cross rpm dependencies. One could force a promotion of one of the components to unlock the dependency.

A better way is to test two components together.

In this example we're taking the tripleo component job and adding the clients component ( component-ci-testing ) to the tripleo component job.

- project:
    check:
      jobs:
        -  periodic-tripleo-ci-centos-8-standalone-tripleo-secondary-clients-master
- job:
    name: periodic-tripleo-ci-centos-8-standalone-tripleo-secondary-clients-master
    parent: periodic-tripleo-ci-centos-8-standalone-tripleo-master
    vars:
      add_repos:
        - type: generic
          reponame: "{{ component }}"
          filename: "{{ component }}-component.repo"
          priority: 1
          baseurl: "https://{{ dlrn_server_url }}/centos8-{{ release }}/component/{{ component }}/component-ci-testing"
          update_container: true
        - type: generic
          reponame: "clients"
          filename: "clients-component.repo"
          priority: 1
          baseurl: "https://{{ dlrn_server_url }}/centos8-{{ release }}/component/clients/component-ci-testing"
          update_container: true

Implications to the Promotion Pipeline

The promotion pipeline is now broken down into 2 parts:

component pipeline (the new layer added to promote individual components)
integration pipeline (former promotion pipeline)

The component pipeline feeds the integration pipeline w/ all the components combined in a single build. Each individual component points to a separate repository containing an unique hash.

component=>start: COMPONENT PIPELINE
integration=>end: INTEGRATION PIPELINE
component->integration

A higher level view of the internal workflow with component and integration can be found here link

commit_hash vs build_id

commit_hash

The commit_hash uniquely identifies a single component repository. It preserves the same functionality from the former promotion pipeline, but only for a single component.

This implies in modifications to how jobs parse commit.yaml to get the hash that identifies the repository to be used.

https://trunk.rdoproject.org/centos8-master/component/compute/0e/d0/0ed00643bff0002f04288e4e208c88ed614f111f_c9bfa013/

commit_hash has a completely different meaning in the new promotion pipeline, and no longer represents the consolidated repository w/ all packages.

Jobs from integration pipeline won't get commit.yaml to setup repos or parse delorean.repo to upload images. Instead, they will use build_id to identify a group of components promoted and ready to consume.

build_id

build_id is a md5 hash generated to uniquely identify a build of components collection.

It is generated everytime a component is promoted. The delorean.repo file is updated with the hash of the promoted component and a new md5 build_id is created.

In the integration pipeline, instead of consuming commit_hash from commit.yaml file, jobs need to parse a new build.yaml file containing the build_id with the consolidated repository with all promoted components.

Assumptions:

infra needs to support delorean report jobs w/ –build-id option (that replaces commit_hash)
infra needs to generate build_id md5
ci needs to update get-hash, upload-cloud-images, and other parts of the code that refer to commit.yaml and commit_hash.

Spec for promoting promoted-components -> tripleo-ci-testing

(spec by wes hayutin, owner rlandy)

A job will handle the promotion of promoted-components -> tripleo-ci-testing. Promotions for current-tripleo and current-tripleo-rdo will be handled by the promotion-server

When md5_url is defined

md5_url is the key that trigger promoting all the components indivdually at once to tripleo-ci-testing.

In a zuul unprotected repo create a task that creates the information required to promote all the components at once. As discussed a new task in https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/get_hash/tasks
Information required for the promotion.
The information will be consumed by the Promote hash label task and promote_hash.sh script.

rfolco, rlandy, and wes discussed the following..

Do as much work as possible in unprotected repos.
- curl/fetch the promoted-components/delorean.repo file
- Extract the full dlrn_hash for each component
- Create a file named $component for each component.

Filename: baremetal ( match the name)

In each component file
- export COMPONENT_NAME ( for logging )
- export COMMIT_HASH
- export DISTRO_HASH

A file per component will allow us to loop and source the component information easily.

Loop through components and promote

An example of how this works can be found in the gist

See: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-primary-distro.yaml for details

Create a version of the task Promote hash label for two use cases.

When a single hash ( centos-7 )
- this should only need an additional when: md5_url not defined
When md5_url is defined ( centos-8 )
- loop through each component w/ a list of components in ansible
  - source the appropriate file
  - execute the tasks as currently written in source

MOLECULE ( test it), owner: rfolco

Monitoring

validation jobs: CI jobs in the Zuul pipeline that compose the criteria to promote a individual component, such as container build, standalone, etc. ↩︎
delorean api: https://trunk-staging.rdoproject.org/api-centos-master-uc ↩︎

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

RDO Components and the Component Promotion Pipeline

tags: Design

Background Info

What are Components?

What is the component pipeline?

Motivation

Goals

Production View

Glossary

Completed Work

Further Development

Overview

Another example with failing components

Zuul Pipelines

Design

Components build

Promotion pipeline workflow

Promotion hashes

consistent

component-ci-testing

promoted-components

Jobs

Promote consistent to component-ci-testing

Component (validation) jobs

Promote component-ci-testing to promoted-components

Speculatively test two unpromoted components together

Implications to the Promotion Pipeline

A higher level view of the internal workflow with component and integration can be found here link

commit_hash vs build_id

commit_hash

build_id

Spec for promoting promoted-components -> tripleo-ci-testing

When md5_url is defined

Loop through components and promote

MOLECULE ( test it), owner: rfolco

Monitoring

tags: `Design`