# Component and Intergation CI for Release Delivery
###### tags: `Documentation`

### What we heard in the sync meeting..

* rdo promoted components -> candidate compose..

* compose is triggered on http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/current-tripleo/

#### Answer from Shreshtha
:::info
* Candidate composes is from OSP promoted components.

* Trunk compose is triggered by RDO current ( http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/current/
:::

### action items
* AI: build jobs that validate changes to rhos-release... However we do not think the coverage is adequate to run a job or two here.

* Attempting to ru w/o candidate repo - using rhos-release args: "-P 17-nightly ceph-4 -r 8.3"

* We heard the nightly compose is built from RDO promoted components.. is it RDO promoted components or OSP promoted components

:::warning
* Should our intergration pipeline pick up the nightly compose?
* If it's built on RDO promoted components, can it be built w/ OSP Promoted Components
:::

### rhos-17
* traditionally we use trunk-candidate ( please define ) [trunk-candidate](http://download.eng.bos.redhat.com/rel-eng/repos/rhos-17.0-rhel-8-trunk-candidate/tagged-builds.lst)
* on 10/13/2020 - lon/rlandy

    rlandy|rover> http://download.eng.bos.redhat.com/rel-eng/repos/rhos-17.0-rhel-8-trunk-candidate/tagged-builds.lst
[14:23:00] <rlandy|rover> <rlandy>  nothing provides python3-network-runner >= 0.2.2 needed by python3-networking-ansible-4.0.1-0.20201008110557.6f56fd4.el8ost.noarch

* at the meeting.. trunk-candidate does not trigger all the time, we instead use the nightly compose.
* rlandy|rover> https://rcm-tools-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Composes II - Nightly/job/nightly-compose-rhosp-17.0
* Also Pungi was mentioned.
* Project Pungi Compose for RHOSP 17.0 nightly (nightly)
* rhos_release_args: "-P 17-nightly ceph-4 -r 8.3"
* using Pungi we failed to build containers

----
## What is the basic information both rel-del and ci needs to understand

### Glossary of terms:

#### Delivery
**brew tags**: Applided during koji/brew builds that help identify and group packages.

**OSP Trunk**: The internal DLRN instance(s) used to sync and perform brew builds internally.

**Trunk Compose**: A pungi compose based on the latest builds from OSP Trunk for a given release.

**Non-Trunk Compose**: A pungi compose based on release candidate brew tags.

**dist-git**: A set of git repositories internally that hold rpm specfiles or the container dockerfiles.

**pungi** a tool to create composes.. makes the build look like what would be available via sub-man. provides repo closure

**odcs** on demand compose service. wrapper around pungi. Single use compose mainly for things like qcow builds. They have a limited lifespan.

**koji** rpm builder, internally know as brew.

#### CI

**Upstream TripleO CI** - tripleo ci deployments that execute against [tripleo repos](https://opendev.org/openstack/?tab=&sort=recentupdate&q=tripleo)

**3rd Party RDO CI** - CI that executes from [rdo softwarefactory](https://review.rdoproject.org/zuul/status). Can vote, but not prevent upstream changes from merging. Can execute against a variety of sources, upstream, github, non-openstack etc..

**Integration Pipeline** - Originally only used to protect upstream jobs from non-tripleo patches breaking tripleo ci. Now is both upstream and downstream. 20+ jobs created to represent *any* voting upstream job.
* upstream: look for "openstack-periodic-integration" in [rdo softwarefactory](https://review.rdoproject.org/zuul/status)
* Monitoring: [link](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
* internal: look for "openstack-periodic-integration" in [internal softwarefactory](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)
* Monitoring: [link](http://tripleo-cockpit.usersys.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1)

**Component Pipeline** - CI to check and test a known good build with changes from ind

### Workflows
* where are the dockerfiles src for the compose. (rlandy)

## rhos-17 plan

* non trunk compose
* run diff
* align centos8 and rhel8 rpms
* run diff
* try to ci

----

* What we heard the workflow is..

digraph hierarchy {

nodesep=0.1
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "RDO compute/consistent"->{ "RDO compute/component-ci-testing" }
    "RDO security/consistent"->{ "RDO security/component-ci-testing" }
    "RDO cinder/consistent"->{ "RDO cinder/component-ci-testing" }
    "RDO compute/component-ci-testing"->{ "RDO compute/promoted-components" }
    "RDO cinder/component-ci-testing"->{ "RDO cinder/promoted-components" }
    "RDO security/component-ci-testing"->{ "RDO security/promoted-components" }
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "RDO compute/promoted-components"->{ "RDO promoted-components" }
    "RDO cinder/promoted-components"->{ "RDO promoted-components" }
    "RDO security/promoted-components"->{ "RDO promoted-components" }
    ->" RDO --> Import to OSP Candidate Compose (where does this go?) "
    ->"wait for " -> "OSP current-tripleo"
}    

-----
# Full CI workflow
Full picture, how things were designed to work.

Pasted to [googlejam board](https://jamboard.google.com/d/1p4ODk5MxuvJZyL42Wzm45v_3KY9_catxuUi4Z1cbYfQ/viewer?f=0) for notes.

digraph hierarchy {

nodesep=0.1
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "RDO compute/consistent"->{ "RDO compute/component-ci-testing" }
    "RDO security/consistent"->{ "RDO security/component-ci-testing" }
    "RDO cinder/consistent"->{ "RDO cinder/component-ci-testing" }
    "RDO compute/component-ci-testing"->{ "RDO compute/promoted-components" }
    "RDO cinder/component-ci-testing"->{ "RDO cinder/promoted-components" }
    "RDO security/component-ci-testing"->{ "RDO security/promoted-components" }
node [color=darkgreen,fontname=Courier,fontcolor=black,shape=box]
edge [color=darkgreen]
    "RDO compute/promoted-components"->{ "RDO promoted-components" }
    "RDO cinder/promoted-components"->{ "RDO promoted-components" }
    "RDO security/promoted-components"->{ "RDO promoted-components" }
    ->{" RDO --> Import to OSP Delorean "}
    ->"OSP DLRN builds from import" ->"OSP compute/consistent" -> "OSP compute/component-ci-testing" -> "OSP compute/promoted-components" ->{ "OSP promoted-components" }
    "OSP DLRN builds from import" -> "OSP cinder/consistent" -> "OSP cinder/component-ci-testing" -> "OSP cinder/promoted-components" ->{ "OSP promoted-components" }
    "OSP DLRN builds from import" -> "OSP security/consistent" -> "OSP security/component-ci-testing" -> "OSP security/promoted-components" ->{ "OSP promoted-components" }
    
    "OSP promoted-components" -> "OSP tripleo-ci-testing" -> "OSP Integration Tests" -> "OSP current-tripleo"
    
    
    "OSP current-tripleo" -> "OSP Trigger Compose"
}    

## RHOS-RELEASE and YUM REPOS

### OSP - 16.2

DLRN = http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/

Dependencies 16.2 [link](http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-16.2.yml)

* rhos-releaes
* osp_release: rhos-16.2
* rhos_release_args: "ceph-4 -r 8.3"

* yum repos

- type: generic
    reponame: osptrunk-candidate-deps
    filename: osptrunk-candidate-deps.repo
    baseurl: http://download.eng.bos.redhat.com/rel-eng/repos/{{ osp_release }}-rhel-8-trunk-candidate/x86_64/
    priority: 30
    enabled: 1
    gpgcheck: 0

  - type: generic
    reponame: rhosp-ansible-2.9
    filename: rhosp-ansible-2.9.repo
    baseurl: http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/layered/rhel8/x86_64/ansible/2.9/os/
    priority: 30
    enabled: 1
    gpgcheck: 0

  - type: generic
    reponame: osp-trunk-deps
    filename: osptrunk-deps.repo
    baseurl: http://download.devel.redhat.com/brewroot/repos/{{ osp_release }}-rhel-8-trunk-build/latest/x86_64
    priority: 30
    module_hotfixes: 1
    enabled: 0
    gpgcheck: 0

### OSP - 17.0

DLRN = http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17/

Dependencies OSP-17 [link](http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-17.yml)

- type: generic
    reponame: osptrunk-candidate-deps
    filename: osptrunk-candidate-deps.repo
    baseurl: http://download.eng.bos.redhat.com/rel-eng/repos/rhos-17.0-rhel-8-trunk-candidate/x86_64/
    priority: 30
    enabled: 1
    gpgcheck: 0

  - type: generic
    reponame: rhosp-ansible-2.9
    filename: rhosp-ansible-2.9.repo
    baseurl: http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/layered/rhel8/x86_64/ansible/2.9/os/
    priority: 30
    enabled: 1
    gpgcheck: 0

  - type: generic
    reponame: osp-trunk-deps
    filename: osptrunk-deps.repo
    baseurl: http://download.devel.redhat.com/brewroot/repos/rhos-17.0-rhel-8-trunk-build/latest/x86_64
    priority: 30
    module_hotfixes: 1
    enabled: 0
    gpgcheck: 0

:::info
A real time view into the above promotions can be found here:
* [high level view of all promotions](http://dashboard-ci.tripleo.org/d/QrkFP-zGz/upstream-and-rdo-promotions?orgId=1)
* [component level promotions](http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline)
:::

# meeting notes

## 10/29/2020

Compare 16.2 builds

- Control
- https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2/c3ecaf6/logs/undercloud/var/log/extra/all_available_packages.txt.gz
- Test
- http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/phase1-16.2_director-rhel-8.3-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/68/undercloud-0/var/log/extra/all_available_packages.txt.gz

Diff: https://termbin.com/ou35

Into the compose..
dump
versions.csv
delorean.repo
delreoan.md5

Ignore list:
*.debuginfo
*.debugsource
*-devel
*-doc

blosc-*
golang.*
hdf5-.*
jjoyce to check erlang
openstack-selinux-*
openstack-tripleo-heat-templates-compat
openstack-tripleo-validations-tests
python2.*
python3-heat-agent
python3-heat-agent-*
python3-kuryr-*
python3-matplotlib-*
pyxattr-*
yaml-cpp-static

Checked into https://code.engineering<foo>/gerrit/#/c/216562/1/compare_rpms/rhos-16_2.txt

## How patches are imported from upstream to internal

DLRN is working as expected here. Due to its design constraints, the import from RDO is done in several steps:

* a) Copy the RDO spec as-is.
* b) Rebase the downstream source git on top of the upstream commit being built, so our downstream-only commits are always up to date.
* c) Add patches from the downstream git via rdopkg (rhospkg actually). This also updates the changelog, and
sets the specific Release number in NVR, which is defined as -<a>.<timestamp>.<commit hash>.el8.ost, where
<a> is a OSP release-specific number (2 for 16.2).
* d) rhpkg import / rhpkg build

This ensures that any spec change done in RDO is brought to OSP without manual intervention, and the downstream
distgit -trunk-patches branch should never be manually modified.