# Content Provider / Parent-Child work items
###### tags: `Design`

[TOC]

# changes
https://projects.engineering.redhat.com/browse/TRIPLEOCI-210

# to-do
- [x] build a poc job ( sagi )
- [x] get review from tripleo, infra-team ( wes )
- [x] socialize w/ ci team ( wes )

# Documentation

## Glossary / Terms / Naming conventions
### Templates
Create a new template w/ an additional **"-pipeline"** template name
### Parent of consumer job
(when creating new parent) tripleo-ci-undercloud-content-consumer-centos-8

## Doc

# Plan
- [x] (sshanidm) Define a job that can expose containers registry link and YUM repo link
- [x] (chandan)Define a test job (i.e. scenario001) that consumes and uses these links
- [x] Create a job that builds RPM and containers and pushing containers to local registry. Make these steps optional with parameters.
- [x] To configure child job (scenario001) to use correctly these artifacts if need. Other branchful jobs should be able still to build/download containers by their own.
- [x] demo DF.
- [x] https://pdfhost.io/v/b8IOld.o6_New_CI_flowpdf.pdf
- [ ] rename standalone-provider -->
- [x] tripleo-ci-centos-8-content-provider (sagi)
- [ ] tripleo-ci-centos-7-content-provider (sagi)
- [x] communicate plan (wes)
- [ ] move all rdo third party jobs to the rdo registry. (wes w/ sagi's review)
- [x] plan out branchful jobs.. ( see below )
- [x] Test multinode jobs w/ a provider job (sagi)
- [x] push ceph containers to openstack zuul registry or quay or both ( manual ) (sagi / chandan)
- [ ] podman pull on the provider for non-tripleo containers (need code ) ( sagi / chandan )
- [ ] Need to add redundant registries to pull from quay, rdo-registry, docker.io, in that order
- [ ] migrate ussuri ( sagi / chandan )
- [ ] migrate train centos8 ( sagi / chandan )
- [ ] migrate train centos7 ( pending decision re: Alex on c7 train )
- [ ] work out provider parent for changes that have both c7 and c8 jobs.
- [ ] migrate stein centos7 ( pending results from just using quay.io)
- [ ] migrate the rest ( pending results from just using quay.io )

## Planning meeting Agenda, Discussion Items:

### address the current issues
* atm we have gate jobs running that are not in check.

#### to-do
- [x] need a patch(s) to fix check vs gate ( sagi )
- [x] Unable to freeze job graph: Job tripleo-ci-centos-8-standalone depends on tripleo-ci-centos-8-content-provider which was not run. ( sagi )
* example https://review.opendev.org/#/c/749710/
* FIX https://review.opendev.org/#/c/752082/

### Branchful parent / child jobs Plan

* Idea
* Add a content provider job for each release
* Put it as a dependencies on the branchful job
* Structure:
* centos-7-train-content provider
* standalone-7-train:
* dependencies:
* centos-7-train-content provider
* centos-7-stein-content provider
* standalone-7-stein:
* dependencies:
* centos-7-stein-content provider

#### meeting notes
* content provider for each release
* question: how complicated is that zuul config.. every branchful job will have a dependency
* really this is just for the tripleo-ci repo.
* concern about time:
* rocky, stein, queens
* rocky.. let's drop all but one job
* remove branchful
* stein.. let's drop all but one job
* remove branchful
* queens.. need to keep what we have.
##### TO-DOs
- [ ] remove centos-7 train container-multinode, -2 from wes atm (chandan)
- https://review.opendev.org/#/q/topic:remove_centos7_multinode+(status:open+OR+status:merged)
* branchful train-c7, train-c8, ussuri, victoria
- [ ]( sagi ) we will have branchful content-provider nodes
* https://review.opendev.org/#/c/752301/
- [ ] ( Chandan ) push docker registry container to rdo-registry
- [x] ( Mariios ) push docker registry container to QUAY
- [ ] (chandan) Use Quay.io docker regitry container
* https://review.opendev.org/#/c/752642/
* https://review.opendev.org/#/c/752643/

### CentOS -7 based content provider jobs
* Idea
* run runv3 playbook to set the env and perform build test packages
then call tipleo-ci container-builds role and push the contents to local registry and pause th job and consume it in same fashion

- [ ] ( Chandan ) https://review.opendev.org/752419
- [ ] ( Chandan ) https://review.opendev.org/752421

#### meeting notes
* c7 is following the same pattern as c8 jobs w/ a content provider, however implementation jobs is different.

* We may choose not to use c7 content providers and just pull containers from key.

### Confirm Promoter Server push docker.io && Quay.io
- [ ] ( marios ) promoter verification re: pushing to both docker.io and quay.io

### update and upgrade jobs [Need discussion]
Before that: How updates and upgrade jobs work?
Ideas:
* There is no issue with update jobs as there will be package updates only? so no changes required there
* for upgrade job, we might to have two content provider for example
* upgrade job from train to ussuri
* we need to build containers for both train and ussuri and put both
jobs as deps
* layout

- c8-content-provider-previous-current-tripleo
- c8-content-provider-current-tripleo
- c8-update job:
    dependencies:
      - c8-content-provider-previous-current-tripleo
      - c8-content-provider-current-tripleo
- c8-standalone job:
    dependencies:
      - c8-content-provider-current-tripleo

#### meeting notes

* deploys w/ previous-current-tripleo updates to current-tripleo
* two content providers for update / upgrade jobs
* n + 1 jobs
* n content provider
* build gerrit changes for n
* provide yum repo for n
* provide containers for n
* n +1
* build gerrit change for n+1
* provide yum repo for n+1
* provide container for n+1
* in CI after deployment, prior to upgrade switch the yum repos, and update the containers-prepare-params.yml

#### update / upgrade TO-DO
Focus here should be soley on CentOS-8. Once we have validated this w/ C8, come back to CentOS-7
- [ ] create two content-provider jobs
- [ ] one for previous-current-tripleo
- [ ] one for current-tripleo ( exists )
- [ ] ensure the switch from previous-build-content-provider
- [ ] yum repo change (this may exist)
- [ ] switch container-prepare-params
- [ ] review and validate this work
- call 22 sep:
- update job should be OK sshnaidm already fixed - he will verify
- standalone-upgrade - pull for tripleo-u deployment and then re-use the built containers for the upgrade (master current-tripleo) from provider
- then can workout the building of tripleo-u too...
- TBD if there will be one provider job with different namespace in same registry for example
- or just build in the standalone-upgrade job... since no-one else will consume tripleo-u (maybe the multinode upgrade job can re-use.)

### For rdo software factory jobs
* ~~push containers to quay.io with different namespace and consuming it in rdo third party jobs~~
* Also pushing non-tripleo containers to rdo registry under ceph namespace
* rdo registry will be THE registry for jobs running in rdo software factory.
- [ ] update rdo env config,
- [ ] WATCH LIST, do not change yet.. update release files move from docker.io to either quay or rdo registry
- [ ] e.g. https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/master.yml#L23
- [ ] https://review.opendev.org/#/c/752685/
- [ ] all rdo env or job config points to rdo registry

#### meeting notes

### Handling jobs defined in zuul layouts outside of tripleo-ci.
* https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/zuul.d/layout.yaml#L14

- [ ] Ensure any job outside of tripleo is using standalone w/ build-containers flag set.
- [ ] victoria, ussuri, train c8
- [ ] stein, rocky, queens
- [ ] set the jobs to use quay.io
- [ ] consider a non-tripleo job template that is the standard for non-standalone jobs.
- [ ] consider asking other projects to drop tripleo jobs if quay.io is a problem

### Any changes to component pipeline?
* these jobs should all use the rdo-registry

- [ ] env override https://review.opendev.org/#/c/752685/

---
# Templates
tripleo-ci templates that need to switch to the new content provider:
- ~~tripleo-ci-build-containers-jobs~~
- ~~tripleo-build-containers-jobs~~
- ~~tripleo-ansible-role-jobs~~
- ~~tripleo-ci-buildimage-jobs~~
- ~~tripleo-buildimage-jobs~~
- ~~tripleo-multinode-baremetal-minimal~~
- https://review.opendev.org/756919
- [x] tripleo-multinode-container-minimal >> ready
- [x] tripleo-periodic >> ready
- [ ] tripleo-undercloud-jobs >> wip
- [ ] https://review.opendev.org/#/c/756128
- [ ] https://review.rdoproject.org/r/#/c/29894/
- [x] tripleo-standalone-scenarios-full >> ready

# tripleo-repos
https://projects.engineering.redhat.com/browse/TRIPLEOCI-210

Switch to content provider jobs/templates
- tripleo-common: https://review.opendev.org/#/c/756382
- python-tripleoclient: https://review.opendev.org/#/c/756516/
- puppet-tripleo: https://review.opendev.org/756554
- tripleo-ansible: https://review.opendev.org/756561
- tripleo-quickstart: https://review.opendev.org/756572
- tripleo-quickstart-extras: https://review.opendev.org/756575

# Q&A (Sprint #34)

### gerrit-topic: new-ci-job
(~~NOT new-ci-jobs~~)
https://review.opendev.org/#/q/branch:master+topic:new-ci-job

### Original provider patch for reference
https://opendev.org/openstack/tripleo-ci/commit/f360f4168f7933bb561093a04c5c7f3c26200cd5

### How does content provider jobs work?
> https://zuul-ci.org/docs/zuul/reference/jobs.html#pausing-the-job

### Where is it implemented in our jobs?
> https://github.com/openstack/tripleo-ci/blob/7acd24cec0d0ab97a7c5d17c5d2ab1dc85df3106/playbooks/tripleo-ci/run-provider.yml#L14

### Which jobs are using content provider parent ?
> standalone* and multinode* jobs parent content provider job (or any job that has dependencies on content provider)

            dependencies:
              - tripleo-ci-centos-8-content-provider

### Clarify what needs to be done in this task: Wire-up content provider pipeline into tripleo repos
> Switch to content-provider jobs in all tripleo/ci repos
> See https://projects.engineering.redhat.com/browse/TRIPLEOCI-210