---
title: CI Content Provider jobs
---
###### tags: `Design`

<!-- Put the link to this slide here so people can follow -->
CI Content Provider jobs: https://hackmd.io/p/YH9xtBmOQbSVqPu7fjgoog

---

# What is the problem

We can no longer use docker.io for storing out current-tripleo containers and using those in the upstream CI jobs. On 01 Nov 100 docker.io will rate limit to [100 containers/ip](https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/) per every six hours. As you know each TripleO deployment uses over 100 containers and each gerrit change launches multiple deployments.

We experienced a lot of issues with pulling containers via upstream proxies with tripleoclient before. Authentication to docker.io that slipped through proxies caused rate limit failures. The DF spent a lot of time and effort debugging and resolving these issues.

We wanted to come with solution that will not require containers to rely on third party container registries.

---
### Curent Status and progress
For the latest status see the [open_reviews](https://review.opendev.org/#/q/topic:new-ci-job)

100+ gerrit patches have been sumitted. Most of these patches are adjusting the zuul job config across the tripleo git repos and in each branch. There are over 30 git repos to modify, with at least 4 branches each.

---

### CI workflow in an old way
https://i.imgur.com/R5pLYUd.png

---

### CI workflow in a new way
https://i.imgur.com/ulb8YAY.png

---

## New templates


- tripleo-ci-centos-8-content-provider
- tripleo-ci-centos-8-containers-multinode:
    vars:
      consumer_job: true
      build_container_images: false
      remove_tags:
        - build
    dependencies:
      - tripleo-ci-centos-8-content-provider

---

## Branchful Jobs

Context: branchful jobs are used to test git repos with only a master branch but the src code is built for multiple release. TripleO-Validations is an example.

* CentOS-8 based content provider
* Running for Master, Victoria, Ussuri and Train release
* Using tcib workflow triggered via [TQE container-build role](https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/container-build)
* CentOS-7 based content provider
* Running for Train and Queens release.
* Using kolla workflow triggered via [TOCI build-containers role](https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers)

---

* [Branchful jobs examples](https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L489):

- tripleo-ci-centos-8-content-provider-train <- parent job
- tripleo-ci-centos-8-containers-multinode-train: <- child job
    irrelevant-files: *multinode_ignored
    vars: *multi_consumer_vars
    dependencies:
      - tripleo-ci-centos-8-content-provider-train

* Behind the scenes Parent job sets

registry_ip_address_branch:
  train: <registry ip address (ip of the paused content provider job)>

* child job uses that var in containers-prepare-parameter.yaml file and sets
namespace for example: **namespace: "104.130.158.105:5001/tripleotraincentos8"**

---

## Non tripleo containers

Context: Containers that are not built by TripleO but used in a deployment like ceph, prometheus etc.

* source of truth for containers - tripleo-common/container-images/
* from train -> [tripleo_container.yaml](https://opendev.org/openstack/tripleo-common/src/branch/stable/train/container-images/tripleo_containers.yaml) for centos-8
* queens, rocky, stein, train-> [overcloud_containers.yaml](https://opendev.org/openstack/tripleo-common/src/branch/stable/train/container-images/overcloud_containers.yaml) for centos-7
* tripleo containers got built using ubi8 base via tcib
* non-tripleo container getting pulled from quay.io to content provider job
* ceph daemon, prometheus, alertmanager, node-exporter, grafana

---

in Child job , for example from standalone scenario 4 job containers-prepare-parameters.yaml file

ceph_alertmanager_image: alertmanager
ceph_alertmanager_namespace: 172.99.75.179:5001/tripleomaster
ceph_alertmanager_tag: v0.16.2
ceph_grafana_image: grafana
ceph_grafana_namespace: 172.99.75.179:5001/tripleomaster
ceph_grafana_tag: 5.4.3
ceph_image: daemon
ceph_namespace: 172.99.75.179:5001/tripleomaster
ceph_node_exporter_image: node-exporter
ceph_node_exporter_namespace: 172.99.75.179:5001/tripleomaster
ceph_node_exporter_tag: v0.17.0
ceph_prometheus_image: prometheus
ceph_prometheus_namespace: 172.99.75.179:5001/tripleomaster
ceph_prometheus_tag: v2.7.2
ceph_tag: v4.0.13-stable-4.0-nautilus-centos-7-x86_64

---

## Upgrades

In a TripleO upgrade two releases are used, the deployed release and then the deployed release +1. Containers and rpms must be built for each release and used at the appropriate time in the job.

Main difference is multiple parents
Deploy release-1, Upgrade to release

- tripleo-ci-centos-8-standalone-upgrade:
vars: *consumer_vars
dependencies:
- tripleo-ci-centos-8-content-provider
- tripleo-ci-centos-8-content-provider-ussuri

---

Each parent sets:

registry_ip_address_branch {{ ci_branch }}:{{ IP }}

After both run:

registry_ip_address_branch:
master: 213.32.75.192
ussuri: 158.69.75.154

[check zuul inventory in consumer upgrade jobs](https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_c49/758443/3/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/c495d87/zuul-info/inventory.yaml)

---

Use registry_ip_address_branch[release]

[During deployment](https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/f92859a7ee88e40727755723097c77aa56eeea3f/roles/standalone-upgrade/templates/standalone_config.yaml.j2#L12-L13)

[During upgrade](https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/f92859a7ee88e40727755723097c77aa56eeea3f/roles/standalone-upgrade/templates/standalone_config.yaml.j2#L12-L13)

release set in the [release configs](https://opendev.org/openstack/tripleo-quickstart/src/commit/e32244d74d1cc70f39e5b6afc6970e0d798245ae/config/release/tripleo-ci/CentOS-8/victoria.yml#L4)

---

Relies on a beautiful mess:
* [emit_releases_file.py](https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/emit_releases_file/emit_releases_file.py)
* [toci_quickstart.sh](https://opendev.org/openstack/tripleo-ci/src/commit/d5514028452f9d427949f5a8fac26b48bd0d7c03/roles/run-test/templates/toci_quickstart.sh.j2#L71-L91)

During run:
* deployment --extra-vars {{ previous release }}
* upgrades --extra-vars {{ current release }}

See the [playbook executions](https://b37c19ac988abbc5b22c-da3f945bb6e5ca02e53e7fc9f3f751d2.ssl.cf5.rackcdn.com/753403/30/check/tripleo-ci-centos-8-standalone-upgrade/e46e5f6/logs/quickstart_files/playbook_executions.log) on any upgrade job to see the release passed to each play.

---

## multiple container registries (fail-over)(experimental)

* What if Quay.io starts hitting API rate limit or went down?
* Here comes the [failover mechanism](https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/container-build/tasks/non_tripleo_containers.yml#L28)
* Keeping non-tripleo container images at multiple registry(quay, rdo registry and docker.io)
* If Quay fails, fallback to rdo registry
* if rdo registry fails, fallback to docker.io
* if docker.io fails, fail the job

---

For RDO third party check jobs and component pipelines
* Use rdo registry for pulling all containers, there is no content provider job there

For Rocky and Stein jobs upstream jobs
* containers will be pulled from Quay.io

---

## Links
* docs: https://docs.openstack.org/tripleo-docs/latest/ci/content_provider_jobs.html
* original commit https://opendev.org/openstack/tripleo-ci/commit/f360f4168f7933bb561093a04c5c7f3c26200cd5
* moar patches https://review.opendev.org/#/q/topic:new-ci-job+(status:open+OR+status:merged)

## Rework templates to minimize the number of content-providers per job

Do to various zuul configs, upstream tripleo-ci as of 11/17 is launching content providers that are not required for the gerrit-review. We are working to correct the problem in the very near future.

* https://review.opendev.org/#/c/761188/
* https://review.opendev.org/#/c/761413/
* https://review.opendev.org/#/c/761412/
* https://review.opendev.org/#/c/761199/
* https://review.opendev.org/#/c/761189/