promoter tests

tags: promoter design

https://tree.taiga.io/project/tripleo-ci-board/us/1705?milestone=264741

WIP

TO-DO

mol promoter - done

~~simplify credentials handling (expect keys on user home dir, not root) (MERGED)~~
- ~~https://review.rdoproject.org/r/28655 New _ensure_credentials (MERGED)~~
- ~~https://review.rdoproject.org/r/28656 Use _ensure_credentials in mol-promoter and staging jobs (not yet in all molecule scenarios) (MERGED)~~
~~https://review.rdoproject.org/r/28683 Run molecule scenarios on promoter changes (MERGED)~~
~~https://review.rdoproject.org/r/#/c/28623/ move provision/staging to py3 (MERGED)~~
~~https://review.rdoproject.org/r/28597 Move mol-promoter to python3 (MERGED)~~
~~https://review.rdoproject.org/r/28587 switch mol-promoter job to voting (MERGED)~~
~~https://review.rdoproject.org/r/28553 Run mol-promoter as unprivileged user (MERGED)~~
~~https://review.rdoproject.org/r/28552 Split promoter scenario test_sequence (MERGED)~~

Isolate legacy code - done

(RFOLCO) rename all current jobs and playbooks to legacy-*
1. ~~https://review.rdoproject.org/r/28353 >> Move existing molecule playbooks to legacy dir (MERGED)~~
2. ~~https://review.rdoproject.org/r/28355 >> Rename molecule-post to mol/post.yml (MERGED)~~

NOTE: The original idea was to rename the jobs, but since we are creating new mol jobs, we just move the legacy playbooks to a legacy/ dir and don; t touch the current molecule files (except the promoter proviosion one that was moved to molecule/promoter as it wasn't running in any job, so nothing has been affected by this move). For the other scenarios, new molecule files need to be created, so we don't touch the old legacy ones.
abandoned:
~~https://review.rdoproject.org/r/#/c/28293/ >> rename to legacy~~
~~https://review.rdoproject.org/r/#/c/28294/ >> update rdo-jobs~~
~~https://review.rdoproject.org/r/#/c/28295/ >> delete original names~~

Initial design discussions (use as reference)

(RFOLCO + SORIN) Refactor/Fix promoter scenario
(RFOLCO + SORIN) Create new empty jobs for the modularized tests.
- add description to the job
(SORIN) create provision playbook that works on centos7.8 with the current duplicated code
- create test jobs for provisioner role on centos 7.8 (local and zuul) - leave the legacy centos7 alone, don't touch them anymore, they will become read only as much as possible.
- the provisioner role should be idempotent. (Continuous Deployment)
(FOLCO) Deduplicate code in provisioner role to make organization more efficient.
- De-dup: https://hackmd.io/kJqHSTWWRMOIfIhvDMGFLg#Code-Deduplication
- consider putting all the scenarios in the same directory
- clone the existing job to new job that will contain unduplicated code.
- copy code from provisioner legacy code into smaller task files
- create a main.yaml that includes all them in right order
- Make provisioner centos7.8 with modularized code job stable and voting
Convert the delegated of the jobs to work with modularized code
- convert promote-images-delegated
- convert container-push-delegated
- convert tripleo-common-integration and re-add tripleo-common test parts removed due to py3 deps missing.
- add correct triggers files to zuul layout centos7.8 jobs
- remove legacy centos7 job with duplicated code.
Make promoter integration centos7.8 with unduplicated code job stable and voting
- Fix staging-single-pipeline job
- fix staging-integration-pipeline job
(PANDA) create configuration engine that works accross all promotion environment

LATER: DO

allow delegated scenarios to work on hosts other than localhost - [not…easy](it makes local testing possible)
create provision playbook that works on centos8
- create test jobs for centos 8 (leave centos7 jobs alone)
- make current provisione code legacy:
  - ~~1 option : mv main.yaml -> main-legacy.yaml, update current job to use main-legacy.yaml~~
  - 2 option clone promoter role into a new promoter-new role
- copy code from provisioner legacy code into smaller task files
- create a main.yaml that includes all them in right order
- Make provisioner centos8 job stable and voting
Make promoter integration centos8 job stable and voting
- Fix staging-single-pipeline job
- fix staging-integration-pipeline job
Convert the rest of the jobs to work on centos8
- convert provision-delegate to centos8
- convert promote-images-delegated to centos8
- convert container-push-delegated to centos8
- convert tripleo-common-integration to cetnos8
- add correct triggers files to zuul layout centos8 jobs
- remove legacy centos7

LATER: Investigate

deploy credentials
making ci-config code deployment idempotent
create configuration engine that works accross all promotion environment

SOMETIME: Things to consider

Why not running promoter as a zuul periodic job? (internal zuul needed for internal promoter)

NEVER: Things to avoid

Code Deduplication

Summary of changes

Create new empty molecule jobs to replace legacy ones
- start from simpler to harder
  - promote-images
  - container-push
  - tripleo-common
  - staging
- make non-voting (its ok to fail)
  - small steps, iterate, fix next step, merge
- make all molecule run with tox
  - follow mol standard: https://github.com/rdo-infra/ci-config/blob/master/zuul.d/jobs-mol.yaml
- consolidate playbooks into a single one
  - molecule-*.yml >> mol-pre.yml
  - remove dups
- - break scenarios into separate files
  - converge.yml -> prepare/converge/verify/cleanup
- create private roles to include in prepare.yml so the molecule local test can run everything outside of zuul
  - inject credential tasks: _ensure_keys
  - docker setup: _ensure_docker
- separate common parts of staging.yml
  - credentials
  - reuse common code
remove all legacy code

Job level: playbooks

BEFORE

molecule-delegated-pre.yml
- install packages
- configure docker-ce repo
- install epel
- remove existing docker
- install docker-ce
- start docker
- setup docker user/group
- reset connection
- setup test-python virtualenv
- install python-tripleoclient from zuul src
- install tripleo-common from zuul src
molecule-delegated-promote-images-pre.yml
- install epel
- install packages
- setup test-python virtualenv
- check if pub key exists
  - create pub key otherwise
- check if pub key is in authorized_keys
  - add pub key to authorized_keys otherwise
molecule-delegated.yml
- source virtualenv
- run molecule command in a shell task
staging.yml (end-to-end promotion integration test)
- override role defaults (credentials and vars)
- set dlrnapi password
- stat docker config.json
- set regstry password
- save uploader key
- include setup_loop from promoter role
- include promoter role
- include update_credentials from promoter role
- add uploader pub key to authorized_keys
- include setup_staging task from promoter role
- include promotion_run from promoter role
- gather stage info
- fetch stage info into zuul executor
- test stage promotion
- check passwords are not leaked in the logs
- check stage registry password is not leaked

AFTER

molecule-pre.yml
- install packages
- install epel
- setup ~~test-python~~ promoter_venv? virtualenv

Keep in the job's pre playbook ONLY what is common to all molecule jobs.

~~molecule-delegated.yml~~
- ~~delegated scenarios are converted to non-delegated~~
- ~~all molecule scenarios are executed by tox~~
- ~~no need to specify command in run playbook~~
~~molecule-delegated-promote-images-pre.yml~~
- ~~DUP: content absorved by other parts~~

(create a new) credentials.yml
- set dlrnapi password
- stat docker config.json
- set regstry password
- save uploader key

staging.yml (end-to-end promotion integration test)
- override role defaults (credentials and vars)
- include credentials.yml
- include setup_loop from promoter role
- include promoter role
- include update_credentials from promoter role
- add uploader pub key to authorized_keys
- include setup_staging task from promoter role
- include promotion_run from promoter role
- gather stage info
- fetch stage info into zuul executor
- test stage promotion
- check passwords are not leaked in the logs
- check stage registry password is not leaked

BEFORE

digraph hierarchy {

nodesep=1.5
node [color=Red,fontname=Courier,shape=box]
edge [color=Blue, style=dashed]

"scenarios"->{"default" "staging" "container push" "tripleo-common" "promote images"}
    "staging"-> { "staging.yml" }
    "staging.yml"-> {"staging-post.yml"}
    "container push"->{ "molecule-delegated-pre.yml" }
    "molecule-delegated-pre.yml"->{ "molecule-delegated.yml" }
    "molecule-delegated.yml"->{"molecule-post.yml"}
    "tripleo-common"->{ "molecule-delegated-pre.yml" }
    "promote images"->{ "molecule-delegated-promote-images.yml"}
    "molecule-delegated-promote-images.yml"->{"molecule-delegated.yml"}
}

AFTER

digraph hierarchy {

nodesep=1.5
node [color=Green,fontname=Courier,shape=box]
edge [color=Black, style=dashed]

"scenarios"->{"functional" "staging" "container push" "tripleo-common" "promote images"}
    "staging"-> { "staging.yml" }
    "staging.yml"-> {"staging-post.yml"}
    "container push"->{ "molecule-pre.yml" }
    "tripleo-common"->{ "molecule-pre.yml" }
    "promote images"->{ "molecule-pre.yml"}
}

molecule-level: scenarios

https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/promoter/molecule

BEFORE

├── molecule
│   ├── container-push
│   │   ├── converge.yml
│   ├── default
│   │   ├── prepare.yml
│   │   ├── converge.yml
│   │   ├── functional_tests.yml
│   ├── promote-images
│   │   ├── converge.yml
│   ├── tripleo-common-integration
│   │   ├── converge.yml
│   +
+

AFTER

├── molecule
│   ├── container_push
│   │   ├── prepare.yml
│   │   ├── converge.yml
│   │   ├── verify.yml
│   │   ├── cleanup.yml
│   ├── promoter_provision
│   │   ├── prepare.yml
│   │   ├── converge.yml
│   │       ├── (functional_tests.yml)
│   │   ├── verify.yml
│   │   ├── cleanup.yml
│   ├── promote_images
│   │   ├── prepare.yml
│   │   ├── converge.yml
│   │   ├── verify.yml
│   │   ├── cleanup.yml
│   ├── tripleo_common_integration
│   │   ├── prepare.yml
│   │   ├── converge.yml
│   │   ├── verify.yml
│   │   ├── cleanup.yml
│   +
+

Add this private role to setup docker-ce for all container scenarios

_ensure_docker role
- configure docker-ce repo
- remove existing docker
- install docker-ce
- start docker
- setup docker user/group
- reset connection
- install python-tripleoclient from zuul src
- install tripleo-common from zuul src

container-push

Split into separate scenarios:

scenario 1: disable target registry push
scenario 2: secure registry, manifest disabled
scenario 3: insecure registry, multiarch manifests
scenario 4: insecure registry, multiarch, ppc disabled

container-push
- prepare.yml
  - include-role _ensure_docker
  - setup experimental docker
  - enable experimental docker
  - launch staging setup
  - include stage setup vars
  - set full_hash var
- converge.yml
  - include containers-promote role
- verify.yml
  1. disable target registry push
    - manifest inspect
    - assert containers pushed to src registry only
  2. secure registry, manifest disabled
    - manifest inspect registries
  3. insecure registry, multiarch manifests
    - manifest inspect
    - get list of local images
    - assert all images have been pushed to registries
    - list files in manifest dir
    - assert no manifest leftovers
  4. insecure registry, multiarch, ppc disabled
    - manifest inspect (excludes ppc)
    - get list of local images
    - assert all images have been pushed to registries
    - list files in manifest dir
    - assert no manifest leftovers
- cleanup.yml
  - teardown

promote-images

promote-images
- prepare.yml
  - setup public key >> moved from molecule-delegated-promote-images-pre.yml
  - launch staging setup
  - include stage vars
  - set full_hash
- converge.yml
  - call promote-images shell script
- verify.yml
  - get link promoted
  - check if link promoted exists and points to hash
  - get previous link
  - check if previous link points to initial hash
- cleanup.yml
  - teardown

tripleo-common-integration

tripleo-common-integration
- prepare.yml
  - include setup-docker.yml
  - pull registry image
  - run registry
  - include stage vars
  - set full_hash
  - generate containers template file
  - populate template w/ staging containers
  - generate prepare-parameter.yaml file
  - copy overcloud_containers.yaml
- converge.yml
  - run container image prepare
- verify.yml
  - check containers in undercloud_registry
  - inspect manifests in undercloud_registry
  - pull containers from undercloud_regstry
- cleanup.yml
  - remove staging files
  - remove local containers
  - stop and remove registries

default promoter_provision

~~default~~ promoter
- prepare.yml
  - install development tools
- ~~converge.yml~~
  - load role defaults
  - set ci-config path
  - set vars for molecule env
  - stat local registry secrets file
  - inject registry password
  - inject dlrnapi password
  - save uploader key
- converge.yml
  - run promoter role
    - include setup_loop task
    - include staging provisioning tasks (tag)
    - include update credentials task
    - install test-requirements in venv
- verify.yml
  - functional tests
  - ~~setup staging~~ can be removed ???
  - ~~(not implemented) check if setup is ok~~
- cleanup.yml
  - teardown

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.