# Validation tests in the component pipeline
###### tags: `Design`

## VF has 3 main packages:
* validations-libs
* validations-common
* tripleo-validations

## First goal is to tests the Validation Framework itself:
* Trigger a Centos8 job

## Validation can be execute in two different ways:
* validation.py or tripleo-validation.py : a Python cli script that can use to run validation. The speficity of the validation.py is that it can be execute without any openstack or tripleo requirements, on a freshly new installed system (CentOS family or Debian). The main goal is to allow customer to run validation, either from another or system, or before starting the Undercloud installation.
* tripleoclient CLI : openstack tripleo validator This command is part of the python-tripleoclient project. It's the main entry point for customer to run validation.

## Upstream CI functional job. The functional job use the validation.py script to execute a set of knwon good validations in order to test the framework (python code):

* https://github.com/openstack/validations-libs/tree/master/playbooks
* https://github.com/openstack/validations-libs/blob/master/roles/validations/tasks/validations.yaml
* https://zuul.opendev.org/t/openstack/build/03fc1a460dc74d18bafaa5497d3fbd77

## jobs:

* One job to execute validations pre/post standalone
* One job to execute validations pre/post standalone upgrade/update
* One job to execute validations pre/post undercloud deployment and pre/post overcloud deployment

## Implementation:

* The main implementation is to use the roles used by upstream functional tests in the VF:
* https://github.com/openstack/validations-libs/tree/master/roles
* Some new features should be added in order to prepare the environment. For now the CI jobs is using upstream openstack roles to install and configure requirements.
* The validations that will be triggered should be stable like the functional does.
* The calls should be both with:
* validation scripts (validation.py and tripleo-validation.py) for the step 1 (non UC installation)
* tripleo cli commands: openstack tripleo validator

## todo:

* implement the workflow in tripleo-quickstart:
* play the VF roles as pre uc step only in component testing cases
* play the VF roles as post uc, then pre oc, post oc.
* create a setup roles in VF for components
* add component validation jobs in rdo-jobs

--------------------------
## reset design session w/ Mathieu

### install
* validations are 100% rpm installed

### execution

* a role
* task
* bindep install or ansible install rpms
* a generic role that can run $x validations
* task will
* vars
* /validations
* a very generic test of validations itself
* /compute
* validations related to compute
* /network

Compute Example
In each var file will be a list of tests:
- validation-$component-tests:
- check-ram
- check-ftype,f12e

* Bascially.. One generic validation role
* Injection of validations in about 3-4 places in the job

* pre undercloud validation
* if component is defined
* pre overcloud validation
* execute based on variable: component
* so only compute validations run when "component: compute"
* any post overcloud
* if component is defined.

### Reviews:

* https://review.rdoproject.org/r/#/c/31281/ -> components job definition
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/755382 -> tqe playbook
* https://review.opendev.org/c/openstack/tripleo-validations/+/755390 -> t-v roles

### Test jobs:

* https://review.rdoproject.org/zuul/build/aed7d3594e54475f95684dc3b8b311f4/log/job-output.txt#4075

### List validations per components

### Component CI documentation

#### Upstream CI:

* https://zuul.opendev.org/t/openstack/jobs

* Check and gate against reviews:
* py3* jobs
* pep8
* mutlinode jobs for TripleO
* jobs are defined on each Projects:
* https://github.com/openstack/python-tripleoclient/blob/master/zuul.d/layout.yaml
* or https://github.com/openstack/validations-libs/blob/master/.zuul.yaml

st=>start: Review XyZ
e=>end: Merged
op=>operation: Check jobs
cond=>condition: zuul/reviewers
op2=>operation: Gates
cond2=>check: Check Gates

st->op->cond
cond(yes)->op2
cond(no)->op

op2->cond2
cond2(yes)->e
cond2(no)->op2

#### RDO CI:

* https://review.rdoproject.org/zuul/jobs
* https://review.rdoproject.org/zuul/job/periodic-tripleo-ci-centos-8-standalone-compute-master-validation

* Components pipeline:
* List of the components: https://trunk.rdoproject.org/centos8-master/component/
* Run daily
* Run only tripleo-ci jobs
* Job definition for components is located here:
* https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/component-jobs.yaml;h=81b54194cd72176bf880b7374b4e37a440b6b643;hb=HEAD
* Each job inherit from tripleo-ci parent jobs and has few variables:
* release
* component
* dlrn_server_url

* Components CI for validations schema:

rdo jobs->quickstart: $component jobs

quickstart-->$project/roles: pull external projects
Note right of $project/roles: oooq requirements*
$project/roles-->quickstart: install TV CI role*
$project/roles-->quickstart: install ansible roles

quickstart-->standalone jobs: run standalone job

standalone jobs->run validations: if $component
Note right of run validations: Run Validations per $component
Note right of run validations: t-validations Components List*
run validations->standalone jobs: passed or failed

* [quickstart requirements](https://github.com/openstack/tripleo-quickstart/blob/master/quickstart-extras-requirements.txt)
* [tripleo-validations CI roles](https://github.com/openstack/tripleo-validations/tree/master/roles/tripleo_validations)
* [tripleo-validations Components List](https://github.com/openstack/tripleo-validations/blob/master/roles/tripleo_validations/vars/main.yml#L2)

---

## Current Results:
* https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-network-master-validation
* https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-compute-master-validation
* https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-tripleo-master-validation
*

## How to specify which validation runs in a component

#### validations config
* https://review.opendev.org/c/openstack/tripleo-validations/+/777248/13/roles/tripleo_validations/vars/main.yml

#### periodic rdo ci config

* https://github.com/rdo-infra/rdo-jobs.git
* example parent standalone job and child validation job

- job:
    name: periodic-tripleo-ci-centos-8-standalone-compute-master
    parent: periodic-tripleo-ci-centos-8-standalone-component-master
    vars:
      component: compute

- job:
    name: periodic-tripleo-ci-centos-8-standalone-compute-master-validation
    parent: periodic-tripleo-ci-centos-8-standalone-component-master-validation
    vars:
      component: compute

* required next steps
* fill out the other components
* socialize validation jobs to component owners
* review validation skip list
* remove all validations from upstream jobs
* one standalone job w/ validations in the name is fine.

* 16.2 / 17 component lines
* every component validation job in upstream/rdo should have an equivilant in OSP.