# Commmon Tempest Role Design
###### tags: `Design`

## What
https://github.com/openstack/openstack-ansible-os_tempest

## WHY

tripleo and openstack-ansible have successfully shared a common tempest role 1.5 years. Let's share more code vs. write our own.
This enable us to have more people checking the code, more functionalities, and work more with upstream community.

:::success
**Specific goals:**

* Write less code, share more
* Drink more beer
* A common skip list would get more tempest errors written up as bugs and keep failing tempest tests out of CI runs.
* Save time, energy and sanity
:::

## Design

### Current examples of TripleO and OpenStack Ansible
* [Console log](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master/e558e46/job-output.txt)
* [Tempest results](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master/e558e46/logs/undercloud/var/log/tempest/stestr_results.html.gz)

### Current infrared
The two teams already share python-tempestconf [link](https://opendev.org/openstack/python-tempestconf)

"test -e ~/.virtualenvs/.tempest_conf/bin/activate && source ~/.virtualenvs/.tempest_conf/bin/activate\n source ~/keystonerc\n /usr/bin/discover-tempest-config  --deployer-input ~/ir-tempest-deployer-input.conf  --debug -v --create   --image http://rhos-qe-mirror-tlv.usersys.redhat.com/images/cirros-0.4.0-x86_64-disk.img    identity.uri $OS_AUTH_URL identity.admin_password $OS_PASSWORD scenario.img_dir ~/tempest-dir/etc            identity.region regionOne    orchestration.stack_owner_role heat_stack_owner       --out ~/tempest-dir/etc/tempest.conf

* jenkins log [link](https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/QE/view/OSP15/job/phase2-15_director-rhel-8.1-virthost-1cont_1comp-ipv4-geneve-lvm-ssl/28/artifact/.sh/07-tempest.log)

* python-tempestconf is used for generating tempest.conf in infrared: https://github.com/redhat-openstack/infrared/blob/master/plugins/tempest/tasks/configure/run.yml#L54

:::info
* question: are we using the same python-tempestconf?

:::

### infrared-plugin
* should be roughly the equivalant of the common-logging infrared plugin
* example [link](https://github.com/openstack/ansible-role-collect-logs/blob/master/infrared_plugin/plugin.spec)
* Will openstack-ansible allow for an infrared plugin file in the role? ( we think so )
* If we expose an infrared based job voting against os_tempest repo, they will allow it.

### shared skip list
* Shared tempest skip list
* Design [link](https://hackmd.io/twHmV2f2SGCMvfHjLMoGoA)
* Documentation [link](https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ade/720434/1/check/openstack-tox-docs/ade701d/docs/)
* Source [link](https://opendev.org/openstack/openstack-tempest-skiplist)

### supported tempest plugins
openstack-ansible supported tempest plugins [link](https://github.com/openstack/openstack-ansible-os_tempest/blob/d866a8f6b99b856b764ca7e9bcd79b36b6f39195/vars/main.yml)

### missing tempest plugin support and tempestconfig generation in tempestconf
* ironic plugin in development
* WIP - https://review.opendev.org/#/c/720705/
* Octavia tempest plugin

### How are tests removed from the skip list?
* Featureset021
* This job run only the failing tests on the skiplist
* If a test start to pass on featureset021, that means, the test is 'fixed'
* In the cockpit there's a session to check these, once the test pass a few times, we can remove it.

### Design requirements
* Core rights on skip list repo
* os_tempest will work from train release onwards.
* Jenkins can read results
* Any polarion requirements?
* Tempest whitebox plugin [link](https://opendev.org/x/whitebox-tempest-plugin)
* It can be enabled just by adding tempest_service_enable_whitebox in ostempest default vars.
* What else??

### Questions / What don't we know yet???
* ADD YOUR THOUGHTS AND QUESTIONS AND CONCERNS HERE!!!
* :question:Installation of tempest (rpm package vs git source code) is this included in this work or the role assumes tempest framework is already on the system ?
* :a: Yes, it is included, you can install by git, pip, rpm, etc
* :question:Does design allows to use Depends-On tempest framework changes / plugins, eg. Depends-On: change from whitebox tempest plugin in development change
* :a: On zuul side, yes, in jenkins, maybe @B_hkIsryS2Sxpv62Mkw57Q can answer
* :question:in IR all post-deployment steps for preparing the env are extraced in 'cloud-config' plugin (like external network and such) - does this overlap? (psedlak)
* :a: All these are parsed setting facts that we passes to os_tempest role
* :question:ir tempest does have support for custom commands also custom package injection (imo mostly used for plugins or component specific tests) (psedlak)
* :a: If this is related to packages to be installed based on plugins, os_tempest have that, custom commands, it doesn't, and it can be implemented as well
* ir tempest has slight support for (psedlak):
* [q] repeated runs (used by upgrade/update jobs - pre, middle, post upgrade)
* :a: We are using this on our upgrade jobs, basically we call the playbook twice, with different options IIRC
* [q] separated test-sets (runs tempest once for set A, then for set B - generates separated result files etc, not sure how much that is used or for what though)
* :a: We don't have this, but doesn't seems to be hard to do
* [q] i would recommend that someone with deep understanding of os_tempest/design goes over infrared tempest plugin spec (and code) and highlights the main differences/featue inparity (could be on some meeting with ir folks to speed it up)
* :a: This is possible to implement in os_tempest I guess, right now we don't have this support.
* :question: ~~possibly **octario integration** should be considered too~~
* ~~https://github.com/redhat-openstack/octario~~
* actually octario already dropped tempest role in past https://github.com/redhat-openstack/octario/commit/e75d63d027435a8fa4f2962bd1ce2e04f86dcc01#diff-2df3916027434b271aa80e3ca19ea382 so if then maybe just check it is used with openstack-ansible-os_tempest role

### INTERESTED IN HELPING?
* weshay
* arxcruz
* psedlak-just-parttime/consultation
* Chandan Kumar
* Amol
* marios
* soniya

### Meeting notes 4/22/2020: