Tempest Skiplist / Allowed list

tags: `skiplist` `tempest`

The idea of this effort is to completly remove the list of tempest tests running on each job, from the featureset or zuul job definition

Coding

Update the validate for tempest allowed list to also add jobs

Right now, we have two ways to use the allowed list, the first one is per job, and the second one is for groups. Right now, the search is done by job or by group, which means, if in the cli you pass –group it will search only for group, howerver, due to cybil efforts, we need a way to also pass –job and return the job as well. so, the groups yaml file that today is like this:

groups:
  - name: default
    tests:
      - smoke
    releases:
      - master
  - name: featureset001
    tests:
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
  - name: featureset001_periodic
    tests:
      - 'tempest.api'
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
jobs:
  - name: standalone
    tests:
      - test1
      - test2
    releases:
      - master
      - xena
      - victoria

Will need to have also the list of jobs:

groups:
  - name: default
    tests:
      - smoke
    releases:
      - master
  - name: featureset001
    tests:
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
    # List of jobs for this group
    jobs:
      - job1
      - job2
  - name: featureset001_periodic
    tests:
      - 'tempest.api'
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
    jobs:
      - job1
      - job2

Of course, the list of the jobs should be optional to keep compatibility

Also, need to ensure that when you pass –group and –job the search will be done properly, which means –group featureset001 –job job4 should return the tempest.scenario.test_network_basic_ops.TestNetworkBasicOps even though job4 is not in the jobs list

Consolidate command

We want to be able to tell which list of tests will be executed passing only the job, and this will need to read both the tempest-skip and tempest-allowed list, for example:

tempest-skip tests –job job1 –allow-file allow.yaml –skip-file skip.yaml –release master

This will return the tests that will be executed, excluding the tests to be skipped, for example, for job1 we have tempest_network_basic_ops and tempest_scenario_ops on the list of jobs to be executed, and on the skip list we have tempest_scenario_ops so the output should be only tempest_network_basic_ops.

Zuul jobs

For the zuul job, right now, all the logic is made on https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L36-L54 so basically, we need to remove the tempest_test_whitelist variable either from the zuul job definition as for example:

- job:
    name: tripleo-ci-scenario000-standalone-options
    vars:
      featureset: '052'
      featureset_override:
        standalone_environment_files:
          - 'environments/low-memory-usage.yaml'
          - 'ci/environments/scenario000-standalone.yaml'
        tempest_services:
          - keystone
        tempest_run_concurrency: 1
        tempest_tempest_conf_overrides:
          'auth.tempest_roles': "Member"
        # Remove this:
        tempest_test_whitelist:
          - 'keystone_tempest_plugin.tests.api.identity.v3'

But also, pay attention that we need to remove from featureset052 as well:

https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset052.yml#L33

In some cases, we have this:

tempest_test_whitelist: >-
  {% if (('periodic' in zuul.pipeline and not job.force_non_periodic|default(false)|bool) or (job.force_periodic|default(false)|bool)) and not tempest_run_skipped | default(false) | bool -%}
  {{ tempest_periodic_regex }}
  {%- else -%}
  {{ tempest_whitelist }}
  {%- endif -%}

That also need to be removed and add on the tempest-allow.yaml the correspondent test.

Pay attention that once you remove from featureset052, ALL the jobs that doesn't have tempest_test_whitelist enabled will run only tempest.api.keystone tests (the default test to be executed if no tests were given)

So, my steps would be:

I would go for a featureset per time, for example, featureset052, and check all the jobs that run featureset052

https://codesearch.opendev.org/?q=featureset%3A+'052'&i=nope&literal=nope&files=&excludeFiles=&repos=

and

https://codesearch.rdoproject.org/codesearch/?q=featureset%3A+'052'&i=nope&files=&repos=

Then check which of those jobs have tempest_test_whitelist defined in zuul job definition, and which don't. For the jobs that doesn't have the tempest_test_whitelist defined in zuul job definition, I would create a group in the tempest-allow (in this case featureset052 and featureset052_periodic, if there are difference between both). Then, I would add on those jobs in the zuul job definition the variable tempest_allowed_group to featureset052 or featureset052_periodic

After that it is safe to remove tempest_test_whitelist from featureset052, and then one by one, add an entry on tempest-allow to each job that runs featureset052 but it does have the tempest_test_whitelist defined in zuul job definition.

List of featureset used in tripleo-ci

featureset: '010' - containers-multinode (soniya)
featureset: '027' - undercloud-containers (soniya)
featureset: '030' - scenario007-multinode-oooq-container (soniya)
featureset: '037' - minor update (use_os_tempest: false) (soniya)
featureset: '062' - scenario010-standalone (no whitelist) (soniya)
featureset: '052' - centos-8-standalone (pooja)
featureset: '056' - standalone-upgrade (pooja)
featureset: '060' - scenario012-standalone (pooja)
featureset: '061' - containers-undercloud-minion (pooja)
featureset: '065' - undercloud-ffu-wallaby - no whitelist @fs config file. (https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset065.yml#L26) List of featureset used in rdo-jobs
featureset: '052' - standalone (whitelist)

After everything is done

We need to get rid of the whitelist and blacklist naming convention, right now, on os_tempest we are still supporting these names, but it will be removed in the future.

Allow List Presentation topics

Introduction.
- What is allowed / skiplist
- The idea of this effort is to completly remove the list of tempest tests running on each job, from the featureset or zuul job definition
- Why use it instead of normal tempest list of tests
Objective.
- Make the list of tests running on tripleo and 3rd party centralized
  - Where the tests for job x is located?
    - job definition
    - default list of tests
    - featureset
  - When you want to test a new test, it requires to use featureset_override
- Quickly changes in the list of tests
  - If the test is in the featureset or in the job definition, we need to run several tripleo jobs (that may not be required)
- Easy for ruck rover maintain
  - Ruck rover time is always meaningfull, the difference between wait 10 minutes and 5 hours to merge a code, is huge in terms of zuul queue.
  - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/772319
    - slowest test takes 7 minutes
      - between check and gate, you have to wait 14 minutes (average) to merge a code
- Can be synchronized what is being executed and skipped between upstream and downstream
  - Train example, where tests were being skipped in upstream but not in downstream
- Can help other projects to collect data (Cibyl)
Overview / Presentation
- Show the repositories
- Show documentation
- Show how everything is being grouped
- Same job name, different releases
Summary/Conclusion.
- That's all folks
- Questions

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Tempest Skiplist / Allowed list

tags: skiplist tempest

Coding

Update the validate for tempest allowed list to also add jobs

Consolidate command

Zuul jobs

After everything is done

Allow List Presentation topics

recheck of all featuresets :

tags: `skiplist` `tempest`