# Tempest Skiplist / Allowed list

###### tags: `skiplist` `tempest`

The idea of this effort is to completly remove the list of tempest tests running on each job, from the featureset or zuul job definition

## Coding

### Update the validate for tempest allowed list to also add jobs

Right now, we have two ways to use the allowed list, the first one is per job, and the second one is for groups. Right now, the search is done by job or by group, which means, if in the cli you pass --group it will search only for group, howerver, due to cybil efforts, we need a way to also pass --job and return the job as well. so, the groups yaml file that today is like this:

groups:
  - name: default
    tests:
      - smoke
    releases:
      - master
  - name: featureset001
    tests:
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
  - name: featureset001_periodic
    tests:
      - 'tempest.api'
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
jobs:
  - name: standalone
    tests:
      - test1
      - test2
    releases:
      - master
      - xena
      - victoria

Will need to have also the list of jobs:

groups:
  - name: default
    tests:
      - smoke
    releases:
      - master
  - name: featureset001
    tests:
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
    # List of jobs for this group
    jobs:
      - job1
      - job2
  - name: featureset001_periodic
    tests:
      - 'tempest.api'
      - 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
    releases:
      - master
      - xena
      - wallaby
      - victoria
      - ussuri
      - train
    jobs:
      - job1
      - job2

Of course, the list of the jobs should be optional to keep compatibility

Also, need to ensure that when you pass --group and --job the search will be done properly, which means --group featureset001 --job job4 should return the tempest.scenario.test_network_basic_ops.TestNetworkBasicOps even though job4 is not in the jobs list

### Consolidate command

We want to be able to tell which list of tests will be executed passing only the job, and this will need to read both the tempest-skip and tempest-allowed list, for example:

tempest-skip tests --job job1 --allow-file allow.yaml --skip-file skip.yaml --release master

This will return the tests that will be executed, excluding the tests to be skipped, for example, for job1 we have tempest_network_basic_ops and tempest_scenario_ops on the list of jobs to be executed, and on the skip list we have tempest_scenario_ops so the output should be only tempest_network_basic_ops.

## Zuul jobs

For the zuul job, right now, all the logic is made on https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L36-L54 so basically, we need to remove the tempest_test_whitelist variable either from the zuul job definition as for example:

- job:
    name: tripleo-ci-scenario000-standalone-options
    vars:
      featureset: '052'
      featureset_override:
        standalone_environment_files:
          - 'environments/low-memory-usage.yaml'
          - 'ci/environments/scenario000-standalone.yaml'
        tempest_services:
          - keystone
        tempest_run_concurrency: 1
        tempest_tempest_conf_overrides:
          'auth.tempest_roles': "Member"
        # Remove this:
        tempest_test_whitelist:
          - 'keystone_tempest_plugin.tests.api.identity.v3'

But also, pay attention that we need to remove from featureset052 as well:

https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset052.yml#L33

In some cases, we have this:

tempest_test_whitelist: >-
  {% if (('periodic' in zuul.pipeline and not job.force_non_periodic|default(false)|bool) or (job.force_periodic|default(false)|bool)) and not tempest_run_skipped | default(false) | bool -%}
  {{ tempest_periodic_regex }}
  {%- else -%}
  {{ tempest_whitelist }}
  {%- endif -%}  

That also need to be removed and add on the tempest-allow.yaml the correspondent test.

Pay attention that once you remove from featureset052, ALL the jobs that doesn't have tempest_test_whitelist enabled will run only tempest.api.keystone tests (the default test to be executed if no tests were given)

So, my steps would be:

I would go for a featureset per time, for example, featureset052, and check all the jobs that run featureset052

https://codesearch.opendev.org/?q=featureset%3A+%27052%27&i=nope&literal=nope&files=&excludeFiles=&repos=

and

https://codesearch.rdoproject.org/codesearch/?q=featureset%3A+%27052%27&i=nope&files=&repos=

Then check which of those jobs have tempest_test_whitelist defined in zuul job definition, and which don't. For the jobs that doesn't have the tempest_test_whitelist defined in zuul job definition, I would create a group in the tempest-allow (in this case featureset052 and featureset052_periodic, if there are difference between both). Then, I would add on those jobs in the zuul job definition the variable tempest_allowed_group to featureset052 or featureset052_periodic

After that it is safe to remove tempest_test_whitelist from featureset052, and then one by one, add an entry on tempest-allow to each job that runs featureset052 but it does have the tempest_test_whitelist defined in zuul job definition.

List of featureset used in tripleo-ci

* featureset: '010' - containers-multinode (soniya)
* featureset: '027' - undercloud-containers (soniya)
* featureset: '030' - scenario007-multinode-oooq-container (soniya)
* featureset: '037' - minor update (use_os_tempest: false) (soniya)
* featureset: '062' - scenario010-standalone (no whitelist) (soniya)

* ***featureset: '052' - centos-8-standalone*** (pooja)
* featureset: '056' - standalone-upgrade (pooja)
* featureset: '060' - scenario012-standalone (pooja)
* featureset: '061' - containers-undercloud-minion (pooja)
* featureset: '065' - undercloud-ffu-wallaby - no whitelist @fs config file.
(https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset065.yml#L26)
List of featureset used in rdo-jobs

* ***featureset: '052' - standalone (whitelist)***
- [x] featureset: '056' - standalone-upgrade - used 2-3 pipelines
- [x] featureset: '062' - scenario010-standalone - used in standalone jobs(upstream/downstream)
- [x] featureset: '060' - scenario012-standalone - used in tripleo-ci-scenario012-standalone-options(only one check job)
- [x] featureset: '001' - ovb + integration cs9 master
- [x] featureset: '035' - ovb + integration cs9 master
- [x] featureset: '039' - ovb + " "
- [x] featureset: '002' - ovb + " "
- [x] featureset: '003' - ovb - used nowhere
- [x] featureset: '020' - ovb + " "
- [ ] featureset: '021' - ovb (special case) - on hold - used no where
- [x] featureset: '063' - ovb - used in check + periodic(rdoproject)
- [x] featureset: '010' - multinode - used in check + neutron/tripleo
- [x] featureset: '030' - multinode - used in periodic + check
- [x] featureset: '037' - multinode (no whitelist) - used in 2 periodic jobs
- [x] featureset: '027' - undercloud-upgrade - used in 1 periodic job
- [x] featureset: '050' - undercloud-upgrade (no whitelist) - used in undercloud/component jobs master/tripleoci
- [x] featureset: '051' - major upgrade (no whitelist) - used no where
- [x] featureset: '066' - used in 1 job - used in 3 jobs(rdo+upstream)
- [x] featureset: '054' - not used anywhere
- [x] featureset: '005' - used nowhere
- [x] featureset: '006' - used nowhere
- [x] featureset: '007' - used nowhere
- [x] featureset: '008' - used nowhere
- [x] featureset: '016' - used in rdoinfo jobs
- [ ] featureset: '017' - used nowhere
- [ ] featureset: '018' - used nowhere
- [ ] featureset: '019' - used nowhere
- [ ] featureset: '028' - used nowhere
- [ ] featureset: '030' - have test_REgex present(multinode jobs in upstream and rdojobs)
- [ ] featureset: '031' - used nowhere
- [ ] featureset: '032' - have test_regex present(used nowhere)
- [ ] featureset: '035' - rdojobs only (integration cs9 master + ovb jobs)
- [ ] featureset: '036' - have test_regex present(used nowhere)
- [ ] featureset: '037' - have test_regex(upgrade_job_templates + multinode_jobs)
- [ ] featureset: '038' - special case(used nowhere)
- [x] featureset: '039' - used in rdojobs - (ovb + integration_jobs_cs9)
- [ ] featureset: '041' - used nowhere
- [ ] featureset: '042' - used nowhere
- [ ] featureset: '043' - used nowhere
- [ ] featureset: '048' - have test_regex..(used nowhere)
- [ ] featureset: '052' - used 3-4 jobs in upstream + rdoproject
- [ ] featureset: '055' - have test_regex...used nowhere
- [x] featureset: '056' - have test_regex...used in 3 rdo pipelines
- [ ] featureset: '058' - used nowhere
- [ ] featureset: '061' - have test_regex(tripleo-ci-centos-8-containers-undercloud-minion)
- [ ] featureset: '064' - used in integration job cs9 + ovb
- [ ] featureset: '099' - used nowhere
## After everything is done

We need to get rid of the whitelist and blacklist naming convention, right now, on os_tempest we are still supporting these names, but it will be removed in the future.

## Allow List Presentation topics
* Introduction.
* What is allowed / skiplist
- The idea of this effort is to completly remove the list of tempest tests running on each job, from the featureset or zuul job definition
* Why use it instead of normal tempest list of tests
* Objective.
* Make the list of tests running on tripleo and 3rd party centralized
* Where the tests for job x is located?
* job definition
* default list of tests
* featureset
* When you want to test a new test, it requires to use featureset_override
* Quickly changes in the list of tests
* If the test is in the featureset or in the job definition, we need to run several tripleo jobs (that may not be required)
* Easy for ruck rover maintain
* Ruck rover time is always meaningfull, the difference between wait 10 minutes and 5 hours to merge a code, is huge in terms of zuul queue.
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/772319
* slowest test takes 7 minutes
* between check and gate, you have to wait 14 minutes (average) to merge a code
* Can be synchronized what is being executed and skipped between upstream and downstream
* Train example, where tests were being skipped in upstream but not in downstream
* Can help other projects to collect data (Cibyl)
* Overview / Presentation
* Show the repositories
* Show documentation
* Show how everything is being grouped
* Same job name, different releases
* Summary/Conclusion.
* That's all folks
* Questions

## recheck of all featuresets :

- [x] featureset: '052' - done
- [x] featureset: '056' - done
- [x] featureset: '062' - done
- [x] featureset: '060' - done
- [x] featureset: '010' - done
- [x] featureset: '030' - done
- [x] featureset: '027' - done
- [x] featureset: '039' - done
- [x] featureset: '001' - whitelist nowhere
- [x] featureset: '035' - done
- [x] featureset: '039' - whitelist nowhere
- [x] featureset: '002' - whitelist nowhere
- [x] featureset: '003' - not used anywhere
- [x] featureset: '020' - ready for review
- [x] featureset: '021' - not used anywhere
- [x] featureset: '063' - ready for review
- [x] featureset: '037' - not used anywhere
- [x] featureset: '050' - not used anywhere
- [x] featureset: '051' - not used anywhere
- [x] featureset: '066' - done
- [x] featureset: '054' - not used anywhere
- [x] featureset: '005' - not used anywhere
- [x] featureset: '006' - not used anywhere
- [x] featureset: '007' - not used anywhere
- [x] featureset: '008' - not used anywhere
- [x] featureset: '016' - not used anywhere
- [x] featureset: '017' - not used anywhere
- [x] featureset: '018' - not used anywhere
- [x] featureset: '019' - not used anywhere
- [x] featureset: '028' - not used anywhere
- [x] featureset: '031' - not used anywhere
- [x] featureset: '032' - not used anywhere
- [x] featureset: '036' - not used anywhere
- [x] featureset: '037' - whitelist nowhere
- [x] featureset: '038' - not used anywhere
- [x] featureset: '041' - not used anywhere
- [x] featureset: '042' - not used anywhere
- [x] featureset: '043' - not used anywhere
- [x] featureset: '048' - not used anywhere
- [x] featureset: '055' - not used anywhere
- [x] featureset: '056' - not used anywhere
- [x] featureset: '058' - not used anywhere
- [x] featureset: '061' - not used anywhere
- [x] featureset: '064' - review on gerrit
- [x] featureset: '099' - not used anywhere