One tempest-skip to rule them all

The idea

Since we are using tempest-skiplist to setup a list of tests to be skipped, we can also have a list of tests that we want to execute. The tempest-skiplist relies on a yaml file with a list of tests that can be skipped per job and / or release. Using the same principle, we can generate a list of tests to be executed per job/release. [1]

1 - https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml

Why?

The scenario today

Today, we need to create a patch on zuul jobs to add or remove a test from the allowed list of tempest tests. This can take several hours, in the best scenario case where there are no failures. This not only increase the amount of resources, but also the amount of time to fix things up.

What about tempest-skip

Add a test to the list of skipped list takes 5 minutes of test running and once the change get merged, all the jobs are ready to consume it. That's simple like that

Effort required

Almost none, the actual yaml file is more skipped test driver, but having an allowed list, should be more job driven, the code is almost ready, all it's required is add a cliff command to list it. Tempest-skip already have a ansible module, that can handle that.

Also, it will be required to edit the jobs to no longer have the list of tests to be executed, handling that to tempest-skip.

So what that means?

This also means we have now a centralized place to those tests, that can be changed faster than the actual scenario, with an already command line, ansible module, and can be imported. This will reduce the resources that tripleo project uses for changes on tempest, since mostly of the time, our ruck rovers needs to add or remove tests from the skiplist.

Let's do it!

Yes, let's do it!