Handling OpenStack tempest skip list

tags: Design

Problem Statement

From train release onwards, We moved to openstack-ansible-os_tempest ansible role for running tempest in TripleO CI Jobs in periodic pipeline, check and gate jobs.
It also leads to deprecation of validate-tempest.
The only bits which is used from validate-tempest TQE role is global skip list shared across all jobs.
It means if a tempest tests starts failing, we file a bug then add the tests to the global skip lists.

By Starting of Ussuri Release, Promotion pipeline got little change.
Now we have:

  • Component Pipeline

    • where component specific jobs gets triggered.
      • for example, If network related projects has content change then Network specific related jobs will be get triggered to test the network bits

  • Periodic Pipeline

    • Where all the components are tested together with a series of jobs then the contents moves to upstream and check gate pipeline.

  • Upstream Check and Gate Pipeline

    • Where a set of tripleo ci jobs get triggered based on their projects.

Now, Since we have a global skip list, if a new failing tests goes to global skip list. It is going to get skipped at all pipeline including integration, periodic and upstream
(check and gate pipeline).

The drawback of global skip list is that we might not able to test a particular feature or scenario at different pipeline leading to faulty content delivery in Upstream and we might
miss unknown issues.

The current skip list has following schema:

known_failures:
  - test: 'neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests'
    reason: 'Remove this blacklist when OVS 2.12 will be released and https://patchwork.ozlabs.org/patch/1102717/ merged.'
    lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'

And

  - test: 'tempest.api.network.test_networks.NetworksTest.test_external_network_visibility'
    reason: 'Running on undercloud'
    undercloud: true

Proposed Solution

Goal is to skip to avoid skipping tests at global level.
Since any pipeline is collection of a set of jobs and here is the current skip list schema:

known_failures:
  - test: 'neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests'
    reason: 'Remove this blacklist when OVS 2.12 will be released and https://patchwork.ozlabs.org/patch/1102717/ merged.'
    lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'

And

  - test: 'tempest.api.network.test_networks.NetworksTest.test_external_network_visibility'
    reason: 'Running on undercloud'
    undercloud: true

So we can add a new key job_name in the above skip list schema.
So the new schema will be:

known_failures:
  - test: 'neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests'
    reason: 'Remove this blacklist when OVS 2.12 will be released and https://patchwork.ozlabs.org/patch/1102717/ merged.'
    lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
    job_name: [<list of job name>] # Note: Job name will be a list.

There might be a case where we want to skip a test based on tempest conf feature enabled or disabled flags.
Then there schema will be:

known_failures:
  - test: 'neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests'
    reason: 'Remove this blacklist when OVS 2.12 will be released and https://patchwork.ozlabs.org/patch/1102717/ merged.'
    lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
    feature_skip: [tempest_tempest_conf_flags]

How to achieve the goal?

  1. Move the current skip list to a new repo:
    -> opendev.org/openstack/openstack-tempest-skiplist
  2. Add the above scheam in the global skip list
  3. Design skiplist parser to parse the skip list and generate tempest skip list based on conditions
  4. Integreating ** openstack-tempest-skiplist ** as a ansible collection as add as a part of os_tempest role?
  5. Create a tool which will read the yaml file, and generate the skipped list based on the criterias (release, job name, installer, etc)
  6. Designing the tool to auto skip the tests from here

Things Done till now

  1. New repo creation reviews

Action Items

  • Arx to talk about openstack-tempest-skiplist in openstack-ansible meeting
  • Finalize the schema 
    ​​​​ - test: 'telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration'
​​​​   reason: 'telemetry tests failed due to One or more ports have an IP allocation from this subnet'
​​​​   lp: 'https://bugs.launchpad.net/tripleo/+bug/1792862'


    ​​​​  - test: 'tempest.api.network.test_networks.NetworksTest.test_external_network_visibility'
​​​​    reason: 'Running on undercloud'
​​​​    undercloud: true
​​​​    jobs:
​​​​      - tripleo-bla-bla-bla
​​​​      - tripleo-ble-ble-ble
​​​​    releases:
​​​​      - master
​​​​      - newton
​​​​      - queens
    • What would be the new schema?
  • Start the cli command
    • Name of the command [ots]
    • Create the args:
      • MVP
        • –installer
        • –release
        • –job
      • Future
        • –installer
        • –feature or keyword
  • Create ansible collection/module
  • Create jobs to verify integrity of the yaml file

Tasks for MVP

  • Define the YAML file structure

    • Yaml file are defined as:
    ​​​​- test: 'test to be skipped'
​​​​  installers:
​​​​    - tripleo
​​​​  releases:
​​​​    - release1
​​​​      lp: launchpad
​​​​      bz: bugzilla
​​​​      reason: reason to skip on release 1
​​​​    - release2
​​​​      lp: launchpad
​​​​      bz: bugzilla
​​​​      reason: reason to skip on release 2
​​​​  jobs:
​​​​    - job1
​​​​    - job2

    Where:

    • test is the full test name to be used.
    • Installers are the type of installer that will execute tempest, right now is just tempest
    • Each test need to have a releases dict with at least one release
      • Each release must have either a launchpad or a bugzilla link, and a reason. You don't need to have both bz and lp but one is required.
    • Optional a test can have a list of jobs where this test will be skipped, if no job list is defined, the test will be skipped everywhere

  • start the cli command

    • Use Cliff

  • Consolidate the yaml file per release in one single file

  • create ansible collection/module

  • hook with os_tempest or fs052 job

Additional Requirments from PM @s

  • ensure the skip lists have a yaml linter job defined and running
  • start designing a molecule job that would ensure functionality. May need to run infrared plugin tests here as well.

Questions/Ideas/FAQs

  • How to make it extensible?
    • (mkopec) extensible how? Can you give an example?
  • How to consume with os_tempest?
  • Does everything is going to be in a single yaml file?
  • Should we also create some check jobs?
    • to verify that every test is mentioned only once
    • to check that all the keywords (release, job, …) are valid (to avoid putting there keywords not recognized by the tool responsible for filtering …)
  • What would be the output/input of the tools?
    • (mkopec) I think the output should be a tempest blacklist like file and the input is a yaml list where every item looks like above mentioned schema

Tempest skip list improvements RFE

Durring the centos-8 bootstrap

  • please only blacklist tests in zuul unless the test fails in
    • component pipeline
    • integration pipeline
    • upstream check / gate

Improve skip list to incorporate job name:

  • by including the job name we will be able to skip the test at the appropriate level w/o duplicating code in zuul.

Improvements for os-ansible-tempest + infrared integration

FEEDBACK from your RUCK AND OR ROVERS

  • install worked fine..

  • validate worked

  • the skip has the right test listed

 [rlandy@localhost openstack-tempest-skiplist]$ tempest-skip list --file roles/validate-tempest/vars/tempest_skip.yml --release master | grep volumes_luks     │························
| barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest.test_encrypted_cinder_volumes_luks          |

  • this should be highlighted https://docs.openstack.org/openstack-tempest-skiplist/latest/listyaml/listyaml.html

  • add a test to skip via the cli

    • no clue, no doc

    • Is one meant to add a skip via the tool or by hand.. I thought the tool

    • same question for removing tests… is the tool only informative or will it remove tests that you select. How does one get into that select menu btw?

    • note that you need to add check jobs and periodic jobs separately. It's a complete name match. (that is what I missed).

    • If the file should only be managed by the tool, we should add a note to the file saying so. Most files managed by tools have a note telling you not to edit them manually.

Select a repo