Tempest Test run info

Problem Statement V1

  • Tempest tests are configured in jobs as whitelist tests.
  • Those tests sometimes get skipped because of skiplist Or those tests are getting override becase of jobs parenting

Solution

  • Create centralized db with api requests, and push test results from zuul jobs to the db.
  • This can help us to integrate it with Cibyl.

Version V2

Problem Statement:

  • Handling Skipped Tempest tests: As a Ruck/Rover, It is the utmost responsability to promote the content every one week for a each release.
    In the promotion pipeline, We have limited resources. We have similar jobs on each release.

  • Case 1:

    • For a each release, few of the ovb jobs randomly started failing with different tempest tests failures. Each time the tempest failures are different.
    • So it is very hard for a ruck/rover to track it. We end up skipping some jobs from the criteria and promote the content.

  • Case 2:

    • When a particular tempest tests started failing and we do not know the root cause. Then we skip the tempest tests to skiplist and created the CIX and promote the content.
    • The bug later on appears in Downstream pipeline and it blocks the content later on.

Solution

BY taking a look at both cases, we need better solution to handle the skip list resolutions.

For Case 1:

  • Needs a dashboard to track the health of a job with running tempest tests over a periodic of time to track the random failures

For Case 2:

  • If a particular tests get skipped, then proper DFG needs to be periodically notified.

https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/tempest-skiplist-jobs.yaml

Select a repo