owned this note   changed 4 years ago  
Published Linked with GitHub

Pro's and Con's of executing different tests on the same job based on where it's runnning

tags: Design

[Sandeep]

Pro:-

  • It make sense to only run tempest run related to a particular change. Save time, Job don't fails on not related failures.

    • For ex. On a network component, Running a storage related tempest test don't make sense.

  • Based on the bugs I have seen so far most of the tempest failure we catch: Is because of issues in individual component.. and not because of tripleo related changes. This gives us capability to run more tests early in component line and integration line and catch issues early.. (Personally, IMHO.. Check jobs should have a light coverage for tempest tests as compared to component and integration line.. Tripleo-ci check should catch deployment related bugs, not catch all the bugs caused by regression in other components)

Con:-

  • If we have different tests for ex> component line vs integration line - Troubeshooting gets difficult - incase one of a test which was not running in component line fails in integration line.
  • If we have less tests in component line than integration line - sometime issue can bypass component line

Component Line: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-network-rhos-16.2/0f011c9/logs/undercloud/var/log/tempest/test_list.txt.gz

Integration Line: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/b6a1a63/logs/undercloud/var/log/tempest/test_list.txt.gz

^^ 293 network related test in integration line, only 9 network test in network component line.

Opinion:

  • I think having capability of running different tests is good, But we need to make sure - We are not missing tests in the component line.

  • Integation line

    • cinder related tests. - 2
    • network related tests. - 2
    • baremetal related tests.
    • General tests

  • Component line

    • Cinder component line.
      • cinder related tests. - 10
      • General tests
    • Network component line - 10
      • Network related tests
      • General tests

[Chandan]

Pro:-

  • If the same job is running at multiple branches
    • Different tests are running there
      • In Case of Upstream, Job time is limited
        • Running a small set of tests there will not catch the issues there but the job might be green always
        • each branch has different version of tempest and tempest plugin
          • might be same of tests are not found in different version of tempest-plugins so we are running different tests to maintain coverage
      • In case of Periodic Line,
        • May be some of the tests are running in upstream and rest of them is running in periodic line, it maintains the coverage
          • But it will not catch issues as there is different set of tests used
            • Which indirectly affect the upstream jobs if all tests are passing in periodic but failing in upstream due to different tests
      • In case of component line,
        • If we are running the same set of tests inheriting from upstream job, then we are still lacking the coverage as it might pass in component line as well as in periodic line, but in upstream it might fail.

Con:-

Proposal

I think we are working on tempest tests consolidation where we define tests based on jobname and branch name, then we can define a policy like:

  • Super set of tests will be running in component line, have all coverage there
    • Cons: Time of the job might increase but the volume of the jobs is less there
  • A same of tests will be running in periodic and same will run in check
  • Benefits
    • Catch issues in component line
      • if missed, will be catched in periodic line and check will run fine,
        • if any patch cause issue, we can catch upstream easily.
Test selection
  • define a set of general tests and then add component specific tests in component line
  • For periodic
    • a sub set of general tests
    • a sub set of each component tests
  • Check will contain the same set of periodic tests
  • Let's create a general tests which comprises of all component tests (a test from each component which has proper coverage to test functionality of a particule component)

Select a repo