Tempest Debugging

tags: Training

Things to cover

Part 1

• Read tempest logs and understand the failure
  • Timeout, real failure, concurrency
    • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/stestr_results.html.gz
  • Understanding what test was doing
    • Where to look for the code for a particular test
    • https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py
      • https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L726
  • Trackdown the logs to where it fails (from tempest -> container -> service log)
    • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/overcloud-controller-2/var/log/containers/neutron/server.log.1.gz
    • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/overcloud-controller-0/var/log/containers/nova/nova-api.log.txt.gz
  • Important logs and features
    • Concurrency
      • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
    • Stackviz
    • tempest_resuts.html
      • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/stestr_results.html.gz
    • tempest_run.log
      • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
    • tempestconf
      • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempestconf.log.txt.gz
    • Tempest tests runned and tempest.conf
      • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/home/zuul/tempest/etc/
  • testproject
    • Execute only the failing tests and skip the skiplist
      • https://review.rdoproject.org/r/c/testproject/+/39944/7/.zuul.yaml
  • Correct Version used
    • Tempest RPM version
      • https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/extra/rpm-list.txt.gz
        • openstack-tempest-30.1.1-0.20220509123356.a7bedbd.el9.noarch
        • wallaby: https://github.com/redhat-openstack/rdoinfo/blob/master/tags/wallaby.yml#L1138
        • for master:
          • https://github.com/redhat-openstack/rdoinfo/blob/master/tags/zed-uc.yml#L1038
            • zed-uc:
              • It means it is built from master
              • if source-branch is specificd, it means it is built from a particular commit
                • https://review.rdoproject.org/r/c/rdoinfo/+/40228
            • From what commit then, take a look at temepst nvr
              • a7bedbd - https://opendev.org/openstack/tempest/commit/a7bedbde46ae2aec796837a7e69fbf35747f75cb
        • for tempest plugins list: https://github.com/rdo-packages/tempest-distgit/blob/rpm-master/openstack-tempest.spec#L123
  • How to skip a test in openstack-tempest-skiplist
    • https://docs.openstack.org/openstack-tempest-skiplist/latest/addtest/addtest.html#examples
    • Basic Skip Example
      • https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/833953/1/roles/validate-tempest/vars/tempest_skip.yml
    • Usage of installer:
      • https://docs.openstack.org/openstack-tempest-skiplist/latest/yaml/formatting.html#with-releases-and-installers

Part 2

• How tempest.yml playbook works
• How to execute tempest locally in your deployment
  • Using os_tempest
  • Manually runing

Part 3

• Debugging tempest test
  • Using pudb