---
# System prepended metadata

title: Tempest Debugging
tags: [Training]

---

# Tempest Debugging
###### tags: `Training`

## Things to cover

### Part 1
* Read tempest logs and understand the failure
    * Timeout, real failure, concurrency
        * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/stestr_results.html.gz
    * Understanding what test was doing
        * Where to look for the code for a particular test
        * https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py
            * https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L726
    * Trackdown the logs to where it fails (from tempest -> container -> service log)
        * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/overcloud-controller-2/var/log/containers/neutron/server.log.1.gz
        * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/overcloud-controller-0/var/log/containers/nova/nova-api.log.txt.gz
    * Important logs and features
        * Concurrency
            * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
        * Stackviz
        * tempest_resuts.html
            * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/stestr_results.html.gz
        * tempest_run.log
            * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
        * tempestconf
            * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempestconf.log.txt.gz
        * Tempest tests runned and tempest.conf
            * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/home/zuul/tempest/etc/
    * testproject
        * Execute only the failing tests and skip the skiplist
            * https://review.rdoproject.org/r/c/testproject/+/39944/7/.zuul.yaml
    * Correct Version used
        * Tempest RPM version
            * https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/extra/rpm-list.txt.gz
                * openstack-tempest-30.1.1-0.20220509123356.a7bedbd.el9.noarch
                * wallaby: https://github.com/redhat-openstack/rdoinfo/blob/master/tags/wallaby.yml#L1138
                * for master:
                    * https://github.com/redhat-openstack/rdoinfo/blob/master/tags/zed-uc.yml#L1038
                        * zed-uc: 
                            * It means it is built from master
                            * if source-branch is specificd, it means it is built from a particular commit
                                * https://review.rdoproject.org/r/c/rdoinfo/+/40228
                        * From what commit then, take a look at temepst nvr 
                            * a7bedbd - https://opendev.org/openstack/tempest/commit/a7bedbde46ae2aec796837a7e69fbf35747f75cb
                * for tempest plugins list: https://github.com/rdo-packages/tempest-distgit/blob/rpm-master/openstack-tempest.spec#L123
    * How to skip a test in openstack-tempest-skiplist
        * https://docs.openstack.org/openstack-tempest-skiplist/latest/addtest/addtest.html#examples
        * Basic Skip Example
            * https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/833953/1/roles/validate-tempest/vars/tempest_skip.yml
        * Usage of installer:
            * https://docs.openstack.org/openstack-tempest-skiplist/latest/yaml/formatting.html#with-releases-and-installers
### Part 2
* How tempest.yml playbook works
* How to execute tempest locally in your deployment
    * Using os_tempest
    * Manually runing

### Part 3

* Debugging tempest test
    * Using pudb