# Tempest Debugging
###### tags: `Training`

## Things to cover

### Part 1
* Read tempest logs and understand the failure
* Timeout, real failure, concurrency
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/stestr_results.html.gz
* Understanding what test was doing
* Where to look for the code for a particular test
* https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py
* https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L726
* Trackdown the logs to where it fails (from tempest -> container -> service log)
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/overcloud-controller-2/var/log/containers/neutron/server.log.1.gz
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/overcloud-controller-0/var/log/containers/nova/nova-api.log.txt.gz
* Important logs and features
* Concurrency
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
* Stackviz
* tempest_resuts.html
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/stestr_results.html.gz
* tempest_run.log
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
* tempestconf
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/tempest/tempestconf.log.txt.gz
* Tempest tests runned and tempest.conf
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/home/zuul/tempest/etc/
* testproject
* Execute only the failing tests and skip the skiplist
* https://review.rdoproject.org/r/c/testproject/+/39944/7/.zuul.yaml
* Correct Version used
* Tempest RPM version
* https://logserver.rdoproject.org/57/39357/20/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/c6f3db6/logs/undercloud/var/log/extra/rpm-list.txt.gz
* openstack-tempest-30.1.1-0.20220509123356.a7bedbd.el9.noarch
* wallaby: https://github.com/redhat-openstack/rdoinfo/blob/master/tags/wallaby.yml#L1138
* for master:
* https://github.com/redhat-openstack/rdoinfo/blob/master/tags/zed-uc.yml#L1038
* zed-uc:
* It means it is built from master
* if source-branch is specificd, it means it is built from a particular commit
* https://review.rdoproject.org/r/c/rdoinfo/+/40228
* From what commit then, take a look at temepst nvr
* a7bedbd - https://opendev.org/openstack/tempest/commit/a7bedbde46ae2aec796837a7e69fbf35747f75cb
* for tempest plugins list: https://github.com/rdo-packages/tempest-distgit/blob/rpm-master/openstack-tempest.spec#L123
* How to skip a test in openstack-tempest-skiplist
* https://docs.openstack.org/openstack-tempest-skiplist/latest/addtest/addtest.html#examples
* Basic Skip Example
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/833953/1/roles/validate-tempest/vars/tempest_skip.yml
* Usage of installer:
* https://docs.openstack.org/openstack-tempest-skiplist/latest/yaml/formatting.html#with-releases-and-installers
### Part 2
* How tempest.yml playbook works
* How to execute tempest locally in your deployment
* Using os_tempest
* Manually runing

### Part 3

* Debugging tempest test
* Using pudb