owned this note
owned this note
Published
Linked with GitHub
# TOCI Review
## What is TOCI?
TripleO CI test framework:
* https://opendev.org/openstack/tripleo-ci/src/branch/master/README.rst
** (this is some ***old*** stuff - history lesson value only)
* It's become the glue that controls the flow of quickstart-based tests
* How quickstart gets run within Zuul tests - keeping in mind what needed to be passed to quickstart in https://hackmd.io/6ldru3bQSiWdP7KR8fT0wg?view
## Where does TOCI live now?
* https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test
** tasks/main.yaml (flow of operations)
** templates (jinja templates - where the work gets done)
## Where do TOCI ***tasks*** come into play within a test workflow?
See the ***tasks*** being executed in **run-v3.yaml**:
Examine the job definition ...
https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml
```
pre-run:
- playbooks/nodepool-provider/pre.yaml
- playbooks/openstack-zuul-jobs/legacy/pre.yaml
- playbooks/tripleo-ci/ceph.yaml
- playbooks/tripleo-ci/pre.yaml
run: playbooks/tripleo-ci/run-v3.yaml
post-run: playbooks/tripleo-ci/post.yaml
```
![](https://i.imgur.com/Zhtie6U.png)
```
2020-01-14 13:10:28.192677 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master]
2020-01-14 13:10:28.193224 | RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
...
2020-01-14 13:10:32.998848 | TASK [common : Include the vars to be available for the playbook]
2020-01-14 13:10:34.100319 | primary | ok
2020-01-14 13:10:34.335057 |
2020-01-14 13:10:34.335347 | TASK [Set legacy log path]
2020-01-14 13:10:34.457115 | primary | ok
...
2020-01-14 13:10:38.387613 | TASK [run-test : Generate role variable file]
2020-01-14 13:10:41.225322 | primary | changed
2020-01-14 13:10:41.286281 |
2020-01-14 13:10:41.286622 | TASK [run-test : Include role variables to be passed to quickstart]
2020-01-14 13:10:41.377597 | primary | ok
2020-01-14 13:10:41.428470 |
2020-01-14 13:10:41.428770 | TASK [run-test : render toci_gate_test script]
2020-01-14 13:10:43.088280 | primary | changed
2020-01-14 13:10:43.142203 |
2020-01-14 13:10:43.142509 | TASK [run-test : render toci_quickstart script]
2020-01-14 13:10:44.733544 | primary | changed
2020-01-14 13:10:44.776311 |
2020-01-14 13:10:44.776605 | TASK [run-test : render common_vars script]
2020-01-14 13:10:46.217795 | primary | changed
2020-01-14 13:10:46.270920 |
2020-01-14 13:10:46.271637 | TASK [run-test : render oooq_common_functions script]
2020-01-14 13:10:47.794898 | primary | changed
2020-01-14 13:10:47.848261 |
2020-01-14 13:10:47.848542 | TASK [run-test : render ready releases.sh file]
2020-01-14 13:10:48.034745 | primary | skipping: Conditional result was False
2020-01-14 13:10:48.085191 |
2020-01-14 13:10:48.085473 | TASK [run-test : run toci_gate_test.sh]
2020-01-14 13:10:48.780774 | primary | + export TOCI_JOBTYPE=standalone-featureset052
2020-01-14 13:10:48.781029 | primary | + TOCI_JOBTYPE=standalone-featureset052
```
******
## Examining the TOCI jinja templates
### toci_gate_test.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2
* **setup for running toci_quickstart.sh**
* sources `common_vars.bash` and `oooq_common_functions.sh`
* keeps track of job time
* basic bootstrapping - for playbooks
** `source /etc/nodepool/provider`
** `source /etc/ci/mirror_info.sh`
* creates the logs directory
* installs python deps
`install python3-setuptools python3-requests python3-urllib3 python3-PyYAML`
* sets up vars for periodic runs
* sets up vars for update/upgrade (see updates/upgrades section below)
* copy nodepool keys to current user
* sets up IP references
* finally, run quickstart
```
./toci_quickstart.sh
```
### toci_quickstart.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_quickstart.sh.j2
* kicks off all the playbook runs and collect logs
* passes variables to quickstart.sh via `yaml` files
```
echo "-- Playbooks Output --------------------------"
for playbook in {{ " ".join(playbooks) }}; do
echo "$QUICKSTART_INSTALL_CMD \
${RELEASE_ARGS[$playbook]:=$QUICKSTART_DEFAULT_RELEASE_ARG} \
{{ nodes_args }} \
{{ featureset_conf }} \
{{ env_vars }} \
{{ role_vars }} \
{{ extra_vars }} \
{{ vxlan_vars }} \
$DEFAULT_ARGS \
$LOCAL_WORKING_DIR/playbooks/$playbook ${PLAYBOOKS_ARGS[$playbook]:-}" \
--extra-vars @{{ workspace }}/logs/zuul-variables.yaml \
--extra-vars @{{ workspace }}/logs/hostvars-variables.yaml \
| sed 's/--/\n--/g' \
| tee -a $LOGS_DIR/playbook_executions.log
echo "# --------------------------------------- " \
| tee -a $LOGS_DIR/playbook_executions.log
done
```
see variables definitions in: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/common/vars/main.yaml
and resolving this in a job log:
*2020-01-13 22:08:06.364052 | primary | +(./tociquickstart.sh:150):
/home/zuul/workspace/.quickstart/bin/ansible-playbook
--tags build,undercloud-setup,undercloud-scripts,undercloud-install,undercloud-validate,images
**--extra-vars** @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-train.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/generalconfig/featureset-multinode-common.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/generalconfig/featureset027.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-ci/toci-quickstart/config/testenv/singlenode.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-ci/toci-quickstart/config/testenv/singlenode-rdocloud.yml
**--extra-vars** @/home/zuul/workspace/logs/role-vars.yaml
**-e **tocivxlannetworking=false
**-e** vxlannetworking=false
**--extra-vars** localworkingdir=/home/zuul/workspace/.quickstart
**--extra-vars** virthost=127.0.0.2
**--inventory** /home/zuul/workspace/.quickstart/hosts
**--extra-vars** tripleoroot=/home/zuul/src/opendev.org/openstack
**--extra-vars** workingdir=/home/zuul
**--skip-tags** tripleo-validations,teardown-all
**--extra-vars** @/home/zuul/workspace/logs/zuul-variables.yaml
**--extra-vars** @/home/zuul/workspace/logs/hostvars-variables.yaml
/home/zuul/workspace/.quickstart/playbooks/quickstart.yml*
#### Reviewing the files passed:
* nodes: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/nodes
* release: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release
* testenv: https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/
* featureset: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config
### oooq_common_functions.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/oooq_common_functions.sh.j2
* Use for functions referenced multiple times
* Or referenced within TOCI
### common_vars.bash
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/common_vars.bash.j2
* contains env variables referenced multiple times
* examples: RELEASE-, REVIEW-, etc. related
### featureset-override
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/featureset-override.j2
* To override settings in the featureset file
* There is a very select list of variable allowed in this list:
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/tasks/main.yaml#L5
* In the past, this list was used for any variable needed to be passed from the job definition to toci_quickstart.sh but preference now is to use `job.{variable}`
* The override works because of the ORDER in which these files are passed - **the last file passed wins**:
```
echo "-- Playbooks Output --------------------------"
${RELEASE_ARGS[$playbook]:=$QUICKSTART_DEFAULT_RELEASE_ARG} \
{{ nodes_args }} \
{{ featureset_conf }} \
{{ env_vars }} \
{{ role_vars }} \
{{ extra_vars }} \
{{ vxlan_vars }} \
$DEFAULT_ARGS \
$LOCAL_WORKING_DIR/playbooks/$playbook ${PLAYBOOKS_ARGS[$playbook]:-}" \
--extra-vars @{{ workspace }}/logs/zuul-variables.yaml \
--extra-vars @{{ workspace }}/logs/hostvars-variables.yaml \
done
```
### role-vars
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/role-vars.j2
* `run_test_role_vars`
## emit-releases and updates/upgrades
https://github.com/openstack/tripleo-ci/tree/master/scripts/emit_releases_file/scripts/emit_releases_file
DOCS: https://docs.openstack.org/tripleo-docs/latest/ci/emit_releases_file.html
* determines whether you need to upgrade/update
* sets up the release variables accordingly
* depends on the featureset number:
`if [[ -f "$RELEASES_SCRIPT" ]] && [[ {{ featureset }} =~ 010|011|037|047|050|056 ]]; then`
* called from `toci_gate_test.sh`
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2#L110
* outputs a emit_releases_file.log
```
Got DLRN hash: ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb for the named hash: current on the master release
Got DLRN hash: f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e for the named hash: tripleo-ci-testing on the master release
Doing an undercloud upgrade
Got DLRN hash: df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045 for the named hash: current-tripleo on the train release
stable_release: master, featureset: {'containerized_overcloud_upgrade': False, 'undercloud_check_sanity': True, 'use_oooq': True, 'tripleo_ci': True, 'ssl_overcloud': False, 'undercloud_undercloud_hostname': 'undercloud.{{ undercloud_cloud_domain }}', 'undercloud_cloud_domain': 'localdomain', 'containerized_undercloud_upgrade': True, 'undercloud_upgrade': True, 'undercloud_enable_swift_encryption': True, 'run_tempest': False, 'undercloud_upgrade_cleanup': True, 'test_ping': False, 'containerized_undercloud': "{% if release not in ['newton','ocata','pike','queens'] -%} true {%- else -%} false {%- endif -%}", 'undercloud_generate_service_certificate': True, 'containerized_overcloud': "{% if release in ['newton', 'ocata', 'pike'] -%} false {%- else -%} true {%- endif -%}", 'undercloud_enable_validations': False}
output releases: {'standalone_target_release': 'master', 'standalone_deploy_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'undercloud_install_release': 'train', 'undercloud_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'overcloud_deploy_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'standalone_deploy_release': 'master', 'standalone_target_newest_hash': u'ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb', 'overcloud_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'overcloud_target_release': 'master', 'overcloud_deploy_release': 'master', 'standalone_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'undercloud_install_hash': u'df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045', 'standalone_deploy_newest_hash': u'ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb', 'undercloud_target_release': 'master'}
```
* See emit releases reference in logs:
```
2020-01-14 17:40:49.686014 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:68): LWD=/home/zuul/workspace/.quickstart
2020-01-14 17:40:49.686168 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:70): export RELEASES_FILE_OUTPUT=/home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:49.686306 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:70): RELEASES_FILE_OUTPUT=/home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:49.686485 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:71): export RELEASES_SCRIPT=/home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py
2020-01-14 17:40:49.686664 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:71): RELEASES_SCRIPT=/home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py
2020-01-14 17:40:49.686816 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:72): export RELEASES_SCRIPT_LOGFILE=/home/zuul/workspace/logs/emit_releases_file.log
2020-01-14 17:40:49.686961 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:72): RELEASES_SCRIPT_LOGFILE=/home/zuul/workspace/logs/emit_releases_file.log
```
and the variables are then exported:
```
2020-01-14 17:40:50.214131 | primary | +(./toci_quickstart.sh:64): declare -A RELEASE_ARGS
2020-01-14 17:40:50.214222 | primary | +(./toci_quickstart.sh:66): [[ -f /home/zuul/workspace/logs/releases.sh ]]
2020-01-14 17:40:50.214305 | primary | +(./toci_quickstart.sh:68): source /home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:50.214415 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): export UNDERCLOUD_INSTALL_RELEASE=promotion-testing-hash-train
2020-01-14 17:40:50.214520 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): UNDERCLOUD_INSTALL_RELEASE=promotion-testing-hash-train
2020-01-14 17:40:50.214665 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): export UNDERCLOUD_INSTALL_HASH=df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045
2020-01-14 17:40:50.214793 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): UNDERCLOUD_INSTALL_HASH=df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045
2020-01-14 17:40:50.214907 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): export UNDERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215015 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): UNDERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215182 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): export UNDERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215308 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): UNDERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215419 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): export OVERCLOUD_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215530 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): OVERCLOUD_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215664 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): export OVERCLOUD_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215790 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): OVERCLOUD_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215905 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): export OVERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216010 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): OVERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216160 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): export OVERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216285 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): OVERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216400 | primary | ++(/home/zuul/workspace/logs/releases.sh:10): export STANDALONE_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216506 | primary | ++(/home/zuul/workspace/logs/releases.sh:10): STANDALONE_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216633 | primary | ++(/home/zuul/workspace/logs/releases.sh:11): export STANDALONE_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216754 | primary | ++(/home/zuul/workspace/logs/releases.sh:11): STANDALONE_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216884 | primary | ++(/home/zuul/workspace/logs/releases.sh:12): export STANDALONE_DEPLOY_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217017 | primary | ++(/home/zuul/workspace/logs/releases.sh:12): STANDALONE_DEPLOY_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217162 | primary | ++(/home/zuul/workspace/logs/releases.sh:13): export STANDALONE_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.217276 | primary | ++(/home/zuul/workspace/logs/releases.sh:13): STANDALONE_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.217415 | primary | ++(/home/zuul/workspace/logs/releases.sh:14): export STANDALONE_TARGET_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217543 | primary | ++(/home/zuul/workspace/logs/releases.sh:14): STANDALONE_TARGET_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217669 | primary | ++(/home/zuul/workspace/logs/releases.sh:15): export STANDALONE_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.217789 | primary | ++(/home/zuul/workspace/logs/releases.sh:15): STANDALONE_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
```
## TOCI is done when the logs show ...
```
2020-01-13 23:00:21.281872 | primary | Quickstart completed.
2020-01-13 23:00:21.282917 | primary | +(./toci_quickstart.sh:171): exit 0
2020-01-13 23:00:21.356261 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:179): echo 'Run completed'
2020-01-13 23:00:21.356627 | primary | Run completed
2020-01-13 23:00:22.047912 | primary | ok: Runtime: 0:54:37.324951
```
* Note that TOCI kicks collect-logs with a slightly different set of variables