TOCI Review

What is TOCI?

TripleO CI test framework:

https://opendev.org/openstack/tripleo-ci/src/branch/master/README.rst
** (this is some old stuff - history lesson value only)
It's become the glue that controls the flow of quickstart-based tests
How quickstart gets run within Zuul tests - keeping in mind what needed to be passed to quickstart in https://hackmd.io/6ldru3bQSiWdP7KR8fT0wg?view

Where does TOCI live now?

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test
** tasks/main.yaml (flow of operations)
** templates (jinja templates - where the work gets done)

Where do TOCI tasks come into play within a test workflow?

See the tasks being executed in run-v3.yaml:

Examine the job definition …
https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml

pre-run:
  - playbooks/nodepool-provider/pre.yaml
  - playbooks/openstack-zuul-jobs/legacy/pre.yaml
  - playbooks/tripleo-ci/ceph.yaml
  - playbooks/tripleo-ci/pre.yaml
run: playbooks/tripleo-ci/run-v3.yaml
post-run: playbooks/tripleo-ci/post.yaml

2020-01-14 13:10:28.192677 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master]
2020-01-14 13:10:28.193224 | RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
...
2020-01-14 13:10:32.998848 | TASK [common : Include the vars to be available for the playbook]
2020-01-14 13:10:34.100319 | primary | ok
2020-01-14 13:10:34.335057 | 
2020-01-14 13:10:34.335347 | TASK [Set legacy log path]
2020-01-14 13:10:34.457115 | primary | ok
...
2020-01-14 13:10:38.387613 | TASK [run-test : Generate role variable file]
2020-01-14 13:10:41.225322 | primary | changed
2020-01-14 13:10:41.286281 | 
2020-01-14 13:10:41.286622 | TASK [run-test : Include role variables to be passed to quickstart]
2020-01-14 13:10:41.377597 | primary | ok
2020-01-14 13:10:41.428470 | 
2020-01-14 13:10:41.428770 | TASK [run-test : render toci_gate_test script]
2020-01-14 13:10:43.088280 | primary | changed
2020-01-14 13:10:43.142203 | 
2020-01-14 13:10:43.142509 | TASK [run-test : render toci_quickstart script]
2020-01-14 13:10:44.733544 | primary | changed
2020-01-14 13:10:44.776311 | 
2020-01-14 13:10:44.776605 | TASK [run-test : render common_vars script]
2020-01-14 13:10:46.217795 | primary | changed
2020-01-14 13:10:46.270920 | 
2020-01-14 13:10:46.271637 | TASK [run-test : render oooq_common_functions script]
2020-01-14 13:10:47.794898 | primary | changed
2020-01-14 13:10:47.848261 | 
2020-01-14 13:10:47.848542 | TASK [run-test : render ready releases.sh file]
2020-01-14 13:10:48.034745 | primary | skipping: Conditional result was False
2020-01-14 13:10:48.085191 | 
2020-01-14 13:10:48.085473 | TASK [run-test : run toci_gate_test.sh]
2020-01-14 13:10:48.780774 | primary | + export TOCI_JOBTYPE=standalone-featureset052
2020-01-14 13:10:48.781029 | primary | + TOCI_JOBTYPE=standalone-featureset052

Examining the TOCI jinja templates

toci_gate_test.sh

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2

setup for running toci_quickstart.sh
sources common_vars.bash and oooq_common_functions.sh
keeps track of job time
basic bootstrapping - for playbooks
** source /etc/nodepool/provider
** source /etc/ci/mirror_info.sh
creates the logs directory
installs python deps
install python3-setuptools python3-requests python3-urllib3 python3-PyYAML
sets up vars for periodic runs
sets up vars for update/upgrade (see updates/upgrades section below)
copy nodepool keys to current user
sets up IP references
finally, run quickstart

    ./toci_quickstart.sh

toci_quickstart.sh

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_quickstart.sh.j2

kicks off all the playbook runs and collect logs
passes variables to quickstart.sh via yaml files

echo "-- Playbooks Output --------------------------"
for playbook in {{ " ".join(playbooks) }}; do
    echo "$QUICKSTART_INSTALL_CMD \
        ${RELEASE_ARGS[$playbook]:=$QUICKSTART_DEFAULT_RELEASE_ARG} \
        {{ nodes_args }} \
        {{ featureset_conf }} \
        {{ env_vars }} \
        {{ role_vars }} \
        {{ extra_vars }} \
        {{ vxlan_vars }} \
        $DEFAULT_ARGS \
        $LOCAL_WORKING_DIR/playbooks/$playbook ${PLAYBOOKS_ARGS[$playbook]:-}" \
        --extra-vars @{{ workspace }}/logs/zuul-variables.yaml \
        --extra-vars @{{ workspace }}/logs/hostvars-variables.yaml \
        | sed  's/--/\n--/g' \
        | tee -a $LOGS_DIR/playbook_executions.log
    echo "# --------------------------------------- " \
        | tee -a $LOGS_DIR/playbook_executions.log
done

see variables definitions in: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/common/vars/main.yaml

and resolving this in a job log:

*2020-01-13 22:08:06.364052 | primary | +(./tociquickstart.sh:150):
/home/zuul/workspace/.quickstart/bin/ansible-playbook
–tags build,undercloud-setup,undercloud-scripts,undercloud-install,undercloud-validate,images
–extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-train.yml
–extra-vars @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/generalconfig/featureset-multinode-common.yml
–extra-vars @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/generalconfig/featureset027.yml
–extra-vars @/home/zuul/src/opendev.org/openstack/tripleo-ci/toci-quickstart/config/testenv/singlenode.yml
–extra-vars @/home/zuul/src/opendev.org/openstack/tripleo-ci/toci-quickstart/config/testenv/singlenode-rdocloud.yml
–extra-vars @/home/zuul/workspace/logs/role-vars.yaml
**-e *tocivxlannetworking=false
-e vxlannetworking=false
–extra-vars localworkingdir=/home/zuul/workspace/.quickstart
–extra-vars virthost=127.0.0.2
–inventory /home/zuul/workspace/.quickstart/hosts
–extra-vars tripleoroot=/home/zuul/src/opendev.org/openstack
–extra-vars workingdir=/home/zuul
–skip-tags tripleo-validations,teardown-all
–extra-vars @/home/zuul/workspace/logs/zuul-variables.yaml
–extra-vars @/home/zuul/workspace/logs/hostvars-variables.yaml
/home/zuul/workspace/.quickstart/playbooks/quickstart.yml

Reviewing the files passed:

oooq_common_functions.sh

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/oooq_common_functions.sh.j2

Use for functions referenced multiple times
Or referenced within TOCI

common_vars.bash

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/common_vars.bash.j2

contains env variables referenced multiple times
examples: RELEASE-, REVIEW-, etc. related

featureset-override

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/featureset-override.j2

To override settings in the featureset file
There is a very select list of variable allowed in this list:
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/tasks/main.yaml#L5
In the past, this list was used for any variable needed to be passed from the job definition to toci_quickstart.sh but preference now is to use job.{variable}
The override works because of the ORDER in which these files are passed - the last file passed wins:

echo "-- Playbooks Output --------------------------"
        ${RELEASE_ARGS[$playbook]:=$QUICKSTART_DEFAULT_RELEASE_ARG} \
        {{ nodes_args }} \
        {{ featureset_conf }} \
        {{ env_vars }} \
        {{ role_vars }} \
        {{ extra_vars }} \
        {{ vxlan_vars }} \
        $DEFAULT_ARGS \
        $LOCAL_WORKING_DIR/playbooks/$playbook ${PLAYBOOKS_ARGS[$playbook]:-}" \
        --extra-vars @{{ workspace }}/logs/zuul-variables.yaml \
        --extra-vars @{{ workspace }}/logs/hostvars-variables.yaml \
done

role-vars

https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/role-vars.j2

run_test_role_vars

emit-releases and updates/upgrades

https://github.com/openstack/tripleo-ci/tree/master/scripts/emit_releases_file/scripts/emit_releases_file
DOCS: https://docs.openstack.org/tripleo-docs/latest/ci/emit_releases_file.html

determines whether you need to upgrade/update
sets up the release variables accordingly
depends on the featureset number:
if [[ -f "$RELEASES_SCRIPT" ]] && [[ {{ featureset }} =~ 010|011|037|047|050|056 ]]; then
called from toci_gate_test.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2#L110
outputs a emit_releases_file.log

Got DLRN hash: ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb for the named hash: current on the master release
Got DLRN hash: f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e for the named hash: tripleo-ci-testing on the master release
Doing an undercloud upgrade
Got DLRN hash: df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045 for the named hash: current-tripleo on the train release
stable_release: master, featureset: {'containerized_overcloud_upgrade': False, 'undercloud_check_sanity': True, 'use_oooq': True, 'tripleo_ci': True, 'ssl_overcloud': False, 'undercloud_undercloud_hostname': 'undercloud.{{ undercloud_cloud_domain }}', 'undercloud_cloud_domain': 'localdomain', 'containerized_undercloud_upgrade': True, 'undercloud_upgrade': True, 'undercloud_enable_swift_encryption': True, 'run_tempest': False, 'undercloud_upgrade_cleanup': True, 'test_ping': False, 'containerized_undercloud': "{% if release not in ['newton','ocata','pike','queens'] -%} true {%- else -%} false {%- endif -%}", 'undercloud_generate_service_certificate': True, 'containerized_overcloud': "{% if release in ['newton', 'ocata', 'pike'] -%} false {%- else -%} true {%- endif -%}", 'undercloud_enable_validations': False}
output releases: {'standalone_target_release': 'master', 'standalone_deploy_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'undercloud_install_release': 'train', 'undercloud_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'overcloud_deploy_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'standalone_deploy_release': 'master', 'standalone_target_newest_hash': u'ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb', 'overcloud_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'overcloud_target_release': 'master', 'overcloud_deploy_release': 'master', 'standalone_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'undercloud_install_hash': u'df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045', 'standalone_deploy_newest_hash': u'ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb', 'undercloud_target_release': 'master'}

See emit releases reference in logs:

2020-01-14 17:40:49.686014 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:68): LWD=/home/zuul/workspace/.quickstart
2020-01-14 17:40:49.686168 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:70): export RELEASES_FILE_OUTPUT=/home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:49.686306 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:70): RELEASES_FILE_OUTPUT=/home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:49.686485 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:71): export RELEASES_SCRIPT=/home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py
2020-01-14 17:40:49.686664 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:71): RELEASES_SCRIPT=/home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py
2020-01-14 17:40:49.686816 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:72): export RELEASES_SCRIPT_LOGFILE=/home/zuul/workspace/logs/emit_releases_file.log
2020-01-14 17:40:49.686961 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:72): RELEASES_SCRIPT_LOGFILE=/home/zuul/workspace/logs/emit_releases_file.log

and the variables are then exported:

2020-01-14 17:40:50.214131 | primary | +(./toci_quickstart.sh:64): declare -A RELEASE_ARGS
2020-01-14 17:40:50.214222 | primary | +(./toci_quickstart.sh:66): [[ -f /home/zuul/workspace/logs/releases.sh ]]
2020-01-14 17:40:50.214305 | primary | +(./toci_quickstart.sh:68): source /home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:50.214415 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): export UNDERCLOUD_INSTALL_RELEASE=promotion-testing-hash-train
2020-01-14 17:40:50.214520 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): UNDERCLOUD_INSTALL_RELEASE=promotion-testing-hash-train
2020-01-14 17:40:50.214665 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): export UNDERCLOUD_INSTALL_HASH=df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045
2020-01-14 17:40:50.214793 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): UNDERCLOUD_INSTALL_HASH=df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045
2020-01-14 17:40:50.214907 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): export UNDERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215015 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): UNDERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215182 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): export UNDERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215308 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): UNDERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215419 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): export OVERCLOUD_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215530 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): OVERCLOUD_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215664 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): export OVERCLOUD_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215790 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): OVERCLOUD_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215905 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): export OVERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216010 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): OVERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216160 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): export OVERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216285 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): OVERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216400 | primary | ++(/home/zuul/workspace/logs/releases.sh:10): export STANDALONE_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216506 | primary | ++(/home/zuul/workspace/logs/releases.sh:10): STANDALONE_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216633 | primary | ++(/home/zuul/workspace/logs/releases.sh:11): export STANDALONE_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216754 | primary | ++(/home/zuul/workspace/logs/releases.sh:11): STANDALONE_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216884 | primary | ++(/home/zuul/workspace/logs/releases.sh:12): export STANDALONE_DEPLOY_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217017 | primary | ++(/home/zuul/workspace/logs/releases.sh:12): STANDALONE_DEPLOY_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217162 | primary | ++(/home/zuul/workspace/logs/releases.sh:13): export STANDALONE_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.217276 | primary | ++(/home/zuul/workspace/logs/releases.sh:13): STANDALONE_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.217415 | primary | ++(/home/zuul/workspace/logs/releases.sh:14): export STANDALONE_TARGET_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217543 | primary | ++(/home/zuul/workspace/logs/releases.sh:14): STANDALONE_TARGET_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217669 | primary | ++(/home/zuul/workspace/logs/releases.sh:15): export STANDALONE_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.217789 | primary | ++(/home/zuul/workspace/logs/releases.sh:15): STANDALONE_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e

TOCI is done when the logs show …

2020-01-13 23:00:21.281872 | primary | Quickstart completed.
2020-01-13 23:00:21.282917 | primary | +(./toci_quickstart.sh:171): exit 0
2020-01-13 23:00:21.356261 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:179): echo 'Run completed'
2020-01-13 23:00:21.356627 | primary | Run completed
2020-01-13 23:00:22.047912 | primary | ok: Runtime: 0:54:37.324951

Note that TOCI kicks collect-logs with a slightly different set of variables

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.