# TOCI Review

## What is TOCI?
TripleO CI test framework:
* https://opendev.org/openstack/tripleo-ci/src/branch/master/README.rst
** (this is some ***old*** stuff - history lesson value only)
* It's become the glue that controls the flow of quickstart-based tests
* How quickstart gets run within Zuul tests - keeping in mind what needed to be passed to quickstart in https://hackmd.io/6ldru3bQSiWdP7KR8fT0wg?view

## Where does TOCI live now?
* https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test
** tasks/main.yaml (flow of operations)
** templates (jinja templates - where the work gets done)

## Where do TOCI ***tasks*** come into play within a test workflow?
See the ***tasks*** being executed in **run-v3.yaml**:

Examine the job definition ...
https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml

pre-run:
  - playbooks/nodepool-provider/pre.yaml
  - playbooks/openstack-zuul-jobs/legacy/pre.yaml
  - playbooks/tripleo-ci/ceph.yaml
  - playbooks/tripleo-ci/pre.yaml
run: playbooks/tripleo-ci/run-v3.yaml
post-run: playbooks/tripleo-ci/post.yaml


2020-01-14 13:10:28.192677 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master]
2020-01-14 13:10:28.193224 | RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
...
2020-01-14 13:10:32.998848 | TASK [common : Include the vars to be available for the playbook]
2020-01-14 13:10:34.100319 | primary | ok
2020-01-14 13:10:34.335057 | 
2020-01-14 13:10:34.335347 | TASK [Set legacy log path]
2020-01-14 13:10:34.457115 | primary | ok
...
2020-01-14 13:10:38.387613 | TASK [run-test : Generate role variable file]
2020-01-14 13:10:41.225322 | primary | changed
2020-01-14 13:10:41.286281 | 
2020-01-14 13:10:41.286622 | TASK [run-test : Include role variables to be passed to quickstart]
2020-01-14 13:10:41.377597 | primary | ok
2020-01-14 13:10:41.428470 | 
2020-01-14 13:10:41.428770 | TASK [run-test : render toci_gate_test script]
2020-01-14 13:10:43.088280 | primary | changed
2020-01-14 13:10:43.142203 | 
2020-01-14 13:10:43.142509 | TASK [run-test : render toci_quickstart script]
2020-01-14 13:10:44.733544 | primary | changed
2020-01-14 13:10:44.776311 | 
2020-01-14 13:10:44.776605 | TASK [run-test : render common_vars script]
2020-01-14 13:10:46.217795 | primary | changed
2020-01-14 13:10:46.270920 | 
2020-01-14 13:10:46.271637 | TASK [run-test : render oooq_common_functions script]
2020-01-14 13:10:47.794898 | primary | changed
2020-01-14 13:10:47.848261 | 
2020-01-14 13:10:47.848542 | TASK [run-test : render ready releases.sh file]
2020-01-14 13:10:48.034745 | primary | skipping: Conditional result was False
2020-01-14 13:10:48.085191 | 
2020-01-14 13:10:48.085473 | TASK [run-test : run toci_gate_test.sh]
2020-01-14 13:10:48.780774 | primary | + export TOCI_JOBTYPE=standalone-featureset052
2020-01-14 13:10:48.781029 | primary | + TOCI_JOBTYPE=standalone-featureset052

******

## Examining the TOCI jinja templates
### toci_gate_test.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2

* **setup for running toci_quickstart.sh**
* sources `common_vars.bash` and `oooq_common_functions.sh`
* keeps track of job time
* basic bootstrapping - for playbooks
** `source /etc/nodepool/provider`
** `source /etc/ci/mirror_info.sh`
* creates the logs directory
* installs python deps
`install python3-setuptools python3-requests python3-urllib3 python3-PyYAML`
* sets up vars for periodic runs
* sets up vars for update/upgrade (see updates/upgrades section below)
* copy nodepool keys to current user
* sets up IP references
* finally, run quickstart

    ./toci_quickstart.sh

### toci_quickstart.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_quickstart.sh.j2

* kicks off all the playbook runs and collect logs
* passes variables to quickstart.sh via `yaml` files

echo "-- Playbooks Output --------------------------"
for playbook in {{ " ".join(playbooks) }}; do
    echo "$QUICKSTART_INSTALL_CMD \
        ${RELEASE_ARGS[$playbook]:=$QUICKSTART_DEFAULT_RELEASE_ARG} \
        {{ nodes_args }} \
        {{ featureset_conf }} \
        {{ env_vars }} \
        {{ role_vars }} \
        {{ extra_vars }} \
        {{ vxlan_vars }} \
        $DEFAULT_ARGS \
        $LOCAL_WORKING_DIR/playbooks/$playbook ${PLAYBOOKS_ARGS[$playbook]:-}" \
        --extra-vars @{{ workspace }}/logs/zuul-variables.yaml \
        --extra-vars @{{ workspace }}/logs/hostvars-variables.yaml \
        | sed  's/--/\n--/g' \
        | tee -a $LOGS_DIR/playbook_executions.log
    echo "# --------------------------------------- " \
        | tee -a $LOGS_DIR/playbook_executions.log
done

see variables definitions in: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/common/vars/main.yaml

and resolving this in a job log:

*2020-01-13 22:08:06.364052 | primary | +(./tociquickstart.sh:150):
/home/zuul/workspace/.quickstart/bin/ansible-playbook
--tags build,undercloud-setup,undercloud-scripts,undercloud-install,undercloud-validate,images
**--extra-vars** @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-train.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/generalconfig/featureset-multinode-common.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/generalconfig/featureset027.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-ci/toci-quickstart/config/testenv/singlenode.yml
**--extra-vars** @/home/zuul/src/opendev.org/openstack/tripleo-ci/toci-quickstart/config/testenv/singlenode-rdocloud.yml
**--extra-vars** @/home/zuul/workspace/logs/role-vars.yaml
**-e **tocivxlannetworking=false
**-e** vxlannetworking=false
**--extra-vars** localworkingdir=/home/zuul/workspace/.quickstart
**--extra-vars** virthost=127.0.0.2
**--inventory** /home/zuul/workspace/.quickstart/hosts
**--extra-vars** tripleoroot=/home/zuul/src/opendev.org/openstack
**--extra-vars** workingdir=/home/zuul
**--skip-tags** tripleo-validations,teardown-all
**--extra-vars** @/home/zuul/workspace/logs/zuul-variables.yaml
**--extra-vars** @/home/zuul/workspace/logs/hostvars-variables.yaml
/home/zuul/workspace/.quickstart/playbooks/quickstart.yml*

#### Reviewing the files passed:
* nodes: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/nodes
* release: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release
* testenv: https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/
* featureset: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config

### oooq_common_functions.sh
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/oooq_common_functions.sh.j2

* Use for functions referenced multiple times
* Or referenced within TOCI

### common_vars.bash
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/common_vars.bash.j2

* contains env variables referenced multiple times
* examples: RELEASE-, REVIEW-, etc. related

### featureset-override
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/featureset-override.j2

* To override settings in the featureset file
* There is a very select list of variable allowed in this list:
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/tasks/main.yaml#L5
* In the past, this list was used for any variable needed to be passed from the job definition to toci_quickstart.sh but preference now is to use `job.{variable}`
* The override works because of the ORDER in which these files are passed - **the last file passed wins**:

echo "-- Playbooks Output --------------------------"
        ${RELEASE_ARGS[$playbook]:=$QUICKSTART_DEFAULT_RELEASE_ARG} \
        {{ nodes_args }} \
        {{ featureset_conf }} \
        {{ env_vars }} \
        {{ role_vars }} \
        {{ extra_vars }} \
        {{ vxlan_vars }} \
        $DEFAULT_ARGS \
        $LOCAL_WORKING_DIR/playbooks/$playbook ${PLAYBOOKS_ARGS[$playbook]:-}" \
        --extra-vars @{{ workspace }}/logs/zuul-variables.yaml \
        --extra-vars @{{ workspace }}/logs/hostvars-variables.yaml \
done

### role-vars
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/role-vars.j2
* `run_test_role_vars`

## emit-releases and updates/upgrades
https://github.com/openstack/tripleo-ci/tree/master/scripts/emit_releases_file/scripts/emit_releases_file
DOCS: https://docs.openstack.org/tripleo-docs/latest/ci/emit_releases_file.html
* determines whether you need to upgrade/update
* sets up the release variables accordingly
* depends on the featureset number:
`if [[ -f "$RELEASES_SCRIPT" ]] && [[ {{ featureset }} =~ 010|011|037|047|050|056 ]]; then`
* called from `toci_gate_test.sh`
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2#L110
* outputs a emit_releases_file.log

Got DLRN hash: ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb for the named hash: current on the master release
Got DLRN hash: f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e for the named hash: tripleo-ci-testing on the master release
Doing an undercloud upgrade
Got DLRN hash: df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045 for the named hash: current-tripleo on the train release
stable_release: master, featureset: {'containerized_overcloud_upgrade': False, 'undercloud_check_sanity': True, 'use_oooq': True, 'tripleo_ci': True, 'ssl_overcloud': False, 'undercloud_undercloud_hostname': 'undercloud.{{ undercloud_cloud_domain }}', 'undercloud_cloud_domain': 'localdomain', 'containerized_undercloud_upgrade': True, 'undercloud_upgrade': True, 'undercloud_enable_swift_encryption': True, 'run_tempest': False, 'undercloud_upgrade_cleanup': True, 'test_ping': False, 'containerized_undercloud': "{% if release not in ['newton','ocata','pike','queens'] -%} true {%- else -%} false {%- endif -%}", 'undercloud_generate_service_certificate': True, 'containerized_overcloud': "{% if release in ['newton', 'ocata', 'pike'] -%} false {%- else -%} true {%- endif -%}", 'undercloud_enable_validations': False}
output releases: {'standalone_target_release': 'master', 'standalone_deploy_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'undercloud_install_release': 'train', 'undercloud_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'overcloud_deploy_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'standalone_deploy_release': 'master', 'standalone_target_newest_hash': u'ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb', 'overcloud_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'overcloud_target_release': 'master', 'overcloud_deploy_release': 'master', 'standalone_target_hash': u'f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e', 'undercloud_install_hash': u'df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045', 'standalone_deploy_newest_hash': u'ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb', 'undercloud_target_release': 'master'}

* See emit releases reference in logs:

2020-01-14 17:40:49.686014 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:68): LWD=/home/zuul/workspace/.quickstart
2020-01-14 17:40:49.686168 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:70): export RELEASES_FILE_OUTPUT=/home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:49.686306 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:70): RELEASES_FILE_OUTPUT=/home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:49.686485 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:71): export RELEASES_SCRIPT=/home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py
2020-01-14 17:40:49.686664 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:71): RELEASES_SCRIPT=/home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py
2020-01-14 17:40:49.686816 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:72): export RELEASES_SCRIPT_LOGFILE=/home/zuul/workspace/logs/emit_releases_file.log
2020-01-14 17:40:49.686961 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:72): RELEASES_SCRIPT_LOGFILE=/home/zuul/workspace/logs/emit_releases_file.log

and the variables are then exported:

2020-01-14 17:40:50.214131 | primary | +(./toci_quickstart.sh:64): declare -A RELEASE_ARGS
2020-01-14 17:40:50.214222 | primary | +(./toci_quickstart.sh:66): [[ -f /home/zuul/workspace/logs/releases.sh ]]
2020-01-14 17:40:50.214305 | primary | +(./toci_quickstart.sh:68): source /home/zuul/workspace/logs/releases.sh
2020-01-14 17:40:50.214415 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): export UNDERCLOUD_INSTALL_RELEASE=promotion-testing-hash-train
2020-01-14 17:40:50.214520 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): UNDERCLOUD_INSTALL_RELEASE=promotion-testing-hash-train
2020-01-14 17:40:50.214665 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): export UNDERCLOUD_INSTALL_HASH=df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045
2020-01-14 17:40:50.214793 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): UNDERCLOUD_INSTALL_HASH=df5db0b000f69dfd92e640a09dc98344ef3be731_79ab6045
2020-01-14 17:40:50.214907 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): export UNDERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215015 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): UNDERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215182 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): export UNDERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215308 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): UNDERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215419 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): export OVERCLOUD_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215530 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): OVERCLOUD_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.215664 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): export OVERCLOUD_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215790 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): OVERCLOUD_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.215905 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): export OVERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216010 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): OVERCLOUD_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216160 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): export OVERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216285 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): OVERCLOUD_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216400 | primary | ++(/home/zuul/workspace/logs/releases.sh:10): export STANDALONE_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216506 | primary | ++(/home/zuul/workspace/logs/releases.sh:10): STANDALONE_DEPLOY_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.216633 | primary | ++(/home/zuul/workspace/logs/releases.sh:11): export STANDALONE_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216754 | primary | ++(/home/zuul/workspace/logs/releases.sh:11): STANDALONE_DEPLOY_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.216884 | primary | ++(/home/zuul/workspace/logs/releases.sh:12): export STANDALONE_DEPLOY_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217017 | primary | ++(/home/zuul/workspace/logs/releases.sh:12): STANDALONE_DEPLOY_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217162 | primary | ++(/home/zuul/workspace/logs/releases.sh:13): export STANDALONE_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.217276 | primary | ++(/home/zuul/workspace/logs/releases.sh:13): STANDALONE_TARGET_RELEASE=promotion-testing-hash-master
2020-01-14 17:40:50.217415 | primary | ++(/home/zuul/workspace/logs/releases.sh:14): export STANDALONE_TARGET_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217543 | primary | ++(/home/zuul/workspace/logs/releases.sh:14): STANDALONE_TARGET_NEWEST_HASH=ffae68805fac442a0d3fa1db08d25957c0741646_6089bcdb
2020-01-14 17:40:50.217669 | primary | ++(/home/zuul/workspace/logs/releases.sh:15): export STANDALONE_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e
2020-01-14 17:40:50.217789 | primary | ++(/home/zuul/workspace/logs/releases.sh:15): STANDALONE_TARGET_HASH=f31e00b220162890eac1de06ba1d1f89e4ec9d44_bbaf1d8e

## TOCI is done when the logs show ...

2020-01-13 23:00:21.281872 | primary | Quickstart completed.
2020-01-13 23:00:21.282917 | primary | +(./toci_quickstart.sh:171): exit 0
2020-01-13 23:00:21.356261 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:179): echo 'Run completed'
2020-01-13 23:00:21.356627 | primary | Run completed
2020-01-13 23:00:22.047912 | primary | ok: Runtime: 0:54:37.324951

* Note that TOCI kicks collect-logs with a slightly different set of variables