# Ansible Dependency pipeline

* Ansible dependency pipeline to test futures release of ansible against tripleo to catch possible issues early.

* Latest ansible version - 2.10 currently(Our target which we want to test)

#### Blockers: [we don't have 2.10 RPM on releases.ansible.com](https://github.com/ansible/ansible/commit/3dedf95dbdf1fa170ad7256bc2dfa2a12228bc32#diff-71669691cb0b0f1bf662939c0db0782fb1a71b103bba2564051bb58bf0e303e6)

## What are our options:-

* Install the latest ansible using pip
* Create latest ansible rpm ourself/ Find a place which is building ansible 2.10 and pull rpm from there

## What we have tried so far:-

1) **Tried removing ansible 2.9 rpm and install via pip only:-**

Lot of rpms(ex: tripleo-ansible) depends on ansible, if we remove ansible - dependent package are also removing.

~~~
stack@undercloud ~]$ rpm -e ansible
error: Failed dependencies:
ansible >= 2.6 is needed by (installed) python3-ansible-runner-1.4.6-3.el8.noarch
ansible >= 2.9.10 is needed by (installed) openstack-tripleo-common-13.1.0-0.20210112061521.02acb41.el8.noarch
ansible >= 2.9.10 is needed by (installed) tripleo-ansible-2.1.0-0.20210106121408.0378977.el8.noarch
python3dist(ansible) is needed by (installed) ansible-pacemaker-1.0.4-0.20201118111731.accaf26.el8.noarch
python3dist(ansible) is needed by (installed) python3-heat-agent-ansible-2.2.0-0.20201118210640.7b525d8.el8.noarch
python3dist(ansible) >= 2 is needed by (installed) ansible-role-openstack-operations-0.0.1-0.20201118162748.3937ea4.el8.noarch
python3dist(ansible) is needed by (installed) ansible-tripleo-ipsec-9.3.1-0.20201118163450.3e445d6.el8.noarch
python3dist(ansible) is needed by (installed) ansible-role-container-registry-1.2.1-0.20201118181146.1f05b2d.el8.noarch
python3dist(ansible) is needed by (installed) ansible-role-tripleo-modify-image-1.2.1-0.20201118135036.9961cf8.el8.noarch
python3dist(ansible) >= 2.9 is needed by (installed) ansible-role-metalsmith-deployment-1.4.0-0.20201222053849.e0e0f13.el8.noarch
python3dist(ansible) >= 2 is needed by (installed) validations-common-1.1.2-0.20210105223343.5ab278b.el8.noarch
python3dist(ansible) is needed by (installed) ansible-role-thales-hsm-0.2.1-0.20201118095233.99b3d39.el8.noarch
python3dist(ansible) is needed by (installed) ansible-role-lunasa-hsm-1.0.1-0.20201215153935.b3981c6.el8.noarch
python3dist(ansible) is needed by (installed) ansible-role-chrony-1.0.3-0.20201118162514.b25e82c.el8.noarch
python3dist(ansible) is needed by (installed) ansible-role-atos-hsm-0.1.1-0.20201118095025.e51c244.el8.noarch
python3dist(ansible) >= 2 is needed by (installed) openstack-tripleo-validations-13.1.0-0.20201216192003.8ef9993.el8.noarch
python3dist(ansible) is needed by (installed) ansible-config_template-1.1.2-0.20201118123122.a6347c3.el8.noarch
~~~

2) **Tried removing ansible 2.9 rpm (with --nodeps) and install via pip only**

Deployment Fails because we don't have /etc/ansible/ansible.cfg file

~~~
ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-52w1zl61

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Exception: [Errno 2] No such file or directory: '/etc/ansible/ansible.cfg'
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1447, in take_action
self._standalone_deploy(parsed_args)
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1318, in _standalone_deploy
transport=parsed_args.transport)
File "/usr/lib/python3.6/site-packages/tripleo_common/actions/ansible.py", line 46, in write_default_ansible_cfg
shutil.copy(base_ansible_cfg, ansible_config_path)
File "/usr/lib64/python3.6/shutil.py", line 245, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/ansible/ansible.cfg'
None
[Errno 2] No such file or directory: '/etc/ansible/ansible.cfg'
~~~

3) **Keeping ansible rpm tried installing latest ansible using pip**

~~~
### ERROR ###
Upgrading directly from ansible-2.9 or less to ansible-2.10 or greater with pip is
known to cause problems. Please uninstall the old version found at:
~~~

4) **Tried Installing latest ansible via pip first and then letting older ansible rpm to install(during deployment workflow)**

This seems to work:-

path of pip binaries (/usr/local/lib/python3.6/site-package) is ahead of rpm binaries (/usr/lib/python3.6/site-packages) in sys.path... so installing via pip keeping older rpm will work.

~~~
[zuul@node-0000819497 ~]$ pip3 show ansible
Name: ansible
Version: 2.10.5
Summary: Radically simple IT automation
Home-page: https://ansible.com/
Author: Ansible, Inc.
Author-email: [e-mail]
License: GPLv3+
Location: /usr/local/lib/python3.6/site-packages --->
Requires: ansible-base
~~~

~~~
[zuul@node-0000819497 ~]$ ansible --version
ansible 2.10.5
config file = None
configured module search path = ['/home/zuul/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
~~~

sudo rpm -ql ansible
~~~
/usr/lib/python3.6/site-packages/ansible/
~~~

~~~
[zuul@node-0000819497 ~]$ /usr/bin/python3.6 -c "import sys;print(sys.path)"
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/zuul/.local/lib/python3.6/site-packages', '/usr/local/lib64/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages'
~~~

* **If we remove ansible rpm and only install via pip - deployment fails complaining ansible-playbook-3 doesn't exists**

https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/main.yml#L121-L139
~~~
PLAY [External deployment step 1] **********************************************
2021-01-21 16:22:09.803745 | fa163ea2-d1aa-9f0e-0635-0000000000fa | TASK | External deployment step 1
2021-01-21 16:22:09.831578 | fa163ea2-d1aa-9f0e-0635-0000000000fa | OK | External deployment step 1 | undercloud | result={
"changed": false,
"msg": "Use --start-at-task 'External deployment step 1' to resume from this task"
}
2021-01-21 16:22:09.832965 | fa163ea2-d1aa-9f0e-0635-0000000000fa | TIMING | External deployment step 1 | undercloud | 0:03:13.104844 | 0.03s
2021-01-21 16:22:09.884947 | fa163ea2-d1aa-9f0e-0635-0000000000fb | TIMING | include_tasks | undercloud | 0:03:13.156813 | 0.03s
2021-01-21 16:22:09.909143 | 48427583-0dca-4a14-970e-ae3ffa839307 | INCLUDED | /home/zuul/standalone-ansible-ek39ivjr/external_deploy_steps_tasks_step1.yaml | undercloud
2021-01-21 16:22:09.946785 | fa163ea2-d1aa-9f0e-0635-000000000d71 | TASK | Container image prepare
2021-01-21 16:24:19.748703 | fa163ea2-d1aa-9f0e-0635-000000000d71 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "[Errno 2]
No such file or directory: 'ansible-playbook-3': 'ansible-playbook-3'", "msg": "Error running container image prepare: [Errno 2] No such file or directory: 'ansible-playbook-3': 'ansible-playbook-3'", "params": {}, "success": false}
2021-01-21 16:24:19.757266 | fa163ea2-d1aa-9f0e-0635-000000000d71 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:05:23.029135 | 129.81s
~~~

### Current Issue:-

With ansible latest rpm, standalone deployment is failing complaining container is not available:-

https://logserver.rdoproject.org/35/31635/1/check/periodic-tripleo-ci-centos-8-standalone-ansible-deps-master/7df4cd0/logs/undercloud/var/log/extra/failed_services.txt.gz

~~~
Jan 21 12:03:56 standalone.localdomain systemd[1]: Starting memcached container...
Jan 21 12:03:56 standalone.localdomain podman[106097]: Error: unable to find container memcached: no container with name or ID memcached found: no such container
Jan 21 12:03:56 standalone.localdomain systemd[1]: tripleo_memcached.service: Control process exited, code=exited status=125
Jan 21 12:03:56 standalone.localdomain podman[106108]: Error: no container with name or ID memcached found: no such container
Jan 21 12:03:56 standalone.localdomain systemd[1]: tripleo_memcached.service: Control process exited, code=exited status=125
Jan 21 12:03:56 standalone.localdomain systemd[1]: tripleo_memcached.service: Failed with result 'exit-code'.
~~~

Container is available:-

https://logserver.rdoproject.org/35/31635/1/check/periodic-tripleo-ci-centos-8-standalone-ansible-deps-master/7df4cd0/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz

~~~
192.168.24.1:8787/tripleomaster/openstack-memcached bea605c792d8717de98b5be2af820faf-updated-20210121113944 a38c487eb9f1 12 minutes ago 682 MB
~~~

## Patches
~~~
https://review.rdoproject.org/r/#/c/31635/

Older Test patches(not active anymore) - https://review.rdoproject.org/r/#/c/26217/ & https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/771590
~~~