Ansible Dependency pipeline

Ansible dependency pipeline to test futures release of ansible against tripleo to catch possible issues early.
Latest ansible version - 2.10 currently(Our target which we want to test)

Blockers: we don't have 2.10 RPM on releases.ansible.com

What are our options:-

Install the latest ansible using pip
Create latest ansible rpm ourself/ Find a place which is building ansible 2.10 and pull rpm from there

What we have tried so far:-

Tried removing ansible 2.9 rpm and install via pip only:-

Lot of rpms(ex: tripleo-ansible) depends on ansible, if we remove ansible - dependent package are also removing.

stack@undercloud ~]$ rpm -e ansible
error: Failed dependencies:
	ansible >= 2.6 is needed by (installed) python3-ansible-runner-1.4.6-3.el8.noarch
	ansible >= 2.9.10 is needed by (installed) openstack-tripleo-common-13.1.0-0.20210112061521.02acb41.el8.noarch
	ansible >= 2.9.10 is needed by (installed) tripleo-ansible-2.1.0-0.20210106121408.0378977.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-pacemaker-1.0.4-0.20201118111731.accaf26.el8.noarch
	python3dist(ansible) is needed by (installed) python3-heat-agent-ansible-2.2.0-0.20201118210640.7b525d8.el8.noarch
	python3dist(ansible) >= 2 is needed by (installed) ansible-role-openstack-operations-0.0.1-0.20201118162748.3937ea4.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-tripleo-ipsec-9.3.1-0.20201118163450.3e445d6.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-role-container-registry-1.2.1-0.20201118181146.1f05b2d.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-role-tripleo-modify-image-1.2.1-0.20201118135036.9961cf8.el8.noarch
	python3dist(ansible) >= 2.9 is needed by (installed) ansible-role-metalsmith-deployment-1.4.0-0.20201222053849.e0e0f13.el8.noarch
	python3dist(ansible) >= 2 is needed by (installed) validations-common-1.1.2-0.20210105223343.5ab278b.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-role-thales-hsm-0.2.1-0.20201118095233.99b3d39.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-role-lunasa-hsm-1.0.1-0.20201215153935.b3981c6.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-role-chrony-1.0.3-0.20201118162514.b25e82c.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-role-atos-hsm-0.1.1-0.20201118095025.e51c244.el8.noarch
	python3dist(ansible) >= 2 is needed by (installed) openstack-tripleo-validations-13.1.0-0.20201216192003.8ef9993.el8.noarch
	python3dist(ansible) is needed by (installed) ansible-config_template-1.1.2-0.20201118123122.a6347c3.el8.noarch

Tried removing ansible 2.9 rpm (with –nodeps) and install via pip only

Deployment Fails because we don't have /etc/ansible/ansible.cfg file

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-52w1zl61

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Exception: [Errno 2] No such file or directory: '/etc/ansible/ansible.cfg'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1447, in take_action
    self._standalone_deploy(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1318, in _standalone_deploy
    transport=parsed_args.transport)
  File "/usr/lib/python3.6/site-packages/tripleo_common/actions/ansible.py", line 46, in write_default_ansible_cfg
    shutil.copy(base_ansible_cfg, ansible_config_path)
  File "/usr/lib64/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/ansible/ansible.cfg'
None
[Errno 2] No such file or directory: '/etc/ansible/ansible.cfg'

Keeping ansible rpm tried installing latest ansible using pip

### ERROR ###
Upgrading directly from ansible-2.9 or less to ansible-2.10 or greater with pip is 
known to cause problems.  Please uninstall the old version found at:

Tried Installing latest ansible via pip first and then letting older ansible rpm to install(during deployment workflow)

This seems to work:-

path of pip binaries (/usr/local/lib/python3.6/site-package) is ahead of rpm binaries (/usr/lib/python3.6/site-packages) in sys.path… so installing via pip keeping older rpm will work.

[zuul@node-0000819497 ~]$ pip3 show ansible
Name: ansible
Version: 2.10.5
Summary: Radically simple IT automation
Home-page: https://ansible.com/
Author: Ansible, Inc.
Author-email: info@ansible.com
License: GPLv3+
Location: /usr/local/lib/python3.6/site-packages --->
Requires: ansible-base

[zuul@node-0000819497 ~]$ ansible --version
ansible 2.10.5
  config file = None
  configured module search path = ['/home/zuul/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

sudo rpm -ql ansible

/usr/lib/python3.6/site-packages/ansible/

[zuul@node-0000819497 ~]$ /usr/bin/python3.6  -c "import sys;print(sys.path)"
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/zuul/.local/lib/python3.6/site-packages', '/usr/local/lib64/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages'

If we remove ansible rpm and only install via pip - deployment fails complaining ansible-playbook-3 doesn't exists

https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/main.yml#L121-L139

PLAY [External deployment step 1] **********************************************
2021-01-21 16:22:09.803745 | fa163ea2-d1aa-9f0e-0635-0000000000fa |       TASK | External deployment step 1
2021-01-21 16:22:09.831578 | fa163ea2-d1aa-9f0e-0635-0000000000fa |         OK | External deployment step 1 | undercloud | result={
    "changed": false,
    "msg": "Use --start-at-task 'External deployment step 1' to resume from this task"
}
2021-01-21 16:22:09.832965 | fa163ea2-d1aa-9f0e-0635-0000000000fa |     TIMING | External deployment step 1 | undercloud | 0:03:13.104844 | 0.03s
2021-01-21 16:22:09.884947 | fa163ea2-d1aa-9f0e-0635-0000000000fb |     TIMING | include_tasks | undercloud | 0:03:13.156813 | 0.03s
2021-01-21 16:22:09.909143 | 48427583-0dca-4a14-970e-ae3ffa839307 |   INCLUDED | /home/zuul/standalone-ansible-ek39ivjr/external_deploy_steps_tasks_step1.yaml | undercloud
2021-01-21 16:22:09.946785 | fa163ea2-d1aa-9f0e-0635-000000000d71 |       TASK | Container image prepare
2021-01-21 16:24:19.748703 | fa163ea2-d1aa-9f0e-0635-000000000d71 |      FATAL | Container image prepare | undercloud | error={"changed": false, "error": "[Errno 2] 
No such file or directory: 'ansible-playbook-3': 'ansible-playbook-3'", "msg": "Error running container image prepare: [Errno 2] No such file or directory: 'ansible-playbook-3': 'ansible-playbook-3'", "params": {}, "success": false}
2021-01-21 16:24:19.757266 | fa163ea2-d1aa-9f0e-0635-000000000d71 |     TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:05:23.029135 | 129.81s

Current Issue:-

With ansible latest rpm, standalone deployment is failing complaining container is not available:-

https://logserver.rdoproject.org/35/31635/1/check/periodic-tripleo-ci-centos-8-standalone-ansible-deps-master/7df4cd0/logs/undercloud/var/log/extra/failed_services.txt.gz

Jan 21 12:03:56 standalone.localdomain systemd[1]: Starting memcached container...
Jan 21 12:03:56 standalone.localdomain podman[106097]: Error: unable to find container memcached: no container with name or ID memcached found: no such container
Jan 21 12:03:56 standalone.localdomain systemd[1]: tripleo_memcached.service: Control process exited, code=exited status=125
Jan 21 12:03:56 standalone.localdomain podman[106108]: Error: no container with name or ID memcached found: no such container
Jan 21 12:03:56 standalone.localdomain systemd[1]: tripleo_memcached.service: Control process exited, code=exited status=125
Jan 21 12:03:56 standalone.localdomain systemd[1]: tripleo_memcached.service: Failed with result 'exit-code'.

Container is available:-

https://logserver.rdoproject.org/35/31635/1/check/periodic-tripleo-ci-centos-8-standalone-ansible-deps-master/7df4cd0/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz

192.168.24.1:8787/tripleomaster/openstack-memcached                    bea605c792d8717de98b5be2af820faf-updated-20210121113944   a38c487eb9f1   12 minutes ago   682 MB

Patches

https://review.rdoproject.org/r/#/c/31635/

Older Test patches(not active anymore) - https://review.rdoproject.org/r/#/c/26217/ & https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/771590

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.