Creating AWX CLI Guide

(note: Some links go to internal Red Had Jenkins jobs and are not accessable to the general community at this point).

Problem

We want to be able to generate and publish the CLI guide to AWX documentation instead of controller docsite.

The current Jenkins job and supporting playbooks are controller-branded.

Also, the tool (awxcli) generates a complete sphinx-based html docsite (stand alone).

Options

The AWX/Controller CLI guide depends on having an active AWX/Controller instance running to pull the details from.

The current code that generates the CLI guide creates HTML for a standalone docsite. We have the following options to create the guide for AWX:

Option 1. Integrate awxkit files and conf.py into AWX docsite generation.
Option 2 Generate RST and integrate into AWX docsite
Option 3 - Create Standalone docsite using existing resources with branding modifications.

NOTE would be beneficial to downstream docs if we can produce some interim output they can bring into their downstream toolchain. Today, downstream publishing supports asciidoc, but moving to DITA. TBD if .md will be supported in the future. Sphinx can generate .xml files in the docutils format so that might be an option to consider (sphinx-build -b xml source build).

1. Integrate awxkit cli files into AWX docsite

There are two steps here:

1. Static doc files are in another directory - we can use a symlink to get them where they need to be. (Tested and this works).
2. Integrate the actual cli generation (requires a running AWX instance to communicate with).

Can test this locally but how would the RTD docs generation instantiate a running AWX instance?? Possibly with GitHub actions?

---

Ideas below here have been rejected…

2. Generate RST and integrate into AWX docsite

Since the existing code is branded and generates html for a standalone site, we cannot use it directly. Instead we could:

1. Modify the code generation to pull out an intermediate step of creating rst files.
2. Modify the AWX docs creation to pull in this additional RST.
3. Publish to the AWX docsite.

NOTE: We need to keep the existing generation working for the Controller CLI Guide so cannot break that rendering option.

Problem: Cannot get RST out of existing extension

The existing code creates a foo.doctree file, but docutils tools like xml2rst can't seem to convert this back to an .rst file (not to mention the tool is python2 based and requires some changes to run).

I've been experimenting with the sphinx extension, sphinx-contrib.autoprogram and the awxcli custom extenstion based on this. I can see they build up a docutils document structure within the code, but similarly cannot see how to get this to an RST file.

Seems sphinx directives produce .doctree files that also include the TOC for the whole site so can't use this directly, but perhaps the next step is to try and move some of the awxkit/cli stuff into the /docs/docsite folder and see how far I get.

3. Create Standalone docsite

The following are basic steps to create a standalone AWX CLI guide (not part of the AWX docsite). It uses existing resources and rebrands for AWX:

The static CLI rst files already exist in the AWX repo, so we could

1. Copy the playbooks to GitHub actions to generate the HTML with AWX branding.
2. Publish to RTD as awx-cli-guide (standalone site).

GitHub Workflow

Drafting out how a new awx_cli_docs workflow would go.

1. Run the awx image (with run_awx_devel action), which will call awx_devel_image action) to generate the awx devel images.

2. Run a new action - awx_cli_docs_generate action with the following:
   CONTROLLER_HOST=https://awx.example.org CONTROLLER_USERNAME=example CONTROLLER_PASSWORD=secret make clean html
CONTROLLER_HOST=https://awx.localhost:8043 CONTROLLER_USERNAME=admin CONTROLLER_PASSWORD=password make clean html

   Where the details come from step 1 on where the awx instance is, password, etc. (seems from the .yaml that username is admin and password is password).

3. Append the tox.ini for docs to combine this HTML with the spinx-build HTML.

Creating the CLI guide locally

You will need a running awx instance. You can either build awx_devel locally or get the image.

Notes on building locally:
If running in a VM, must have 16G memory to build UI etc

docker-compose is already part of the docker install

1. pip install -r requirements/requirements.txt
2. Set up docker for nonroot user https://docs.docker.com/engine/install/linux-postinstall/
3. Ceate devel image https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md
4. (optional) Build the ui (samccann - can't get this step working)
   $ docker exec tools_awx_1 make clean-ui ui-devel
5. create awx user -
   $ docker exec -ti tools_awx_1 awx-manage createsuperuser

Generate local copy of awx cli docsite

You can build the CLI locally if you have an AWX instance running.

1. Install requirements for awxkit -
   pip install sphinx sphinxcontrib-autoprogram
   pip install awxkit (dunno how to run from source)

2. cd awxkit/awxkit/cli/docs/

3. CONTROLLER_HOST=https://localhost:8043 CONTROLLER_USERNAME=awx CONTROLLER_PASSWORD=awx CONTROLLER_VERIFY_SSL=f make clean html

Generating the AWX CLI guide locally

Just samccann's scribble-notes on how she got it running locally…

Creating awx_devel image

If running in a VM, must have 16G memory to build UI etc

• docker-compose is already part of the docker install

1. (may not be required if running the latest container image see step 3) pip install -r requirements/requirements.txt
2. set up docker for nonroot user https://docs.docker.com/engine/install/linux-postinstall/
3. create devel image https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md
   NOTE: You can create the image locally for awx, or you can just get the latest from https://github.com/ansible/awx/pkgs/container/awx_devel
4. run awx instance - make docker-compose

In a separate terminal:

1. (optional) build the ui
   $ docker exec tools_awx_1 make clean-ui ui-devel
2. create awx user -
   $ docker exec -ti tools_awx_1 awx-manage createsuperuser

Generate local copy of awx cli docsite

1. install requirements for awxkit -
   pip install sphinx sphinxcontrib-autoprogram
   pip install awxkit (dunno how to run from source)

2. cd awxkit/awxkit/cli/docs/

3. CONTROLLER_HOST=https://localhost:8043 CONTROLLER_USERNAME=awx CONTROLLER_PASSWORD=awx CONTROLLER_VERIFY_SSL=f make clean html

From the cli docs:

The preferred way to install the AWX CLI is through pip:

pip install "git+https://github.com/ansible/awx.git@$VERSION#egg=awxkit&subdirectory=awxkit"

…where $VERSION is the version of AWX you’re running. To see a list of all available releases, visit: https://github.com/ansible/awx/releases

Common steps on reboot:

sudo systmectl start docker
docker run hello-world (just to verify)
source ~/venvs/awx/bin/activate
make docker-compose

Further experiments

Going from doctrees to xml:

Create a small unpickle.py script:
import pickle

with open("install.doctree", "rb") as f:
result = pickle.load(f)

print(result)

then python unplickle.py > xx to see the unpickled XML generated during the sphinx docs build.

xml2rst

clone or fork https://github.com/docutils/docutils
follow readme - https://github.com/samccann/docutils/blob/master/sandbox/xml2rst/README

• pip install lxml
• run 2to3 against sandbox/xml2rst and sandbox/docutils_xml
• Add this to ~/.bashrc : export PYTHONPATH=/home/samccann/docutils/sandbox/docutils_xml:
• if needed in VSCode - type F1 Terminal: Select Default Shell and set to bash

At this point - python3 xml3rst.py –help should work

Anwesha's notes

Generate the AWX CLI Docs locally

1. Create VM and ssh to new VM

   • Create fedora 40 Virtual Machine with Core 2 and 20GB RAM
   • Python 3.12.4
   • get the ip address of the newly created VM ip a s
   • add ssh key in the local machine and ssh
     • ssh-copy-id user@ipofvm
     • ssh -L 8080:127.0.0.1:8000 user@ipofvm
ssh -l helps us to open the browser in the local machine.

2. Start the docker container . Fedora 40 comes with a preinstalled docker

   sudo systemctl start docker
sudo systemctl status docker

3. Install sphinx sphinxcontrib-autoprogram

   python3 -m pip install sphinx sphinxcontrib-autoprogram
python3 -m pip install awxkit
cd awxkit/awxkit/cli/docs

4. CONTROLLER_HOST=https://localhost:8043 CONTROLLER_USERNAME=awx CONTROLLER_PASSWORD=awx CONTROLLER_VERIFY_SSL=f make clean html

5. Install the http.server module and activate the virtual environment

   python3 -m http.server
python3 -m venv .venv
source .venv/bin/activate

6. Get the official awx_devel image from the ghcr repository

   docker images
docker pull ghcr.io/ansible/awx_devel:devel

7. Go to the awx directory

   cd awx

8. run the docker compose command

   make docker-compose

9. Open the build html in the chosen port of the local browser.
   localhost:8080/build/html

Steps followed in Generating the AWX CLI docs

1. Create VM and ssh to new VM

   • Create fedora 40 Virtual Machine with Core 2 and 20GB RAM
   • Python 3.12.4
   • get the ip address of the newly created VM ip a s
   • add ssh key in the local machine and ssh
     • ssh-copy-id user@ipofvm
     • ssh user@ipofvm

2. Install the dependencies

   ​sudo dnf install docker docker-compose ansible openssl 
   ​sudo systemctl start docker
   ​sudo dnf install git make gcc python-devel xmlsec1 python3-ldap xmlsec1-openssl openldap-devel
   ​
   

3. Clone the awx repo

git clone https://github.com/ansible/awx.git

4. Start the Python virtual env

python3 -m venv .venv
source .venv/bin/activate

5. Go to the awx dir cd awx

6. make docker-compose

7. Install dependencies

python3 -m pip install setuptools_scm xmlsec 
sudo dnf install xmlsec1-devel 

8. Export the environment variable
   export CFLAGS="-Wno-error=incompatible-pointer-types"

9. Install dependecies

sudo dnf install libtool-ltdl-devel libtool-ltdl

10. The docker engine on Fedora CoreOS does not support the “buildx” subcommand which is needed to build container images for multiple architectures. To solve this issue follow the following steps (in your local browser)

    1. download buildx from Releases · docker/buildx · GitHub 304
    2. Get the the code in the VM from local machine.
       scp Downloads/docker-desktop-x86_64.rpm userid@ipofvm
    3. move the binary in ~/.docker/cli-plugins/docker-buildx directory
       ​​​​​mkdir -p ~/.docker/cli-plugins
       ​​​​​mv ~/buildx-v0.16.2.linux-amd64 ~/.docker/cli-plugins/docker-buildx
       ​​​​​chmod +x /.docker/cli-plugins/docker-buildx
       ​​​​​sudo chgrp wheel /var/run/docker.sock
       
       

11. Build the doc with the make command
    make docker-compose-build

Use the ghcr image

1. Start new vm
2. Follow https://github.com/ansible/awx/pkgs/container/awx_devel to download the awx image
3. run the image to

How is the doc created?

We have the awxkit this along with sphinx-contrib creates the cli docs. What

• Install desktop for fe

• What is happening this

• document the flow of the jenkins jobs.

Brainstorming Ideas

• If we have a API guide can we create a CLI Guide based on that info?
• AWX and Tower repo connection. (Boiler plate seems to be the same)
• Detailed md page -> Controller AWX Cli doc is generated now (for Ansible Engineering Handbook)

Steps

• Detailed md page -> Controller AWX Cli doc is generated now (for Ansible Engineering Handbook) - Jira Issue under the AWX Cli epic
• Fork the tower repo to show Toshio and Sandra.

Links

1. https://github.com/docker/buildx/issues/262