###### tags: `Documentation`
# Ansible community docs 2023: Decoupling core and package docs

If we look at `docs.ansible.com` today we can see there are two categories, or domains, for the set of community docs:

- `docs.ansible.com/ansible/`
- `docs.ansible.com/ecosystem.html`

The Ansible community team proposes various high-level actions that seek to improve organization of the content set along those categories.

## Decoupling of core/package docs - Intended outcomes

- Position Ansible community documentation for sustainable long-term growth.
-- Drive adoption and sustain human-to-human knowledge sharing.
-- Simplify the contribution process for community documentation.
-- Create more opportunities for documentation contributions.
-- Reduce technical debt with focused initiatives.
-- Use CI workflows to handle mundane publishing tasks.

- Provide the Ansible community with more ownership of the documentation.
-- Ansible core release cycles no longer gate documentation.
-- Establish community-driven release cycles.
-- Help resolve issues such as [Move Ansible Community docs into the ansible-community org](https://github.com/ansible-community/community-topics/issues/240)

- Reduce fragmentation for users in the community while navigating the Ansible ecosystem.
-- Provide a trusted, authoritative source of automation knowledge.
-- Build content journeys that support project adoption in the community.
-- Provide clear pathways to platform content.

- Bring consistency and coherence to the documentation set.
-- Make it easy for new projects in the Ansible ecosystem to get their content on `docs.ansible.com`.
-- Use the Diataxis framework to organize content in the doc set by type, establishes shared best practices across projects for faster doc iterations and more easily identifiable doc deliverables.

## Ansible ecosystem on Read The Docs

Projects in the Ansible ecosystem should host their documentation on ReadTheDocs.

The main project provides a lightweight overlay navigation with each project in the ecosystem added as a subproject. Consider the following proof-of-concept:

https://oranod-docsite.readthedocs.io/en/latest/
https://oranod-docsite.readthedocs.io/projects/builder/en/latest/
https://oranod-docsite.readthedocs.io/projects/core/en/latest/
https://oranod-docsite.readthedocs.io/projects/lint/en/latest/
https://oranod-docsite.readthedocs.io/projects/rulebook/en/latest/

### Move core to the ecosystem

The current docsite `docs.ansible.com` subdomain has these two contexts:

- https://docs.ansible.com/ansible/ Ansible package docs
- https://docs.ansible.com/ansible-core/ Ansible Core docs

We can move `/ansible-core/` to the ecosystem as follows:

1. Merge the `.readthedocs.yaml` configuration to `devel` and build.
2. Evaluate with the community and announce for a release cycle or two.
3. Configure [redirects to migrate from /ansible-core to Read The Docs](https://docs.readthedocs.io/en/stable/user-defined-redirects.html#migrating-your-documentation-to-another-domain) to achieve:

RedirectMatch permanent 
"https://docs.ansible.com/ansible-core/devel/*"
"https://ansible.readthedocs.io/projects/core/en/latest/*"

### Ansible subdomains

From an SEO perspective, `docs.ansible.com` is the authoritative source for Ansible documentation. We should preserve that as a key strength for the Ansible community. We also don't want to break user help.

> The heart of this proposal to is ensure that the Ansible community documentation remains a distinct, thriving project within the Ansible ecosystem. Change can be difficult, especially when you need to draw lines. But the docs project needs to remove duplication and establish clear paths to continue growing.

We can update the DNS entry for `docs.ansible.com` to point to Read The Docs and add our domain in the project, see [Custom domains](https://docs.readthedocs.io/en/stable/custom-domains.html#).

This will result in a structure such as the following:

https://docs.ansible.com/en/latest/
https://docs.ansible.com/projects/builder/en/latest/
https://docs.ansible.com/projects/core/en/latest/
https://docs.ansible.com/projects/lint/en/latest/
https://docs.ansible.com/projects/rulebook/en/latest/

Unfortunately we cannot put the package docs as the main project (see below for reasons). We also need to preserve search engine indexes for `docs.ansible.com` as much as possible. We don't want to break everyone's bookmarks, lose SEO authority that has been built up over time, or seriously annoy and frustrate the community and Ansible user base.

Therefore it seems reasonable to propose that we keep `docs.ansible.com` for the package docs and try not to alter it too radically. We can discuss a new subdomain such as `ecosystem.ansible.com` for ecosystem docs hosted on RTD, using `ansible.readthedocs.io` until we reach a decision.

#### Building package docs on Read The Docs

We cannot build the package docs (Ansible Core + Collections and all modules and plugins) on Read The Docs without considerable expense.

The package docs build was tested on Read The Docs for Business, which offers a limit of [7 GB of memory](https://docs.readthedocs.io/en/stable/builds.html#build-resources) and 30 minutes of build time. However the package docs build either timed out (30 mins of build time by default) or failed with `Build exited due to excessive memory consumption` messages in logs.

From discussion with the Read The Docs support team, there is scope to increase the default limits with the Read The Docs Enterprise plan. However that falls in the range of 1k USD per month.

### Read The Docs configuration

Projects are required to have a `.readthedocs.yaml` configuration file in the base of the repository.

There are three types of builder configuration, Sphinx, MKDocs, Custom. Available functionality in Read The Docs, such as search and theme, depends on the builder for the project.

#### Sphinx

sphinx:
  builder: dirhtml
  configuration: docs/conf.py
  fail_on_warning: true

#### MKDocs

mkdocs:
  fail_on_warning: true
  configuration: mkdocs.yml

#### Custom

build:
  os: ubuntu-22.04
  tools:
    python: "3.11"
  commands:
    - build commands go here
    - mkdir -p _readthedocs/html/
    - cp -r docs/docsite/_build/html/* _readthedocs/html/

### Themes

- Projects that build with MKDocs should use the [mkdocs-ansible](https://github.com/ansible/mkdocs-ansible)
- Projects that build with Sphinx should use the [sphinx-ansible-theme](https://github.com/ansible-community/sphinx_ansible_theme)

> There is additional work required to modernize the Sphinx Ansible theme and overcome issues where functionality is not complete for certain areas like API docs. The effort to modernize the Sphinx Ansible theme is tracked in this issue: https://github.com/ansible-community/sphinx_ansible_theme/issues/41

- Projects that use custom builds also require custom themes that follow the Ansible community colour palette and style guidelines. However, we should limit custom builds where possible.

### Search

Read The Docs offers [server-side search](https://docs.readthedocs.io/en/stable/server-side-search/index.html) across all projects and subprojects.

Search capabilities are based on Elastic Search so that pages are [indexed](https://dev.readthedocs.io/en/latest/server-side-search.html#server-side-search) and queryable.

#### Sphinx builder

Projects that use the RTD Sphinx builder include a search box. For example, https://oranod-docsite.readthedocs.io/projects/builder/en/latest/

You can locate the search box at the lower right of the window.



This takes you to the following search page: https://readthedocs.org/search/?q=project%3Aoranod-builder+galaxy-keyring

You can search the current project with `project:<projects/project-name> <query>` for example:

- https://readthedocs.org/search/?q=project%3Aoranod-builder+galaxy-keyring
- https://readthedocs.org/search/?q=project%3Aoranod-lint+galaxy-keyring

You can search across all projects with `subprojects:<main-project-name> <query>` for example:

- https://readthedocs.org/search/?q=subprojects%3Aoranod-docsite+galaxy-keyring&type=file
- https://readthedocs.org/search/?q=subprojects%3Aoranod-docsite+retract_fact&type=file

For more information, see [query syntax](https://docs.readthedocs.io/en/stable/server-side-search/syntax.html).

#### MKDocs and custom builders

Projects that use the RTD MKDocs builder, or a custom build process, are not currently supported with the RTD seach box. However, RTD does mention some overrides for search integration with [MKDocs builds](https://dev.readthedocs.io/en/latest/search-integration.html#mkdocs).

### Migrating existing projects

Many projects are already imported and hosted on Read The Docs, for example: https://ansible-lint.readthedocs.io/

The process to migrate an existing project is as follows:

1. Ensure the main project is created.
2. Include admins from the main project as maintainers to the existing project and vice versa.
3. Add the existing project as a subproject.

In the following screengrab, Ansible SDK can be added as a subproject to `oranod-docsite.readthedocs.io` even though it already exists as a project and is hosted at `ansible-sdk.readthedocs.io`


Adding an existing project as a subproject automatically creates a redirect. With the preceding example, `ansible-sdk.readthedocs.io` would automatically redirect to `oranod-docsite.readthedocs.io/projects/sdk`.

You can observe this behaviour with the following URL http://oranod-builder.readthedocs.io/ that automatically redirects to https://oranod-docsite.readthedocs.io/projects/builder/en/latest/

### Ansible Core docs on Read The Docs

To build Ansible Core content on Read The Docs the following configuration is required:

# RTD API version
version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.11"
  # Build with make coredocs
  commands:
    - python -m pip install --upgrade --no-cache-dir pip setuptools six wheel
    - python -m pip install --exists-action=w --no-cache-dir -r ./requirements.txt
    - python -m pip install --exists-action=w --no-cache-dir -r ./test/sanity/code-smell/docs-build.requirements.txt
    - cd docs/docsite && make coredocs
    - mkdir -p _readthedocs/html/
    - cp -r docs/docsite/_build/html/* _readthedocs/html/

This configuration uses the `make coredocs` command which is necessary to create the `conf.py` symlink for the Sphinx build.

Because this is a custom build command, it does not use the Read The Docs Sphinx extension that enables RTD search. The resulting output uses the Sphinx Ansible theme, as defined in `core_conf.py`, as well as Sphinx search. However Sphinx search is inconsistent with the other projects and leads to a more disjointed search experience for users.

#### RTD Sphinx build

Ansible Core docs were tested with the RTD Sphinx build using the following configuration:

python:
   install:
   - requirements: ./requirements.txt
   - requirements: ./test/sanity/code-smell/docs-build.requirements.txt
sphinx:
   builder: dirhtml
   configuration: docs/docsite/rst/conf.py

After some testing it seems RTD builders cannot successfully build Ansible Core docs. The likely culprit is likely to be complexity the structure of the `ansible/docs/docsite` directory. You can find details in [RTD Sphinx build of Ansible Core](https://hackmd.io/l-1-wd1GTESlRjpueWGWPA).

However this complexity exists because package and core docs are built from the same project. Separating core docs from the package docs allows us to simplify things down and eventually start using the RTD Sphinx build so we get the nice things (RTD features like search).

#### Ansible core content

Projects in the Ansible ecosystem should always include some level of documentation. We should reduce Ansible core docs to a lightweight subset of the current docs. Possible candidates include:

- Installing Ansible (new details that focus on `ansible-core` and briefly explain the package then point users to `docs.ansible.com`)
- Ansible core porting guides

After we further decouple package docs from Ansible core docs in `ansible/ansible` we can simplify the contents of the `ansible/docs/docsite` directory and use the RTD Sphinx build instead of a custom build process.

## Creating a package docs project

If Ansible core docs are hosted on `docs.ecosystem.ansible.com` and Ansible package docs are hosted on `docs.ansible.com` it makes sense for the source for each content set to have a separate repository.

- Ansible core docs reside in `ansible/ansible`.
- Ansible package docs reside in `ansible/docsite`.

At a high-level the proposal to create a package docs project means doing another lift and shift. In this case it is the RST source and other assets in `ansible/ansible/docs/docsite`.

Things that do not change:

- Continue publishing the package docs at `docs.ansible.com`.
- Zero impact to anything outside `ansible/ansible/docs/docsite/`. Documentation that is dynamically generated from the `ansible/ansible` code source stays in place.

Benefits:

- Ansible community can take more direct ownership of the docsite development and operations.
- Removes Ansible core freeze dates as merge blockers.
- Package docs can be versioned independent of core and use a different branching strategy.
- Decreases workload on the Ansible core team. Both in the volume of traffic for doc contributions as well as for operations like sanity testing doc PRs.
- Leads to the removal of Red Hat internal Jenkins instances and other infrastructure that requires maintenance.

How will the build work?

The package build job calls into different parts of the `ansible/ansible` repository for intersphinx references and to use other scripts and dependencies. The build will need to fetch a git ref of `ansible/ansible` and build on top of that.

High-level steps to get there:

1. Bring this plan to the community for review and acceptance.
2. Establish a timeline for the lift and shift to take place.
3. Develop a plan to redirect doc PRs and support community through the transition period. This change will likely cause some disruption. Need to be sure there is clear messaging and guidance for doc contributions.
4. Ensure the Ansible core team remain as the technical stewards of the content and actively review documentation contributions. Ansible core should remain an authoritative voice on technical accuracy and completeness of the documentation.
5. Establish what content should constitute Ansible core documentation. See more below.

### Diataxis and content types

With the proposed changes to hosting and repositories, Ansible community docs can have two main categories:

- Ansible package
- Ansible ecosystem

At this point it is useful to have some kind of framework that organizes technical content. This helps the community determine where documentation belongs and identify content requirements.

The [Diataxis framework](https://diataxis.fr/) provides a good approach for characterizing content by type to support both the acquisition and the application of knowledge.

Ecosystem documentation should focus more on tutorials (install, configure) and reference. Package documentation expands the content set with How to's that span projects in the ecosystem. Being able to easily identify and describe types of content also facilitates the creation of contributor opportunities.

### Opening up the package doc builds to community

The Ansible community should have access to and ownership of the build and deployment of the package docs. Providing more ownership of the publishing process strengthens the Ansible community while reducing operations and maintenance for Ansible at Red Hat.

High-level steps to get there:

1. Create an AZP project for Ansible community docs CI.
2. Use ephemeral VMs to build and deploy package docs on demand.
3. Use an S3 bucket for static hosting. This also allows us to safely create PR previews for package docs.