# Pulp "Unified Docs" Project - Part 1 ## Abstract The goal of this project is to create an unified and well-structured documentation for Pulp Project. This document is organized as follows: - **Content Design**: structuring content systematically - **Presentation Layer**: presenting aggregated content in a website hierarchy - **Implementation**: choosing and implementing the tech - **Publish Process**: releasing/publishing the build - **Content Migration**: migration priorities and delivarables (ongoing) - **Future Work**: ideas for future work (ongoing) --- ## Content Design ### Overview The goal of this project is to have an unified docs. But in order to aggregate from various sources and build something reasonable, we need some basic standarization. Because of that, we've choose to addopt [Diataxis](https://diataxis.fr/) as a basis for building such a standard. It defines a user-driven approach which outlines [4 content-types](https://diataxis.fr/map/#expectations-and-guidance), which we'll use as our basic *documentation* **content types**. Furthermore, we'll define **personas**, **repository-types** and a base **folder structure**, and how those structure can be translated in the **presentation layer** (the final website hierarchy, which doesn't have to be the same as the base folder-structure). Here are the schematics of how these should work togheter: ```mermaid graph A["` **persona** X **content-type** `"] B["` **folder structure** `"] C["` **presentation layer** `"] B -- strictly based on --> A C -- flexibly based on --> A C -- predictably fetches from --> B ``` ### Personas We've identified three different user profiles for our product. These profiles will guide how we'll create and organize content. - **User**: "I just want to create sync and publish repositorioes" - **Admin**: "I need to get this instance configured and keep it running" - **Dev**: "I need to add features, troubleshoot and fix bugs" ### Content Types For each persona, we'll use the 4 [Diataxis categories](https://diataxis.fr/map/#blur): - **Tutorial**: learning-oriented phase - We begin by learning, and learning a skill means diving straight in to do it - under the guidance of a teacher, if we’re lucky. [DRF Tutorial Example](https://www.django-rest-framework.org/tutorial/1-serialization/) - **Guide**: goal-oriented phase - Next we want to put the skill to work. [Pytest How-to Example](https://docs.pytest.org/en/latest/how-to/usage.html) - **Reference**: information-oriented phase - As soon as our work calls upon knowledge that we don’t already have in our head, it requires us to consult technical reference. [Pytest Reference Example](https://docs.pytest.org/en/8.0.x/reference/index.html) - **Learn**: explanation-oriented phase - Finally, away from the work, we reflect on our practice and knowledge to understand the whole. [Pytest Explanation/Learn Example](https://docs.pytest.org/en/latest/explanation/fixtures.html) ### Plugin Ecosystem Pulp Project has a complex set of plugins. We've categorized them to help organize content presentation. 1. **content-repos**: Content repositories that are shipped [within OCI-images](https://docs.pulpproject.org/pulp_oci_images/single-process-images/#pulp-minimal). - **Contains**: ansible, container, deb, file, maven, python, rpm, ostree, certguard - **Obs**: Special care to Normal users. 1. **general-repos**: Non-content and non-pulpcore - **Contains**: oci-env, oci-images, operator, cli, glue, k8s-resources, squeezer, selinux, openapi-generator - **Obs**: More advanced or general purpose (oci-env, cli). 1. **pulpcore-repo**: The pulpcore repo only. - **Contains**: pulpcore - **Obs**: Common and overview content ### Folder Structure In the project, we have the requirement of aggreating content from different repositories into a single build. To make this work, each respository should **know what content to create/update**, so we've defined folder structure for each repository-type. Note that this does not mean every one of these folder must be populated, it means they can. Later on we'll elaborate a priority plan of deliverables for orienting this. #### 1. All repositories ```bash CHANGES.md # expected to be here staging_docs/ user/ tutorials/ guides/ learn/ reference/ admin/ tutorials/ guides/ learn/ reference/ dev/ tutorials/ guides/ learn/ reference/ ``` > [Changed in feb/24] Remove the global "reference" and use it as the other contents. #### 2. Pulp-Docs ```bash CHANGELOG.md staging_docs/ (...) # same as other + index.md # landing page sections/ user/index.md admin/index.md dev/index.md help/index.md ``` ## Presentation Layer There are many ways in which these "Content-types x Personas" matrix can be arranged into a concrete documentation website (see [complex hierarchies](https://diataxis.fr/complex-hierarchies/)). The first Live Demo was [this](https://pedro-psb.github.io/pulp-docs). The actual one being used is [this](https://staging-docs.pulpproject.org/). The current demo structure can be summarized in plain text as below. Other ideas on organizing the content in a hierarchy are very welcomed. ```bash Home User Manual Overview Pulpcore Overview # optional ${tutorials,guides,learn,reference} Plugins ${plugin-repo} Overview # optional ${tutorials,guides,learn,reference} Extra ${extra-repo} Overview # optional ${tutorials,guides,learn,reference} Admin Manual (...) # same as previous Developer Manual (...) # same as previous Help Get Involved PulpCon Documentation Usage/ Changelogs/ Governance/ ``` ## Implementation The following choices were based primarily in popularity/maintanability: - Main framework: [mkdocs](https://www.mkdocs.org/) - Theme: [mkdocs-material](https://squidfunk.github.io/mkdocs-material/) - also provides some plugin out-of-the-box - Reference Generator: [mkdocstrings](https://mkdocstrings.github.io/) - generates doc from code/docstrings - Aggregation: [pulp-docs](https://github.com/pedro-psb/pulp-docs) For the aggregation, we starrted with [mkdocs-multirepo-plugin](https://github.com/jdoiro3/mkdocs-multirepo-plugin), but after some experimentation it proved itself not enough flexible to get content from the proposed **content type** structure to the **presentation layer** structure. Because of that, we've developed *pulp-docs*, a python package to help build and serve the docs. It uses a [mkdocs-macros-plugin](https://mkdocs-macros-plugin.readthedocs.io/en/latest/) hook to inject a preparation script into the mkdocs processing workflow. Additionally, it should help develop and distribute future doc tooling and automations across all repos, such as configured linting and other kind of automations, while also making it easier to run the docs locally regardless of a full oci-env setup. ## Publish/Release Process The publishing workflow is [define in pulpcore](https://github.com/pulp/pulpcore/blob/main/.github/workflows/staging-docs.yml) and should be configured to run nightly and publish to https://staging-docs.pulpproject.org/. ## Content Migration ### Workflow Here's a recommended workflow: 1. Copy the existing docs to a folder named `tmp_docs` 2. Convert all of `tmp_docs` to markdown all at once using `rst2myst` 3. Checkin the `tmp_docs` to version control so you can work on it over time 4. Move files and/or sections from `tmp_docs` to `staging_docs` 5. When `tmp_docs` is empty, the work is done and it can be removed ### Setup Here's some helpful commands to setup the repository: ```bash #!/bin/bash # 1. create structure mkdir -p staging_docs/{admin,user,dev}/{guides,tutorials,learn,reference} touch staging_docs/{admin,user,dev}/{guides,tutorials,learn,reference}/.gitkeep cp -r docs tmp_docs # 2. convert to rst->markdown (optional) pip install rst-to-myst[sphinx] find tmp_docs -name '*.rst' -exec rst2myst convert --replace-files {} ';' # 3. extra automated cleaning DOCDIR=tmp_docs find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/:::{note}/!!! note/' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/:::{tip}/!!! tip/' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/:::{warning}/!!! warning/' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/:::{hint}/!!! tip/' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/:::{glossary}//' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/{term}//' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/{github}//' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/{ref}//' {} ';' find $DOCDIR -type f -name "*.md" -exec sed -i -re 's/^\([a-zA-Z-]+\)=$//' {} ';' # remove (something)= find $DOCDIR -type f -name "*.md" -exec sed -i -re 's/`([a-zA-Z ]*)<\w*>`/`\1`/' {} ';' # remove `some thing <RemoveThis>` find $DOCDIR -type f -name "*.md" -exec sed -i -e 's/:::$//' {} ';' ``` (Taken from [this gist](https://gist.github.com/pedro-psb/b933a5abce2797b939c230b4d4020cd1)) The motivation for having `tmp_dir` is to make it easier to determine what files were already migrated and what remains. If there is not much content in the repo, that may not be necessary. ### Guidance Some guidance on how to choose the right place to put some content. - Tutorial or How-to Guide? - A tutorial’s purpose is to help the pupil acquire basic competence. - A how-to guide’s purpose is to help the already-competent user perform a particular task correctly. - Read more [here](https://diataxis.fr/tutorials-how-to/) - Admin or User? - Generally, if it is related to an API it's for a user. "Related to" includes the API itself or accessing it via the CLI, bindings, etc. - There are some exceptions, specifically, the following are probably APIs that are for admins: - Access Policies, Groups, Users, Signing Services, Importers, Exporters, Repair. - All other info is for admins, which likely include topics like: - Installation, Upgrading, Configuration (anything realted to settings), Logging, Custom Authentication e.g. LDAP or django-social, architecture, performance, tuning, monitoring, OTEL, etc. ### Markdown style guide Mkdocs provides a superset of features over markdown. Here we'll mention only fundamental decisions that should be followed. For more extensive reference on other markdown "components", see the [live cheatsheet](https://staging-docs.pulpproject.org/pulp-docs/docs/dev/reference/markdown-cheatsheet/). #### Internal links ``` this is a [link](site:{repo}/docs/{persona}/{content-type}/page.md). ``` - use custom absolute links (with [mkdocs-site-urls](https://github.com/octoprint/mkdocs-site-urls) plugin): - see tradeoffs of *absolute vs relative* [here](https://github.com/pedro-psb/pulp-docs/issues/2) #### File includes We won't support file includes for now. Move their content to markdown files. ## ggainey's notes - General: - drop any "create naked artifacts" workflows, we are trying to dissuade users from doing that - force order by naming files NN-name, where "NN" is monotonically-increasing integer - Markdown tips: - don't forget to indent anything under `!!note` or `!!warning` et al - example: https://github.com/pulp/pulp_rpm/pull/3443/files#diff-93bac9e5a8b6eabb3d7da39b7b588e1af561f8a2b2ed501fb7facaa30613262dR3-R4 - use tab-formating for code-samples and their output - example: https://github.com/pulp/pulp_rpm/pull/3443/files#diff-9d8af4ff676337d58f6e1860bdcc3abe47622b59f0095f44111c19dba0dc8fcbR8-R28 - Placement tips: - **overviews:** `{persona}/index.md` (e.g. `user/index.md`) - **rbac:** `admin/guides` - **settings:** `admin/reference` (its extensive and descriptive material) - **glossary:** `{persona}/reference` (for the persona it makes more sense) - **plugin ecosystem concepts:** `user/reference` - example: https://github.com/pulp/pulp_rpm/pull/3443/files#diff-5949196cbd442dad6a7cd1eb2b4bd878f59b3634e5f2db24d0ae7d757c68bde2 - the example is not in reference because it was before we added the reference fetching. ## Future work - every plugin should include an "upload content to a repo" example - every plugin should have "upload naked artifacts and turn into content" example, **in the admin/guide (admin/reference?) section** Q1 Increase the adoption of Pulp upstream by improving the documentation for new and existing users. - [x] Create a unified documentation site - [x] Make contributing to the documentation easier - [x] Systematically structure the documentation content Q2 - Proposal Increase the adoption of Pulp upstream by improving the documentation for new and existing users. - Make the staging-docs stable and production ready - integrate changelog links - See https://staging-docs.pulpproject.org/pulp-docs/docs/sections/user/ - changelog needs to be generated to .md not .rst - integrate REST API links - See https://staging-docs.pulpproject.org/pulp-docs/docs/sections/user/ - maybe add pointer at the top of each plugin section? - version needs to be filled in - fix ordering mechanism problem (no renaming files) - move pulp-docs to pulp organization - Content improvement and de-duplication - fill in the "voids" in our pulpcore staging-docs (eg, "Getting Started") - start migrating content from pulpproject.org into the staging-docs hierarchy - poss use this content to fill the holes above - Ultimately - replace current pulpproject.org with the output of staging-docs efforts - Add base CI and automation infrastructure for auto-build/test/publish of new-docs-location - doc-only changes should run *only* docs-CI (ie, not full code test suites) - setup the docs-CI (e.g, test builds) - pipeline needs to account for "plugin Foo releases, docs-site rebuilds"