Backstage Docs Assessment

Prepared by: Dave Welsch (@dwelsch-esi)

Date: 2023-09-01

Introduction

This document assesses the quality and completeness of the Backstage project's documentation and website.

This document:

• Measures existing documentation quality against the CNCF’s standards
• Provides examples of great documentation as a reference
• Recommends a program of key improvements with the largest return on investment
• Provides guidance to making the recommended improvements actionable TBD

How this document works

The assessment is divided into three sections:

• Project documentation: for end users of the project, aimed at people who intend to use it
• Contributor documentation: for new and existing contributors to the project
• Website: for branding, website structure, and maintainability

Each section begins with a rating based on a rubric with appropriate criteria for the section.

Project documentation

Criteria	1	2	3	4	5
Information architecture
New user content
Content maintainability
Content creation processes
Inclusive language

Scale:

• 1 = (Is not present or requires significant work)
• 3 = (Is present, but needs work)
• 5 = (Is executed extremely well or no improvement required)

Comments

Backstage is a platform for organizing and managing software projects. This complicates the documentation, because it can be difficult to distinguish among several levels of involvement with Backstage:

1. Use of Backstage as an organizational contributor (coder, DevOp, tech writer) to catalog, create, or document software.
2. Administation of a Backstage instance ("app" in Backstage nomenclature), including installation, deployment, software upgrades, configuration, and customization using plugins.
3. Extending a Backstage app, including writing plugins and modifying existing plugins.
4. Modification or extension of the Backstage platform.

This complexity necessitates clear definitions of roles so that use cases can be documented, indexed, found, and used. The four levels described above suggest that documentation should be organized under at least four roles:

1. Using Backstage for software development: end user.
2. Installing and maintaining a Backstage app for an organization: Backstage admin.
3. Extending a Backstage app: DevOp or internal tooling developer.
4. Extending the Backstage platform: contributing developer.

The main issues with the Backstage documentation are:

1. Lack of organization around user roles, and
2. Lack of instructional content that defines tasks for each role and provides explicit instructions to accomplish these tasks.

The Overview contains valuable high-level conceptual information, including a list of benefits to particular user roles (engineering managers, developers, platform engineers). This is a good demonstration of awareness of user roles that needs to be extended throughout the documentation.

There are installation and configuration instructions in Getting Started. However:

1. The instructions describe a local installation.
2. The user documentation, for the most part, lacks procedural information.

The following sections contain brief assessments of each element of the Project Documentation rubric.

Information architecture

Conceptual content is abundant throughout the Backstage documentation. This is good, because using Backstage effectively requires a thorough understanding of its information model.

The documentation seems feature complete. All software features are documented.

Instructional content is a weakness of the Backstage docs: tasks required of each role are missing, as are instructions for completing these roles. Instructional material for at least the most common use cases ("happy path") should support learning (tutorials) as well as doing (procedures).

Escalation options (FAQs, Troubleshooting docs) exists for most Backstage functionality. As well, the contributor community provides robust support (GitHub issues and discussion channels).

Reference information exists, including for APIs, but is scattered throughout the documentation.

It's not clear that the documentation is completely up to date, although release notes exist and are easily findable.

Examples:

• https://prometheus.io/docs/introduction/overview/
• https://kubernetes.io/docs/home/

New user content

New user content is present and findable ("signposted"), including installation instructions for all applicable OSes. However, the Getting Started instructions don't distinguish between administrator and developer end-user roles. Deployment describes various scenarios for administrators, but doesn't provide end-to-end instructions. There is no clear workflow path after installation.

There is example content available, including tutorials for a variety of tasks and a very nice demo server.

Examples:

• https://falco.org/docs/getting-started/
• https://prometheus.io/docs/introduction/first_steps/

Content maintainability & site mechanics

The documentation is searchable. There does not seem to be a localization framework. There doesn't currently seem to be any kind of localization effort.

There does not seem to be any mechanism for versioning documentation content.

Examples:

• https://kubernetes.io/docs/contribute/

Content creation processes

The procedures for duplicating the documentation locally and for contributing documentation are documented but are but not easily findable.

These procedures are presumably included in any updates to (and subsequent release of) the project code (since the doc is in the same GitHub repo). Reviewers and approvers are presumably the Backstage OSS community at large. It would be nice if this situation were made clear explicitly.

The website does not have a clear individual owner or maintainer.

Examples:

• https://github.com/nats-io/nats-site/blob/master/MAINTAINERS.md (clearly documented maintainers)
• https://thanos.io/tip/contributing/how-to-contribute-to-docs.md/

Inclusive language

I found no content that uses non-recommended words as documented by the Inclusive Naming Initiative website.

The website makes occasional use of words like "simple" and "easy", but avoids explicitly ableist language.

Recommendations

The following sections contain recommendations for improvements to the Backstage project documentation. These recommendations are for broad improvements that would greatly increase documentation effectiveness. Further collaboration between technical writers and project leadership is needed to analyze the recommendations into actionable, estimable work efforts.

Clarify user roles and objectives

Any systemic improvement to the Backstage documentation hinges on clearly defining and documenting user roles. This is a first step in defining any documentation set, but nature of Backstage makes it especially important here.

The example roles given in the comments can be used as a starting point for discussion, but should not be adopted without extensive analysis. The roles need to be defined by product experts, not by my hot take.

Discussion participants should include technical writers and all types of users. It should specifically include both Backstage product experts and Backstage OSS project managers because there appears to be no bright line between running a large Backstage instance and contributing to the Backstage codebase.

Develop instructional documentation

Most of the Backstage documentation seems to have evolved from architecture and design documentation. This makes it very rich in conceptual material and reference documentation, but weak in user-oriented instructional documentation.

"Instructional documentation" is a broad category that includes such traditional documentation artifacts as tutorials; getting started guides; procedural recipes or "cookbooks"; runbooks; and how-to guides.

For every user role:

Define use cases: Define the common use cases for each role. Some typical examples:

Administrator: Installation; deployment; configuration; maintenance; upgrades; extension (plugins); disaster recovery.

User (developer): Development environment setup; Adding an existing element; creating a new element; creating a template; viewing projects; searching; creating documentation; CI/CD; deploying; reverting a deployment.

Write procedures: For each use case, develop instructional content, including: "happy path" procedures that can be followed by anyone familiar with the product generally; examples; troubleshooting guides; and for new users, tutorials.

Reorganize the documentation site

Right now different types of documentation (conceptual/architectural; instructional; reference) for different user roles are intermixed throughout the documentation site.

Organize by role and task

The site should be reorganized based on an overarching principle of grouping together documentation needed by a particular user role for a particular set of tasks. This sounds daunting, but it's the schema behind a traditional developer documentation suite, which can be used as a model. For an example of such a doc suite, see this blog post. Or this one on how to think about a doc specification.

Provide adequate navigation signals

Reorganizing the site will make the documentation more usable. Not to be overlooked is the companion task of making the documentation findable. This involves creating adequate tables of contents (TOCs), indexes, and a glossary to help navigate the site. Much of this is automated by the static site generator (currently Docusaurus), but it's the writer's responsibility to assure that these navigation aids are adequate.

Contributor documentation

Criteria	1	2	3	4	5
Communication methods documented
Beginner friendly issue backlog
“New contributor” getting started content
Project governance documentation

Scale:

• 1 = (Is not present or requires significant work)
• 3 = (Is present, but needs work)
• 5 = (Is executed extremely well or no improvement required)

Comments

A unique problem with Backstage is that user and contributor roles exist on a spectrum; plugins designed or modified to serve a particular organization can be contributed to the Backstage project, or can originate there in anticipation of a particular organization's need. As a result, documentation for project contributors is intermingled, and easily confused, with instructions for admin users trying to expand functionality by modifying or adding code, especially plugins.

Communication methods documented

Communication channels are clearly documented on the Community page of the website, including message channels, newsletters and adoption meetings. There is some minor conflation of commercially sponsored content on the community page.

The GitHub repository is directly linked from the main menu. There are 22 related repositories listed on the Backstage project page. A little digging is required to suss out the purpose of some of these projects.

Examples:
https://kubernetes.io/community/

Beginner friendly issue backlog

The backlog is robustly triaged and documented. A "good first issue" label is available and prominently recommended. Issues are well documented with complete descriptions.

There are quite a few stale backlog items open in the backlog list. Many, if not a majority, of these seem to be plugin requests.

New contributor getting started content

The Backstage OSS community is visible and accessible on a community page reached via a top-level menu item on the website. There is an active Discussions board in the GitHub repo.

Contributor guidelines are provided.

Help is available in any of the discussion groups and through a community page on GitHub. Help resources are not linked from the Getting Started documentation.

Examples:

• https://github.com/helm/community
• https://github.com/backstage/community

Project governance documentation

Goverance is clearly documented in GOVERNANCE.md

Examples:

• https://github.com/kubernetes/steering

Recommendations

As an open source project, Backstage looks healthy and well run.

The only recommendation here is to disentagle the contributor documentation from the product documentation, as described in the Information architecture recommendations.

Website

Criteria	1	2	3	4	5
Single-source for all files
Meets min website req. (for maturity level)
Branding and design
Case studies/social proof
Maintenance planning
A11y plan & implementation
Mobile-first plan & impl.
HTTPS access & HTTP redirect
Google Analytics 4 for production only
Indexing allowed for production server only
Intra-site / local search
Account custodians are documented

Scale:

• 1 = (Is not present or requires significant work)
• 3 = (Is present, but needs work)
• 5 = (Is executed extremely well or no improvement required)

Comments

Single-source requirement

The source files for the technical documentation reside in a single directory of the Backstage GitHub repo. There is no separate website repo. It's not immediately clear whether this includes the entire website or just the Docs section. TBD

The strategy for generating the docs is documented but obscure. Again, it's not clear if this is for the *Documentation section only. TBD

Minimal website requirements

Listed and acknowledged below are the (cumulative) minimal website requirements for projects based on their maturity level: sandbox, incubating, graduated and archived.

Maturity	Requirement	Met?
Sandbox	Majority of Website guidelines satisfied
Sandbox	Docs assessment submitting a request completed.
Incubating	All Website guidelines satisfied
Incubating	Request docs (re-)assessment through CNCF service desk
Incubating	Project doc: stakeholders (roles) identified and doc needs documented
Incubating	Project doc: Hosted directly
Incubating	Comprehensive, addressing most stakeholder needs
Graduated	Docs assessment: all assessment follow-through actions are complete
Graduated	Project documentation fully addresses needs of key stakeholders
Archived	The website repo is in an archived state
Archived	Archived status of the project is obvious to site visitors
Archived	Link to successor project website and/or migration documentation (if it exists)

Usability, accessibility and devices

The website is usable from mobile. Doc pages are readable.
Most navigation features are accessible from mobile.

Backstage is probably not a good candidate for a mobile-first design.

Accessibility has not yet been assessed. TBD

Branding

The Backstage brand is easily recognizable through the logo and color scheme. The scheme is used consistently across the website and docs.

The website typography is easy to read.

Examples:

• https://helm.sh/

Case studies/social proof

Case studies and testimonials are not prominent and are not readily findable from the website. There is no logo wall of participating organizations.

The project has an active blog,Community discussions are available on the website.

Examples:

• https://www.fluentd.org/testimonials (user testimonials)
• https://goharbor.io/ (logo wall)
• https://blog.rook.io/ (blog)

SEO, Analytics and site-local search

TBD

SEO helps users find your project and it's documentation, and analytics helps
you monitor site traffic and diagnose issues like page 404s. Intra-site search,
while optional, can offer your readers a site-focused search results.

We evaluate on the following:

• Analytics:
  • Is analytics enabled for the production server?
  • Is analytics disabled for all other deploys?
  • If your project used Google Analytics, have you migrated to GA4?
  • Can Page-not-found (404) reports easily be generated from you site
    analytics? Provide a sample of the site's current top-10 404s.
• Is site indexing supported for the production server, while disabled for
  website previews and builds for non-default branches?
• Is local intra-site search available from the website?
• Are the current custodian(s) of the following accounts clearly documented:
  analytics, Google Search Console, site-search (such as Google CSE or Algolia)

Maintenance planning

The website tooling (Docusaurus static site build) is well supported.

Cultivation of website maintainers from within the community is unknown. TBD

As far as I know, site build times are reasonable and site maintainers have sufficient permissions. TBD

Examples:

• http://kubernetes.io

Other

The website is served via HTTPS. I don't know if HTTP access redirects. TBD

Recommendations

TBD

Recommendations

The top documentation concerns for this project are:

• Defining stakeholder roles and disentangling amibuous roles.
• Completing instructional documentation for all stakeholder roles, including project contributors.
• Organizing and "signposting" documentation by role and task so that stakeholders can find documentation that supports their roles' activities.

(Analysis to actionable tasks and issues TBD)