# regodoc - Self describing policy metadata/documentation

An enhancement proposal for Open Policy Agent / Rego.

Authors: Itay Shakury, Aqua Security <<[e-mail]>>
Reviewers:
Timeline:
- 2020-09-26: First draft
- 2020-09-27: Revise the "notes" field into "custom" field which is more flexible
- 2020-09-28: Limit the metadoc to be static JSON
- 2020-09-30: Make entrypoint into a list
- 2021-02-02: Add rule level code documentation, json schema

## About this proposal

This proposal describes a conventional pattern to describe policy metadata and code documentation as part of Rego code. Policy metadata allows a Rego policy to declare it's charicteristics and commentary for any tool that consumes it, or any user who uses it. Code documentation allows annotating the Rego code for other developers, similar to other programming languages for example godoc, javadoc, etc. These two use cases are separate but related, and share implementation details and considerations, and therefore are discussed together in this proposal.

This is a follow up of the Github issue: https://github.com/open-policy-agent/opa/issues/2180.

### Terminology
The following terms are used throughout this document:

1. User persona - evaluates a policy, either explicitly using a tool (e.g conftest), or implicitly via a proxy tool (e.g admission controller).
2. Author persona - writes the policy (Rego).
3. Developer persona - builds tools/systems that integrates with Rego to evaluate policies.
4. Policy - unit of decision making, which is written as one or more Rego files, which contains one or more Rules that are evaluated to make the decision.
5. Metadoc - short for metadata/documentation, both are relevant use cases for this proposal.

### Use cases
1. As a User, I know where to find additional information that describes the policy that I'm using.
2. As an Author, I know how to document to Users the domain context that relates to the policy.
3. As an Author, I know how to document to Users the intended usage of the policy.
4. As an Author, I know how to document to Authors the way that the policy works and is written.
6. As a Developer, I know where to find usage and technical documentation for a policy that I'm using.
7. As a Developer, I can programmatically consume metadata from policies that I'm using.

### Scope
In:
- Define the specification: format and conventions for declaring policy metadata and policy code documentation
- Discuss implications of this proposal on policy authoring experience

Out:
- How to consume the metadoc
- Tooling (for example documentation generation)
- Recommending best practices for authoring policies

## Prior art
- [John Harris](https://github.com/johnharris85) of VMWare did a great demonstration at a previous OPA community call of a Rego code documentation tool that he created. [July 7 2020 recording (demo starts at 21 minute mark)](https://zoom.us/rec/share/ps12CJSo7EFJbM_NtRrxHakqJ4_IX6a81nUb-aVZnh4SxdXCJjpGadFIAjuGWMN7).
- [Konstraint](https://github.com/plexsystems/konstraint) project is using structured comments to generate Gatekeeper constraint definitions. [Example](https://github.com/plexsystems/konstraint/blob/main/examples/container-deny-privileged/src.rego).
- [Fugue](https://github.com/fugue) is using specially named rules to describe controls which the policy relates to. [Example](https://github.com/fugue/regula/blob/master/rules/aws/ebs_volume_encrypted.rego).
- [AppShield](https://github.com/aquasecurity/appshield/). Is using javadoc inspired comments to generate documentation for the library of policies. [Example](https://github.com/aquasecurity/appshield/blob/master/policies/kubernetes/policy/runs_as_root.rego).
- Many programming languages have code documentation tools that we can learn from. For example, Golang has the `go/doc` package that's described [here](https://golang.org/pkg/go/doc/). Golang code documentation also integrates with Golang tests, by means of [Examples](https://golang.org/pkg/testing/#hdr-Examples).

## Spec
This section presents the proposed specification.

> For a detailed discussion of the decision making process, please refer to the "Considerations" section blow.

### Policy metadata

1. A policy can declare metadoc information by declaring a **rego rule**.
2. The rule must be in the **same package** as the policy it's describing.
2. The rule name must be **`__rego__metadoc__`**.
4. The rule must be a **JSON literal** that may contain any of the supported fields described below.
5. It's recommended but not required that the metadoc will be defined at the top of the file.

Supported fields:

field | description | type (validation) | example
--- | --- | --- | ---
id | Unique identifier for the policy | string | XYZ
title | Descriptive name for the policy | string | Pod has privileged container
description | Full description of the policy goal and business logic. Ideally not covering it's usage (which is described separately) | string (markdown) | This policy checks for containers in the Pod that have the `securityContext.privileged` field set to `true`. Privileged containers can allow almost completely unrestricted host access.
custom | Additional information about the policy. For example: remediation advice, references, etc. | object | <table><tr><th>key</th><th>value</th></tr><tr><td>severity</td><td>High</td></tr><tr><td>links</td><td><ul><li>[Kubernetes Docs: Privileged mode for pod containers](https://kubernetes.io/docs/concepts/workloads/pods/pod/#privileged-mode-for-pod-containers)</li></ul></td></tr></table>
entrypoints | Description of existing rules that should be queried. Further information can be described in the entrypoints ruledoc | array of string (path to rule) | app.xyz

Example:

package app.xyz

__rego__metadoc__ := {
  "id": "XYZ",
  "title": "Pod has privileged container",
  "description": "This policy checks for containers in the Pod that have the `securityContext.privileged` field set to `true`. Privileged containers can allow almost completely unrestricted host access.",
  "custom": {
    "severity": "High",
    "links": [
      {
        "title": "Kubernetes Docs: Privileged mode for pod containers",
        "href": "https://kubernetes.io/docs/concepts/workloads/pods/pod/#privileged-mode-for-pod-containers)"
      }  
    ]
  },
  "entrypoints" : [
      "app.xyz.deny"
  ]
}

deny[ret] {
  ...
}

### Rule documentation

1. A policy can declare rule level code documentation by declaring an **incremental object rule**. By using an increments rule, each rule's documentation can be adjacent to it's defintion.
2. The object keys represent existing rules, and the value is that rule's documentation.
3. The rule must be in the **same package** as the policy it's describing.
4. The rule name must be **`__rego__ruledoc__`**.
5. The rule must be a **JSON literal** that may contain any of the supported fields described below.
6. It's recommended but not requried that the ruledoc will be defined right before the rule it's documenting.

Supported fields:

field | description | type (validation) | example
--- | --- | --- | ---
description | Full description of the rule's goal and business logic | string (markdown) | containers returns all the containers in a Kubernetes resource (Pod, Deployment, etc.), if any. This includes init containers and regular containers.
input | Description of this rule's expecations of the input document, or incase of a function, it's arguments as well | string | the input docuemnt should be a valid Kubernetes resouce, or a valid Kubernetes Admission Review resource, of Pod, or another resource that defines a Pod, such as Deployment, ReplicaSet, StatefulSet, etc.
input_schema | URL of JSON schema describing the input | http://path/to/jsonschema
output | Description of this rule's result | string | set of container objects
examples | Full paths to tests that demonstrate basic usage of the rule | array of string (path to rule) | <ul><li>app.xyz.test_privileged_denied</li><li>app.xyz.test_unprivileged_allowed</li></ul>

Example:

package app.xyz

__rego__ruledoc__["deny"] = {
    "description": "containers returns all the containers in a Kubernetes resource (Pod, Deployment, etc.), if any. This includes init containers and regular containers",
    "input": "the input docuemnt should be a valid Kubernetes resouce, or a valid Kubernetes Admission Review resource, of Pod, or another resource that defines a Pod, such as Deployment, ReplicaSet, StatefulSet, etc.",
    "input_schema": "https://raw.githubusercontent.com/instrumenta/kubernetes-json-schema/master/v1.18.1-standalone/pod.json",
    "output": "set of container objects",
    "examples": [
        "app.xyz.test_privileged_denied",
        "app.xyz.test_unprivileged_allowed"
    ]
}

deny[res] {
  ...
}

### Language syntax

As a second phase, it would be nice to extend the basic spec described here with "syntactic sugar" that will make metadoc authoring more streamlined, and documented policies more readable. This section describes a syntax that can be transformed into the primitive metadoc/ruledoc described above. The proposed syntax takes the following form:

Syntax spec:
1. A policy author can add a documentation block instead of defining documentation using a rule. the result is the same.
2. Each block defines either a `__rego__metadoc__` object or a `__rego__ruledoc__` object which is determined by the position of the block:
1. If the block is located right before the `package` declaration, it is defining policy metadata.
2. If the block is located right before the rule head, it is defining that rule's documentation.
4. A block is a collection of lines. Each line must start with the character `@`.
6. Each line sets one field in the generated object, by using a `key: value` format, where:
1. `key` is a path-selector string, without enclosing qoutes (`"`), to one of the metadoc/ruledoc supported fields.
9. `value` is rego value of the type of the field as defined in the spec.

Example:

@id: "XYZ"
@title: "Pod has privileged container"
@description: "This policy checks for containers in the Pod that have the `securityContext.privileged` field set to `true`. Privileged containers can allow almost completely unrestricted host access."
@custom.severity: "High"
@entrypoints: ["deny"]
package app.xyz

@description: "containers returns all the containers in a Kubernetes resource (Pod, Deployment, etc.), if any. This includes init containers and regular containers"
@input: "the input docuemnt should be a valid Kubernetes resouce, or a valid Kubernetes Admission Review resource, of Pod, or another resource that defines a Pod, such as Deployment, ReplicaSet, StatefulSet, etc."
@output: "set of container objects"
@examples: ["app.xyz.test_privileged_denied", "app.xyz.test_unprivileged_allowed"]
deny[ret] {
  ...
}

> This syntax can also be written as-is inside a Rego comment, which is a least-friction path for adoption

---

## Considerations
This section discusses different implementation details, and explains how they contribute the the proposed design.

### Comments vs expressions
The metadoc information could be added to the policy as Rego comments, or as actual Rego code.

- Comments need some kind of structure to be parsable. This is essentially another syntax within Rego.
- Comments are always untyped (or serialized) text, which is harder to work with programmatically.
- Using Rego code for the metadoc means other rules within the policy can use this information, for example to produce a summary report.
- Metadoc identifiers might conflict with actual business code, in which case metadoc naming conventions will need to be reserved (this is unnecessary with comments).
- Comments are not parsed into the AST so not encapsulated with the policy. In theory, the comments might get lost in a complex parsing/compilation process.
- Authoring in comments is more error prone, and are harder to build tools for.
- Using comments is easier for external tools to process - no need involve Rego library, can rather use simpler static text processing methods.

✍︎ This proposal choose to use Rego code over comments.

### Supporting static parsers
Allowing the metadoc to be authored in arbitrary Rego means the only way for another tool to read it would be to load and evaluate the policy. This is especially concerning since the Rego library is only available in Golang.
If every consuming tool had to use Rego Golang library just to read the metadoc, it would have limited its usability and acceptance.

To address this concern, we could limit the metadoc section to a subset of Rego which is parsable without Rego - A JSON literal, so that the metadoc object will be both valid Rego and Valid JSON

- By restricting to JSON we gain all the aformentioned benefits of code over comments, while at the same time enabling staticly parsing the policy as text.
- Imposing any special treatment to the metadoc might be a source for confusion for new users.

✍︎ This proposal choose to limit the metadoc Rego code to be a static JSON literal.

### Object vs scalars
Some of the existing implementation that describe metadoc in Rego, do so by declaring multiple scalars with meaningful names. This approach is also suggested in the Github issue discussion. As an alternative the entire metadoc information could be grouped in a single object with different fields for each piece of information. For example:

title := "Runs as root user"
description := "Force the running image to run as a non-root user to ensure least privileges."
...

metadata := {
  "title" : "Runs as root user",
  "description" : "Force the running image to run as a non-root user to ensure least privileges.",
  ...
}

- Introducing more conventional metadoc identifiers is more likely to cause conflicts with other policy code.
- An object is easier to consume as a group, and is easier to discover it's fields.

✍︎ This proposal choose to use an object to group policy metadata under.

### Naming
Using a Rego Rule to describe the metadoc information means that the Rule's name should not be usable by other Rules in the policy. To minimize potential conflicts we need to pick a name for this rule that is unique and uncommon.

Based on the Rego language grammar, valid rule names which are limited to `( ALPHA | "_" ) { ALPHA | DIGIT | "_" }`.

✍︎ This proposal choose to use `__rego__metadoc__` for the name of the object.

### Domain specific fields
Many Rego policies are written to identify issues in the input, and for those policies it is common to annotate findings with fields such as "severity", "remediation", or other domain specific information. This proposal however targets Rego upstream and cannot assume the business domain of the policy. Therefore we should find a generic way to capture this information.

✍︎ This proposal choose to include a "custom" field which generically describes additional information.

### Usage documentation
In addition to the obvious metadoc fields like id, title, description, etc., there is another important use case for code documentation which is documenting the intended usage techniques. In javadoc for example, we can describe what arguments a function expects, and it's return value. In John Harris' demo, we saw a section for passing and failing examples. In godoc, we can designate tests as Examples to be showcased in generated documentation.
For Rego, we consider the following usage documentation:

#### Entrypoints
A policy can be composed of many rules and span many files. Usually a policy will expect it's consumers to evaluate specific rules which yield the policy decision. This is a decision that the Author makes which isn't easily communicated to the User. In the authorization domain there are some conventions like naming this rule `deny` but even there we see variations, like `violation` (e.g [conftest's `isFailure`](https://github.com/open-policy-agent/conftest/blob/73cfc1c6586a7f9f480e135d3388a803ca42ca05/policy/engine.go#L320)) so it's needed to be definitively documented and communicated.

✍︎ This proposal choose to add an "entrypoints" field to the metadoc.

#### Input/Output
Once a we know which Rules to evaluate there is still the question of what they expect as input, and what they return. Javadoc for example lets us document the function's "arguments", and "return" value (e.g [Javadoc "@param" and "@return"](https://docs.oracle.com/javase/8/docs/technotes/tools/windows/javadoc.html#CHDHJECF)). In Rego, the metadoc is at the policy level rather then the Rule level, which shifts the attention to the `input` document and the evaluation result (as defined by the "entrypoint" Rule).

✍︎ This proposal choose to add "input" and "output" fields to the entrypoint.

#### Examples
Oftentimes an actual working example is the best usage documentation. We have seen this in John Harris' demo which specified "Pass" and "Fail" sections. Another reference is from Golang which picks up examples from tests into documentation.

✍︎ This proposal choose to use tests as examples rather then as text documentation.

There are two approaches for implementing this:
1. Link from tests to metadoc.
a. as a comment on the test.
b. as a convention to the test name i.e `example_testname`.
2. Link from metadoc to tests.
a. as a field in the metadata object that points to the test.

Linking from test to metadoc will complicate example discovery because consuming tool will need to crawl the Rego code/AST to find examples, and tests are often maintained in a separate file which is not always available at runtime. On the other hand, linking from metadoc to tests is straightforward for discovery as we maintain the example list explicitly.

✍︎ This proposal choose to link from the metadoc to the test name.
✍︎ This proposal choose to add an "examples" field to the metadoc.

## Implications on authoring experience
This section discusses the impact that accepting this proposal might have on policy authoring experience.

### Policy and packages
Although not strictly required, it is widely accepted that Rego package names are hierarchical, and that each policy is contained in one package.
With this proposal, we are in fact setting this practice in stone, since the metadoc information is defined at the package level.

### Rule results
It is very common for Rego rules to evaluate to a set of string messages, which tries to convey additional information about the policy. For example:

  deny[msg] {
    ...
    msg := "XYZ: container %s in namespace %s should not include 'SYS_ADMIN' in securityContext.capabilities.add"
  }

This proposal relates to that pattern because it the information that's conveyed in the string message might be more accurately described in the metadoc. The policy Author could either use the metadoc to construct the string message, or preferably return an object that contains information from the metadoc (or the metadoc itself).

## Tooling
Follow up contributions should introduce tooling support this proposal's spec. This section presents some ideas:

1. `opa explain` command that displays metadata information about a policy.
2. Documentation generating tools that generates end-user consumable documentation by parsing the metadoc.
3. Improvement to authoring tools, like the VS Code plugin, that supports this experience.