Title: Emitting comments in ytt
Status: Scoping (Scoping | Pre-Alpha | In Alpha | In Beta | GA | Rejected)
Originating Issue: [ytt#63](https://github.com/carvel-dev/ytt/issues/63)
- [Problem Statement](#problem-statement)
- [Customer Impact](#customer-impact)
- [Proposal](#proposal)
- [Emitting YAML Comments](#Emitting-YAML-Comments)
- [Emitting ytt templating](#emitting-ytt-templating)
- [Preserve existing comments](#preserve-existing-comments)
- [Use Cases](#use-cases)
- [Open Questions](#open-questions)
- [Answered Questions](#answered-questions)
- [Implementation Considerations](#implementation-considerations)
## Problem Statement
Configuration Authors want ytt to be able to emit comments from YAML templates in the resulting YAML.
Today, users who need to read and understand YAML configuration files have hard time finding the source information, understanding complex or not self-explanatory logic when certain overlays or custom libraries are used.
Authors miss the opportunity to keep this important information in the template and the consumers may spend time in finding it from original yaml. This is the time lost to making progress in their first-order work.
### Customer Impact
1) Emitting comments would bring a huge benefit to our customers. They could not only leave more explicit breadcrumbs, but name the source of a change or rationale for a given value. This not only reduces the guesswork, but eliminates much of the toil in gathering the facts.
2) Without the ability to weave in context (in the form of YAML comments), our customers must burn hours sleuthing back through whatever process(es) executed ytt, to find their inputs.
3) Also, folks who want to automate building ytt templates also got it rough. Without the ability to include ytt templating directives as output of a ytt template, they are forced to hack a makeshift templating system together. At best it makes the whole system avoidably complex and unnecessarily brittle as they must bring in other tools (e.g. sed/awk/yq/patch).
4) The ability to emit ytt annotations and code in the form of comments in the output, would allow folks to design ytt templates that evaluate in phases. Each phase is a ytt invocation with data values specific to that phase. All along the way, they continue to reap the benefits of structural templating. The resulting system is conceptually simpler: one tool.
5) In the Kubernetes space, as cluster deployments become increasingly complex, the need to emit comments becomes critical. Users are rightfully expecting their tooling to help them see the comments as they were in the original template and also add few for their own configuration logic.
**Current handling of comments in ytt:**
1) YAML comments with # and ytt comments with #! are ignored in the templated output.
Input:
```yaml
#! This is a ytt comment
# This is a yaml comment
kind: Pod
apiVersion: v1
metadata:
name: test-comments
labels: test-label
```
Output:
```yaml
kind: Pod
apiVersion: v1
metadata:
name: test-comments
labels: labels
```
2) Comments are shown only with the **ytt fmt** option when there is no templating involved.
Input:
```yaml
# source: "abc"
# author: "xyz"
#! generated by ytt
#! yaml fragment function
#@ def labels():
app: echo
org: test
#@ end
kind: Pod #! yaml comment
apiVersion: v1
metadata:
name: test-comments
labels: #@ labels()
```
ytt gives an error:
```yaml
ytt: Error: Compiling YAML template 'config.yml':
Failed to parse line 4: '# source: "abc"':
Expected ytt-formatted string (use '#@' for annotations or code, '#!' for comments)
```
With ignore-unknown-comments flag, this runs with an output as below:
```yaml
kind: Pod
apiVersion: v1
metadata:
name: test-comments
labels:
app: echo
org: test
```
## Proposal
### Emitting YAML Comments
Authors can template in comments via the @yaml/yaml-comment annotation.They will appear in the final output as yaml comment (# with a leading space)
### Preserve existing/new comments
Preserve single-line, multi-line and inline yaml comments (#) and the ytt comments (#!)
### Disallow emitting ytt templating through the @yaml/yaml-comment
yaml annotations like @overlay won't be allowed to use through comment
### Opt-out of emitting the leading space in corner cases
Add #cloud-config at the start of the configuration
### Generate ytt with ytt
- This is useful in the package world by automating the process of creating packages.
- Doing package installs in packages so that the customers install just one package.
- Demo magic — create scripts inside comments and some functions and your file has your whole demo scripted.
### Preserve ytt annotations through a ytt run
- automating PackageInstall bump in tap-pkg
## Use cases
**1. Emitting yaml comments via the @yaml/yaml-comment annotation.**
```yaml
By default, the yaml comment is prefixed with a leading space:
databases:
#@yaml/yaml-comment "selected '{}' because env = '{}'".format(data.values.db_conn[data.values.env].name, data.values.env)
name:
#! alias for environment
alias: #@ data.values.db_conn[data.values.env].name
hostname: #@ data.values.db_conn[data.values.env].host
```
Comments are evaluated and preserved
```yaml
databases:
# selected 'dev1' because env = 'dev'
name:
# alias for environment
alias: dev1
hostname: myexampledb.a1b2c3d4wxyz.us-west-2.rds.amazonaws.com
```
**2. Preserve existing comments or add new yaml comments in templates :**
Preserve single-line, multi-line and inline yaml comments (#) and the ytt comments (#!)
```yaml
# source: "abc"
# author: "xyz"
#! generated by ytt
---
#! ytt comment
# yaml fragment function
#@ def labels():
app: echo
org: test
#@ end
kind: Pod # yaml comment
apiVersion: v1
metadata:
name: test-comments
labels: #@ labels()
```
Comments preserved as below:
```yaml
# source: "abc"
# author: "xyz"
#! generated by ytt
---
#! ytt comment
# yaml fragment function
kind: Pod # yaml comment
apiVersion: v1
metadata:
name: test-comments
labels:
app: echo
org: test
```
**3. Authors may opt-out of emitting the leading space in corner-case**
e.g. Add #cloud-config at the start of the configuration. Notice the no leading space
A cloud-config file must contain a header:
Either #cloud-config for processing as cloud-config (suggested)
or #! for processing as a shell script (advanced).
Input
```yaml
#cloud-config
#!cloud-config
#@ def labels():
app: echo
org: test
#@ end
kind: Pod # yaml comment
apiVersion: v1
metadata:
name: test-comments
labels: #@ labels()
```
Output
```yaml
#cloud-config
#!cloud-config
kind: Pod
apiVersion: v1
metadata:
name: test-comments
labels:
app: echo
org: test
```
**4. Disallow emitting ytt templating through the @yaml/yaml-comment:**
```yaml
databases:
#@yaml/yaml-comment "@overlay"
name:
alias: #@ data.values.db_conn[data.values.env].name
hostname: #@ data.values.db_conn[data.values.env].host
```
**5. Emitting ytt templating - Generate ytt with ytt:**
```yaml
#@overlay/match ...
#@yaml/ytt-annotation "overlay/match by= "
#@yaml/ytt-code "if "
#@yaml/comment "because env = "+ data.values.env + "; disabling bar."
foo:
#@yaml/comment "null set by not-bar.yaml"
bar: #@yaml/ytt-value ""
#@overlay/match ...
```
A Carvel user wanted to template their PackageInstall CR so that they could include it in a collection of packages to be installed as a larger system:
Build Step: during "build time", populate some fields with values determined by the published of the package:
telemetry-package-install.yml
```yaml
#@ load("@ytt:data", "data")
#@ #@ load("@ytt:data", "data")
#@ #@ load("@ytt:yaml", "yaml")
#@ #@ load("_profiles.star", "profiles")
#@ #@ if profiles.is_any_enabled([profiles.full]):
#@ #@ if profiles.is_pkg_enabled("telemetry.platform.com"):
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
name: #@ data.values.name
namespace: platform-install
spec:
serviceAccountName: #@ data.values.serviceAccount
packageRef:
refName: #@ data.values.name + "." + data.values.domain
versionSelection:
constraints: #@ data.values.version
prereleases: {}
---
apiVersion: v1
kind: Secret
metadata:
name: telemetry-values
namespace: platform-install
stringData:
values.yml: #@ #@ yaml.encode(data.values.telemetry)
#@ #@ end
#@ #@ end
```
such that the lines that start with #@ #@ ... are desired to pass through to the output:
```yaml
#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")
#@ load("_profiles.star", "profiles")
#@ if profiles.is_any_enabled([profiles.full]):
#@ if profiles.is_pkg_enabled("telemetry.platform.com"):
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
name: telemetry
namespace: platform-install
spec:
serviceAccountName: telemetry-acct
packageRef:
refName: telemetry.platform.com
versionSelection:
constraints: 1.1.0
prereleases: {}
---
apiVersion: v1
kind: Secret
metadata:
name: telemetry-values
namespace: platform-install
stringData:
values.yml: #@ yaml.encode(data.values.telemetry)
#@ end
#@ end
```
Install Step: while the system is being installed, the system operator provides installation values to complete the PackageInstall CR:
```yaml
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
name: telemetry
namespace: platform-install
spec:
serviceAccountName: telemetry-acct
packageRef:
refName: telemetry.platform.com
versionSelection:
constraints: 1.1.0
prereleases: {}
```
```yaml
---
apiVersion: v1
kind: Secret
metadata:
name: telemetry-values
namespace: platform-install
stringData:
values.yml: |
```
## Open Questions
1. If we preserve existing ytt comments does customer want the output to show yaml comment(# ) or ytt comment(#! )?
2. What if comment itself needs # without space as in "#cloud-config" or #@yaml-comment type annotation
3. What can be the impact of --ignore-unknown-comments=true
4. What kind of comment errors can we think of?
5. Use of --strict for throwing the error.
6. Using #% which would tell YTT to emit the line to the yaml file
7. How does this affect the ytt fmt command and ignore-unknown-comments flag?(They don't work together)
8. Can overlays be written to add comments?
## Answered Questions
1.
## Implementation considerations
Currently, ytt is built around the v2 release of the yaml module which does not support emitting comments.
1) Implement the "plumbing" code for using yaml v3 parser in certain scenarios where comments are required to emit.
2) Existing yaml comments (#) and/or ytt comments (#!)should be preserved in the final output as yaml comment? (# )
3) Emitting comments can be static or dynamic. e.g. using data values to populate some information in comments.
4) Authors should be able to generate ytt with ytt using the comments/code annotation
5) Authors may opt-out of emitting the leading space in corner-case.
6) Disallow to emit ytt templating through the @yaml/yaml-comment.
7) Analyze and remove the impact of ignore-unknown-comments flag to not throw error for comments
Google Drive
Gist
Clipboard
Sign in with Wallet
