# Cari's brain on ytt

Note: This is a working doc meant for writing down concepts as I see them. Please see the [docs](https://carvel.dev/ytt) for official definitions.

Input
---
Users:
- Configuration Authors: Authors of templates and data values. In Kubernetes world, these are often application developers, devops, or PAs. The authors then distribute the templates to those that want to install/use them.
- Configuration Consumers: Consumers of the templates that the Configuration Authors wrote. In k8s world, they use the templates to install the applications onto their clusters.

Types:
- NewBulkFilesSource: Used to process JSON input, from the ytt playground
- NewRegularFilesSource: Primary input source. Processes files input with `-f` and accepts other flags.

- Directory Structure / file system contract:
- Libraries:
- Like a virtual file system built from the relative file path of the inputs. Some libraries can be 'private'.
- Private library:
- is one that is in the `_ytt_lib` directory.
- Other libraries cannot access a private library.
- It can be thought of as a separate ytt invocation.

- File Marks: Used to override the default behavior of ytt file processing. Can override the file extension.
- Data Values: Provide a way to inject input data into a template via parameterized variables.

Concepts
---
- Overlays:
- Additional templates that are merged with the previous templates. Templates and data values are merged separately.
- Left: being modified
- Right: the overlay describing the modification.

- Single unit template: Each yaml document is its own single unit. One unit can overlay another, and does not relate to scope. For other file types, a file may be a single unit of configuration.

Strive for:
---
- Readability:
- Important for new maintainers
- Descriptive & consistent variable names
- Extract code to functions when it's not helpful to have the context at the current level.
- Don't extract code to function when it does not add to complexity.
- Refactors are necessary to get here but should be done separate from features.

- "Single Point Of Truth": Goal is to not have duplicate code to reduce inconsistency and organization. Don't pay the 'complexity cost' twice.

- Compactness: the property that a design can fit inside a human being's head.

- Be different when we have a reason to be. / Dont conform to patterns when it doesn't fit

- Order test variables consistently across tests.

- If I remove one line/thing in the code only ONE test should fail. In other words, test only one unit per test.

:bee: :bee: :bee:
Go patterns
---
- Explicit error checks
- Discourage use of named returns generally
- generally, yes, adds work to the reader to determine what's being returned.
- when small, might be nice to use to trim
- no reason can't "document" return values with named returns (don't have to use them)
- Keep methods private by default
- this makes sense in general, however it may be worth considering what the package 'api' should expose. Could the package be imported by other tools?
- Function names plurality match what they return (ie: return list, name is plural)
- Constructors start with 'New'

When reviewing think about:
---
- Does an if make more sense as a switch (and viceversa)
- When you want to return a boolean to indicate something to the parent method, consider combining the two methods instead
- Understand the contributers intention, be a guide to them to find the right spot or with what is already there
- begin with the tests, agree on the ux first
- When building Starlark builtin functions, you must return Starlark values, not nil or golang values. See ip patched PR for details.

When Patching
---
For unreleased code, dont have to make just the minimal code change, do it right.

Doc opportunities
---
How to write a builtin function
- mention that you must return starlark values.

Intentional ytt 'patterns'
---
- library patterns
- A file (schema or data value) uses it's `libRef` to indicate what library it belongs to. No libRef? Then it's intented for the current library.
- A private library is mostly 'self contained' and can be thought of as a separate ytt invocation.
- Many of data value and schema code follows a similar pattern.. Is this the goal or an intermediate stage?
- Factory
- LibraryExecutionFactory
- Envelope
- DocumentSchemaEnvelope
- Parameterized tests
- yamltemplate package: tests that can be in a single file. Similar to e2e tests, asserts on output. (used for overlays, ytt starlark addtions, file positions via `--output`)
- TestSchema_With_fuzzed_inputs ?
- schema (recent changes & intentional choices)
- Prototypes
- Helped unexport internals of nodes and prefer mutation thru getters and setters. It behaves as an intermediate type to decouple yamlmeta and schema packages.
- `func NewDocumentSet(val *DocumentSetProto)`
- Change Node `type` to `meta` `map[string]interface{}`. Allows the schema package to set this info rather than yamlmeta
- Do all typing and checking in schema package
- previously `TypeWithValues` is now a `yamlmeta.Node` that contains `schema/type` meta
- Data Values and Schema are separate packages. Schema package contains typing and validations, and logic to connect dv and schema, and send it to libraries.
- Data values is renamed Envelope. It contains and addresses dvs
- dvpp and dvSpp are mirrors
-

AST Structures
---
What is it?
* Composite of the node
* tree structure
* A scalar isn't a Node, it's the value of a Node
* Abbreviation for "Abstract Syntax Tree"

How're they created?
* how do they go from bytes to ast?
* Scalar types are parsed using our forked yamlv2 parser
* workspace package templateloader.go -> ParseYAML()
* yamlmeta package document_set.go -> NewDocumentSetFromBytes() unstructured data becomes structured data
* parser.go -> ParseBytes() -> parseBytes()
* Decode() yields raw yaml structures (no annotations yet)
* yaml.v2 package yaml.go unmarshals the data
* yaml.v2 defines a struct that represents the structure of the type and the decoder parses based on that
* out is what we write the value to
* n is the data we're decoding
* yamlmeta package parser.go -> parser()
* output is either a map, with map items or array with array items
* we use a mapSlice instead of a map to preserve map ordering
* assignMetas() takes yaml comments and converts to metas
* comments directly to the left of a node OR comments directly after a node are added to the node as meta
* We check that a map items' value is null if it is followed by #@ value
* Since yaml.v2 does not set the position of maps, we manually set the position of a map to the position of its parent

Do they contain scalars?
* Yes, but not as a separate node.
* They are stored in the value of where they are contained, (ie. Document, MapItem, and ArrayItem)this is where they get their position as well.
* Scalars implement the TypeWithValues interface, which has a GetPosition() function.

What do they store?
* Positions
* GetValues returns the children
* metas

How are they output?
* template.replace
*

### Representation of a computed node value:

---
foo:
- { a: null } #@ 41+1

doc
 -> map
    -> mapitem("foo")
       -> array
          [0] map
              -> mapitem("bar")
                   value: null
                   meta: "@ 41+1"

skills knowledge and ability

know code coverage, modular design ...

if you saw this error message can you find where it is?
with the code in front of you, can you talk thru the layers/packages?

what problems are you interested in?

Have you had an idea of a new improvement in ytt?

End goal: to be an appendix to contributer models

john and cari together

## template.replace()
What: It is a built in starlark module