Overlay error messages

--- tags: improving-overlay-err-msgs --- # Overlay error messages ## Problem When an overlay matcher's expectation fails, the error message `ytt` gives falls short of helping most users determine what to do next. For example, `test.yml` ```yaml= --- clients: - client1: secret: blah1 - client2: secret: blah2 #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all --- clients: #@overlay/match by=overlay.all,expects="1+" - client1: secret: foo ``` results in this error message: ``` ytt: Error: Overlaying (in following order: test.yml): Document on line test.yml:10: Map item (key 'clients') on line test.yml:11: Array item on line test.yml:13: Map item (key 'client1') on line test.yml:13: Expected number of matched nodes to be 1, but was 0 ``` ## Analysis Specifically, we've identified the following deficiencies: 1. **missing "left" context** — there is no information about which document was being matched and where the expectation failed. 2. **information overload** — the error message contains a lot of contextualizing information which can be overwhelming, at least at first. 3. **buried the leid** — the most important bit of information — which matcher failed expectations — is not the most prominent/obvious piece of information. 4. **implied implicit context** — in this particular case, there's an implied matcher ... and _it's the one that's failing!_ In addition to filling those gaps, we observed... - **name the types** — given the overlapping nature of node types, and how crucial it can be to understand which node is involved in a failure. ## Proposals Examples refer to files named in [Use Cases](#Use-Cases). ### "Minimalist" _In fewest words possible, without losing clarity._ Variant A: ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: bar-nemi.yml:6 (map item) to match 1 in foo.yml:5 (map) but matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: bar-tmai.yml:6 (array item) to match 1 in foo.yml:2 (array) but matched 2 (foo.yml:3, foo.yml:5). ``` Variant B: _French quotes around types._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: bar-nemi.yml:6 «map item» to match 1 in foo.yml:5 «map» but matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: bar-tmai.yml:6 «array item» to match 1 in foo.yml:2 «array» but matched 2 (foo.yml:3, foo.yml:5). ``` Variant C: _In columns._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: bar-nemi.yml:6 (map item) to have matched 1 in foo.yml:5 (map) but has matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: bar-tmai.yml:6 «array item» to have matched 1 in foo.yml:2 «array» but has matched 2 (foo.yml:3, foo.yml:5). ``` Variant D: _Terse "match" tenses._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: bar-nemi.yml:6 (map item) to match 1 in foo.yml:5 (map) matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: bar-tmai.yml:6 «array item» to match 1 in foo.yml:2 «array» matched 2 (foo.yml:3, foo.yml:5). ``` ### "Failure separate from context" ``` ytt: Error: Expected overlay to match 1 map item in target, but matched 0. Overlay: bar-nemi.yml:6 (map item) Target: foo.yml:5 (map) ``` ``` ytt: Overlay error: Expected to match 1 map item in target, but matched 0. overlay: bar-nemi.yml:6 (map item) target: foo.yml:5 (map) ``` ``` ytt: Overlay error: Expected to match 1 map item in target, but matched 0. Overlay bar-nemi.yml on line 6 (map item) Inside foo.yml on line 5 (map) ``` ``` ytt: Overlay error: Expected to match 1 document in target, but matched 0. overlay: bar-nemi.yml:2 (document) target: <all documents> ``` ### XYZ ``` ytt: Overlay error: Expected to match 1 map item in target, but matched 0. overlay: bar-nemi.yml on line 6 (map item) base: foo.yml on line 5 (map) matcher: (implicit) "#@overlay/match by=<key equality>, expects=1" ``` ``` $ ytt .... ytt: Overlay error: Expected to match 1 map item in target, but matched 0. overlay: bar-nemi.yml on line 6 (map item) base: foo.yml on line 3 (map) matcher: '#@overlay/match by=overlay.subset({...}), expects="1+"' ``` ``` ytt: Overlay error: Expected to match 1 map item in target, but matched 0. matcher: '#@overlay/match by=overlay.subset({...}), expects="1+"' overlay: bar-nemi.yml:6 (map item) base: foo.yml:3 (map) ``` ``` ytt: Error: Overlay expected to match 1 map item in target, but matched 0. overlay: bar-nemi.yml on line 6 (map item) base: foo.yml on line 5 (map) matcher: (implicit) "#@overlay/match by=<key equality>, expects=1" ``` ``` ytt: Error: Overlay expected to match 1 map item in target, but matched 0. overlay: bar-nemi.yml on line 6 (map item, 'client1') base: foo.yml on line 5 (map, 'clients[1]') matcher: (implicit) "#@overlay/match by=<key equality>, expects=1" ``` ``` ytt: Error: Overlay expected to match 1 map item in target, but matched 0. overlay: bar-nemi.yml on line 6 (map item) base: foo.yml on line 5 (map) matcher: (implicit) "#@overlay/match by=<key equality>, expects=1" --- clients: - <expected to match 1 map item, matched 0> ``` ### "Named" _[Minimalist](#“Minimalist”) w/ names of items._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: bar-nemi.yml:6 (map item, 'client1') to match 1 item in foo.yml:5 (map in 'clients[1]') but matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: bar-tmai.yml:6: array item 'clients[0]' to match 1 in foo.yml:2: array 'clients[]' but matched 2 (foo.yml:3, foo.yml:5). ``` Note: - when calculating paths, walk up tree until first map item key. ```yaml= template: spec: containers: - name: nginx - - name: sidecar ``` - map item on line 3 is `'containers'` - array on line 4 is `'containers[]'` - array item on line 4 is `'containers[0]'` - map item on line 4 is `'containers[0].name'` - array item on line 5 is `'containers[1]'` - array on line 5 is `'containers[1][]'` - array item on line 5 is `'containers[1][0]'` - map item on line 5 is `'containers[1][0].name'` ### "Complete" _[Named](#“Named”) w/ failing matcher_ Variant A: _Name matcher function (including implicits)._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: bar-nemi.yml:6 (map item, 'client1') to match 1 (by <key equality>) item in foo.yml:5 (map in 'clients[1]') but matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: bar-tmai.yml:6: array item 'clients[0]' to match 1 (by `overlay.all`) in foo.yml:2: array 'clients[]' but matched 2 (foo.yml:3, foo.yml:5). ``` Note: - implied matcher indicated by angled brackets (`<key equality>`) - literal Starlark expressions indicated by back-tics (`overlay.subset(...)`) - map key short-hand expanded (`#@overlay/matcher by="name"` ==> ``by `overlay.map_key("name")` ``) Variant B: _Include entire annotation._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): Expected: "@overlay/match by=<key equality>, expects=1" (implicit) annotating bar-nemi.yml:6 (map item, 'client1') to match 1 item in foo.yml:5 (map in 'clients[1]') but matched 0. ``` ``` ytt: Error: Overlaying (in order: bar-tmai.yml): Expected: "#@overlay/match by=overlay.all,expects=1" annotating bar-tmai.yml:6: array item 'clients[0]' to match 1 in foo.yml:2: array 'clients[]' but matched 2 (foo.yml:3, foo.yml:5). ``` ### "Parsed" _Include trees of both overlay and target from match failure, up._ ``` ytt: Error: Overlaying (in order: bar-nemi.yml): bar-nemi.yml (overlay): 3: «document» — `overlay.all` over <all documents> 4: «map item» 'clients' — <key equality> over «map» @ foo.yml:2 5: «array item» [0] — `overlay.all` over «array» @ foo.yml:3 5: «map item» 'client1' — <key equality> over «map» @ foo.yml:5 foo.yml (target): 1: «document» 2: «map» 2: «map item» 'clients' 3: «array» 5: «array item» [1] 5: «map» Expected: bar-nemi.yml:6 «map item» 'client1' to match 1 (by <key equality>) item in foo.yml:5 «map» in 'clients[1]' but matched 0. ``` Note: - «array» and «map» are excluded from overlays since those kinds of nodes cannot be annotated. ### "Process Orientation" _Illustrate which step in execution failed._ ``` Progress: ✓ enumerate files ✓ prepare data values ✓ render templates ✗ overlay documents • render output ytt: Error: Overlay matcher failed to meet expectations: Expected: bar-nemi.yml:6 (map item, 'client1') to match 1 (by <key equality>) item in foo.yml:5 (map in 'clients[1]') but matched 0. ``` ### "Full" _Some form of all elements mentioned above, all together._ ``` ytt: Error: Progress: ✓ enumerate files ✓ prepare data values ✓ render templates ✗ overlay documents • render output bar-nemi.yml (overlay): 3: «document» — `overlay.all` over <all documents> 4: «map item» 'clients' — <key equality> over «map» @ foo.yml:2 5: «array item» [0] — `overlay.all` over «array» @ foo.yml:3 5: «map item» 'client1' — <key equality> over «map» @ foo.yml:5 foo.yml (target): 1: «document» 2: «map» 2: «map item» 'clients' 3: «array» 5: «array item» [1] 5: «map» Expected: bar-nemi.yml:6 «map item» 'client1' to match 1 (by <key equality>) item in foo.yml:5 «map» in 'clients[1]' but matched 0. ``` Variant B: ``` ytt: Overlay error: Expected to match 1 map item in target, but matched 0. Overlay bar-nemi.yml on line 6 (map item) Inside foo.yml on line 5 (map) bar-nemi.yml 3: --- 4: clients: 5: #@overlay/match by=overlay.all,expects="1+" 6: - client1: foo.yml 1: --- 2: clients: 5: - client2: ``` ``` ytt: Overlay error: Expected to match 1 map item in target, but matched 0. Overlay bar-nemi.yml on line 6 (map item) Inside foo.yml on line 5 (map) bar-nemi.yml 5: #@overlay/match by=overlay.all,expects="1+" 6: - client1: foo.yml 2: clients: ... 5: - client2: ``` Variant C: ``` ytt: Overlay error: Expected to match 1 map item in target map, but matched 0. (implicit) Overlay <key equality> matcher on map item bar-nemi.yml:6 | (implicit) #@overlay/match by=<key equality>, expects=0 bar-nemi.yml:6 | - client1: Inside map foo.yml:5 | - client2: ``` ``` ytt: Overlay error: Expected to match 1 map item in target map, but matched 0. Overlay matcher on map item bar-nemi.yml:5 | #@overlay/match by=overlay.all, expects=0 bar-nemi.yml:6 | - client1: ~ Inside map foo.yml:5 | - client2: ``` Variant D: ``` ytt: Overlay error: Expected to match 1 map item in target map, but matched 0. overlay: bar-nemi.yml:6 base: foo.yml:5 matcher: (implicit) #@overlay/match by=<key equality>, expects=0 ``` ## Use Cases ### Input Input for all use-cases is the same: `foo.yml` ```yaml= --- clients: - client1: secret: blah1 - client2: secret: blah2 ``` ### "Not Enough Documents" `bar-ned.yml` ```ymal= #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all, expects=2 --- ``` ### "Too Many Documents" `bar-tmd.yml` ```yaml= #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all, expects=0 --- ``` ### "Not Enough Map Items" `bar-nemi.yml` ```yaml= #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all --- clients: #@overlay/match by=overlay.all - client1: ~ ``` ### "Too Many Map Items" `bar-tmmi.yml` ```yaml= #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all --- #@overlay/match by=overlay.all, expects=0 clients: ~ ``` ### "Not Enough Array Items" `bar-neai.yml` ```yaml= #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all --- clients: #@overlay/match by=overlay.all,expects=3 - client1: ~ ``` ### "Too Many Array Items" `bar-tmai.yml` ```yaml= #@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.all --- clients: #@overlay/match by=overlay.all,expects=1 - client1: ~ ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.