---
# System prepended metadata

title: Overlay error messages
tags: [improving-overlay-err-msgs]

---

---
tags: improving-overlay-err-msgs
---
# Overlay error messages

## Problem

When an overlay matcher's expectation fails, the error message `ytt` gives falls short of helping most users determine what to do next.


For example,

`test.yml`
```yaml=
---
clients:
- client1:
    secret: blah1
- client2:
    secret: blah2
    
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
clients:
#@overlay/match by=overlay.all,expects="1+"
- client1:
    secret: foo
``` 

results in this error message:

```
ytt: Error: Overlaying (in following order: test.yml):
  Document on line test.yml:10:
    Map item (key 'clients') on line test.yml:11:
      Array item on line test.yml:13:
        Map item (key 'client1') on line test.yml:13:
          Expected number of matched nodes to be 1, but was 0
```


## Analysis

Specifically, we've identified the following deficiencies:

1. **missing "left" context** — there is no information about which document was being matched and where the expectation failed.
2. **information overload** — the error message contains a lot of contextualizing information which can be overwhelming, at least at first.
3. **buried the leid** — the most important bit of information — which matcher failed expectations — is not the most prominent/obvious piece of information.
4. **implied implicit context** — in this particular case, there's an implied matcher ... and _it's the one that's failing!_

In addition to filling those gaps, we observed...

- **name the types** — given the overlapping nature of node types, and how crucial it can be to understand which node is involved in a failure.



## Proposals

Examples refer to files named in [Use Cases](#Use-Cases).


### "Minimalist"

_In fewest words possible, without losing clarity._


Variant A:

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  bar-nemi.yml:6 (map item) to match 1 in
  foo.yml:5 (map) but matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  bar-tmai.yml:6 (array item) to match 1 in
  foo.yml:2 (array) but matched 2 (foo.yml:3, foo.yml:5).
```



Variant B:

_French quotes around types._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  bar-nemi.yml:6 «map item» to match 1 in
  foo.yml:5 «map» but matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  bar-tmai.yml:6 «array item» to match 1 in
  foo.yml:2 «array» but matched 2 (foo.yml:3, foo.yml:5).
```


Variant C:

_In columns._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  bar-nemi.yml:6 (map item) to have matched 1 in
       foo.yml:5 (map)      but has matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  bar-tmai.yml:6 «array item» to have matched 1 in
       foo.yml:2 «array»      but has matched 2 (foo.yml:3, foo.yml:5).
```

Variant D:

_Terse "match" tenses._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  bar-nemi.yml:6 (map item) to match 1 in
       foo.yml:5 (map)       matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  bar-tmai.yml:6 «array item» to match 1 in
       foo.yml:2 «array»      matched  2 (foo.yml:3, foo.yml:5).
```


### "Failure separate from context"

```
ytt: Error: Expected overlay to match 1 map item in target, but matched 0.

Overlay: bar-nemi.yml:6 (map item)
 Target: foo.yml:5 (map)
```

```
ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

overlay: bar-nemi.yml:6 (map item)
 target: foo.yml:5 (map)
```


```
ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

Overlay bar-nemi.yml on line 6 (map item)
Inside foo.yml on line 5 (map)
```

```
ytt: Overlay error: Expected to match 1 document in target, but matched 0.

overlay: bar-nemi.yml:2 (document)
 target: <all documents>
```


### XYZ

```
ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

overlay: bar-nemi.yml on line 6 (map item)
   base: foo.yml on line 5 (map)
matcher: (implicit) "#@overlay/match by=<key equality>, expects=1"
```


```
$ ytt ....

ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

overlay: bar-nemi.yml on line 6 (map item)
   base: foo.yml on line 3 (map)
matcher: '#@overlay/match by=overlay.subset({...}), expects="1+"'

```


```
ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

matcher: '#@overlay/match by=overlay.subset({...}), expects="1+"'
overlay: bar-nemi.yml:6 (map item)
   base: foo.yml:3 (map)
```



```
ytt: Error: Overlay expected to match 1 map item in target, but matched 0.

overlay: bar-nemi.yml on line 6 (map item)
   base: foo.yml on line 5 (map)
matcher: (implicit) "#@overlay/match by=<key equality>, expects=1"
```

```
ytt: Error: Overlay expected to match 1 map item in target, but matched 0.

overlay: bar-nemi.yml on line 6 (map item, 'client1')
   base: foo.yml on line 5 (map, 'clients[1]')
matcher: (implicit) "#@overlay/match by=<key equality>, expects=1"
```

```
ytt: Error: Overlay expected to match 1 map item in target, but matched 0.

overlay: bar-nemi.yml on line 6 (map item)
   base: foo.yml on line 5 (map)
matcher: (implicit) "#@overlay/match by=<key equality>, expects=1"


---
clients:
- <expected to match 1 map item, matched 0>

```




### "Named"

_[Minimalist](#“Minimalist”) w/ names of items._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  bar-nemi.yml:6 (map item, 'client1') to match    1 item in
       foo.yml:5 (map in 'clients[1]') but matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  bar-tmai.yml:6: array item 'clients[0]' to match    1 in
       foo.yml:2: array 'clients[]'       but matched 2 (foo.yml:3, foo.yml:5).
```

Note:
- when calculating paths, walk up tree until first map item key.
  ```yaml=
  template:
    spec:
      containers:
      - name: nginx
      - - name: sidecar
  ```
  - map item on line 3 is `'containers'`
  - array on line 4 is `'containers[]'`
  - array item on line 4 is `'containers[0]'`
  - map item on line 4 is `'containers[0].name'`
  - array item on line 5 is `'containers[1]'`
  - array on line 5 is `'containers[1][]'`
  - array item on line 5 is `'containers[1][0]'`
  - map item on line 5 is `'containers[1][0].name'`


### "Complete"

_[Named](#“Named”) w/ failing matcher_

Variant A:

_Name matcher function (including implicits)._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  bar-nemi.yml:6 (map item, 'client1') to match    1 (by <key equality>) item in
       foo.yml:5 (map in 'clients[1]') but matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  bar-tmai.yml:6: array item 'clients[0]' to match    1 (by `overlay.all`) in
       foo.yml:2: array 'clients[]'       but matched 2 (foo.yml:3, foo.yml:5).
```

Note:
- implied matcher indicated by angled brackets (`<key equality>`)
- literal Starlark expressions indicated by back-tics (`overlay.subset(...)`)
- map key short-hand expanded (`#@overlay/matcher by="name"` ==> ``by `overlay.map_key("name")` ``)


Variant B:

_Include entire annotation._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):
Expected:
  "@overlay/match by=<key equality>, expects=1" (implicit) annotating
  bar-nemi.yml:6 (map item, 'client1') to match    1 item in
       foo.yml:5 (map in 'clients[1]') but matched 0.
```

```
ytt: Error: Overlaying (in order: bar-tmai.yml):
Expected:
  "#@overlay/match by=overlay.all,expects=1" annotating
  bar-tmai.yml:6: array item 'clients[0]' to match    1 in
       foo.yml:2: array 'clients[]'       but matched 2 (foo.yml:3, foo.yml:5).
```


### "Parsed"

_Include trees of both overlay and target from match failure, up._

```
ytt: Error: Overlaying (in order: bar-nemi.yml):

bar-nemi.yml (overlay):
3: «document» — `overlay.all` over <all documents>
4:   «map item» 'clients' — <key equality> over «map» @ foo.yml:2
5:     «array item» [0] — `overlay.all` over «array» @ foo.yml:3
5:       «map item» 'client1' — <key equality> over «map» @ foo.yml:5

foo.yml (target):
1: «document»
2:   «map»
2:     «map item» 'clients'
3:       «array»
5:         «array item» [1]
5:           «map»
        
Expected:
  bar-nemi.yml:6 «map item» 'client1'  to match    1 (by <key equality>) item in
       foo.yml:5 «map» in 'clients[1]' but matched 0.
```

Note:
- «array» and «map» are excluded from overlays since those kinds of nodes cannot be annotated.


### "Process Orientation"

_Illustrate which step in execution failed._

```
Progress:
✓  enumerate files
✓  prepare data values
✓  render templates
✗  overlay documents
•  render output

ytt: Error: Overlay matcher failed to meet expectations:
Expected:
  bar-nemi.yml:6 (map item, 'client1') to match    1 (by <key equality>) item in
       foo.yml:5 (map in 'clients[1]') but matched 0.
```

### "Full"

_Some form of all elements mentioned above, all together._

```
ytt: Error: 

Progress:
✓  enumerate files
✓  prepare data values
✓  render templates
✗  overlay documents
•  render output

bar-nemi.yml (overlay):
3: «document» — `overlay.all` over <all documents>
4:   «map item» 'clients' — <key equality> over «map» @ foo.yml:2
5:     «array item» [0] — `overlay.all` over «array» @ foo.yml:3
5:       «map item» 'client1' — <key equality> over «map» @ foo.yml:5

foo.yml (target):
1: «document»
2:   «map»
2:     «map item» 'clients'
3:       «array»
5:         «array item» [1]
5:           «map»


Expected:
  bar-nemi.yml:6 «map item» 'client1'  to match    1 (by <key equality>) item in
       foo.yml:5 «map» in 'clients[1]' but matched 0.
```

Variant B:

```
ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

Overlay bar-nemi.yml on line 6 (map item)
Inside foo.yml on line 5 (map)

bar-nemi.yml
 3: ---
 4: clients:
 5: #@overlay/match by=overlay.all,expects="1+"
 6: - client1:
 
foo.yml
 1: ---
 2: clients:
 5: - client2:
```

```
ytt: Overlay error: Expected to match 1 map item in target, but matched 0.

Overlay bar-nemi.yml on line 6 (map item)
Inside foo.yml on line 5 (map)

bar-nemi.yml
 5: #@overlay/match by=overlay.all,expects="1+"
 6: - client1:
 
foo.yml
 2: clients:
 ...
 5: - client2:
```

Variant C:

```
ytt: Overlay error: Expected to match 1 map item in target map, but matched 0.

(implicit) Overlay <key equality> matcher on map item 
  bar-nemi.yml:6 |  (implicit) #@overlay/match by=<key equality>, expects=0
  bar-nemi.yml:6 |  - client1:

Inside map
  foo.yml:5 | - client2:
```

```
ytt: Overlay error: Expected to match 1 map item in target map, but matched 0.

Overlay matcher on map item
  bar-nemi.yml:5 |  #@overlay/match by=overlay.all, expects=0
  bar-nemi.yml:6 |  - client1: ~

Inside map
  foo.yml:5 | - client2:
```

Variant D:

```
ytt: Overlay error: Expected to match 1 map item in target map, but matched 0.

overlay: bar-nemi.yml:6
   base: foo.yml:5
matcher: (implicit) #@overlay/match by=<key equality>, expects=0

```




## Use Cases


### Input

Input for all use-cases is the same:

`foo.yml`
```yaml=
---
clients:
- client1:
    secret: blah1
- client2:
    secret: blah2
```

### "Not Enough Documents"

`bar-ned.yml`
```ymal=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all, expects=2
---
``` 

### "Too Many Documents"

`bar-tmd.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all, expects=0
---
``` 


### "Not Enough Map Items"

`bar-nemi.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
clients:
#@overlay/match by=overlay.all
- client1: ~
```

### "Too Many Map Items"

`bar-tmmi.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
#@overlay/match by=overlay.all, expects=0
clients: ~
```


### "Not Enough Array Items"

`bar-neai.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
clients:
#@overlay/match by=overlay.all,expects=3
- client1: ~
``` 

### "Too Many Array Items"

`bar-tmai.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
clients:
#@overlay/match by=overlay.all,expects=1
- client1: ~
```