---
tags: improving-overlay-err-msgs
---
# Overlay Error Message User Research
## Hypothesis
If we present an overlay error message in format ([Solution 1](#Solution-1) | [Solution 2](#Solution-2)),
A user with at least basic exposure to Overlays:
1. can more quickly understand what and where the problem is
2. is more likely make a change to the overlay that solves the problem
## Candidates
### Possible Sources
- CAPI
- Tim Downey; 12/1, 2:15-3:00 PM PST
- (Nat Bennett, not working with ytt anymore) ([Eli mentioned in Slack](https://vmware.slack.com/archives/C010XR15VHU/p1599859665020300))
- TAS Team
- CF for K8s team
- OSS Community Members
- (search #carvel for anyone saying, "I'm new")
- Connor Braa @braa; 12/2, 2:00-2:45 PM PST
- Tim Rupp @trupp; 12/2, 3:30-4:15 PM PST
- Jorge Morales @jorgemoralespou; 12/1, 9:45-10:30 AM PST
- Our own team members
- Dennis Leon
- From the #carvel channel
- Scott Rosenberg (member id: UGWTERGQJ) 10 hrs ahead
- Olivier (member id: U01CNSLNAGG) 10 hrs ahead
- Zebradil (German Lashevich, member id: U01AFLB3AA1) 10 hrs ahead
### Slack DM Templates
Slack message to send to people who have expressed interest in helping us:
Hi ____! Garrett here from the Carvel team. Thanks for expressing interest in providing feedback for our new ytt overlay error messages.
I'm looking to conduct interviews during a 30-45 minute zoom call. Is there a spot in your calendar sometime after November 30th that we could schedule between 9:00AM and 5:00PM PT?
In this session, we will look at some ytt overlay scenarios and their corresponding candidate error messages, and talk through them. Your feedback will shape the upcoming work we do on ytt error messages.
Some followup questions for those who say YES:
- How would you characterize your proficiency with yaml templating?
- Have you used ytt before? Specifically overlays?
Sketch: what is in it for them?
- taking out some of the pains and making it more enjoyable
- improve general adoption (larger community and better support)
- having a hand in the tools that you use, first class input in how the tools that you use, works. (because your opinion does matter
### Categorizing Facets
_(with questions that might produce data that we can assess; likely part of a pre-test survey)_
- proficiency with YAML templating
- what kinds of YAML templating/patching have you done to date?
- proficiency with `ytt`
- what are the more sophisticated things have you done with `ytt` in particular, so far?
- proficiency with domain (here, Kubernetes)
- what sorts of things have you experienced with Kubernetes?
## Candidate Solutions
### Solution 0: Current
For reference, a current typical overlay error message looks like this:
```
ytt: Error: Overlaying (in following order: overlay.yml):
Document on line overlay.yml:3:
Map item (key 'clients') on line overlay.yml:4:
Array item on line overlay.yml:6:
Map item (key 'client1') on line overlay.yml:6:
Expected number of matched nodes to be 1, but was 0
```
---
### Solution 1
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:6 (map item, 'client1') to match 1 item in
config.yml:5 (map in 'clients[1]') but matched 0.
```
#### Hypothesis
The solution is ...
- more concise (eliminating all but the two lines involved in the mismatch),
- eliminates jargon (no mention of `node`, but user-friendly concrete types), and
- provides context in the target file,
... which will improve the ability of a novice user to solve the problem on their own in a timely manner.
#### Assumptions
- this solution is less confusing. Displays the important information clearly and concisely.
- we are not introducing any 'jargon'. 'map item' and 'map' in the error message are understood by the user.
- by providing the map in the template file, users are able to fix unintended matches quicker or matching on the incorrect kind. (this additional info in the error message won't be distracting. it will hopefully provide insight into what the overlay process is matching against)
- fewer words and lines hopefully result in reduced time to visually parse the error message.
---
### Solution 2
```
ytt: Error: Overlaying (in following order: overlay.yml):
Expected to match 1 map item (client1) in config.yml, but matched 0.
overlay.yml:6
3: ---
4: clients:
5: #@overlay/match by=overlay.all,expects="1+"
6: - client1:
config.yml:5
1: ---
2: clients:
5: - client2:
```
#### Hypothesis
The solution ...
- eliminates jargon, and
- provides visual context in overlay and target files.
... which will improve the ability of a beginner or novice user to solve the problem on their own in a timely manner.
#### Assumptions
- this solution is less confusing. Displays the important information clearly and concisely.
- we are not introducing any 'jargon'. 'map item' and 'map' in the error message are understood by the user.
- the yaml context displays enough information for the user to 'fix' the ytt error. The line which is causing the error is displayed.
- in the event there are 'lots' of client elements in the array, or there are 'lots' of target files, there exists a way to present a subset of yaml context without losing the ability for a user to debug / fix the ytt error.
---
## Scenarios
### Scenario A:
**Intention:** Change the secret in client1 to "foo"
`config.yml`
```yaml=
---
clients:
- client1:
secret: blah1
- client2:
secret: blah2
```
`overlay.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
clients:
#@overlay/match by=overlay.all,expects="1+"
- client1:
secret: foo
```
```shell
$ ytt -f config.yml -f overlay.yml
```
#### Solution 0.0
```
ytt: Error: Overlaying (in following order: overlay.yml):
Document on line overlay.yml:3:
Map item (key 'clients') on line overlay.yml:4:
Array item on line overlay.yml:6:
Map item (key 'client1') on line overlay.yml:6:
Expected number of matched nodes to be 1, but was 0
```
#### Solution 1.0
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:6 (map item, 'client1') to match 1 item in
config.yml:5 (map in 'clients[1]') but matched 0.
```
#### Solution 2.0
```
ytt: ytt: Error: Overlaying (in following order: overlay.yml):
Expected to match 1 map item (client1) in config.yml, but matched 0.
overlay.yml:6
3: ---
4: clients:
5: #@overlay/match by=overlay.all,expects="1+"
6: - client1:
config.yml:5
1: ---
2: clients:
5: - client2:
```
#### Solution 0.1 (for reference)
```
Overlaying (in following order: overlay.yml):
Document on line overlay.yml:3:
Map item (key 'clients') on line overlay.yml:4:
Array item on line overlay.yml:6:
Expected number of matched nodes to be {0,1}, but was 2 (lines: config.yml:3, config.yml:5)
```
#### Solution 0.2 (for reference)
```
Overlaying (in following order: overlay.yml):
Document on line overlay.yml:3:
Map item (key 'clients') on line overlay.yml:4:
Array item on line overlay.yml:6:
overlay.map_key_matcher:
Expected to find mapitem with key 'client1', but did not
```
#### Plays
1. Using Vim replace syntax
1. overlay.yml:5 — (s /expects="1+"// | s/expects="1+"/expects=1/ ) ==> X.1
1. overlay.yml:5 — s/expects=".*"/expects=0// ==> X.1
1. overlay.yml:5 — s/by=overlay.all/by="client1"/ ==> X.2
1. overlay.yml:5 — s/by=overlay.all/overlay.subset({"client1": {}})/ ==> Correct
#### Solution 1.1
_(if expects=0)_
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:6 (array item, 'clients[0]') to match 0 items in
config.yml:2 (array in 'clients') but matched 2 (config.yml:3, config.yml:5)
```
_(if expects=1)_
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:6 (array item, 'clients[0]') to match 1 items in
config.yml:2 (array in 'clients') but matched 2 (config.yml:3, config.yml:5)
```
#### Solution 1.2
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:6 (array item, 'clients[0]') to find map item with key 'client1' in
config.yml:5 (map, 'clients[1]') but did not
```
#### Solution 2.1
_(if expects=0)_
```
ytt: Error: Overlaying (in following order: overlay.yml):
Expected to match 0 array item (clients[0]) in config.yml, but matched 2.
overlay.yml:6
3: ---
4: clients:
5: #@overlay/match by=overlay.all,expects=0
6: - client1:
config.yml:3,5
1: ---
2: clients:
3: - client1:
5: - client2:
```
_(if expects=1)_
```
ytt: ytt: Error: Overlaying (in following order: overlay.yml):
Expected to match 1 array item (clients[0]) in config.yml, but matched 2.
overlay.yml:6
3: ---
4: clients:
5: #@overlay/match by=overlay.all,expects=1
6: - client1:
config.yml:3,5
1: ---
2: clients:
3: - client1:
5: - client2:
```
#### Solution 2.2
```
ytt: ytt: Error: Overlaying (in following order: overlay.yml):
Expected to find map item (client1) in config.yml, but did not.
overlay.yml:6
3: ---
4: clients:
5: #@overlay/match by="client1"
6: - client1:
config.yml:5
1: ---
2: clients:
5: - client2:
```
---
### Scenario B:
**Intention:** Change the image of the container with the `name: my-app` to 'some-other-image'.
`config.yml`
```yaml=
---
spec:
app:
containers:
- name: my-app
image: some-image
resources:
- path.sh
- name: foo
image: bar-image
```
`overlay.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
spec:
app:
containers:
#@overlay/match by="name"
name: my-app
image: some-other-image
```
```shell
$ ytt -f config.yml -f overlay.yml
```
#### Solution 0.0
```
ytt: Error: Overlaying (in following order: overlay.yml):
Document on line overlay.yml:4:
Map item (key 'spec') on line overlay.yml:5:
Map item (key 'app') on line overlay.yml:6:
Map item (key 'containers') on line overlay.yml:7:
Expected map, but was *yamlmeta.Array
```
#### Solution 1.0
```
ytt: Error: Overlaying (in order: overlay.yml):
Type mismatch:
overlay.yml:7 (map item, 'containers') expects a map
config.yml:4 (map item, 'containers') has an array.
```
#### Solution 2.0
```
ytt: Error: Overlaying (in following order: overlay.yml):
Type mismatch: map item (containers) expected to contain a map in config.yml, but found an array
overlay.yml:7
4: ---
5: spec:
6: app:
7: containers:
8: #@overlay/match by="name"
9: name: my-app
config.yml:4
1: ---
2: spec:
3: app:
4: containers:
5: - name: my-app
```
#### Solution 0.1 (for reference)
```
Overlaying (in following order: overlay.yml):
Document on line overlay.yml:4:
Map item (key 'spec') on line overlay.yml:5:
Map item (key 'app') on line overlay.yml:6:
Map item (key 'containers') on line overlay.yml:7:
Array item on line overlay.yml:9:
Expected number of matched nodes to be 1, but was 2 (lines: config.yml:5, config.yml:9)
```
#### Plays
1. Using Vim replace syntax
1. overlay.yml:8 — s/by="name"/by=overlay.all/ ==> X.0
1. overlay.yml:8 — s/by="name"/by=overlay.all/ &&
overlay.yml:9 - s/name: my-app/- name: my-app/ ==> X.1
3. overlay.yml:9 — s/name: my-app/- name: my-app/ ==> Correct
#### Solution 1.1
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:9 (array item, 'containers[0]') to match 1 items in
config.yml:4 (map item, 'containers') but matched 2 (config.yml:5, config.yml:9).
```
#### Solution 2.1
```
ytt: Error: Overlaying (in following order: overlay.yml):
Expected to match 1 array item (conatiners[0]) in config.yml, but matched 2.
overlay.yml:9
4: ---
5: spec:
6: app:
7: containers:
8: #@overlay/match by=overlay.all
9: - name: my-app
config.yml:5,9
1: ---
2: spec:
3: app:
4: containers:
5: - name: my-app
9: - name:
```
---
### Scenario C:
**Intention:** Change lastName of all clients, add one if not there.
`config.yml`
```yaml=
---
clients:
- client:
firstName: foo
lastName:
- client:
firstName: nar
```
`overlay.yml`
```yaml=
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.all
---
clients:
#@overlay/match by=overlay.all, expects="1+"
- client:
lastName: bar
```
```shell
$ ytt -f config.yml -f overlay.yml
```
#### Solution 0.0
```
ytt: Error: Overlaying (in following order: overlay.yml):
Document on line overlay.yml:4:
Map item (key 'clients') on line overlay.yml:5:
Array item on line overlay.yml:7:
Map item (key 'client') on line overlay:7:
Map item (key 'lastName') on line overlay.yml:8:
Expected number of matched nodes to be 1, but was 0
```
#### Solution 1.0
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:8 (map item, 'lastName') to match 1 item in
config.yml:6 (map in 'clients[1]') but matched 0.
```
#### Solution 2.0
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected to match 1 map item (lastName) in config.yml, but matched 0.
overlay.yml:8
4: ---
5: clients:
6: #@overlay/match by=overlay.all,expects="1+"
7: - client:
8: lastName: bar
config.yml:6
1: ---
2: clients:
6: - client:
7: firstName: nar
```
#### Solution 0.1 (for reference)
```
Overlaying (in following order: overlay.yml):
Document on line overlay.yml:4:
Map item (key 'clients') on line overlay.yml:5:
Array item on line overlay.yml:7:
Expected number of matched nodes to be >= 3, but was 2 (lines: config.yml:3, config.yml:6)
```
#### Solution 0.2 (for reference)
```
Overlaying (in following order: overlay.yml):
Document on line overlay.yml:4:
Map item (key 'clients') on line overlay.yml:5:
Array item on line overlay.yml:7:
Map item (key 'client') on line overlay.yml:7:
Map item (key 'lastName') on line overlay.yml:9:
Expected number of matched nodes to be 1, but was 2 (lines: config.yml:4, config.yml:5)
```
#### Plays
1. Using Vim replace syntax
1. overlay.yml:6 — (s /expects=".\*"/expects="\*<3"/) ==> X.0
1. overlay.yml:8 — (s /expects=".\*"/expects="\*>2"/) ==> X.1
1. overlay.yml:5 — s/by=overlay.all/by=overlay.subset({"client": {}})/ ==> X.0
2. overlay.yml:8 — insert #@overlay/match by=overlay.all ==> x.2
3. overlay.yml:8 — insert "#@overlay/match missing_ok=True" ==> Correct
#### Solution 1.1
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:7 (array item, 'clients[0]') to match 3 item in
config.yml:2 (map in 'clients') but matched 2 (config.yml:3, config.yml:6).
```
#### Solution 1.2
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected:
overlay.yml:9 (map item, 'lastName') to match 1 item in
config.yml:3 (map item 'client') but matched 2 (config.yml:4, config.yml:5).
```
#### Solution 2.1
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected to match 1 map item (lastname) in config.yml, but matched 2.
overlay.yml:9
4: ---
5: clients:
7: - client:
8: #@overlay/match by=overlay.all
9: lastName: bar
config.yml:4,5
1: ---
2: clients:
3: - client:
4: firstName: foo
5: lastName:
```
#### Solution 2.2
```
ytt: Error: Overlaying (in order: overlay.yml):
Expected to match 3 array item (clients[0]) in config.yml, but matched 2.
overlay.yml:7
4: ---
5: clients:
6: #@overlay/match by=overlay.all,expects="1+"
7: - client:
config.yml:3,6
1: ---
2: clients:
3: - client:
6: - client:
```
## Metrics
- how close they got to a solution
- duration to solution
- number of steps/missteps
## Plan
This document should provide:
- For each scenario, what each solution would look like.
- A script for how to set the stage for each scenario.
- A small set of possible outputs/responses, depending on what the participant does.
For each candidate, our goal is to collect information about how the candidate attempts to solve the overlay error. This information will allow us to confirm or deny our hypothesis about a given error message. Both qualitative data, such as what candidate say when 'thinking out loud', and quantitative data, like number of steps, should be captured.
Both error message solutions outlined in this document are meant to be tested on candidates of all experience levels.
This experiment will be held over a zoom call, likely with host of the experiment sharing their screen. The host will show the template file, the overlay file, and the current error message on their screen. When a user makes a change and wants to see the new error message, the host will provide the updated error message. In the case that the user makes a move that was not anticipated, the host could get the current error message using the ytt playground. We don't want show the user this error message, so the host would convert that error message into the proper proposed solution for a new error message.
Depending on how long the interview will last (suggested between 30-45 mins), we want to get through as many scenarios as we can, however we do not want to rush the user. Think about the user's experience level and try to make sure that users at different levels are seeing the same scenarios. That way, we didn't only show scenario 1 to beginners and only show scenario 3 to advanced.
## Script
We estimate that from start of call to end of call, it will take about 45 minutes.
1. **Setup + Intro + Summary** (~5 mins)
1. Ask them if they are okay with us recording the session;
on agreement begin recording.
2. Thank them for their help
3. Tell them that the best answers are what they genuinely think and feel.
The worst answers are what they believe we want them to say.
2. **Preview the steps** (~5 mins)
1. We are going to present you with some sample inputs into a `ytt` use.
2. As we start, the initial result will be an error message.
3. We'd like you to solve the problem (thinking out loud as you go).
4. There are no trick questions: just scenarios inspired by real-life situations.
5. They may not solve the problem, and that's okay.
- We are **not** looking for the correct solution.
- We **are** looking for how _they_ are experiencing troubleshooting.
7. They can take as much time as they like, so long as they think out loud.
4. **Run Scenario A** (<=10 mins)
1. Here are the inputs (show `config.yml` and `overlay.yml`). The goal here is (read intent).
2. Please think aloud as you read these files.
- if they seek your validation as to whether they "read it right",
let them know it's more valuable if you didn't participate.
3. When we run this command, here's the output (show the selected solution)
4. Are there any words in the error message that do not make sense.
5. Please think out loud.
6. (silence is golden) wait for them to settle on making a change.
- poker face: resist the urge to hint (verbally or non-verbally)
- if they attempt to engage you in troubleshooting, just smile.
- if you need to remind them that it's more valuable if you didn't participate.
7. When they make a change, find that step in the Scenario "Plays"
5. **Run Scenario B** (<=10 mins)
6. **Run Scenario C** (<=10 mins)
7. **Close Out** (~5 mins)
1. Thank them again for their participation: focus, thinking out loud, willingness.
2. Ask them to reflect on this session: what seemed to work for them? why?
3. Is there anything else that they want to share?
4. Final thanks.
## Results
[Document containing notes and video recordings of interview.](https://docs.google.com/document/d/11MkGFgFoRd7dK3bwYEvERLMQRQyr_4CzVCWwKxooERQ/edit?usp=sharing)
Interviewed four users, all who were familiar with ytt, and had seen overlays before:
- Connor Braa
- Tim Downey
- Jorge Morales
- Tim Rupp
:::warning
### Hypothesis
If we present an overlay error message in format ([Solution 1](#Solution-1) | [Solution 2](#Solution-2)),
A user with at least basic exposure to Overlays:
1. can more quickly understand what and where the problem is
2. is more likely make a change to the overlay that solves the problem
:::
Recalling our hypothesis, the data suggests that we should accept (1) and reject (2).
Based on the notes and discussion, we have synthesized that our work has made improvements with pointing where the problem is in an overlay, however the users need more help with moving forward.
For some users, the fundamentals of how the Overlay mechanism works escapes them. It was unclear what steps `ytt` takes to process an overlay. This prevented them from being able to navigate the overlay, itself, forget about the error message.
Based on the notes and feedback collected during the interviews, we decided to move forward with implementing and iterating on these error messages. In order to collect more results for iteration, we must first implement the changes to `ytt`, then conduct user testing to ensure our changes provided value, lastly we propose new changes to the error messages and start the next iteration.
Sign in with Wallet
Connect another wallet