# LM Bias Project

## Pilot 2
Collected 100 examples. Results are [here](https://docs.google.com/spreadsheets/d/1s_h0-Epxg8gZe5lAa0ulLJ-Jc9qFBx0PTUcx02-yedI/edit?usp=sharing).
Some takeaways,
- 100% of examples were minimal pairs. We seem to have communicated this well
- 94% do express a stereotype. 96% have the correctly labeled stereotype
- This gives us 90 acceptable examples out of 100.
- There were some examples (at least 2) that are minimal pairs and express some stereotype but are wrongly labeled. These examples could be kept in the dataset if they are relabeled correctly during validation.

## New setup
The two options on the table right now,
- Label as stereotype or anti-stereotype
- - Stereotype first

### Labeling (anti)stereotype,
Labeling pros,
- We know our current setup works pretty well!
- People seem to grasp the idea of sterotypes and anti-stereotypes so labeling these might not be noisy
- Explicitly asking people to think about anti-stereotypes encourages examples that have them

Labeling cons,
- Asking workers to do one more thing
- We will want to validate the direction of bias

### Stereotype first
Stereotype pros,
- Layout will be simpler, we don't have to ask the workers to do yet another thing
- We will implictly get the direction of bias we're looking for

Stereotype cons,
- Pretty big overhaul of instructions which will need to be piloted.
- May not as strongly encourage writing anti-stereotype examples.