---
# System prepended metadata

title: Research Experiment Specs
tags: [research]

---

---
tags: research
---

# Research Experiment Specs
## phase zero

test the testing environment

what:
- google forms
    1.  log records
        - [direct form link](https://docs.google.com/forms/d/1vviUeRb2WxkpW6FF1FQyQQJBs8E1fDvxY48XhVCdvM0/edit)
        - [fill-out-the-form link](https://forms.gle/d4sQgSHH5qtkCNPx5)
    3.  schema cultivation: add, delete, merge category
        - [direct form link](https://docs.google.com/forms/d/10v96HsOuTc53kUOtQ_3vH2C1xCIpDTnBoSJboxX4Sm8/edit)
        - [fill-out-the-form link](https://forms.gle/xiKCRxo89FfaE33Y9)
- google docs
    - keep an up-to-date version of the taxonomy here:
        - [google docs links](https://docs.google.com/document/d/1A5e6v09aQiiFKTgx7deHw8Lb9YSySx6gGyCH6a5M1So/edit)
  

scenario:
- proposed scenario: colloborating on a lemonade stand
- the team? Govrn core contributors :) 
- any real incentives? Not directly, but the scenario is plausible enough for people to understand the context & we'll learn about Govrn's process from it.

experiment set up:
- research team creates initial conditions

ideal testing group:
- govrn core, non research team
- aaron, stefen, lara, keating, jp, etc

ideal outcome:
- flag bugs
- identify friction in the process
- capture reflection on what worked well and didn't work well
- the data itself won't be conclusive, but we want to look at the quality and quantity of data generated - aka good data coverage
- solve the cold start problem

deliverables:
- data simulation (code)
- one pager on learnings based on participant feedback (qualitative)
- cursory EDA summary stats on taxonomy/schema changes, participation rates (quantitative)
- create revised testing environment/flow
- create improved copy/text for the forms
- create very rudimentary data viz for each participant's contribution to the taxonomy & get feedback on people's interest in understanding their marginal contribution

outstanding questions:
- should we allow subcategories (aka trees?)

phase zero a

- emergent initial condition setting
    - freeform text which research team aggregates into contribution types

phase zero b

- proposing deltas
- research team aggregates deltas based on agreed upon preference aggregation mechanism

phase zero c
- add relationships between contribution types (connecting nodes)

* b and c can happen concurrently

reflections, data viz, etc

phase zero d
- add value assignment 
    - there are actually many value schemas (ranking, points, words - like "added beauty", "creating a sense of belonging")

## How the experiment works

## Overview

1. **taxonomy** 
    - emergent initial condition setting
2. **taxonomy and value** 
    - taxonomy modification through proposed deltas
    - emergent value assignment initial condition setting
3. **value:** 
    - value assignment modification through proposed deltas 
4. **recap**
### Session 1 (30 minutes)

Phase of experiment: Emergent initial condition setting

Agenda:
- Introduce the scenario & docs
- Ask the team to submit freeform contribution text
- Team is encouraged to further add emergent contributions async


### Research Team Regroup 1

- Aggregate emergent contributions into initial contribution types canonical taxonomy
    - what aggregation algorithm are we using?
    - It might be worth proposing a non-ideal canonical taxonomy to create something for the group to modulate. We can still aggregate the "ideal one" but see if the modulation process leads to this one from a non-ideal taxonomy.
- Additional prep
    - We also may want to come up with boundary pushing contribution ideas that don't nest well into the canonical taxonomy and send those to participants to encourage them to form deltas
    - We could create a visualization to show each person's marginal contribution to the taxonomy


### Session 2 (30 minutes)

Phase of experiment: Submit modification proposals and value assignment proposals to the initial canonical taxonomy

Agenda:
- Show the team the canonical taxonomy
- Request everyone to submit new contributions
- Request everyone to submit requested changes to the taxonomy 
- Request everyone to submit emergent value assignments 

Open questions
- Do we want people to do taxonomy delta proposals and value assignment proposals at the same time?
- Should the emergent value assignments be based on individual contributions or on contribution types of the taxonomy?

### Research Regroup 2 

Deliverables:
- Aggregate taxonomy deltas into new proposed canonical taxonomy
    - what algorithm shoul we use? Maybe we add this to the agenda for session 2
- Aggregate emergent value assignments into an intial condition value framework

Prep:
- Create edge case situations (i.e. suggested contributions that nest under a contribution type but should be much more or less valuable than the assigned contribution)
- Create new complications (i.e. there was a complaint from the city and now the group needs to obtain a permit) to perturb the existing value framework

Outstanding question:
- Do we want the value framework based on contribution type or do we want each person's contributions to be individually assigned value?

### Session 3 (30 min) 

Phase of experiment: Submit modification proposals to the value assignments

Agenda:
- Show everyone the aggregated changes to the canonical taxonomy
- Show everyone the aggregated initial conditions value framework
- Explain how to submit proposed changes to the value framework

### Research Regroup 3

Phase of experiment: Value distribution



### Session 4

Phase of experiment: Recap, Learnings & Feedback

What worked well?
What didn't work well?
What should we change for next time?

Deliverables:
- share an EDA on each person's 

## Feedback on Google Doc Flow

- Remaining design decisions: 
    1. Should we have people report the taxonomy in the govrn app? We could potentially create a LemonadeDAO. 
        - For phase zero, this would look like having no activity types and having people create their own categories that the research team then aggregates into drop down menu options. 
        - Implication: we'll need to manually update the categories in the drop down menu.
    2. Do we want people to be able to see each other's contributions? 
         - Thinking no in the emergent setting of initial conditions but yes afterwards?

- Additional docs we may need to create
    - Majority vote docs 
    - Doc to see everyone's contributions so far


- On the [v0: Contribution Records Link](https://docs.google.com/forms/d/1vviUeRb2WxkpW6FF1FQyQQJBs8E1fDvxY48XhVCdvM0/viewform?edit_requested=true&pli=1)
    - [ ] The form looks really pretty! I'm not able to fill out though? Also wasn't able to click on 'request edit access'?
    - [ ] For the emergent phase (before the canonical contribution taxonomy exists) we'll need to delete the category field at the bottom. OR we can ask people to propose a category name or tags.
    - [ ] Let's add the lemonade stand description on this form for reference.
    - [ ] Maybe we should add a description of what phase the experiment is in on the form as well. E.g. "This is the emergent phase where we crowdsource free text contributions into a proposed initial canonical taxonomy."
    - [ ] I can get you a list of emails to grant access to the Contribution Taxonomy form. Should it be read only so as to funnel proposed edits through the proper form?

- The "Direct Form Link" and "Fill out the Form Link" for logging records look the same to me?

- On the [v0: Cultivation Link](https://docs.google.com/forms/d/10v96HsOuTc53kUOtQ_3vH2C1xCIpDTnBoSJboxX4Sm8/viewform?edit_requested=true)
    - [ ] Love the context blurb at the beginning
    - [ ] Let's add a link to the the Canonical Taxonomy for easy reference?
    - [ ] 














