Research Experiment Specs

--- tags: research --- # Research Experiment Specs ## phase zero test the testing environment what: - google forms 1. log records - [direct form link](https://docs.google.com/forms/d/1vviUeRb2WxkpW6FF1FQyQQJBs8E1fDvxY48XhVCdvM0/edit) - [fill-out-the-form link](https://forms.gle/d4sQgSHH5qtkCNPx5) 3. schema cultivation: add, delete, merge category - [direct form link](https://docs.google.com/forms/d/10v96HsOuTc53kUOtQ_3vH2C1xCIpDTnBoSJboxX4Sm8/edit) - [fill-out-the-form link](https://forms.gle/xiKCRxo89FfaE33Y9) - google docs - keep an up-to-date version of the taxonomy here: - [google docs links](https://docs.google.com/document/d/1A5e6v09aQiiFKTgx7deHw8Lb9YSySx6gGyCH6a5M1So/edit) scenario: - proposed scenario: colloborating on a lemonade stand - the team? Govrn core contributors :) - any real incentives? Not directly, but the scenario is plausible enough for people to understand the context & we'll learn about Govrn's process from it. experiment set up: - research team creates initial conditions ideal testing group: - govrn core, non research team - aaron, stefen, lara, keating, jp, etc ideal outcome: - flag bugs - identify friction in the process - capture reflection on what worked well and didn't work well - the data itself won't be conclusive, but we want to look at the quality and quantity of data generated - aka good data coverage - solve the cold start problem deliverables: - data simulation (code) - one pager on learnings based on participant feedback (qualitative) - cursory EDA summary stats on taxonomy/schema changes, participation rates (quantitative) - create revised testing environment/flow - create improved copy/text for the forms - create very rudimentary data viz for each participant's contribution to the taxonomy & get feedback on people's interest in understanding their marginal contribution outstanding questions: - should we allow subcategories (aka trees?) phase zero a - emergent initial condition setting - freeform text which research team aggregates into contribution types phase zero b - proposing deltas - research team aggregates deltas based on agreed upon preference aggregation mechanism phase zero c - add relationships between contribution types (connecting nodes) * b and c can happen concurrently reflections, data viz, etc phase zero d - add value assignment - there are actually many value schemas (ranking, points, words - like "added beauty", "creating a sense of belonging") ## How the experiment works ## Overview 1. **taxonomy** - emergent initial condition setting 2. **taxonomy and value** - taxonomy modification through proposed deltas - emergent value assignment initial condition setting 3. **value:** - value assignment modification through proposed deltas 4. **recap** ### Session 1 (30 minutes) Phase of experiment: Emergent initial condition setting Agenda: - Introduce the scenario & docs - Ask the team to submit freeform contribution text - Team is encouraged to further add emergent contributions async ### Research Team Regroup 1 - Aggregate emergent contributions into initial contribution types canonical taxonomy - what aggregation algorithm are we using? - It might be worth proposing a non-ideal canonical taxonomy to create something for the group to modulate. We can still aggregate the "ideal one" but see if the modulation process leads to this one from a non-ideal taxonomy. - Additional prep - We also may want to come up with boundary pushing contribution ideas that don't nest well into the canonical taxonomy and send those to participants to encourage them to form deltas - We could create a visualization to show each person's marginal contribution to the taxonomy ### Session 2 (30 minutes) Phase of experiment: Submit modification proposals and value assignment proposals to the initial canonical taxonomy Agenda: - Show the team the canonical taxonomy - Request everyone to submit new contributions - Request everyone to submit requested changes to the taxonomy - Request everyone to submit emergent value assignments Open questions - Do we want people to do taxonomy delta proposals and value assignment proposals at the same time? - Should the emergent value assignments be based on individual contributions or on contribution types of the taxonomy? ### Research Regroup 2 Deliverables: - Aggregate taxonomy deltas into new proposed canonical taxonomy - what algorithm shoul we use? Maybe we add this to the agenda for session 2 - Aggregate emergent value assignments into an intial condition value framework Prep: - Create edge case situations (i.e. suggested contributions that nest under a contribution type but should be much more or less valuable than the assigned contribution) - Create new complications (i.e. there was a complaint from the city and now the group needs to obtain a permit) to perturb the existing value framework Outstanding question: - Do we want the value framework based on contribution type or do we want each person's contributions to be individually assigned value? ### Session 3 (30 min) Phase of experiment: Submit modification proposals to the value assignments Agenda: - Show everyone the aggregated changes to the canonical taxonomy - Show everyone the aggregated initial conditions value framework - Explain how to submit proposed changes to the value framework ### Research Regroup 3 Phase of experiment: Value distribution ### Session 4 Phase of experiment: Recap, Learnings & Feedback What worked well? What didn't work well? What should we change for next time? Deliverables: - share an EDA on each person's ## Feedback on Google Doc Flow - Remaining design decisions: 1. Should we have people report the taxonomy in the govrn app? We could potentially create a LemonadeDAO. - For phase zero, this would look like having no activity types and having people create their own categories that the research team then aggregates into drop down menu options. - Implication: we'll need to manually update the categories in the drop down menu. 2. Do we want people to be able to see each other's contributions? - Thinking no in the emergent setting of initial conditions but yes afterwards? - Additional docs we may need to create - Majority vote docs - Doc to see everyone's contributions so far - On the [v0: Contribution Records Link](https://docs.google.com/forms/d/1vviUeRb2WxkpW6FF1FQyQQJBs8E1fDvxY48XhVCdvM0/viewform?edit_requested=true&pli=1) - [ ] The form looks really pretty! I'm not able to fill out though? Also wasn't able to click on 'request edit access'? - [ ] For the emergent phase (before the canonical contribution taxonomy exists) we'll need to delete the category field at the bottom. OR we can ask people to propose a category name or tags. - [ ] Let's add the lemonade stand description on this form for reference. - [ ] Maybe we should add a description of what phase the experiment is in on the form as well. E.g. "This is the emergent phase where we crowdsource free text contributions into a proposed initial canonical taxonomy." - [ ] I can get you a list of emails to grant access to the Contribution Taxonomy form. Should it be read only so as to funnel proposed edits through the proper form? - The "Direct Form Link" and "Fill out the Form Link" for logging records look the same to me? - On the [v0: Cultivation Link](https://docs.google.com/forms/d/10v96HsOuTc53kUOtQ_3vH2C1xCIpDTnBoSJboxX4Sm8/viewform?edit_requested=true) - [ ] Love the context blurb at the beginning - [ ] Let's add a link to the the Canonical Taxonomy for easy reference? - [ ]