Meeting Minutes: Fedora Annual Contributor Survey

# Meeting Minutes: Fedora Annual Contributor Survey Informal polls to be developed and implemented (by the Revamp team at first and later by other teams) to measure awareness. ## Meeting June 1 - Attendees - Aleksandra - Vipul - Marie - Isa - Discussion - Marie to review and let Vipul know: https://pagure.io/Fedora-Council/council-docs/pull-request/146#request_diff - Badge has been pushed and added to LimeSurvey - Marie follow up with Ashlyn - Communications - Aleksandra to send to devel-announce - Vipul to make discussion post - Marie to publish tweet ## Meeting May 25th - Attendees - Aleksandra - Sumantro - Marie - To Do - Design ticket - https://pagure.io/design/issue/814 - close this out - Open web & apps ticket - https://pagure.io/fedora-websites (Sumantro) - https://pagure.io/fedora-websites/issue/1150 - Ping Ashlyn (Marie) - Badges - https://pagure.io/fedora-badges/issue/800#comment-791032 - Update the art (Marie) - Ping badge team to get it pushed or Vipul (Sumantro) - Survey - Make changes from last year (Aleksandra) - Getting into limesurvey, contact Vipul (Sumantro) - Blog post - need a blog post (Isa) - Add Isa to the chat on Element - Go/No-Go Meeting - Set for June 1st. ## Meeting April 6th - Attendees - Aleksandra - Marie - Discussion - Open tickets - Badges ticket - https://pagure.io/fedora-badges/issue/800#comment-791032 - Council ticket - https://pagure.io/Fedora-Council/tickets/issue/396 - Design ticket - Last year https://pagure.io/design/issue/754#comment-732394 - How to add survey banner hackmd: https://hackmd.io/mko8NHloTVyjPMug9_LCvw ## Meeting March 10th - Attendees - Aleksandra - Marie - Discussion - Program management request: https://pagure.io/fedora-pgm/pgm_team/issue/31 - https://hackmd.io/fGi94k5dQWOb5UnS-WFFbA?both ## Meeting February 23rd - Attendees * Marie * Aleksandra - Discussion - Draft of documentation completed by Marie, reviewed and edited by Aleksandra - Ask Web & Apps team to get this up on their Docs page: - https://hackmd.io/mko8NHloTVyjPMug9_LCvw?view#How-to-add-the-Fedora-Annual-Survey-Banner - Review schedules with support teams (badges, web & apps, design) - Program Management Team - we should make a request ## Meeting February 10th - Attendees - Marie - Aleksandra - Vipul - Discussion - How to format what we store on Council docs - we have a couple options - storing a version in the limesurvey format - lss format, not indexed, just sitting in repository- stored, read only backup - markdown format version which people can collaborate on - these lss files can be edited in a GUI on limesurvey.com - workflow for additions - archived lss, this gets edited only once a year (updated after survey is live) - changes get made in a markdown file - git diff to see what changes were made before the new survey goes live (to ensure all changes are implemented) - changes get implemented to lss file - badge thoughts - we need to update this to say "21" https://badges.fedoraproject.org/badge/community-survey-taker-i - will run this by year - call for volunteers to help with survey - analyzing 2021 data - communications - presentation of results - development of tooling to analyze the data - promotion - To Do: - [ ] Aleksandra to put update on PR and to relay our process plan - [ ] Docs - [ ] Marie draft call for volunteers ## Meeting January 27th - Attendees - Marie - Aleksandra - Discussion - https://pagure.io/Fedora-Council/council-docs/pull-request/134 - Aleksandra is going to follow up - Analysis - We should document what questions we are looking at for 2021 so we can look at the same questions for 2022 - Taking February 10th meeting as a work session for documentation - Looking at greg's analysis - Marie to analyze questions.. were they useful? what more questions should we be asking - Aleksandra to understand how to create graphs, work with Greg ## Meeting December 16th - Attendees - Marie - Aleksandra - Discussion - updates - Aleksandra updated questions and sent to Greg - PR https://pagure.io/Fedora-Council/council-docs/pull-request/134 - Sent to Vipul to review - TO DO - Marie - Draft process doc - Schedule a meeting for mid/end January (done) - Aleksandra - follow up on PR - upload changes after that ## Meeting December 2nd - Attendees - Marie - Aleksandra - Questions to analyze: Sent to Greg Dec 2nd 1. How does knowledge about community engagement correlate with folks role in Fedora? - How familiar are you on how Fedora is organized as a project? - And those that chose "Mindshare Commitee" or "Regional Fedora Ambassador group" from Who would you first try to contact if you wanted to run a Fedora event/needed swag? 2. What is correlation between - How familiar are you on how Fedora is organized as a project? - What are your current role(s) in Fedora Project? 3. How language barrier correlates with role in Fedora? - Is English your first language? If not, do you find language as a barrier for participation in Fedora? - What are your current role(s) in Fedora Project? 4. How satisfaction with Fedora correlates with role? - What are your current role(s) in Fedora Project? - How satisfied you are with Fedora as a developer workstation? - How satisfied are you with the Fedora Project overall? - How likely are you to recommend contributing to Fedora to a friend? 5. How satisfaction with Fedora correlates with Linux experience? - How satisfied are you with the Fedora Project overall? - How likely are you to recommend contributing to Fedora to a friend? 6. How comfortable are people speaking up correlation with roles? - What are your current role(s) in Fedora Project? - How comfortable do you feel speaking up about your ideas/opinions/theories or asking questions in Fedora spaces? 7. What is the correlation between roles and peoples preference between default or obscure tools? - What are your current role(s) in Fedora Project? - What is your preferred desktop environment? - Which text editor do you use (for random small edits)? 8. What is the correlation between how familiar people are with Fedora and their likeliness to recommend to a friend - How familiar are you on how Fedora is organized as a project? - How likely are you to recommend contributing to Fedora to a friend? 9. Does Flatpack adoption correlates with modules adoption? Do people generally don't like new things, or there people treat flatpack and module technologies differently? - Would you recommend Fedora Modules? - Would you recommend Flatpaks? - Documentation - Write up process/instructions - Update the text of the survey - Questions to add to survey - How would you rate your involvement in the project ## Meeting November 24th - Attendees - Greg - Aleksandra - Marie - Analysis Questions - We need to develop questions for what we want to know - Previous notes from survey question development: - We need to understand the usage pattern in various Fedora flavours and tooling - We should check if we see clusters created in the community based on particular tools, or media or role. - What languages do our contributors know? Do they feel that Fedora Community is accessible to them in a language they understand? Or is it a barrier to have to use English? - Do you feel like you are included as a member of the Fedora Community? (In the context of other answers, this is interesting. What do people who feel like they “belong” do in Fedora? What do people who feel excluded do or not do in Fedora?) - How do people feel about accessibility in general, but also mindshare specifically - Satisfaction rating for Mindshare - Knowledge of CoC, aware of community standards of behavior - How comfortable are people to speak up in public, or private about issues in the community - Do people feel recognized/appreciated - Do people feel empowered to propose ideas or work on things - awareness about the revamp, and also how the new commops structure is working/improving - To Do: - Next step: let's start with the easy graphs, throw questions into the Survey chat - As we go: work up some more complex questions - Mid december check-in - Marie to schedule - Initializing the council docs ## Meeting November 18th - Commblog post - Discussion - Follow the thread - pin - Analysis - Follow up with Greg - Podcast - It would be great to talk about analysis there - Promote that the csv is available - Updating the survey for 2022 - Sustainability - Add the survey to the Council pagure repo - Documention on Council docs - Page on how to run annual survey - Page with survey text - Can we ask Vipul to document limesurvey process? TO DO: - Marie: - [ ] Finish up blog post - [ ] Draft documentation for Council docs - Aleksandra - [ ] Put the survey up on the repo - [ ] Work on "roles" analysis ## Meeting November 9th - Review processed data - finished prep - Review blog post - To Do: - Send to Council for final review ## Meeting November 4th - Quick check in - blog post draft: https://hackmd.io/fGi94k5dQWOb5UnS-WFFbA?view ## Meeting October 27th & October 28th - We are almost through cleaning up the data - Meeting with Greg - Nest Session: - https://docs.google.com/presentation/d/1KK25_XRFxxeuDT-c7rkQlZOeb4UEsYVR-_bABycPsYI/edit#slide=id.gd404890b01_0_410 - Data WIP: - https://docs.google.com/spreadsheets/d/1xJQtTh56kPBH9v-vwGYaVaX8qFRUb5EugwmM5gTZQ1Y/edit#gid=0 - unique answers can reveal who is giving the response - has this come up in other situations for you? is there any reason it would be problematic? - is there a standard way to handle these types of responses? - Greg's response: - programatically, for any given answer you can look at the proportions, or numerically. Doesn't matter, when a category drops below a threshold, use "Other" - Doesn't have to be very high, there isn't a standard per se. If you can figure out who a person is, the threshold isn't high enough. - 1% might be reasonable? - I have been able to gather some insights from the data, but I am still a newbie - are there standard questions/ways to analyze the data - specific to open source communities would be great - nothing standard about surveys ;) - Good places to start: - Pick a question that use as a lens to look at some other questions. - For ex: Comparing "How satisfied are you with Fedora?" - you can split between "1-2" and "3-5" as no or yes they are satisifed with Fedora and then look at other questions against those two groups - it would be interesting to look at the 1-2 responses - thoughts about a second iteration of the survey - are there specific things we should be looking at in order to improve for next years survey? - Greg's response: - Looks pretty good, but need to get into the data to understand fully - Some questions can have quite specific answers, how detailed should our answer choices be? - we got a quite a few "other" for desktop that seems to fall under options we provided - Differentiating between variations of Google for docs for example - We are getting information that we don't necessarily need? - Greg's response: - Will it be too much from the perspective of the survey respondent? is a good way to measure - - Is it problematic to publish the data on the Community Blog - PII isn't an issue here - People could determine, but there is plausible deniability - We can do some text based analysis on the open ended questions - Map that back to how satisfied are you question - see what comes up the most (word map?) - Modeling is also an option if data doesn't seem to show much - https://www.theforeman.org/2019/05/2019-foreman-survey-analysis.html#plugins - https://emeraldreverie.org/2021/02/23/docs-survey-2020-results/ - Document what we have done to the data - this is good policy to track the steps we take with the data - Time based data is a bit tough to handle - Make sure to keep these types of things the same year to year on the survey - Wrap Up: - How does Greg want to help? - Modeling, graphs, charts - Review the data - Chat on Element to dicuss survey analysis - Greg's going to set this up - Let's set up a checkpoint - End of November - Marie will set up ## Meeting September 20th Nest wrap up - people will have fun with the data, we showed just a teaser - we should prioritize getting out the raw data - council review - want to send to council-private to review before we release - promotions (marie) - write a blog post - discussion.fpo - mailing lists - mindshare, council, devel-announce, d&i team - get the data ready - Marie to work on community engagement section, Aleksandra to tackle the rest - check each question with write ins - remove comments entirely - what to do with throwaway answers - Email Greg (marie) - October 14th meeting - try to send data over before the week of the 11th - Podcast - should coordinate with Grayson about timing ## Meeting July 13th Nest preparation * By the end of July * Is the survey representative? ** Total number of contributors and their roles. *** Compare number of FAS accounts active in the last year ** How many participated in the Coding and Desktop sections * Choose interesting questions from the survey and draw the graphs for them and comment * Choose high-level questions: ** Where do get Fedora news from? Highlight Ask Fedora * Correlations: ## Meeting July 1st * Scheduling a meeting with Greg * For the second or third week of July * How do we want to publish the results? * Nest * Presentation: focus on results * Podcast * After Nest: focus on process * Community Blog * Publish this right as our talk ends * Check with Ben in advance * Tweet * Satisfaction rating is xyz% * 800 results * how many badges * Publishing the data? * The data needs to be edited, remove the comments * To be published in a blog post around Nest * Nest session * Aleksandra started some notes, is going to play with the data for the next week * To Do: * Analyze results * Come up with preso outline * Create slides * Rehearse * Present! * Analysis Process * Set up a hackmd to take notes on each question * Aleksandra has a google slide deck as well * https://docs.google.com/presentation/d/1xAvqFj7WCEaYqzm94MV1IDWPfQWr3Sar_5m_BviSp14/edit?usp=sharing ## Meeting May 20th https://pagure.io/fedora-websites/issue/1079 ### Blog Post June 1st The Fedora Council is running the first Annual Fedora Contributor Survey and we want to hear from you! The survey will be open to take for the month of June, and there is a shiny Fedora Badge to earn. Our goal is to gather authentic and valuable feedback to better support the Fedora contributor communnity. We plan to analyze the results and share findings at Nest with Fedora, 2021. Take the Annual Contributor Survey today! The survey was proposed and developed by Council member, Aleksandra Federova, with support from Marie Nordin(promotion & coordinating feedback) and Vipul Siddharth(LimeSurvey wrangler). The Council as well as the Mindshare Committee gave input and feedback on the survey several times as it was being developed. The Community Outreach Revamp Objective team also pitched in on the Community Engagement section. The development of a yearly survey falls under the Revamps activites and we want to keep survey fatigue at a minimum so it made sense to tie these two initiatives together. ## Meeting May 6th Attendees: Marie & Aleksandra Reviewed and edited engagement questions and added a couple to "About You" ## Meeting April 22nd Attendees: Marie, Aleksandra #### Promo materials - Marie to review badge design - Marie to review banner design Blog Post #### Content Edits - what level are we analyzing - Aleksandra to add in satisfaction question to each section Community questions are being developed - https://pagure.io/mindshare/issue/263 ## Meeting April 22nd Attendees: Marie, Sumantro, Mariana #### Community Engagement Questions Information we want to gather with the survey questions - What languages do our contributors know? Do they feel that Fedora Community is accessible to them in a language they understand? Or is it a barrier to have to use English? - Do you feel like you are included as a member of the Fedora Community? (In the context of other answers, this is interesting. What do people who feel like they "belong" do in Fedora? What do people who feel excluded do or not do in Fedora?) - How do people feel about accessibility in general, but also mindshare specifically - Satisfaction rating for Mindshare - Knowledge of CoC, aware of community standards of behavior - How comfortable are people to speak up in public, or private about issues in the community - Do people feel recognized/appreciated - Do people feel empowered to propose ideas or work on things - awareness about the revamp, and also how the new commops structure is working/improving #### Questions * Who would you first try to contact if you wanted to run a Fedora event/needed swag? - Fedora Council - FCAIC - Mindshare Committee - Regional Fedora Ambassador group - Local User Group * On a scale of 1-5 how accessible do you find Fedora resources to be? (such as docs, wiki pages, pagure repo, IRC/chatroom) #### Added * How comfortable do you feel speaking up about your ideas/opinions/theories or asking questions in Fedora spaces?(chat platforms, pagure, discussion) - Completely comfortable - Somewhat comfortable - Neutral - Uncomfortable - Varies depending on space - Other * Is English your first language? If not, do you find language as a barrier for participation in Fedora? - Yes - No + field for freeform response * Are you aware that Fedora provides concrete resources to further Fedora's vision and initiatives? (event sponsorships, travel assistance, swag for local projects/events) - Yes, and I have taken advantage of those resources - Yes, but the process did not go well for me - Yes, but I am not sure how to request resources - No, I was previously unaware - Other * How confident are you on how Fedora is organized as a project? - I am confident - I have a good idea - I only understand the team I work with - I have little understanding ## Meeting April 8th Attendees: Aleksandra, Marie Goal: Come up with to do list to get survey published. #### Promotion ##### Badge design - Badge design ticket: - https://pagure.io/fedora-badges/issue/800 - Talk to Vipul about best way to include this - permanent claim link? this is for a series tho ##### Banner design List of websites to put it on: - discussion - community blog - mailing list - new account system - start.fedoraproject.org - docs.fedoraproject.org - fedora wiki - irc/telegram/matrix/discord Ticket for design: - https://pagure.io/design/issue/754 Ticket for adding to websites: - dependent on design ##### Blog post Need to write post for community Blog Draft: #### Content Edits Divide questions into these sections - About you section - Desktop section - Coding section - Community Engagement section - need to come up with 6 questions - Marie will open a Mindshare ticket for this - https://pagure.io/mindshare/issue/263 - Add in skip logic - Adding in "ratings" question to each section - Add in a question around motivation ## Meeting March 18th Attendees: Aleksandra, Marie, Ben, & Gregory Greg: Example set of questions that will all feed into each other: What role do you have What is your goal when you came to Fedora How long have you been involved How would you rate X thing? How many questions are ok? - conflicting answers to this - depends on the community - sometimes more info in each question is helpful, to get the correct responses Survivor bias, Who did we miss? How do we get them next time? Can limesurvey do skip logic? (should be) - this will help filter out inauthentic responses Check the Community Central Session from RH OSPO - https://www.youtube.com/watch?v=_ToDcKzeXBc Front intro page is important Sometimes its more effective to break out multiple choice options so you get authentic responses Incentives: - Fedora Survey Badge - Get a special piece of swag after completing X number of surveys One open ended question at the end? (after folks have spent time thinking about Fedora) Open ended questions at the beginning (to capture some unbiased thoughts) Banner to add to different web pages - Different entry urls for different platforms? - Helps show some correlation between users on different platforms and their relationship with Fedora Ask a question you know the answer to: - We can cross-link "Which Fedora Editions do you use?" with DNF stats we get from the independent source. Community questions & technical questions together or separate? - depends bc each community is different - if you split the survey up, there is no way to relate the two surveys together - if you think the people are two different groups, makes sense to separate them Ordinal models are good to use (because they assume you hit the levels below) Example Analysis: https://emeraldreverie.org/2021/02/23/docs-survey-2020-results/ ## Meeting February 18th my main idea was to understand the footprint, the usage pattern for tools and apps so it is not just the total average numbers for the entire userbase, but we can have different views on which types of users prefer which types of apps and tools another thing is split it up, with three or five (or whatever) most important questions, and then "would you like to answer more" tbh, i think that the selling point of this particular survey is the possibility to tell us that you hate nano, or emacs, or flatpak, or modules One thing we talked about when we were talking about this in... prague, I think? was doing one survey a year that basically repeats the same key questions, and then also one survey a year on (six months appart) that focuses on something specific (like, "how much do you love modularity?") With that in mind, I think the flatpaks/modules/silverblue questions cuold be moved from this to a survey specifically about those kind of usage things A question I have is where the decentralized pockets of our community are. We know Fedora is big and there are small Fedora communities around the world, but where are they and how do we connect with them? Knowing more about platforms and preferences for using FOSS would be very insightful to me Yes, some word-smithing needed :) But getting at the idea of understanding where people are, how we can connect with them, and to make sure we are leveraging our communication channels more effectively Giving options with most generally used communication mediums can be a good start with the rephrasing. i need to understand if we have separate groups in the community with nonoverlapping intersets, or we have a more random thing without explicit clusters bookwar: i guess that raises two questions for me. 1) how do we group users into types, and 2) what do we do with that information once we have it? bookwar mattdm: that's what "about you" section supposed to be bookwar: ah, I see. so, are those the right "about you" questions to get to that point? given that we have to stay away from pii type questions... idea Q: Which types of users prefer which types of apps and tools? idea Q: What platforms does our community use and where can we best engage them?