USER Research - Radicle CI Integrations

As part of our work in the Radicle & Integrations & Tooling team, we want to gain a deeper understanding of our users and validate our assumptions about their needs, through user research. Specifically, we're exploring how existing Continuous Integration (CI) engines can be integrated with Radicle, using our tools.

Goals

*   Establish a process for continuously collecting feedback
*   Identify problems through user research
*   Validate solutions through iterative testing
*   Prepare user stories to add in our backlog
*   Inform other teams about our findings so to incorporate them in their own tools

Methodology

Our methodology includes the following plan. We start with defining our goals which are depicted above.

1.  Define Our Goals: Identify what we want to achieve through user testing (e.g., improve usability, reduce errors).
2.  Recruit Participants: Decide on our target audience and identify potential participants.
3.  Prepare Test Materials: Develop a test plan, create test cases, and prepare necessary materials.
4.  Conduct User Tests: Schedule sessions, observe interactions, and take notes.
5.  Analyze Results: Review recordings, categorize findings, and prioritize issues.
6.  Iterate and Refine: Use results to inform design decisions, fix bugs, or improve the user experience.

Tools

First we picked two user archetypes which we discovered after discussing with Radicle users and contributors that are the most important for our tools since they are more likely to need an Integration between Radilce and their CI.

OSS Contributor

An OSS contributor is an individual who primarily writes code for one or more open-source projects. This role can range from full-time software developers in OSS projects to hobbyists seeking to contribute to such projects. A key aspect of their job involves utilizing CI/CD tools to leverage the entire software development lifecycle, including testing, coding, and creating pull requests or documentation.

Given their technical expertise and familiarity with CI/CD workflows, we assume that OSS contributors are more likely to adopt Radicle as it aligns with their existing values and workflow. By leveraging their existing knowledge and experience, they can seamlessly integrate Radicle into their development process.

OSS Maintainer

An OSS maintainer has distinct responsibilities and requirements that differ from those of an OSS contributor. Their primary focus is on ensuring the continuity of the software development lifecycle, CI/CD infrastructure, and processes. This involves reviewing and approving pull requests from contributors, as well as accepting or rejecting patches to the codebase.

In addition to these technical tasks, OSS maintainers also play a crucial role in community management, coordinating with other developers and projects, and implementing long-term plans. Given the significant impact of CI/CD on their work, OSS maintainers are likely to adopt Radicle only if it can seamlessly integrate with their existing CI workflow and processes, without disrupting their current operations.

We started by creating empathy maps for each user archetype based on conversations with Radicle users. However, we soon realized that understanding the user's pains, gains, and jobs to be done was more critical to our goal. To address this, we chose the Value Proposition Canvas as a valuable tool to help us better understand and prioritize these key aspects of our users' experiences.   

The Value Proposition Canvas is a business model tool that helps ensure that a company’s product or service is positioned around customers’ values and needs.   

We also used an online form to book slots for interviews and finally we ran the interviews using online meetings tools so we can have both audio and video communication and observe user reactions during the interview.

Procedure

*   We reached out to users on Zulip and Discord to fill out a brief form, which helped us schedule online meetings to discuss our project. We have, so far, only had 4 forms submitted, and we were able to connect with 3 of them. The meeting invitation link can be found here: https://docs.google.com/forms/d/1ENCHKQk_DQu3NZ5glBS4GoWeThapwXU3zRDpKfLPyLU.

*   To ensure a smooth conversation, we created a meeting scenario that allowed for flexibility in case users wanted to discuss specific topics. We also took into account the diverse needs, interests and level of experience of our participants, who held different roles such as OSS contributors, project maintainers, and node operators. To accommodate these differences, we tailored our discussion approach to each role.

*   Finally, we prepared a repository with all necessary files to facilitate a seamless interview process. Additionally, we made it easy for CI integration by having everything setup in one place.

The scenario was the following:

1.  Intro: What is RIT (our org), who we are, why CI integrations exist

2.  Ask them if they know what is the "CI broker" and the adapters and how they work in general

3.  Ask if they know where to find the documentation. If not ask them to try to find it. Provide the link if they can't. (https://app.radicle.at/nodes/seed.radicle.gr/rad:z4Uh671FzoooaHjLvmtW9BtGMF9qm)

4.  Give them the pre-configured Github repo (rad:z4HyZP9FLJweV8LnF3LrLY6rZ9vTf) with a simple workflow and ask them to

*       Fork github repo (Github actions are disabled by default)
*       Publish to radicle
*       Add adapter configuration
*       Create a file
*       Git push both
*       Create a patch because you don't get any comments on commits (Note: our CI integrations report run results as Comments on Patches, as an intermediate solution until there is support in the protocol level for that)
*       Push to both (git push both test-both -o patch.message='test adapter' HEAD:refs/patches)
*       Get comment with -v

Notes:
*   Ask them to run the maintainer scenario according to what CI they use
*   Pay attention if they understand what they do either by observing or asking quick questions
*   If time permits ask them to run the same pipeline for a second time to understand if their issues are related to the setup or the everyday run

Analysing the results

Each user brought a unique perspective and set of challenges to the table, providing valuable insights into their experiences with Radicle. Leveraging the data collected from the registration form, which included information on user type and frequency of use, we were able to gain a better understanding of our target audience's needs. To further inform our discussion, we also created a dynamic form that the interviewer filled out during each conversation. By combining this data with other insights gathered from the interviews, we were able to paint a comprehensive picture of the user needs and pain points.

Feedback contains the following:

*   All of them had a prior experience in Radicle
*   They were also aware of CI Integration tools and one of them had already tried  
     
*   Two of them found where the documentation is easily because they've done it in the past
*   They had issues to setup the repo and run the adapters
*   They said that it was relatively easy to setup the repo  
     
*   And the also offer some suggestions for improvement

Refinement based on the results

The user feedback and analysis provided valuable insights that informed our CI integrations for Radicle. Although the response rate from our survey was limited, the data collected still offered several key takeaways that highlighted areas where we can enhance our tools. Specifically, based on the results, we identified opportunities to improve in the following areas:

*   Enhancing documentation to provide more granular and precise step-by-step guides

*   Expanding the repo setup guide to better equip project maintainers with a deeper understanding of its purpose and usage

*   Developing more robust tools to automate the process of setting up repositories

*   Refining adapters to reduce spamming when adding comments to patches, including progress updates and workflows' results

*   Streamlining or automating the management of repositories that require explicit operations for adapter functionality