# [Notes] Narratives about designers/developers/scientists challenges with OSS usability

### Background and Introduction for blog post
Over the course of four days in early April, a group of us at the **Open Science Retreat** [link to event recap] spent time discussing **challenges and opportunities with usability in Open Source Scientific Software** [link to breakout recap].

As a way to wrap up the time spent together, a few group members wrote personal narratives (including self-assigned catchy titles) that highlight some of the individual and collective experiences in usability.

Those narratives are listed below.

- By giving ourselves catchy titles, the idea is to speak openly and directly about challenges and potenital ways to overcome so that others my feel seen in these experiences.

As the "usability person" in the room, I heard many stories

While you can read about the specific workshop outputs in the othdesigners, scientists and engineers

Springboard stories: needs and goals
'Points of pain' stories: barriers to overcome
Key scenarios: context for actions
Narrative scenarios: what happens and why
Design Maps: perceptions, actions and barriers
Flow diagrams: actions and decisions
Use cases: detailed sequences of actions

### Y. ECR (using packages to move forward scientific discoveries)
Needs/Goals:
- be able to use pre-existing packages in the workflows I am developing.
- proper documentation. examples do help to know the capabilities. does not need ot be specific to my domain. better to learn by example rather than text alone.
- if the workflow can be generalized and work well with other packages [something about dependencies too], I can run my workflow and re-run in the future. This helps set me up for analysis that leads to a methodology.
- I need a workflow <> algo <> package.
- Eventually build you your own package
- I need concrete guidelines on how to build a package that won't only be used by me, but will survive in the field

'Points of pain' stories: barriers to overcome
- Bad documentation makes figuring out the package/function hard
- I am less inclined to use the package and find an alternative, even if less-straightforward
- Packages which are known to be unstable -> might break my code in the future -> don't like such dependencies
- Identify what the best practices for creating a package are, if that is a pre-existing document even better

Key scenarios: context for actions "what do you need to do in order to do what you want to do"
- Understand what makes up a robust package
- Answer questions along the lines of: What is the bare minimum documentation required for early phase? What makes up a user-friendly, re-usable package? Differences across programming languages?

Narrative scenarios: what happens and why
Design Maps: perceptions, actions and barriers
Flow diagrams: actions and decisions
Use cases: detailed sequences of actions

### K.
Springboard stories: needs and goals
* As a researcher, I depend on being able to execute/use the latest research software to provide state-of-the-art analysis results
* Ressources (Time) is a major constraint.
* Results need to be primarily robust, reliable and reproducible
* Each new software adaptation is primarily justified by "accuracy" of their results.
'Points of pain' stories: barriers to overcome
* The best tools are not always the most usable, specifically they are:
* difficult to install and to resolve dependencies,
* have "novel" file formats such that data needs to be prepared,
* difficult to understand how to use,
* often buggy and not robust,
* without a well written documentation or examples,
* without a license.
Key scenarios: context for actions
Narrative scenarios: what happens and why
Design Maps: perceptions, actions and barriers
Flow diagrams: actions and decisions
Use cases: detailed sequences of actions

### A.
Linear mixed models are statistical models widely applied in animal research and thus part of my daily work. The `MIXED` procedure of the proprietary software *SAS* is very flexible and allows to model all kinds of correlated data and can be used to check the performance of the model. The OSS *R*, in contrast, provides a similar/comparable level of statistical analysis only when using several R packages, like `lme4`, `nlme` and/or `performance`. As *R* is continuously under development, this is understandable, but makes the usibility of *R* for mixed model analysis not very straight forward. During my work as a statistical consultant, it is in somce cases difficult to argue why scientist should move from *SAS* to *R*.

Springboard stories: needs and goals
'Points of pain' stories: barriers to overcome
Key scenarios: context for actions
Narrative scenarios: what happens and why
Design Maps: perceptions, actions and barriers
Flow diagrams: actions and decisions
Use cases: detailed sequences of actions

### M. UX designer
#### ... my dilemma
How do we align on the universe of problems to solve in usability in order to move confidently into the solution space and start making usability investments?

#### ...
-
#### ... a few barriers to overcomes
- What is usabiilty? What is the relationship between usability and maintainabilit?
- Software for whom? Most interestingly, software for software engineers can be even more unasable than software for other users.

---

## Full Draft post
Submitted draft: https://hackmd.io/AlzAXuvrRrSRXcldqT9T0A?both

Over the course of four days in early April, a group of us at the **Open Science Retreat** [link to event recap blog post] spent time discussing **challenges and opportunities with usability in Open Source Scientific Software** [link to breakout recap blog posr].

As a way to wrap up the time spent together, a few group members wrote personal narratives (including self-assigned catchy titles!) that highlight some of the individual and collective experiences in usability.

The narrative structure included the following questions:
* What is your role and context?
* What is the core issue regarding usability in your daily work?
* What are the critical barriers you need to overcome? (or maybe you don't know yet!)
* Who can help you and the community over this critical barrier?

### The Lost Early-Career Researcher

As an early career researcher, I have primarily had the experience of using pre-existing packages and integrating them into the workflows I am developing for my analysis with spatio-temporal data derived from multiple-resources.

Having only tinkered around with building my own package, I am lost in a flurry of information. I would like to create workflows that are guaranteed to still run in the future, but it is not uncommon to experience errors with broken scripts due to the package-dependencies. Generally, I try to steer away from unstable packages, but one isn't so lucky all the time. I also have a hard time incorporating certain functions from individual packages into my workflow, on more than one occasion, as sufficient documentation is hard to come by all the time. I am less inclined to use such a package and seek alternatives. **The best packages to use for me are ones that go a step further and provide a test-example of how certain functions work.** It is always easier to learn by example rather than by text alone.

And if I magically happen to get so far as to generalize my algorithm and workflows into complete ready-to-go packages that represent my methodology, **I do not know where to start to ensure that someone beyond me would use it and benefit from it.** I need concrete guidelines on what makes up the robustness and usability of the package and the type, amount and extent of through documentation. To me, it seems arbitrary most times and a function of personal taste rather than a standard. It doesn't help that sometimes these standards differ across domains/communities.

This compilation of issues could easily discourage early-scientists (likely scientists, in general) from openly sharing their code, as the lack of best practices guidelines in combination with the lack of acknowledgement of the effort that goes into is high investment but low reward. I discovered that their exists a gap between my ambitions and training, that they do not align perfectly to meet their goals. That is partially due to **the ambiguity of what the scientific community up to this point defines as good software/packages, even though we know a good one when we see one**. Without the formal training of a software developer, I ask myself what could be done to make this journey easier and less complicated than it seems?

### The Accidental Research Software Engineer

I'm an RSE coming from substantive research, using software as a medium to make the methods I develop more accessible to fellow scientists and students. For me, that frequently means creating graphical user interfaces that reduce the reliance on code and coding skills.

I don't have a formal background in usability, so **while I care deeply about making the software useful and approachable, I am limited to haphazard attempts and intuitive solutions.** The bulk of my user feedback comes from teaching students, running workshops, observing colleagues and answering their questions, so I rarely get to observe entirely naive first-time users systematically. My impression is that **(my) interfaces are designed around the underlying technical design of the software, and may not serve users' interests in the best possible way.**

I wish there were institutional support for usability research and improvement. My contributions are largely evaluated by their scientific merit and substantive feature set, and **I can't always justify spending time on interfaces alone without expanding functionality.** This does not mean that researchers don't care about usability -- to the contrary, I believe that usable tools are a great way to overcome barriers and pave the road towards better research practices, and that better-designed software has a larger impact. However, this is not reflected in the current incentive system.

**I would love there to be design- and usability-related guidance for the entirety of the research software lifecycle.** Starting early on, feedback could help shape the tools and interfaces we build, and better align them with scientists' needs without having to scrap and overhaul existing work. Later, it could point out areas for improvement, and help successful open source tools compete with proprietary alternatives. Throughout, I believe this would be most helpful if it could keep in mind the scientific goals and resource constraints in RSE work, and support developers and researchers in implementing the proposals, possibly by including the user community. Ideally, rather than merely adding to the list of requirements for a research software tool, this could help reduce the support load that many RSEs currently face.

[half-baked aside, not for the post] I wonder whether several of the issues we report are due to an artificial split between 'users' and 'developers', and really what everybody is asking for are multi-disciplinary teams (including not only RSEs but infrastructure/sysadmin staff too) that support one another, rather than requiring that any researcher should be able to pick up any tool without further help, and that developers should put so much work into every package and tool they create prior to publishing that they basically obviate any need for collaboration with them later. To what degree are we trying to uphold a broken structure, and work around a systemic lack of resources and energy on part of everyone involved? [/unproductive bitterness]

### The Pragmatic Bioinformatician
As a Bioinformatician in an academic setting, I often have the task of analysing high-dimensional data using the latest state-of-the-art methods. The decision which method to use is driven by the expected novelty of the results, the overall performance, matching assumptions and plausible theoretical or algorithmic foundation but not usability. At the same time, we also strive to contribute novel approaches. Both tasks involve running similar methods to either decide on a best method to apply or to benchmark my own approach against existing methods.

**Getting other researcher’s methods to run is a time-consuming process** that already stalls at installation procedures and dependency resolutions, input formatting, usage examples as well as documentation and versioning. In some cases, the underlying method and published results suggest convincingly a superior performance but the accompanying software is unusable. This sometimes requires me to re-implement the approach from scratch.

In a research setting, the final product is not the software but the dissemination of the novelty (e.g. publication).**Therefore, there is little reward for good software engineering and long-term maintenance.** Maintaining relevant software packages past publication takes away time from future research and oftentimes the developer has moved on. Bioinformatics researchers are not software engineers, however methods and models from this community have arguably played a part in getting biomedical research to where it is now. The dependency on these contributions paired with software development without software engineering support certainly plays a large part in the current state of such software.

Adding **training may alleviate the situation.** I wonder though whether it is fair to expect researchers to also become software engineers. Providing broad software engineering support seems the right thing to do and while it seems like an additional expense to an institution, I believe that overall productivity and quality may significantly benefit with such expertise that may in the long run justify these expenses.

### The Eagerly Support-Guy

"Working on a High Performance Compute Cluster? - Well, if I have to …" The lament is long and ranges from overly bureaucratic access regulations to various technical details rendering one's research on an official cluster impossible. I overheard this HPC-no-thank-you remark frequently in hallway tracks. And working to support cluster users, I have to admit: **Usability and HPC-clusters do not go along well.**

Adapting life science workflows is particularly challenging. A single such "workfow" (designed to analyse some sort of data, e.g. genetic material) may require several dozen applications. All of these software tools need to be deployed, I/O-issues need to be solved. Some applications clearly are "runs on my system"-only software packages. Not all HPC clusters allow using “[Bioconda](https://doi.org/10.1038/s41592-018-0046-7)”, a package management systems tailored for the life sciences which can be used by any user to install needed software. As a consequence, a frustrated user base ("No, we cannot support _your_ software.") is inevitable.

This HPC attitude forces entire user groups to build and maintain their own compute base. As a result, we see a waste of money and resources poured into redundant infrastructure.

Is there any silver lining?

In order to meet user expectations, **the HPC community mindset has to change**. Away from [chasing FLOPS and providing ever faster computers,](https://www.top500.org/) towards the attempt to support all scientific domains. This would require to reach out and listen. We would stop providing ["true HPC"](https://en.wikipedia.org/wiki/No_true_Scotsman) clusters and gain a pleased user base.

### The persistent biostatistician

Being the statistical consultant in a medium-size research institute, I support the scientists in their data processing and statistical analysis. As I'm an advocate of open and reproducible science, the software equipment I need for my work is beyond statistical software. Using containers for reproducible workflows (like Docker or Singularity) has become a wide-spread approach. Me as a scientist, I see the many benefits of utilizing software containers. But I'm also aware that security risks come along when using it on an institute's server. As I see it,**it should be the task of the IT to set up rules and to find a good balance between security and usability.** Unfortunately, the answer often is: "NO, we will not install Docker on the server because of security issues." Arguing with the IT engineers often feels unfair - I mean I'm a biologist and statistician by training, I'm not a computer scientist or engineer - and I leave such conversations as a loser.. I wish I could have argument papers ("How To convince the IT to set-up Docker on the server") which I can use to prepare for such conversations with well thought out arguments.
[A similar story could be about R Shiny Apps]

### The Altruistic Author

I work in an interdisciplinary academic setting where it is already a challenge to find a common vocabulary. On top of that, many people in my fields don't have a very strong background in computer science. As the author of a scientific software package, I quickly realized that **only documenting every single feature is not enough when facing such obstacles.** But step-by-step examples of how to use features of a piece of software can give a good starting point for other people's research endeavours. By seeing the results of small examples, users get a much better understanding of how to use the software. Besides the actual learning, examples can also be a neat way of explaining certain pit falls or sensible value ranges.

There are **not much incentives for investing time and resources into such usability improvements.** But when users have questions or problems, it actually saves a lot of time when you're able to simply refer to an already existing example which shows how to solve the problem. Providing such a reference is surprisingly often enough help users need. Apart from that, getting feedback from new users who are happy because they could set up their research workflows quickly and without much frustration is great.

### The not so unhappy Research Software Developer/Product Owner

I am working in Research Data Management of a medium sized, non academic environmental research facility as a developer in several smaller software packages and the product owner of a larger scale data infrastructure project. I am in the lucky situation to be a full time software research developer, without the need to intermingle two very distinct careers paths, namely being a scientist **and** a software developer.

Usability is an issue for us, with two distinct sides to it. On the one hand it is an important measure, as user experience is one of the main driving factors for adoption of our products and services and adoption is one of the most important assessment criteria for our work. On the other hand **we are lacking knowledge, experience and tools on how to asses and improve the usability of our artifacts.**

Currently we are addressing the issue mainly by starting out with our own, but heavily developer and software design oriented ideas on the topic,
bring a project into the state of a minimal viable product before we address the broader public with user/stakeholder workshops and introductory courses. While this approach is working so far it is still rear facing and **addresses usability after the fact instead of putting it into the center of the software development process.**

I guess, **what we really need is more internal and/or external formal knowledge on the topic.** Training for developers and probably also external consultants, working with the development teams during the critical phases of a project, would likely help a lot. However, I also think **this would mean another cultural shift in the science-software relationship, before the necessary first one is completed.** We are still in a transition phase, were the idea, that the development of scientific software is a dedicated profession in its own right and not something scientists can or should do during their spare time, is not entirely settled yet.

So **we need to see a further professionalization of the software development process in science first**, before we are able to finally integrate more of the ideas of the commercial software development processes into our own culture.

### An anxious research software engineer outside the comfort zone

I'm currently working on the development and maintenance of a Python package for high-dimensional interpolation. There are recent results in this topic from applied mathematics that we think are beneficial to bring to the wider scientific audience. Although I'm working closely with mathematicians in the project, I'm not a trained mathematician myself. While this can be frustrating at times (due to my lack of some theoretical foundations), this difference in the background can be an asset as well, especially when it comes to designing public interfaces for the non-mathematicians audience.

There are papers, there are software implementations, and there are usable (to the target audience) software implementations. Creating bridges between them is non-trivial. Translating results from papers to (any) software implementation is probably the most straightforward for us, we write the software for ourselves and we use it ourselves to produce even more results. Even if the software is badly designed, one can get used to that and still be (somewhat) productive with it.

Creating a usable public software implementation, on the other hand, is tricky. It requires additional knowledge about the potential users, their needs, and their backgrounds. **All the public interfaces and the documentation must be well considered, not to mention taking into account users' feedback from different channels. We want people to use the software, after all.**

I think the effort that we put into usability depends on the goal of the project itself. If it's for internal consumption, and it's just about new features and producing the next results, then I think any ways of writing software are good enough as long as the product serves its (internal) purposes. But if one of the goals is also to disseminate the development to a wider audience, then it entails certain responsibility of adhering to the **community standards and best practices of usable open research software.** It's certainly more than, say, simply putting the source code on an online public repository. Adhering to those standards--from properly naming things to writing public documentation--takes time, and most probably would hinder the development of brand-new features.

It would be great if the goal of a (presumably open) scientific software project is clarified from the beginning: is it actually just for internal consumption within a small group or indeed for public consumption? If it's for the latter, then **usability should be part of the requirement.** The project lead should understand this, and together with the team, must juggle between different priorities: adding new features, maintaining the old ones, and developing more usable software.

_.. Do you see yourself in these narratives? Do you have ideas on how to make things better? Join [the discussion and share your share experience](https://github.com/meagdoh/ssi-fellowship/discussions/12)._

## Authors (alphabetical order)
* Meag Doherty ([e-mail])
* Anja Eggert ([e-mail])
* Kjong-Van Lehmann ([e-mail])
* Christian Meesters ([e-mail])
* Lennart Schüler ([e-mail])
* Yomna Eid ([e-mail])
* Damar Wicaksono ([e-mail])