# [Under Review - SSI Blog] The Sorry State of Usable Software in Open Science: Results from Open Science Retreat

By Meag Doherty, Anja Eggert, Yomna Eid, Kjong-Van Lehmann, Christian Meesters, Lennart Schüler

##### Misalignment between incentives for software developers and expectations of scientific software users. License is CC-BY-SA 4.0.

At the recent [Open Science Retreat](https://https://open-science-retreat.gitlab.io/), a group of us from across scientific domains dove into our issues of usability in scientific software. Below is a recap of the discussions and next steps.

### The Sorry State of Usability in Open-Source Scientific Software

A large fraction of science depends on the reliability of community open-source software. Without reliably maintained software, without software which can be ported across systems, results are - by definition - not reproducible. All group members expressed frustrations about usability of their current software. To find a way out, we asked ourselves:

> What are the conditions that enable scientific open source teams to prioritize and focus on usability successfully? And who can effectively help change the status quo?

### What constitutes usability in this context
As we started our discussion, it became clear that not everyone saw usability the same way. Is usable the same as maintainable? Does it only apply to GUIs? If I'm the only user of my software and I think it's usable, is it so?

To find common ground for our three days together, we collected aspects of what constitutes usable free open science software based on our own experiences.

And while many of the following aspects seem obvious and are solved in many communities, they are often missing and do constitute a challenge in FOSS, particularly in smaller software projects that however provide key elements for analysis. Critical elements for usable software:
* __License__: A free permissive license is a prerequisite for the community to be able to use existing software packages.
* __Documentation__: Usage examples are an integral part of the documentation that is often ignored, but can make a significant difference for the user. Well-structured documentation such as (https://documentation.divio.com/) is essential for software to be usable. This allows the user to utilize the software according to the specifications and to prepare the input appropriately.
* __Versioning__: Code should be bundled in releases accompanied by changelog and release notes. This ensures reproducibility and makes the user aware of altered software behavior.
* __Deployability__: For software to be usable, it should be deployable. In other words, there needs to be an installation routine available that takes care of compilation and dependency resolution in an automated fashion. Additionally, it would be desirable to have executable containers available so the user can get started immediately.
* __Issue Tracker__: Usable software should have a mechanism by which users can provide feedback and record potential bugs and issues, and ideally request support.
* __Avoiding proprietary code base__: For various reasons, some of which are historic, proprietary code bases are still popular (e.g., Matlab, Mathematica, SAS). However, any new software that depends on licensed code bases creates severe barriers to portability and adoption.

### Pathways to achieving "Usable Software"
We will continue to raise awareness about these issues. By sharing personal stories [link to personal narrative posts] that we drafted during the retreat, funders, PIs, and open science practitioners will more clearly see the usability problems facing scientists.

We also believe it is worthwhile to advocate for basic usability review criteria similar to those in the Journal of Open Source Software (JOSS) [link to JOSS criteria] to be used on papers describing scientific software releases. In the coming weeks, some members of the group will circulate a letter advocating for this change.

If you are interested in this topic, follow the [‘Usable Software Ecosystems Research’ (USER) study by Superbloom Design Github repo](https://https://github.com/simplysecure/USER_project) for forthcoming research results on the considerations and practices around usability and design in open source scientific and research software.

# V.01

This blog will be published here: https://www.software.ac.uk/blog

## Authors (alphabetical order)
* Meag Doherty ([e-mail])
* Anja Eggert ([e-mail])
* Kjong-Van Lehmann ([e-mail])
* Christian Meesters ([e-mail])
* Lennart Schüler ([e-mail])
* Yomna Eid ([e-mail])
* Damar Wicaksono ([e-mail])

## Relevant links
- https://hackmd.io/UR0k4Iz9QAm1RSZXaZUO8g
- [[Notes] Narratives about designers/developers/scientists challenges with OSS usability](https://https://hackmd.io/zXs41Xr5QdOH49fQhM5Dow)

## Blog post outline
* Why is this issue important?
* Summary of the discussion topic
* Preliminary List of Items defining "Usable Software"
* Software Maintenance
* Recommendations
* Pathways to achieving "Usable Software"
* Examples of Usable OS Scientific Software
* Other significant points outside the main topic of discussion
## Header image
Misalignment between incentives for software developers and expectations of scientific software users. License is CC-BY-SA 4.0.

## Intro
[the sad state of affairs overall]
A HUGE fraction of science depends on the reliability of community open-source software. For without reliably maintained software, without software which can be ported across systems, results are - by definition - not reproducible.

[the sad state of affairs specific to package management]
The world of scientific software is extremely diverse. It varies from domain to domain, and software is usually provided for research in so-called 'packages'. We did realize that people maintaining packages are facing a number of reoccurring issues. These include the lack of a free license and automated installers.

Without any licence, however, installation of a software package might not be legal. Without a version tag, nobody reliably knows whether a change has been introduced that changes the behaviour of a software.

###### Results of a small publication sampling in December 2020, where the 10 most recent software publications in the indicated journals were considered. A substantial fraction of this software does not show any licence, no installation routine, no versioned release and/or is not maintained (here: at publication time, the last contribution to this software is > 6 months ago). For some software programs, no repository ("no repo") has been indicated in the respective publication.

We consider a publication practice which totally neglects to check the most basic software engineering criteria a potential contribution to the reproducibility crises. The underlying cause is the mismatch of incentives for scientific software developers (publish quickly and move on to the next project) versus the need for reliable software.

[the goal]
It is therefore paramount to scrutinize papers describing scientific software releases during the review process. To facilitate this, reviewers should be provided with a list of basic review criteria similar to those in the Journal of Open-Source Software (JOSS) (link to criteria - here)

investigating what are the conditions that enable scientific Open-Source teams to prioritize and focus on usability successfully. A grass root initiative might achieve exactly this goal.

No grass root initiative can effectively change the status quo on its own; journal editors, research funding organizations, and other entities need to increase their awareness.

## What constitutes usability

The question around what constitutes usable software is frequently discussed and many communities have some definition that does agree on many core elements. Usability can be described as the capacity of a system to provide a condition for its users to perform the tasks safely, effectively, and efficiently while enjoying the experience. (source: https://en.wikipedia.org/wiki/Usability).

Within this interdisciplinary group we have collected aspects of what constitutes usable free open science software that are currently absent in the majority of FOSS. This is based on the different experiences represented in this group. While many of these aspects seem obvious and are solved in many communities, they do constitute a challenge in FOSS, particularly in smaller software projects that however provide key elements for analysis.
We believe that certain elements are a necessary requirement for software to be considered usable, while other elements are very desirable but not absolutely necessary.

* Required elements for usable software:
* __License__: A free permissive licence is a prerequisite for the community to be able to use existing software packages.
* __Documentation__: Usage examples are an integral part of the documentation that is often ignored. A well structured documentation such as (https://documentation.divio.com/) are requirements for usable software such that the user can utilize the software according to the specifications and input can be prepared appopriately.
* __Versioning__: Code needs to be bundled in releases accompanied by changelog and release notes. This ensures reproducibility and enables the user to be aware of altered software behavior.
* __Deployability__: For software do be usable it needs to be deployable. This means that there needs to be an installation routine available that takes care of compilation and dependency resolution in an automatic fashion. Additionally it would be desirable to have executable containers available such that the user can get started immediately
* __Issue Tracker__: Usable software requires a mechanism by which users can provide feedback and record potential bugs and issues and ideally request support.
* __Avoiding proprietary code base__: For various reasons, some of which are historic, proprietary code base is still popular (e.g., Matlab, Mathematica, SAS). However, any new software that depends on licensed code bases creates severe barriers to overall portability and adoption.
* __Transparent prioritization and decision-making processes__

* Elements that are strongly advisable:
* __Test suite__: The source of a package should include a test suite that demonstrates the correctness of the package according to its intended usage. Using a test suite, package maintainers and developers explicitly specify and verify the expected behaviors of their package. From the users' perspective, the availability of a test suite results in additional confidence in using the package.
* ?Efficiency
* Scalability: Scalability is often a usability requirement but can of course not be guaranteed in specific problem setting where the algorithms are limited or when dealing with problems that need to explore solution spaces that scale exponentially. However, in many cases, scalability can be significantly improved with relatively simple means.
* Tutorials: Extensive Tutorials and examples that walk through the different capabilities of the software are highly desirable but not always practical.

We started with a discussion on the pre-print findings from the ‘Usable Software Ecosystems Research’ (USER) study by Superbloom Design. This research - supported by the Sloan Foundation - aims to define considerations and practices around usability and design in open source scientific and research software.