# Improving reviewing experience in nf-core ## Context One main factor in the quality of nf-core pipelines is the emphasis of code and documentation reviews before merging into main pipeline code. By doing so, mistakes and errors can be picked up early, as well as act as a 'standardisation' mechanism to ensure similarity in code styles and implementations across all pipelines. Currently, reviewers come from two main 'sources', amongst team members themselves, and from other members of the community. Excluding single-institution projects, these other members are in most cases volunteers. In addition to this, the recent maintainers team was established to try to give further credit and 'formalise' the work of regular and expert reviewers and contributors to the wider nf-core initiative, with the hope that this would help support keep on top of the steadily rising number of pull requests that comes with the growing community. ## Problem Despite the establishment of the maintainers team with around 20 members, reviews are often performed by a few very dedicated group of people within this group. Furthermore, the number of wider-communtiy reviews appear (qualitatively) to be decreasing. In both cases, we feel that there is not currently sufficient incentive to boost reviewing across members - either for the maintainers team nor wider community - in a balanced manner. Factors in this could be a result of multiple factors: a core team regularly doing reviews so other members of the community don't feel they need to ('someone else will do it'), or newer members feeling like they don't currently have sufficient expertise to perform reviews. Therefore unreviewed pull requests and increasing, and in an imbalanced manner (larger/more active pipelines vs smaller teams) - both in number and in time to review. ## Concept ### General concept To increase incentivisation of reviewing from all areas of the community, we would like to introduce a 'gamification' or 'currency' system, in which community members can 'earn' and/or 'trade' access to reviews. This could be implemented roughly as performing reviews earn currency, which can then be used to 'pay' or 'trade' for reviews. ### Considerations Within this system, there are a variety of considerations that need to be taken into account. #### Technical - To what extent can such a system be automated? - How to calculate - How to log? - How to earn/pay? - How is the amount of 'reward' and/or 'cost' calculated? - By number of lines? - By 'discrete' objects (e.g. config, PR module PR, subworkflow PR, vs pipeline PR)? - By number of comments? - How should complexity be accounted for? #### Practise - There needs to be a certain level of automation, it's unlikely contributors will want to spend time on calcualting this manually - How to prevent abuse? - Ensure people do 'summary' reviews without actually reading the code, just to earn the 'currency' - Ensure people don't over-estimated the value of their review or under-estimate their contribution - How much can we rely on 'trust' and 'goodwill' that people would follow - How to prevent siloing - I.e., people in larger teams just earning/trading amongst themselves - defeating the point - Could it be that 'pipelines' earn currency - How to contributors to nf-core modules, but don't develop an nf-core pipeline earn/pay? - How to 'on-board' new members and/or how to account for people not being comfortable with reviewing (yet) - Start off with X number of credits when joining? - Everyone at start of new year gets a base amount? - Should contributing PRs also earn points? - Should such earning points from contributing PRs only happen on common repos (not pipelines, to ensure contribution to community as a whole) - What if someone accumulates surplus points and stops reviewing? - Could this be spent on 'swag'? - How to prevent people just accumulating points for swag (e.g. if earning comes from contributing) - How should multiple-reviewers on a PR be treated? - Should they get same points? - Split in half? - Each subsequent reviewer gets a slight penalty (e.g. multiple by 0.7) - How to prevent PR 'packing'? - As in, very complex PR which gets many functionality or multiple modules added - which takes a long time to review but costs not so much to add - Documentation is equally important, how to incorporate this as this often does not result as 'compartmentalised' content as e.g. modules - Should 'new' PRs be worth than 'updating' PRs - This mostly affects modules/subworkflows etc ## Implementation ### General Strategy Here we propose setting up a system of earning through performing reviews and spending credits by recieving reviews. To keep the system relatively simple and easy to implement, we scale based on the type of nf-core component that is under review. Points can only be _earned_ on common repositories, however can be _spent_ on pipeline repositories. Points can be earnt by performing: | Review Type | Points | |-------------|-------:| | Config | 10 | | Module | 10 | | Subworkflow | 20 | | Pipeline | 40 | Points can be spent on recieving reviews on PRs: | Pull Reuquest Type | Points | |--------------------|-------:| | Config | 5 | | Module | 5 | | Subworkflow | 10 | | Pipeline | 20 | By focusing on single PRs for each nf-core component, we also hope this will promote more _atomic_ PRs, i.e., PRs that are easier to review because it contains a single module, or single bit of functionality. Furthermore,we could consider having additional multipliers, best on how far the PR or review follows best practises: - Is the repository on a common repo (configs/modules/subworkflows/website) - Is the organisation of the PR nf-core? - Is it closing an issue? - Is the PR atomistic (i.e., contains a single module or subworkflow) We could also consider having diminishers - If multiple reviewers review the same PR, for example - each subsequent reviewer gets a 0.8 - the PR comes with a fixed number of points to be distributed among the reviewers ### Examples @maxulysse can you fill in here? Maybe using tables above as guidance to start? ## Timeline - Try a 3 month experiment with 3-4 of most active pipelines ## Alternative solutions Instead of (or in combination with) the above solution, we could simply go for an earning points for review that can be spent on other things (i.e. no spending) For example, with X number of reviews you can get an award such as stickers or other merchandise. <div> <img src="https://hackmd.io/_uploads/rJ7fhE5MA.png" height="150px"> <img src="https://hackmd.io/_uploads/HyQz3EcGR.png" height="150px"> <img src="https://hackmd.io/_uploads/BJ7MhV9MC.png" height="150px"> <img src="https://hackmd.io/_uploads/By7G3NcM0.png" height="150px"> <img src="https://hackmd.io/_uploads/BJmz3V9M0.png" height="150px"> </div> <br> The downside of this is that this will cost nf-core money to make the merchandise, and also many people may not be motivated to gather merchanidise.