---
# System prepended metadata

title: CODE CHECK

---

---
title: "CODE CHECK"
author:
- affiliation: Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, Cambridgeshire, GB
  corresponding: sje30@cam.ac.uk
  email: sje30@cam.ac.uk
  name: Stephen Eglen
  orcid: 0000-0001-8607-8025
- affiliation: Institute for Geoinformatics, University of M"unster, M"unster, Germany
  email: daniel.nuest@uni-muenster.de
  name: Daniel Nüst
  orcid: 0000-0002-0024-5046
date: "`r format(Sys.time(), '%d %B, %Y')`"
abstract: |
          | ...
#author_summary: |
#  TBD
bibliography: bibliography.bib
#output: rticles::plos_article
#csl: plos.csl
output: pdf_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

<!-- Let's stick to one sentence per line! -->
<!-- Citations: @Figueredo:2009dg, citep: [@Figueredo:2009dg], multiple citep: [@Figueredo:2009dg; @other_2018] -->
<!-- add " {-}" after headings to remove numbering -->

# Background

- role of data and code, data science
- reproducible resarch
- data enclaves
- CASCAID
- ROpenSci/PyOpenSci
- time capsules > https://twitter.com/DougBlank/status/1135904909663068165?s=09

**main contributions**

- a concept for integrating a minimal code review into common scholarly communication processes around peer review
- set of shared principles (recognition value) that allow researchers to quickly understand the level of evaluation across journals/publishers/conferences, and allows helps these stakeholders to establish workflow checking in their domain/solution/product
- a common language and rubric for classifying different levels of code and data evaluation as part of a peer-review

# What is CODE CHECK?

- **Principles**: https://codecheck.org.uk/
- relation to "proper" citation of data and software, and depositing of data and software in suitable repos _besides_ the bundle for CODE CHECK: do it, only provide the concrete analysis script for the check
- **Why is it useful?**
  - reduce the barrier to evaluating non-text parts of research
  - acknowledges different skill sets and existing high load on reviewers
  - CC breaks problem that publishers staff expertise today is not likely to suffice
  - engaging ECRs is a perfet fit (they are up-to-date in methods, are introduced to peer review)
  - relation to research compendia? point it out for authors unsure about how to structure their workflow
  - CrossRef and Publons and ORCID have features around reviews
  - shifting burden from reviewer to author (but not too much)
- keynote https://doi.org/10.7557/5.4963: in discussion, the speaker mentioned value of peer review for "baseline reassurance", i.e. a paper has at least been checked by someone with an understanding, to increase trust especially if looking at papers from other disciplines
- **report** must fulfil some requirements to make clear what is really checked and not mislead - readers will have very different understandings of a green checkmark!
- reproducibility is hard https://twitter.com/annakrystalli/status/1144176149859377152?ref_src=twsrc%5Etfw > code check gives at least some benefits

# Related work

- https://paperpile.com/shared/rVNwBS
- https://medium.com/bits-and-behavior/a-modern-vision-for-peer-review-d5f73f0fae07 - work by Amy J. Ko: https://faculty.washington.edu/ajko/publications
  - is CODE CHECK's process too small, too incremental a change?
- _Successes and struggles with computational reproducibility: Lessons from the Fragile Families Challenge_ > https://osf.io/preprints/socarxiv/g3pdb/
- community standards for software, cf. slide 6 in https://figshare.com/articles/_/5675104
  - "good enough practice" is "get a colleague to try using it", slide 19 in https://figshare.com/articles/_/5675104
- journals doing code reviews
  - _"We know of 16 journals that take this valuable step (ie TOP Level 3): AEA as already mentioned, plus others in polisci, psychology, biostats, and org chem: see Data Rep Policies here https://t.co/plOF8j6ADU "_ (https://twitter.com/EvoMellor/status/1202692360456589339?s=09); see also whole thread!
  - https://medium.com/@NeurIPSConf/call-for-papers-689294418f43
  - https://www.journals.elsevier.com/information-systems/editorial-board/
  - SIGMOD Reproducibility Review?
  - Biostatistics "AER" (https://academic.oup.com/biostatistics/article/10/3/405/293660)
- open science platforms/cloud services > https://www.nature.com/articles/d41586-019-03366-x
- live code in articles > o2r, eLife RDS https://www.nature.com/articles/d41586-019-00724-7, ...
- Artefact Evaluation - https://www.artifact-eval.org/about.html
- continuous analysis - https://www.nature.com/articles/nbt.3780
- "A Universal Identifier for Computational Results" > https://www.sciencedirect.com/science/article/pii/S1877050911001256
- Discussion about using Docker containers during JOSS reviews > https://github.com/openjournals/joss/issues/498#issuecomment-462046912
- https://scigen.report, an independent site for registering reproductions, via https://annakrystalli.me/talks/ro-reprohack.html#32
  - example https://scigen.report/reviews/get?doi=10.1063/1.1823034
  - seems to be closed infrastructure

# CODE CHECK implementation concepts

- need to get wording straight: process/implementation/workflow (DN: suggest to use "workflow" for research-analysis workflow)
- https://codecheck.org.uk/process/
- _dimensions_ along which a check can vary
- independent/pre-submission (Peer_Coding) by author + colleagues
- AGILE
- premier (OA) publisher
- community OA / scholarly society with public peer review
- invited reproducibility paper track, cf. https://www.elsevier.com/journals/information-systems/0306-4379/guide-for-authors
- on giving credit: public databases
  - https://www.reviewercredits.com/
  - Publons
  - ORCID
- compare with existing processes that are publicly documented
  - AJPS Verification Policy: https://ajps.org/ajps-verification-policy/
  - ...
- flexible level of detail? a check can be enhanced to (potentially semi-automatically) check for good practices
  - "A group of us in the @force11rescomm Software Citation Implementation Working Group produced a "Software Citation Checklist for Authors" https://t.co/4Noe66FsoX providing guidelines on how to cite software and describing why it is important and what software should be cited" via https://twitter.com/alegonbel/status/1207664922932453376?s=09)

# Annotated Examples
<!-- see examples.md -->

- **the default implementation**
- critically discuss our/the default implementation vs. the principles
- HPC example
- if we see the corpus of examles as a scientific dataset, it should be published in a citable way (maybe even submit to a data journal)

# Open problems

- need publishers build up data science experts who can conduct checks?
- integration of software/data citation checks benefitial?
- bot-support and tools (linter)
- Does nudging towards better practices work (highlight the successes) or is any check with opt-in doomed to fail?
- Do we need more than one codechecker? Is there a similar diversity as in a peer review where more opinions are needed?
- can CODE CHECK work for preprints? how? is a re-check required?
- risk: is the hurdle too low? are "levels" of checks are solution to increase transparency of check's level of detail
- code check in the context of innovations and disruptions in scientific publishing
  - https://doi.org/10.1042/ETLS20180172
  - https://doi.org/10.3390/publications7020034
- see https://github.com/codecheckers/discussion

# Acknowledgements

Collaborators and potential collaborators:

- eLife
- GigaScience
- AH, SciData
- Mozilla
- PLOS?

# Contribution

..
