House Keeping:

• No Fire Alarm test today
• Toilets & Fire Exit
• Water point - downstairs

Introductions

Who am I?

Dr Anna Krystalli (@annakrystalli)

• Research Software Engineer University of Sheffield
• 2019 Fellow Software Sustainability Institute
• Software Peer Review Editor rOpenSci
• Co-organiser Sheffield R Users Group

Why am I here?

I believe there's lots to learn about Reproducibility from working with real published projects.

Who is my favorite animated character?

Stitch!

Who are you?

Why are you here?

Who is your favorite animated character?

Welcome Back

Introduction & tips for reviewing

ReproHack Objectives

1. Practical Experience in Reproducibility

2. Feedback to Authors

3. Think more broadly about opportunities and challenges

Plan of Action

Reproduce paper

1. Project review and team formation

2. Select and register your project

3. Work on your project!

4. Re-group part-way through.

5. Feedback at the end (group & authors)

Materials

• Repository Instructions and links
• Paper list List of submitted papers
• Hackpad Event collaborative notebook
• Feedback form Record feedback for authors

Code of Conduct

Event governed by ReproHack Code of Conduct

Additional Considerations

• Reproducibility is hard!

• Submitting authors are incredibly brave!

Thank you Authors!

• Without them there would be no ReproHack.

• Show gratitude and appreciation for their effort and bravery.

• Constructive criticism only please!

Reproduce and Review

Selecting Papers

• No. attempts: No. times reproduction has been attempted
• Mean Repro Score: Mean reproducibility score (out of 10)
  • lower == harder!

Review as an auditor

Access

• How easy was it to gain access to the materials?

Installation

• How easy / automated was installation?
• Did you have any problems?

Data

• Were data clearly separated from code and other items?
• Were large data files deposited in a trustworthy data repository and referred to using a persistent identifier?
• Were data documented …somehow…

Documentation

Was there adequate documentation describing:

• how to install necessary software including non-standard dependencies?
• how to use materials to reproduce the paper?
• how to cite the materials, ideally in a form that can be copy and pasted?

Analysis

• Were you able to fully reproduce the paper?
• How automated was the process of reproducing the paper?
• How easy was it to link analysis code to:
  • the plots it generates
  • sections in the manuscript in which it is described

Analysis

If the analysis was not fully reproducible

• Did results (e.g. model outputs, tables, figures) differ to those published? By how much?
• Were missing dependencies?
• Was the computational environment not adequately described / captured?

Review as a user

What did you find easy / intuitive?

What did you find confusing / difficult

What did you enjoy?

Feed back

Feedback as a community member

Acknowledge author effort

Give feedback in good faith

Focus on community benefits and system level solutions

Additional activities

Finished early?

Explore the work more deeply:

• Try and run additional analyses.
• Create new plots.
• Combine materials with your own or other open materials available on the web!

Replicate your paper

• Consider attempting replication!
• Replications could be considered for publication in ReScience C Journal

ReScience C

ReScience C is an open-access peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible.

Resources

• Example Compendium: Demo Research compendium
• The Turing Way: a lightly opinionated guide to reproducible data science.
• Statistical Analyses and Reproducible Research: Gentleman and Temple Lang's introduction of the concept of Research Compendia
• Packaging data analytical work reproducibly using R (and friends): how researchers can improve the reproducibility of their work using research compendia based on R packages and related tools

Let's go!

1. Paper review

• Have a look at the papers available for reproduction

2. Team formation / project registration

• Fine to work individually
• Add your details to the hackpad.
• Register your team and paper on the hackpad

3. Mid-point regroup

• Which paper have you selected? Briefly describe what it's about.
• Briefly describe the approach to reproducibility the paper has taken.
• Anything in particular you like about the paper's approach so far?
• Anything you're having difficulty with?

4. Feedback to authors

• Please complete the feedback form for authors
• Feel free to record general findings the hackpad

5. Final regroup

• So, how did you get on?
• Final comments.
• If there's time, tackle some discussion topics (see hackpad).
• On post-its: One thing you liked, one thing that can be improved.

Get involved!

reprohack-hq repository

Check out our issues

Chat to us:

THANK YOU ALL!

• Thank you PARTICIPANTS for coming!

• Thank you AUTHORS for submitting!

• Thank to the N8 CIR for sponsoring!