## Welcome to the N8 CIR ReproHack Series! <!-- Put the link to this slide here so people can follow --> <br> ### Event Repository: ### [bit.ly/n8-reprohacks](http://bit.ly/n8-reprohacks) Contains all event information and links to materials --- ### House Keeping: - No Fire Alarm test today - Toilets & Fire Exit - Water point - downstairs --- # Introductions <!-- Use this section as an ice-breaker. Introduce yourself, then allow others to go around the room and introduce themselves too --> --- <!-- Add details about yourself the organiser here: --> ## Who am I? ### Dr Anna Krystalli (@annakrystalli) - Research Software Engineer _University of Sheffield_ - 2019 Fellow _Software Sustainability Institute_ - Software Peer Review Editor _rOpenSci_ - Co-organiser _Sheffield R Users Group_ --- ## Why am I here? I believe there's lots to learn about Reproducibility from working with real published projects. --- ## Who is my favorite animated character? Stitch!  --- <!-- Open it up to participants --> ## Who are you? ## Why are you here? ## Who is your favorite animated character? --- # Welcome Back ## Introduction & tips for reviewing --- ## ReproHack Objectives 1. **Practical Experience in Reproducibility** 3. **Feedback to Authors** 5. **Think more broadly about opportunities and challenges** --- # Plan of Action --- ### Reproduce paper 1. **Project review and team formation** 2. **Select and register your project** 3. **Work on your project!** 4. **Re-group part-way through.** 6. **Feedback at the end (group & authors)** --- ## Materials - [**Repository**](https://github.com/annakrystalli/n8cir-reprohacks) Instructions and links - [**Paper list**](https://sheffield-university.shinyapps.io/n8cir-reprohacks/) List of submitted papers - [**Hackpad**](https://hackmd.io/6nhNpD0VTHaCLuXPBy-dFw?both#Leeds) Event collaborative notebook - [**Feedback form**](https://forms.gle/wvXfjNT7HbkqH2Rj7) Record feedback for authors --- ## Code of Conduct Event governed by [**ReproHack Code of Conduct**](https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html) <br> ### Additional Considerations - #### Reproducibility is hard! - #### Submitting authors are incredibly brave! --- ## Thank you Authors! :raised_hands: - #### Without them there would be no ReproHack. - #### Show gratitude and appreciation for their effort and bravery. :pray: - #### Constructive criticism only please! --- # Reproduce and Review # :mag: --- ## Selecting Papers - **No. attempts:** No. times reproduction has been attempted - **Mean Repro Score:** Mean reproducibility score (out of 10) - lower == harder! --- ## Review as an auditor :bookmark_tabs: --- ## Access - How easy was it to gain access to the materials? ## Installation - How easy / automated was installation? - Did you have any problems? --- ## Data - Were data clearly separated from code and other items? - Were large data files deposited in a trustworthy data repository and referred to using a persistent identifier? - Were data documented ...somehow... --- ## Documentation Was there adequate documentation describing: - how to install necessary software including non-standard dependencies? - how to use materials to reproduce the paper? - how to cite the materials, ideally in a form that can be copy and pasted? --- ## Analysis - Were you able to fully reproduce the paper? :white_check_mark: - How automated was the process of reproducing the paper? - How easy was it to link analysis code to: - the plots it generates - sections in the manuscript in which it is described --- ## Analysis ### If the analysis was not fully reproducible :no_entry_sign: - Did results (e.g. model outputs, tables, figures) differ to those published? By how much? - Were missing dependencies? - Was the computational environment not adequately described / captured? --- ## Review as a user :video_game: <br> #### What did you find easy / intuitive? #### What did you find confusing / difficult #### What did you enjoy? --- # Feed back # :speech_balloon: --- ## Feedback as a community member <br> #### Acknowledge author effort #### Give feedback in good faith #### Focus on community benefits and system level solutions --- # Additional activities --- ## Finished early? ### Explore the work more deeply: - Try and run additional analyses. - Create new plots. - Combine materials with your own or other open materials available on the web! ### Replicate your paper - Consider attempting **replication**! - Replications could be considered for publication in **ReScience C** Journal --- ## [ReScience C](http://rescience.github.io/) > **ReScience C** is an open-access peer-reviewed journal that targets computational research and encourages the **explicit replication** of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible. --- # Resources - [**Example Compendium**](https://github.com/annakrystalli/rrcompendiumDTB): Demo Research compendium - [**The Turing Way**](https://the-turing-way.netlify.com/introduction/introduction): a lightly opinionated guide to reproducible data science. - [**Statistical Analyses and Reproducible Research**](): Gentleman and Temple Lang's introduction of the concept of Research Compendia - [**Packaging data analytical work reproducibly using R (and friends)**](https://peerj.com/preprints/3192/): how researchers can improve the reproducibility of their work using research compendia based on R packages and related tools --- # Let's go! :checkered_flag: --- ## 1. Paper review + Have a look at the papers available for reproduction ## 2. Team formation / project registration + Fine to work individually + Add your details to the [**hackpad**](https://hackmd.io/6nhNpD0VTHaCLuXPBy-dFw?both#Leeds). + Register your team and paper on the [**hackpad**](https://hackmd.io/6nhNpD0VTHaCLuXPBy-dFw?both#Leeds) --- ## 3. Mid-point regroup - Which paper have you selected? Briefly describe what it's about. - Briefly describe the approach to reproducibility the paper has taken. - Anything in particular you like about the paper's approach so far? - Anything you're having difficulty with? --- ## 4. Feedback to authors - **Please complete the feedback form for authors** - Feel free to record general findings the hackpad --- ## 5. Final regroup - So, how did you get on? - Final comments. - If there's time, tackle some discussion topics (see hackpad). - On post-its: One thing you liked, one thing that can be improved. --- # Get involved! ### [reprohack-hq](https://github.com/reprohack/reprohack-hq) repository #### Check out our [issues](https://github.com/reprohack/reprohack-hq/issues) #### Chat to us: [](https://reprohack-autoinvite.herokuapp.com/) --- # THANK YOU ALL! :pray: - ### Thank you PARTICIPANTS for coming! - ### Thank you AUTHORS for submitting! - ### Thank to the N8 CIR for sponsoring! # :wave: