## Welcome to the AIMOS Mini-ReproHack! <!-- Put the link to this slide here so people can follow --> <br> ### Event Page: ### https://reprohack.org/event/13/ Contains all event information and links to materials --- # Introductions <!-- Use this section as an ice-breaker. Introduce yourself, then allow others to go around the room and introduce themselves too. If online, consider using break-out rooms of 5-6 people --> --- <!-- Add details about yourself the organiser here: --> ## Who am I? ### Dr Anna Krystalli (@annakrystalli) - Research Software Engineer _University of Sheffield_ - 2019 Fellow _Software Sustainability Institute_ - Software Peer Review Editor _rOpenSci_ - _ReproHack_ Core Team member --- ## Why am I here? I believe there's lots to learn about Reproducibility from working with real published projects. --- <!-- Open it up to participants --> # Your turn - ## Who are you? - ## Why are you here? --- # Welcome Back --- ## ReproHack Objectives 1. **Practical Experience in Reproducibility** 3. **Feedback to Authors** 5. **Think more broadly about opportunities and challenges** --- # Plan of Action --- ### Reproduce paper 1. **Paper review and team formation** 2. **Select and register your paper on hackpad** 3. **Work on your paper!** 6. **Feedback at the end (group & authors)** --- # Tips for reviewing --- <!-- Remind participant of code of conduct and basic expectations. Bring attention to the additional considerations involved in giving feedback to authors --> ## Code of Conduct Event governed by [**ReproHack Code of Conduct**](https://reprohack.org/code-of-conduct) <br> ### Additional Considerations - #### Reproducibility is hard! - #### Submitting authors are incredibly brave! --- ## Thank you Authors! :raised_hands: - #### Without them there would be no ReproHack. - #### Show gratitude and appreciation for their effort and bravery. :pray: - #### Constructive criticism only please! --- # Reproduce and Review # :mag: --- ## Selecting Papers - **Author comments:** paper descroption and why you should choose to reproduce. - **Tags:** Tools, languages & domains - **No. attempts:** No. times reproduction has been attempted - **Mean Repro Score:** Mean reproducibility score (out of 10) - lower == harder! --- ## Review as an auditor :bookmark_tabs: ### Looking for FAIR principles - Findable - Accessible - Interoperable - Reusable --- ## Access - How easy was it to gain access to the materials? ## Installation - How easy / automated was installation? - Did you have any problems? --- ## Data - Were data clearly separated from code and other items? - Were large data files deposited in a trustworthy data repository and referred to using a persistent identifier? - Were data documented ...somehow... --- ## Documentation Was there adequate documentation describing: - how to install necessary software including non-standard dependencies? - how to use materials to reproduce the paper? - how to cite the materials, ideally in a form that can be copy and pasted? --- ## Analysis - Were you able to fully reproduce the paper? :white_check_mark: - How automated was the process of reproducing the paper? - How easy was it to link analysis code to: - the plots it generates - sections in the manuscript in which it is described --- ## Analysis ### If the analysis was not fully reproducible :no_entry_sign: - Did results (e.g. model outputs, tables, figures) differ to those published? By how much? - Were missing dependencies? - Was the computational environment not adequately described / captured? --- ## Review as a user :video_game: <br> #### What did you find easy / intuitive? #### What did you find confusing / difficult #### What did you enjoy? --- # Feed back # :speech_balloon: --- ## Feedback as a community member <br> #### Acknowledge author effort #### Give feedback in good faith #### Focus on community benefits and system level solutions --- # Let's go! :checkered_flag: --- ## 1. Paper review / Team formation + Have a look at the papers available for reproduction + Add your details to the **hackpad**. + Fine to work individually or tackle a papers as group --- ## 2. Project registration + Register your team and paper on the **hackpad** + Register for an account on the Hub. + Feel free to work here or in break-out rooms. --- ## 3. Reproduce Paper - Inspect the review form. - Try to reproduce the paper using the materials provided. - Record your responses to review questions according to your experiences. --- ## 4. Feedback to authors - **Please complete the feedback form for authors** - Feel free to record general findings the hackpad --- ## 5. Final regroup - So, how did you get on? - Final comments. - On hackpad: One thing you liked, one thing that can be improved. --- # Get involved! ### Visit ReproHack Hub <https://reprohack.org> - [**Submit a paper for review**](https://reprohack.org/paper/new/) - [**Organise your own event**](https://reprohack.org/event/new/) _Check out our [Resources](https://reprohack.org/resources) for more details_ ### Chat to us: [](https://reprohack-autoinvite.herokuapp.com/) --- ## THANK YOU ALL! :pray: - ### Thank you PARTICIPANTS for coming! - ### Thank you AUTHORS for submitting! --- # Resources - [**Example Compendium**](https://github.com/annakrystalli/rrcompendiumDTB): Demo Research compendium - [**The Turing Way**](https://the-turing-way.netlify.com/introduction/introduction): a lightly opinionated guide to reproducible data science. - [**Statistical Analyses and Reproducible Research**](): Gentleman and Temple Lang's introduction of the concept of Research Compendia - [**Packaging data analytical work reproducibly using R (and friends)**](https://peerj.com/preprints/3192/): how researchers can improve the reproducibility of their work using research compendia based on R packages and related tools --- # :wave: