## Welcome to the ICCS ReproHack! <br> <!-- Put the link to the event page here so people can access all materials --> ### Event Page: ### http://reprohack.org/event/27 ### HackPad: ### https://hackmd.io/A2XgwVyFQDG4TxeD8jn5HQ Contains all event information and links --- <!-- Add any housekeeping details here. If running an online event, you might want to include an introduction to the platform here --> ### Welcome and Why a ReproHack? - The Importance of Reproducibility - Is there a Reproducibility Crisis? - https://www.nature.com/articles/533452a - Why a ReproHack? --- ### Institute of Computing for Climate Science - Schmidt Futures - Virtual Institute for Scientific Software (VISS) - Virtual Earth System Research Institute (VESRI) --- ### House Keeping: - We are using Zoom, Slack and HackMD - Feel free to step away from your computer. - You can message us (Paul, Tom, Marion, Christian) privately anytime if you do not feel comfortable to ask a question openly - We will have two coffee breaks - Add your name and affiliation to the Hackpad --- ### Photo Permission: - We will take photos/screenshots during the event which we might use on social media and in our newsletter to post about this event, and potentially advertise future events - Please list your name under "photo consent" in the hackpad if you want us to NOT include you/edit out your face. --- <!-- Remind participant of code of conduct and basic expectations. Bring attention to the additional considerations involved in giving feedback to authors --> ## Code of Conduct Event governed by [**ReproHack Code of Conduct**](https://reprohack.org/code-of-conduct) --- ### Agenda part 1 - 13:00 Welcome and Orientation - 13:10 Ice breaker session - 13:40 **Alejandro Coca-Castro: The Turing Way** - 14:00 Tips & Tricks for ReproHacking - 14:10 Select papers, chat and coffee - 14:30 Round I of ReproHacking --- ### Agenda part 2 - 15:30 Re-group and sharing of experiences - 15:50 Coffee break - 16:15 Round II of ReproHacking - 17:15 Re-group and sharing of experiences - 17:45 Feedback and Closing - 18:00 End --- # Introductions <!-- Use this section as an ice-breaker. Introduce yourself, then allow others to go around the room and introduce themselves too. If online, consider using break-out rooms of 5-6 people --> --- <!-- Add details about yourself the organiser in the next few slides: --> ## Who am I? ### Christian Fernandez Perotti (@christianfernandez) - MPHil in Digital Humanities, _University of Cambridge_ - Programme Manager, _Institute of Computing for Climate Science (ICCS)_ - Research Assistant, _EDIP Project, Faculty of Education, University of Cambridge_ --- ## Why am I here? <!-- Edit with your own answer --> > I believe that reproducibility is an essential quality standard for published research, and ICCS is committed to advocate for reproducibility in computing for climate science. --- ## Who is my favorite animated character? <!-- Edit with your own answer --> Pucho  <!-- Let the rest of the organisers do the same --> --- <!-- Open it up to participants --> - ### Who are you? - ### Why are you here? - ### Who is your favorite animated character? --- ## Talk **Alejandro Coca-Castro: *The Turing Way*** --- ## ReproHack Objectives 1. **Practical Experience in Reproducibility** - Learn the Do's and Don'ts for your own research/software - Learn from others how to tackle other people's software 3. **Feedback to Authors** 5. **Think more broadly about opportunities and challenges** --- # Plan of Action 1. **Paper review and team formation** 2. **Select and register your paper on hackpad** 3. **Work on your paper!** 4. **Re-group part-way through.** 5. **Feedback at the end (group & authors)** --- # Tips for reviewing --- ## Remember: - #### This is not a competition! - #### Team work is key! - #### Reproducibility is hard! - #### Submitting authors are incredibly brave! --- ## Thank you Authors! :raised_hands: - #### Without them there would be no ReproHack. - #### They are doing the right thing. - #### Show gratitude and appreciation for their effort and bravery. :pray: - #### Constructive criticism only please! --- # Reproduce and Review # :mag: --- <!-- ## Selecting Papers - **Author comments:** paper description and why you should choose to reproduce. - **Tags:** Tools, languages & domains - **No. attempts:** No. times reproduction has been attempted - **Mean Repro Score:** Mean reproducibility score (out of 10) - lower == harder! --> --- ## Review as an auditor :bookmark_tabs: ### Looking for FAIR principles - Findable - Accessible - Interoperable - Reusable --- ## Findability and Access - Do software and data have a DOI or other persistent identifier? - How easy was it to gain access to the materials? --- ## Installation and Running - How easy / automated was installation? - How easy was it to find out how to run the software? - Did you have any problems? --- ## Data - Were data clearly separated from code and other items? - Were large data files deposited in a trustworthy data repository and referred to using a persistent identifier? - Were data documented ...somehow... --- ## Documentation Was there adequate documentation describing: - how to install necessary software including non-standard dependencies? - how to use materials to reproduce the paper? - how to cite the materials, ideally in a form that can be copy and pasted? --- ## Analysis - Were you able to fully reproduce the paper? :white_check_mark: - How automated was the process of reproducing the paper? - How easy was it to link analysis code to: -- the plots it generates -- sections in the manuscript in which it is described --- ## Analysis ### If the analysis was not fully reproducible :no_entry_sign: - Did results (e.g. model outputs, tables, figures) differ from those published? By how much? - Were there missing dependencies? - Was the computational environment not adequately described / captured? --- ## Review as a user :video_game: <br> #### What did you find easy / intuitive? #### What did you find confusing / difficult #### What did you enjoy? --- # Feedback # :speech_balloon: --- ## Feedback as a community member <br> #### Acknowledge author effort #### Give feedback in good faith #### Focus on community benefits and system level solutions --- ## Finished early? - Explore the work more deeply: -- Try and run additional analyses. -- Create new plots. -- Combine materials with your own or other open materials available on the web! ......**or** - Pick another paper / join another group --- ## Todos during coffee --- ### Paper review / Team formation + Have a look at the papers available for reproduction + Add your details to the **hackpad**. + Fine to work individually or tackle a papers as group --- ## Todos after coffee --- ### Project registration + Register your team and paper on the **hackpad**. + Register for an account on the Hub. + Feel free to work here or in break-out rooms. --- ## Reproduce Paper - Inspect the review form. - Try to reproduce the paper using the materials provided. - Record your responses to review questions according to your experiences. --- # Coffee & Paper Selection --- # Welcome Back --- # Let's go! :checkered_flag: **Please find a room and start hacking!** --- ## Mid-point regroup - Which paper have you selected? Briefly describe what it's about. - Briefly describe the approach to reproducibility the paper has taken. - Anything in particular you like about the paper's approach so far? - Anything you're having difficulty with? --- ## Feedback to authors - **Please complete the feedback form for authors** - Feel free to record general findings and discussion points in the hackpad --- # Coffee break --- # ReproHacking continued... --- ## Final regroup - How did you get on? - Final comments. - Discussion topics from hackpad. - One thing you liked, one thing that can be improved. --- # Get involved! ### Visit ReproHack Hub <https://reprohack.org> - [**Submit a paper for review**](https://reprohack.org/paper/new/) - [**Organise your own event**](https://reprohack.org/event/new/) _Check out our [Resources](https://reprohack.org/resources) for more details_ ### Chat to us: [](https://reprohack-autoinvite.herokuapp.com/) --- # THANK YOU ALL! :pray: - #### Thank you PARTICIPANTS for coming! - #### Thank you AUTHORS for submitting! - #### Thank you Alejandro for your talk! # :wave: