ReproHack Hub Launch - Collaborative Hackpad
===

###### tags: `Reprohack` `hackpad`

:::info
- :earth_africa: **Remote**
- :calendar: **2021-11-18**
- :watch: **10:00 - 17:00 UTC**
- :arrow_forward: **Slides:** https://bit.ly/reprohack-hub-launch-slides
- :page_with_curl: **Paper List:**
- Central List: https://reprohack.org/paper/
- Event List: https://reprohack.org/event/11/
- :scroll: **Participant Guidelines:** https://reprohack.org/participant_guidelines
- :purple_heart: **Code of Conduct:** https://reprohack.org/code-of-conduct
- :left_speech_bubble: **Chat to us on Slack:** https://reprohack-autoinvite.herokuapp.com/
- :Bird: **Twitter:** #ReproHackHub @ReproHack
- **hackmd docs:** https://hackmd.io/s/features

***

## Participants:
**Please sign in below:**
- ***Jane Doe (pronouns/ Affiliation / twitter / etc)***
- Anna Krystalli, (she/her), University of Sheffield, @annakrystalli
- Linda Nab, (she/her), Leiden University Medical Center, @lindanab1
- Harry Chown, (he/him), University of Manchester, [e-mail]
- Sander van Rijn, (he/him), Leiden university, @sjvrijn, [e-mail]
- Robert (Bob) Turner (he/him), University of Sheffield, @bobatron, [e-mail]
- Ilja Kocken (he/him), PhD candidate/Data Steward Earth Sciences at Utrecht University, the Netherlands, [@Japhir](https://twitter.com/japhir), [e-mail]
- Paul Richmond (he/him), University of Sheffield, @gpu_mondus, [e-mail]
- Esther Plomp (she/her), Delft University of Technology, @PhDtoothFAIRy
- Igors (Igor) Dubanevics, (he/him), Research Assistant, University of York, @igordub, [e-mail]
- Alessandro Gasparini (he/him), Karolinska Institutet, [@ellessenne](https://twitter.com/ellessenne), [e-mail]
- Elisavet (Betty) Syriopoulou (she/her), Karolinska Institutet, @betty_syriop, [e-mail].
- Yuliya Leontyeva (she/her), Karolinska Institutet,PhD, [e-mail]
- Sheeba Samuel (she/her), University of Jena, Germany, @sheebasamuel
- Carlos Cámara-Menoyo, RSE at [Centre for Interdisciplinary Methodologies](https://warwick.ac.uk/fac/cross_fac/cim/), [University of Warwick](https://warwick.ac.uk/), :earth_africa: https://carloscamara.es/en, :bird: [@drccamara](https://twitter.com/drccamara) :elephant: [@[e-mail]](https://scholar.social/@ccamara)
- Kevin Lang, RDM Contact Point, Bauhaus-Universität Weimar, Germany, @kev_lan
- Joe Heffer, The University of Sheffield, @joe_heffer

---

# Icebreaker Room Names
- **Group 1:** Fire exits :fire_engine:
- **Group 2: Alps**
- **Group 3:** **Mars** :stars:
- **Group 4:** Fika :coffee::cake:
- **Group 5:** :sheep:
- **Group 6:**
- **Group 7:**
- **Group 8:**
- **Group 9:**
- **Group 10:**

***
## Agenda
**10:00 - Welcome and Orientation
10:10 - Ice breaker session in groups
10:20 - TALK Dr Esther Plomp: _The Turing Way Community_.
10:45 - Tips and Tricks for Reproducing and Reviewing.
11:10 - Select Papers, Chat and :coffee:
11:30 - Round I of ReproHacking (break-out rooms)
12:30 - Re-group and sharing of experiences
12:45 - LUNCH :pizza: :stew: :strawberry:
13:45 - Round II of ReproHacking (break-out rooms)
14:45 - Coffee break :coffee:
15:00 - Round III of ReproHacking (break-out rooms) - Complete Feedback form
16:00 - TALK Prof Stephen Eglen: _Code Check_.
16:25 - Re-group and sharing of experiences
16:50 - Feedback and Closing**

:::

# Plan of Action
## Welcome and Introduction

- Into Slides (https://zenodo.org/record/5675926)

:computer: Form teams
---
Feel free to tackle papers individually or as teams.

:dart: Select papers
---
- Choose paper from list of proposed papers

- Register the paper selected and the participants reproducing below. You can copy, paste and edit the following template:

### **Paper:** <Title of the paper reproduced>
**Reviewers:** Reviewer 1, Reviewer 2 etc.

### **Paper:** [Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study](https://www.reprohack.org/paper/46/)
**Reviewers:** Carlos Cámara-Menoyo, Elisavet Syriopoulou, Alessandro Gasparini, Yuliya Leontyeva.

_We're hanging out in the "Fika" room!_

### **Paper:** [The Viewing Angle In AGN SED Models, A Data-Driven Analysis](https://www.reprohack.org/paper/51/)
**Reviewers:** Igors Dubanevics, Sheeba Samuel, Ilja Kocken and Harry Chown
**Room:** Mars :space_invader:

### **Paper:** [A multiscale Bayesian inference approach to analyzing subdiffusion in particle trajectories](https://www.reprohack.org/paper/5/)
**Reviewers:** Sander van Rijn
**Room:** Sheep :sheep:
**Review:** https://www.reprohack.org/review/53/

## Questions that are asked in the reviewer portal of ReproHack:
### reproducibility
1. Did you manage to reproduce it?
2. How much of the paper did you manage to reproduce?
3. Briefly describe the procedure followed/tools used to reproduce it.
4. Briefly describe your familiarity with the procedure/tools used by the paper.
### operating system
5. Operating System -- Which type of operating system were you working in?
6. What additional software did you need to install?
7. What software did you use?
8. What were the main challenges you ran into (if any)?
9. What were the positive features of this approach?
10. Any other comments/suggestions on the reproducibility approach?
### documentation
11. How well was the material documented? /10
12. How could the documentation be improved?
13. What do you like about the documentation?
14. After attempting to reproduce, how familiar do you feel with the code and methods used in the paper? /10
15. Any suggestions on how the analysis could be made more transparent?
### reusability
16. Rate the project on reusability of the material.
17. Are materials clearly covered by a permissive enough license to build on?
- [ ] Permissive license for DATA included
- [ ] Permissive license for CODE included
18. Any suggestions on how the project could be more reusable?
19. Any final comments?

### Answers for the Paper: [The Viewing Angle In AGN SED Models, A Data-Driven Analysis](https://www.reprohack.org/paper/51/):
(putting it here for easy collaboration)

:confetti_ball: This review is live at: https://www.reprohack.org/review/51/! :exclamation:

1. *Did you manage to reproduce it?*
- partially reproducible
2. *How much of the paper did you manage to reproduce?*
- 9/10
3. *Briefly describe the procedure followed/tools used to reproduce it.*
- First we ran the code via the included binder. Then we tried to delete rawdata and download them and re-create figures from scratch. We also tried to install the package locally.
4. *Briefly describe your familiarity with the procedure/tools used by the paper.*
- Different for each reviewer:
- Ilja's not familiar with python or jupyter. Transferrable skills from R, org-mode, GNU/Linux.
- Harry: Faniliar with python, a little with jupyter, zero experience with physics modules
- Sheeba familiar with python, Jupyter, and Binder but zero experience with physics modules
- Igor: familiar with Python and Jupyter notebook, but not Binder and used physics Python packages.
5. *Operating System -- Which type of operating system were you working in?*
- Ilja's on Arch Linux x86_64 5.14.16, Harry's on Ubuntu, Igor's on Ubuntu (WSL2), Sheeba's on Ubuntu.
6. *What additional software did you need to install?*
- To run locally, we needed to install:
- git
- jupyterlab
- anaconda
- and then install the environment packages with `conda env create -f environment.yml`
7. *What software did you use?*
- See above.
8. *What were the main challenges you ran into (if any)?*
- Notebook 5 is intentionally missing, but this was unclear from the binder. Lack of master script linking notebook code. Unable to reproduce data without raw (deleted original data and tried to reproduce from pulldown)
9. *What were the positive features of this approach?*
- Great use of the binder framework for sharing the code and documentation, we loved the pre-rendered html files. Setting it up as a binder that doesn't require the user to install anything locally was a great way to make the analysis quickly reproducible for everyone! The fact that the files were neatly split into separate analysis chunks was useful, and the step-by-step file provided a great overview linking everything together.
10. We didn't find the binder button immediately, so it was not obvious that we could inspect everything in an online pre-configured environment. There were no guidelines on how to run the analysis locally. Which commands do you need to run to install the dependencies via the `envornment.yml` file? How do I start the jupyter lab after installing the dependencies locally?
11. *docs* 8/10
12. *docs improvements* Figuring out which codebook does what, and why. The README nicely describes all the tools that were used, but did not include a big-picture description of what problem the project is solving. The step-by-step was a nice file, but lacked the more general introduction as well.
- It would be nice to 0-pad the notebook filenames so that they are sorted correctly by default.
- Adding a master file that runs the different codebooks would be nice.
- It could perhaps be merged with the step-by-step file.
- Furthermore, this would make it more clear that the code for step 5 is not in this repo. It would also be possible to add it to the repo, but set all the code chunks to not evaluate, or to provide a warning that running this part of the analysis requires HPC.
- Each notebook could provide more details on the goal and background of the notebook.
- It was not immediately clear how to run the environment (we found the binder button only after some time). The hyperlink in the text points to binder's homepage.
- We enjoyed the step-by-step document, the way the analysis is split into separate notebooks with a single task. The use of autopep8 in the notebooks for formatting the code is good.
- We like the inclusion of rendered html files and intermediate output, which allowed us to quickly inspect everything without needing to run the chunks.
13. *docs likes*: Documentation was good, they've shown how each script should be pieced together with a brief description of each step
14. How familiar are we after playing around? 7/10
15. *analysis more transparent:* Very transparent paper/code, no corners have been cut or hidden from the public view (except the HPC code)
16. *reusability score* 2/10
17. license permissible enough for code + data (checkboxes) both yes
18. *suggestions to make project more reusable*:
- Subdirectories inside `Data/{Raw,Interim,Final,Complementary}` directories, e.g. `Data/Interim/CIGALEOutputs`, must be created while the sripts are running, in case user deletes them.
- Hard-coded filenames with dates Notebook 2_Clean_Sample.ipynb: Cell 3
- Some galaxies are empty causing value errors, if-statements should be applied to prevent this Notebook 2_Clean_Sample.ipynb: Cell 15
- Switch to [semantic versioning](https://semver.org/spec/v2.0.0.html)
- Add a [Changelog](https://keepachangelog.com/en/1.0.0/)
20. *final comments*: We'd like to thank the authors for putting so much effort into making their workflow reproducible! They went above and beyond, and the comments above are mostly nitpicks that prevent the workflow from being fully reproducible from scratch.

:books: Reproduce
---
- Attempt to reproduce papers from available materials and documentation
- Make notes about your experiences, in particular with respect to how easy it is to:
- :earth_africa: navigate the materials
- :repeat: reproduce the analysis
- :recycle: reuse the materials

:memo: Feedback to authors
--
<br>

---

# Questions for Speaker(s)
<!-- Please add any questions for our speaker below -->

## Questions for Stephen Eglen
Who is going to do the work of code-checking projects? Will this be another burden placed on scientists?

What are your thoughts on researchers who still do not use code, but rather use e.g. excel files or SPSS or other proprietary software?

# Paper Registration

_Use the following template (also provided in "Plan of Action") and register your review below._

### **Paper:** [Title of the paper reproduced]
**Reviewers:** Reviewer 1, Reviewer 2 etc.

<!-- Please register your paper in the space below using the template provided -->

***

# Discussion topics

<!-- Feel free to use this space to add topics for discussion and make notes on those discussions. You can remove if you don't plan to have any guided discussion sessions -->

***

# Regroup notes
<!-- Any other notes you'd like to add. -->

***

# Feedback

## :green_book: One thing you enjoyed:

- I enjoyed spending some time away from my research to collaborate with different people from around the world to help work on these papers. I've learnt a lot from the different people within my group
-
-

## :red_circle: One thing that could be improved:

- The markdown for the hub was difficult to write into. Maybe there could be a "notes" style page for each group
-
-
-