# (AM Session) Foundational Open Science Skills (FOSS) Lesson 6: Repeatability I
:::info
**Date**: `2022-10-20`
**Today's Lead Instructor:** Michele
**Today's Helpers:** Michael Culshaw-Maurer, Tina
**Course Website:** https://cyverse-learning-materials.github.io/foss
**Zoom Link:** https://arizona.zoom.us/j/86152278453

**Instant Feedback:** (please complete before you leave class) [Complete Form](https://docs.google.com/forms/d/e/1FAIpQLSeVyEB8sU99Mn4IuzQ561Crp7v_wDl-yEcD2iutBxXRfrHo-Q/viewform?usp=sf_link)

:::

## :stopwatch: Agenda

### Warm-up:

#### Questions & Comments about Open Science left over from last week?

When trying to commit changes to github I realized I had to do some weird configuration by using a "token" as a password. But after that it was easy to use. This was on Cyverse.
- For longer-term use, I'd check this out to get SSH set up: https://swcarpentry.github.io/git-novice/07-github/index.html

---

## Discussion Q/A (Breakouts)

### Discussion Set 1

**How do you define reproducible science?**

Taking someone elses work (data, code, tool) and making it run correctly. That being said, there is not much incintive to 'reproduce' someone else's work in my research field.

In the best of scenarios, that youu have acess to the datset, and all the code and necessary information to reproduce the results (including software version and operative sytsem used)
. If the dataset is not available because in HIPAA protected, then being able to reproduce results with at least a simulated dataset.

Others can reproduce your results.

Describing methods in a way that will allow others to be able to recreate your findings; making code, data, and tools open and accessible and not closed behind a paywall or license

When my reviews come back and I have to change a figure, I can just run my code again.

When I can recreate the work done by others and vice versa.

Given a raw dataset, one is able to completely replicate results using a clearly defined pipeline.

Accessing datasets and all needed information to be able to go thorugh the process. Also some extra information if large space is needed.

When I do my code in a manner that others can run it, understand it. A good documentation (and a good code) are required.

### Discussion Set 2

**How do these definitions apply to your research/teaching?**

I keep track of all the analysis using RMarkdown documents

I try to leave myself a lot of comments in my code so that I can remember what I'm doing when I come back to a project or if it needs to be shared with a collaborator

I do my analysis in RMarkdown notebooks so I can re-run them. I also have scripts for heavier-duty data processing, but I'd like for that step to be more reproducible as well.

I also write reports in RMarkdown... also lots of comments in my code...

Right now, I'm very much focused on making things reproducible for my future self and for other lab members. So I use MD files to keep track of the pipeline I create/use with detailed annotations for the logic and reasoning for using specifice types of analyses, methods, etc. I also annotate where I am getting my methodology, where the data is currently stored... kind of like a lab notebook.

I dont...

Never used markdown :)
The culture of 'reproduce' seems to be in its infancy in my fields (remote sensing for rangelands)
I am still new to coding (Python) so lots of extra comments, links, contact info etc. in the work to reference to, mostly for myself.

### Discussion Set 3

**Have you ever had any hurdles to reproducing your work?**

- Have you ever run into a problem that prevented you from generating the same results, figures, analyses as before?

software versions!! +1+1
What did I do about it? Update my code.
interoperability in general. This is why I started using docker for everything

All the time, when I am trying to replicate someone else's methodology that I found in a paper

Yes, problems to identify last verions of my code (not doing version control :-( )

Having to spend time figuring out if a package has updated, or if something has changed about the way data gets read into a package, or if there is a problem on my end in the way that data are saved or formatted; spending time reading through issues on github to see if someone has already run into the same problem i'm experiencing

Keeping track of methods used

- Have you ever lost time trying to figure out how you (or a collaborator) got a particular result?

Constantly
Yes, although getting better
Many times this has happened.

I do collaborate with others on work (mainly text-based, not much code usually), so when I *do* run into issues with trying to make sense of someone else's code I always take forever

- What were the issues you ran into, and how might you have solved them?
Forgetting to download results from HPC in time...need to have a data management plan

Any type of documentation would have been helpful, sometimes I am given a figure and some data and asked do re-do someone else's analysis to make better figures. I'm put on a lot of projects to provide bioinformatic support and most of my time is trying to figure out 1) what they want and 2) whay they have been doing.

If I am using a method with paramenter settings I will take a screen capture of the setting used. Very inefficient but...

Ending with several pieces of test code, and not being well organized. Also, updating a package or tried to use a package that was not anymore available

Trying to sort out *where* the data that collaborators used to generate a figure in a shared markdown file lives.

### Discussion Set 4

**What are some tasks you have automated or want to automate?**

- Have you ever successfully automated a task?
I wrote some software to run sequences of GIS tasks that used different software

This is something I struggle with in the sense that I do a lot of copying/pasting of old code to do things like plotting. So instead of writing a function to create a bunch of similar looking plots i tend to create a big piece of code for the 1st plot and the copy + modify to make the subsequent plots
-> I started writing plotting functions because this was happening, instead of copy-and-paste.

Photogrammetry processing of drone imagery (python scripts using Software API)

Basedata type products from LiDAR (USGS/Drone) for geospatial use would be nice to have with an automated system.

I've never written software, only bash scripts with a series of nested sctipts.

- Found a way to make something scale or take less time?
Yes, I found Earth Lab and ESIIL! At least partnering with them to help with geospatial (LiDAR) issues with EDS instead of old school GIS methods.

Yes, parallelizing a script on the HPC which made things run 23 times faster

- What was the task, and how did you do it?

I use caching functions all the time, so I'm constantly saving my data at different stages so I don't have to run the entire notebook while debugging.
- Are there any things you wish you could automate?
I would like to be able to automate submitting tasks to HPC - especially as we haven't been able to figure out how to use more than one node at a time so then the task has to be restarted multiple times... +1

I wish and strive to automate everying

- What are some barriers to automating them?

Loops/functions/conditional statements scare me ;P

When automating tasks that take a long time, it's hard to debug and we often decide it's faster to do it manually.
Technical chops to implement it. +1

Short duration projects. The choice is always 1. do it manually and finish it quickly, or 2. try to automate which will take much longer to figure out.

Having to work in different environments or platforms due to starting with incredibly large files (HPC) and then also working with PHI or sensitive information such as genetic sequences and electronic health records.

---

### Tutorial

::: info
Link to VM Spreadsheet:

https://docs.google.com/spreadsheets/d/1ocg2g-uLgEB5jtBw85tSIhc0WpZjcyd15ibwDWTeb70/edit?usp=sharing

**Please put your name next to the VM you self assign.**
:::

::: warning

These machines will be shut off during non-FOSS hours. **Please let us know if you'd like to use it so that we can turn them on for you!**
:::

### Links shared in class
https://osf.io/preprints/metaarxiv/cd5j9/

https://journals.sagepub.com/doi/10.1177/2515245917747646

[Ross-Ibarra Lab Expectations document](https://rilab.ucdavis.edu/expectations.html)

[Iteration in R workshop](https://www.youtube.com/watch?v=pd8CLdI6kqM) (featuring Michael with a mustache)

[Customize an R/RStudio Docker image](https://github.com/MCMaurer/custom_rocker_example)