USDA Long Term Agroecosystem Research Network

Data Carpentry

22-24 Aug 2018, CSU Morgan Library

Workshop website: https://btmoyers.github.io/2018-08-22-USDA/

Welcome!

This is a document that any and all of us can use to take notes and share difficult to type information (e.g. links). To start editing, click on the icons above that look either like a pencil (editing only) or a book (both editing and viewing). You can just start typing anywhere in the dark panel.

Apologies for the issue with sign-ins! The permissions have been reset so that you can edit without signing in.

Names

• Pat Nash
• Bryan Carlson
• Abby Jackson
• Gerardo Armendariz
• Mara Sedlins
• Stephen Teet
• Laura O'Gan
• Greg Sonnier
• Nicole Kaplan
• Bruce Vandenberg
• Kevin Cole
• Dan Arthur
• Jane Okalebo
• Kerrie Geil
• Lindsey Yasarer
• Steve Van Vactor
• Steve Wagner
• Jennifer Carter
• Earl Keel

What is tidy data?

1. Each column is one variable.
2. Each row is one observation.
3. Each cell contains one data type.
4. Each table contains one type of observation.

To learn more about tidy data, check out this great paper.

22 August AM

OpenRefine

OpenRefine is an open source tool for data cleaning. It can be very useful for dealing with typos, whitespace, and other common issues with data cleaning. Even better: it does all of its work without modifying the original data file, which is ideal for data management. Here is the lesson and here are setup instructions.

Exercise 1

1. Using faceting, find out how many years are represented in the census.

2. Is the column formatted as Number, Date, or Text? How does changing the format change the faceting display

3. Which years have the most and least observations?

Exercise 2
Rename the species column. Did any issues arise?

Exercise 3

1. You might like to look for trends in your data by month of collection across years.
2. How do you sort your data by month?
3. How would you do this differently if you were instead trying to see all of your entries in chronological order?

Review (or intro) to R basics

Here is the lesson and here are setup instructions.

An example of how to embed code:

foo <- c(1,3)

What will happen in each of these examples? (hint: use class() to check the data type of your objects):

num_char <- c(1, 2, 3, "a")
num_logical <- c(1, 2, 3, TRUE)
char_logical <- c("a", "b", "c", TRUE)
tricky <- c(1, 2, 3, "4")

What happens when you combine numbers and characters?
character wins
What happens when you combine numbers and logical values?
numbers win
What happens when you combine characters and logical values?
character wins

Why do you think it happens?

22 August PM

Starting with data in R

download.file("https://ndownloader.figshare.com/files/2292169", "data/portal_data_joined.csv")

Manipulating data with dplyr

Here is a dplyr cheatsheet.

Here is a tidyr cheatsheet.

Pipe exercise
Using pipes, subset the surveys data to include animals collected before 1995 and retain only the columns year, sex, and weight.

Mutation exercise
Create a new data frame from the surveys data that meets the following criteria: contains only the species_id column and a new column called hindfoot_half containing values that are half the hindfoot_length values. In this hindfoot_half column, there are no NAs and all values are less than 30.
Hint: think about how the commands should be ordered to produce this data frame

split-apply-combine exercises

1. How many animals were caught in each plot_type surveyed?
2. Use group_by() and summarize() to find the mean, min, and max hindfoot length for each species (using species_id). Also add the number of observations (hint: see ?n).
3. What was the heaviest animal measured in each year? Return the columns year, genus, species_id, and weight.

For tomorrow

• install the R packages tidyverse and markdown and test load using library()
• install git (gitbash counts)
• create a GitHub account
• make a lunch order (below)

Lunch orders

Menu: https://www.jimmyjohns.com/downloadable-files/jj_menu_no_prices.pdf
Please type your name, the sandwich you would like to order, and any notes for your order

Order your sandwich

1) URL:

https://online.jimmyjohns.com

2) Select Pickup

3) Select this location:

1335 W Elizabeth St
Fort Collins, CO 80521

4) Set you pickup time for 11:30am to 11:45am (8/23/2018)

23 August AM

Visualization with ggplot2

Here is the base R cheatsheet. Here is an RStudio IDE cheatsheet that includes many keyboard shortcuts.

Here is the ggplot2 cheatsheet. Here is the ggplot2 website.

Here is where we made the subset data file for plotting: https://datacarpentry.org/R-ecology-lesson/03-dplyr.html#exporting_data

Plot Exercise
Use what you just learned to create a scatter (point) plot of weight over species_id with plot_type showing in different colors. Is this a good way to show this type of data?

Plot Exercise 2

1. Boxplots are useful summaries, but hide the shape of the distribution. For example, if the distribution is bimodal, we would not see it in a boxplot. An alternative to the boxplot is the violin plot, where the shape (of the density of points) is drawn.
   Replace the box plot with a violin plot; see geom_violin().
2. In many types of data, it is important to consider the scale of the observations. For example, it may be worth changing the scale of the axis to better distribute the observations in the space of the plot. Changing the scale of the axes is done similarly to adding/modifying other components (i.e., by incrementally adding commands). Try representing weight on the log10 scale; see scale_y_log10().
3. So far, we’ve looked at the distribution of weight within species. Try making a new plot to explore the distribution of another variable within each species.

• Create a boxplot for hindfoot_length. Overlay the boxplot layer on a jitter layer to show actual measurements.
• Add color to the data points on your boxplot according to the plot from which the sample was taken (plot_id).

Hint: Check the class for plot_id. Consider changing the class of plot_id from integer to factor. Why does this change how R makes the graph?

One great resource for colors: http://www.cookbook-r.com/Graphs/Colors_(ggplot2)/

Here are two more:
http://colorbrewer2.org/
https://coolors.co/

R markdown

We're going to be building a document that looks like this:

https://github.com/maglet/rmacc-rmarkdown/blob/master/Markdown_Demo.pdf

The script files I worked with are in this git repository:
https://github.com/maglet/rmacc-rmarkdown

But if you want, you can take another graph you made during the ggplot session and document it.

Here is the RStudio rmarkdown cheatsheet.

Git in RStudio

Slides for the git lesson are here. Here is the Happy Git and GitHub tutorial.

git setup

git config --global user.name "Vlad Dracula"
git config --global user.email "vlad@tran.sylvan.ia"