Lab 4: Generating Data

Learning Objective

The objective of this lab is to gain familiarity with sanitizing, visualizing, and gaining insights from data.

In all past assignments, the data that you worked with had been sanitized for you (i.e. you could expect Pyret to process the data without any extra work). The data in this lab, however, is being ported directly from Google Forms submissions, so there exists data that either isn't valid or isn't useful, and you need to find ways to effectively sort through such cases so that you can gather the insights you need.

If you don't feel more comfortable with sanitizing, visualizing, and interpreting data after working on this lab, come to TA hours! Your TAs would love to help in any way they can.

Backstory

You’ve just arrived at Pluto University to meet with one of the resident psychology professors.

The professor wants to learn more about human behavior. Given your rare qualifications as an Earth-dwelling human, they thought they could use your help.

They need your help to collect meaningful data and analyze it for them so they can publish their next paper!

Your Mission

Brainstorm. What defines you as a person? What kind of information would be good for a report on human behavior? Some types of data are better for different things. For example, line graphs are often best at representing data in the form of numbers, while pie charts may be better for presenting tallies and votes.

For their paper, the professor would like a variety of data types, from numbers to strings. After thinking for a while, the professor decides that they want to study the life of a student – namely, Brown University students (how convenient!).

Your lab TAs will give you a link to a Google Form with the professor's questions.

Fill out the form to contribute to the study!

Go to the response spreadsheet (also from your TAs), make a copy of it (select "Make a Copy" under the "File" tab), and import it into your program. This copy of the spreadsheet is yours to tweak manually. There are some values that Pyret will not allow you to read in, such as empty cells and other lovely surprises for you to discover. Pyret will let you know when it finds a value it doesn't like, and you will have to correct these by hand directly in the spreadsheet.

To import the spreadsheet, include this at the top of your program:
(Replace the ssid with that of your own copy.)

include tables
include image
include shared-gdrive("dcic-2021", "1wyQZj_L0qqV9Ekgr9au6RX2iqt2Ga8Ep")
include gdrive-sheets
import data-source as ds

# spreadsheet id from Google Sheets
ssid = "<Copy of Your SSID>" # aka that weird sequence of letters and numbers in the url
data-sheet = load-spreadsheet(ssid)
student-data =
  load-table: timestamp, average-sleep, weekend-sleep, concentration, classes, schoolwork-hours, extracurricular-activities, extracurricular-hours, on-campus, avg-commute
    source: data-sheet.sheet-by-name("Form Responses 1", true)
 end

Note: The highlighted portion is what your ssid looks like:

Preparing the Data

Now that we have the data collected, we need to make sure that the data is clean and processed before analysis.

1. Talk about why it will be difficult to draw insights from the columns about concentration name and hours of extracurricular obligations. Be able to explain your reasoning to a TA. Create a new table (within Google Sheets) that doesn't include these columns, and use this new table for the rest of the lab.

2. Once you import your table of student data, you will notice that the values in certain columns are in the format some(x) or none. In order to turn these values into the data types we all know and love (String, Number, etc.) use the sanitize functions upon import. Read the documentation here.

Before witing your sanitizers, remember: There are some values that Pyret will not allow you to read in, such as empty cells and other lovely surprises for you to discover. Pyret will let you know when it finds a value it doesn't like, and you will have to correct these by hand directly in the spreadsheet.

3. Look at the other columns. Discuss with your partner which columns might need to be cleaned, and come up with a plan to do this. In particular, come up with functions to clean up the columns about the number of classes, the hours spent doing schoolwork, the number of extracurriculars, and the commute time. Think about some times when it might be better to clean the data using sanitizers and other times when it might be more prudent to do so manually.

4. Write these functions, and create a new table with cleaned columns.

---

CHECKPOINT: Call over a TA once you reach this point.

---

Analyze the data

Now that the data has been tidied up, we need to put it together and analyze it in a way that will be meaningful for a research paper! What interesting ways can we represent this information? How should we visualize it?

1. Filter your table so that you only have students with high numbers of extracurriculars (you can decide which value on the scale defines the cutoff).
2. Filter your table to people who get 6-8 hours of sleep on an average weeknight.
   1. What is the average number of classes these people are taking?
   2. What percentage of them live on campus?
3. Filter your table so that you only have students with low numbers of extracurriculars who get at least 7 hours of sleep. Once again, you can choose the cutoff.
4. Create a scatterplot to compare hours of sleep on weeknights against minutes spent traveling to class.
5. Create a scatterplot to compare hours spent working vs. hours spent sleeping.
6. Create a scatterplot to compare average hours of sleep on weekends to average hours of sleep on weekdays.
7. Create a new column that sums the number of hours spent sleeping and doing homework.
   1. Do you notice any impossible results?
   2. With your partner, discuss the pros and cons of filtering out responses that seem impossible.
8. Create a new column that calculates the ratio of hours doing homework to hours sleeping (on weeknights).
9. Find the average number of minutes people spend on their commute.

---

CHECKPOINT: Call over a TA once you reach this point.

---

Reporting back…

You get a message from your space laboratory out in the city and they're wondering what is taking so long – they need your help in the laboratory! Consult with your lab partner on how to explain what you have been up to. Telling your boss and co-workers that you're "relaxing on Pluto's beautiful campus" probably isn't going to cut it. Show them something cool to persuade them that your trip is worthwhile!

1. We want to transpose the table (flip rows and columns) to look at the average commute times and average hours of sleep on weekdays of students living on campus and students living off campus. Since we don't yet know how to transpose an entire table through code, we'll build the table we need manually.

   1. Filter the main table to compute average commute times and average hours of sleep on weekdays of students living on campus.
   2. Do the same for students not living on campus.
   3. Create a new table (in Pyret) named campus-living-table with the columns location, avg-commute, and avg-sleep-weekdays.
   4. Fill in the table with the appropriate data.
      Note: there should only be two rows – one for those who are on meal plan and one for those who are not. It will look something like the following:

      Location	Commute time	Sleep on weekdays
      On campus	Average	Average
      Off campus	Average	Average

2. Plot any two columns that you think might be related to one another. Do you notice any cool correlations?

---

CHECKPOINT: Call over a TA once you reach this point.

---

Presentation Time!

It's now time for the professor's paper to be published and presented to a bunch of other smart professors! The professor gets a little shy when it comes to answering questions and presenting – so they have asked you to do all of it.

Discuss the answers to the following questions with your partner:

1. What data preparation did you have to do before you could start your analysis?
2. Why does messy data exist?
3. When can you safely assume your data matches a specific format?
4. How can we expand what we did today to work on even larger, even messier data sets?
5. In what scenarios do people have to sort through messy data?
   1. Do you have any suggestions for how to clean data?
   2. What should they watch out for?
6. How can we prevent "messy data"? What could we have changed in the form to make the data / results nicer?

---

CHECKPOINT: Call over a TA to check you off for lab.

---

Great job!

Thanks to your excellent work, the presentation was a success and the professor is very pleased. Who knew aerospace engineers could make such good psychologists?