Homework 4: Python Practice

Due: Friday, April 8th, 2022 at 11:59pm ET

Collaboration Policy: This assignment uses the default course collaboration policy. Roughly, you may discuss conceptual issues, but you must code alone. See the policy for details and examples of what is and isn't allowed.

This HW4 FAQ post will be maintained on Ed.

This handout may look long, but that's because we're giving you a lot of notes and guidance on how to develop the required functions in Python. The amount of code you are writing is much less than the handout might suggest. Most of the time needed on this assignment will be in practicing notation and figuring out how to write list comprehensions.

Learning Objectives

This assignment mostly is to help you get practice writing Python functions using dictionaries and list comprehensions. You will not work with objects on this assignment (though you will in this week's lab).

Assignment Link

Use this GitHub Classroom link to get the stencil code for this assignment.

Setup

If you're having trouble with VSCode, check out our First Time Setup Guide and these folder setup instructions.

Please make sure to read our Gradescope submission guide before handing in your assignment!

Style Expectations

For all future assignments, including this one, we expect your code to conform to the CS200 Python Style Guide.

If you know how to do something in Java but need help translating it into Python, then you can reference our Going to Python from Java Guide.

For more information on testing and formatting tests in Python, please see our Python Testing Guide.

Working with datasets as dictionaries

We have a dataset of information about the Billboard Hot 100. The data contain information about songs from 1950-2015, including analysis of the lyrics. (Feel free to read about the methodology in collecting the data here.)

The data are in a file called songs.json. The .json suffix indicates that the file is in a format called JSON, which has become a standard format (across programming languages) for representing structured data with strings (for ease of sharing between tools and over networks). More information about JSON is available at the end of the handout.

When we load songs.json into Python, we get a list of dictionaries. Each dictionary represents one song. The dictionary maps string keys to Python values and objects (such as lists and other dictionaries). Here is a concrete outline of many fields of the dictionary that you will get for one song (where <value> denotes a string, number, or boolean).

{
    "artist": <value>,         # Artist of the song.
    "num_lines": <value>,      # Number of lines in the song.
    "sentiment": {
        "neg": <value>,        # Negativity assoc. w/ lyrics. (between 0-1 inclusive, 1 being 100% negative).
        "neu": <value>,        # Neutrality assoc. w/ lyrics. (between 0-1 inclusive, 1 being 100% neutral).
        "pos": <value>,        # Positivity assoc. w/ lyrics. (between 0-1 inclusive, 1 being 100% positive).
        "compound": <value>    # A single summarizing value of sentiment
    },
    "tags": <list>,            # Genre tags associated with artist of the song.
    "f_k_grade": <value>,      # Flesch–Kincaid grade level of lyrics.
    "year": <value>,           # Release year of the song.
    "pos": <value>,            # Position of Billboard's Top 100 for year <year>.
    "title": <value>,          # Title of the song.
}

Look at the JSON file for examples of what all of the entries look like!

Task 1: Open hw4.py in the stencil. It contains the code needed to load songs.json into Python (as a list of dictionaries).

For this assignment, you are writing Python functions. You are not writing a Python class. Your hw4.py file will simply consist of a sequence of function (and perhaps variable) definitions. We will deduct points for solutions that create classes.

Task 2: In hw4.py, write a function called most_positive that takes in an artist name and returns their most positive song, according to the compound sentiment metric in the JSON. If there is a tie, return the song with the higher Flesch-Kincaid grade. You can assume the artist is in the JSON file.

Keep this simple: just write a for loop that goes through every song in json_data, keeping track of the one with the highest compound score.

Note: Given a song, you get the compound sentiment by writing song['sentiment']['compound'].

Note: None is the Python term for null.

Built-in operations, list comprehensions, types, and exceptions

Now we will do some analysis of the songs that requires you to work with the data in more complicated ways.

Task 3: Start a function called artist_average_pos that takes an artist as input. Within the function, use a for loop to compute a list of all of the positions the artist has ever held on the Billboard Top 100 (including duplicates), in any order.

Example: the list for 'Moody Blues' should be [50, 32, 91]

This code should use the data from the JSON file as-is (i.e. should not perform any type conversions).

Assume that the input artist will be present in the JSON file. The function should not return anything at this point, but you can print the list or look at it in the interactive terminal for debugging.

Note: You create an empty list with [].

Note: The append method on lists adds an item to a list (with mutation).

Task 4: Now we need to finish average_artist_pos by returning the average of the list you computed in task 3. Add a return statement that computes the average in a single expression using the standard sum/length formula. Use the built-in functions document to find the names of these functions in Python. Don't try anything fancy for computing the average (e.g, no additional loops, list comprehensions, lambdas, etc).

Now try running this function for the artist Jay-Z? (we do NOT expect this to run successfully). What happened? Could a similar problem have happened in Java? Why or why not? Write your answer in a triple-quoted string immediately underneath the artist_average_pos function.

Task 5: Revise the return statement so that it successfully computes the average position. The statement should still be a single expression but should now use a combination of list comprehensions and built-ins. This function should work for all artists. You can assume that the input artist will be present in songs.json.

Note: we are only using the list comprehension for the final comptuation of the average. Your code should still contain the for loop for extracting the pos values for the given artist.

Note: There are several examples of list comprehensions at the bottom of the Going to Python from Java Guide.

Note: You convert a numeric string to an integer by using the function int, as in int("5").

Task 6: Make a copy of this function, but replace "pos" with "year" and change the function name to artist_average_year. What happens when you try to run this function for the artist "Katy Perry"? (don't turn in anything in for this question, just think about it.)

Revise your code to also work with songs whose year is given with 2k as an abbreviation for the year 2000. If you search songs.json (in your favorite text editor or terminal tool), you'll find years such as 2k5 (for 2005), and 2k13 (for 2013). While you might handle this with an if statement, it is better to do this using exceptions instead (since the use of 2k is an exceptional rather than a normal case).

As part of your solution, create a helper function called convert_year that takes a year-string and tries to convert it to an integer. If that conversion fails with a ValueError, replace the k in the year string with the needed number of zeros to make it a normal year, then convert the string to an int (the exception handling should be inside the convert_year function).

Note: you can use the following to replace k with the needed number of zeros in a string named yr:

yr.replace('k', '0'*(5 - len(yr)))

Task 7: Extend artist_average_year once more, this time raising a LookupError if the artist is not in the JSON or the input is None. This check only needs to be done on artist_average_year, NOT also in artist_average_pos.

Global variables, tuples, sets, and more comprehensions

Now we will try to compute the top non-pop genres in a small timespan. First, create an empty dictionary called genre_counts_by_year as a global variable. ({} is an empty dictionary.) This dictionary will be keyed on a year (int). Each year will map to a dictionary of genres (string) to counts (int).

• The count will be the number of times that the genre has appeared as a tag for a top song in that year.
• If a genre doesn't appear for some year, it will not appear in the dictionary for that year.
• Each year will also have an inner dictionary entry that maps from "none" to 0 (this will be useful for avoiding later errors in case some year only had pop songs – we include it for ALL years by default, however).

Task 8: Use for loops (not list comprehensions or lambdas) to write a function populate_genre_counts that populates this dictionary based on the Billboard JSON data. The function takes no inputs and returns no outputs. It's only job is to set up the genre_counts_by_year dictionary for use in later functions.

Include a line in your file that runs this function after you define it so that genre_counts_by_year will be populated for use in other functions.

Note: Before you start coding, draw out a small example of what you expect the resulting dictionary to look like, say on the first 5 songs of the songs.json file.

Note: Recall that you can use in to check whether a key exists in a dictionary. You can also use not in to check for the absence of a key.

Note: Remember type conversions!

Hint: This function provides a great opportunity to practice using the debugger: step through your code a bit and watch the dictionary get populated. This will help you learn the debugger even if your code seems to be running fine.

Task 9: Write a function top_non_pop_genre that takes in a dictionary that maps genres to counts (i.e., one of the inner dictionaries) and returns the non-pop genre with the highest count. Assume there is at least one key in the dictionary that is not "pop".

This function should use list comprehensions and built-in functions, so that the entire function is a single return statement with the expression to compute the answer (plus documentation as per the style guide).

Note: You can get a list of key, value tuples of a dictionary d by using d.items(). You can use this as you would any other list, including within a list comprehension.

Note: You can access each element of a tuple using indexing, for example, if t = ("key", "value"), t[0] would return "key" and t[1] would return "value".

Note: To use max() on a list of tuples, you can use the optional named argument key, which takes in a function that extracts the value you want to sort based on. Lambdas in Python work similarly to how they do in Pyret and Racket. To find the maximum of a list of tuples based on the first value in each tuple, you would write:

max(l, key=lambda x: x[0])

(In our case, x[0] refers to the key in the inner dictionary.)

Testing

Create a file named test_hw4.py.

Since many of the functions we have asked you to write here are hard-coded to use the Billboard data (which is large), it's difficult for us to ask you to develop tests that demonstrate your understanding of how to craft tests. Instead, we will use this assignment mainly to practice setting up tests in Python, with only one function graded for the set of tests that you came up with.

Refer to the Python Testing Guide for guidance on setting up tests in Python.

We will NOT do wheat/chaff grading on this assignment.

Task 10: Here are several statements that should hold true once you have implemented your functions. Convert each of these to a pytest assertion (and make sure these hold true of your code!).

• For the artist named "Jive Bunny":
  • the title of the most positive song is "Swing The Mood"
  • the average positivity rating is 97.0
  • the average year is 1990.0
• For the artist named "Coldplay":
  • the title of the most positive song is "Speed Of Sound"
  • the average positivity rating is 54.3333
  • the average year is 2008.5
• For the artist named "Frank Sinatra":
  • the title of the most positive song is "Strangers In The Night"
  • the average positivity rating is 29.4
  • the average year is 1957.8

Task 11: Develop a test function with assertions about convert_year. We will grade this on the thoroughness of your tests for inputs that are numeric strings and strings that would be numeric if k were replaced with three zeros.

Task 12: Develop a collection of smaller example dictionaries to use for testing top_non_pop_genre and write a series of assertions with those examples to test your function. We will grade this on

• whether you identified a good collection of scenarios to check (evidenced by your example dictionaries)
• whether your assertions explicitly check a variety of interesting data scenarios using those examples.

By "interesting", we are looking for whether you thought about meaningful different patterns within your data (e.g., how many genres, how genres split across years, etc).

If you want to know more about JSON (Optional)

More Info on JSON (for those who are interested)

Fun with JSON

JSON stands for JavaScript Object Notation. It is used all over the web to represent and package data.

A JSON object is a dictionary with strings as keys and values of the following data types:

• Number
• String
• Boolean
• Array (of any JSON data type), represented as a comma separated list surrounded by square brackets.
• Object, a collection of key-value pairs, where the key is a string and the value is any JSON data type, represented with curly brackets, colons, and commas.
• null

A JSON can be packaged up as any combination of these types, so the following are all valid JSONs!

Object:

{
    "title" : "JSONs are cool!"
}

Array:

[1, 2, 3]

Object with values of arrays and objects:

{
  "vegetables": ["carrot", "celery", "cucumber"],
  "colors": { "orange": ["carrot"], "green": ["celery", "carrot"] }
}

Array of objects:

[
  {
    "name": "asparagus",
    "color": "green"
  },
  {
    "name": "broccoli",
    "color": "green"
  },
  {
    "name": "eggplant",
    "color": "purple"
  }
]

The possibilities are endless!

They also reflect the structure of dictionaries and lists in Python, as JSON objects can be easily represented as a dictionary, and JSON arrays can easily be expressed as lists.

You can easily load JSONs into Python dictionaries and lists with the following code:

import json
with open("<file-name>.json") as file:
    json_data = json.load(file) # variable used for JSON data

Grading Expectations

In grading this assignment, we will be looking at:

• whether you have produced solutions that use the constructs indicated in each question. For example, if a question says "use a list comprehension" and you instead write the solution using a for loop, you will not earn many points. (This also covers creating classes, which you should not do on this assignment.)
• whether you have followed the Python Style Guide
• whether you have met the criteria listed in the testing tasks

Handing In

To hand in your files, submit two files to Homework 4 on Gradescope:

• hw4.py
• test_hw4.py

---

Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CSCI0200 document by filling out the anonymous feedback form.