---
tags: hw, spr22
---

# Homework 4: Python Practice

### Due: Friday, April 8th, 2022 at 11:59pm ET

**Collaboration Policy:** This assignment uses the default course [collaboration policy](https://hackmd.io/wStcp9Z4TH641sgw7r6Zyw?view). Roughly, you may discuss conceptual issues, but you must code alone. See the policy for details and examples of what is and isn't allowed.

This [HW4 FAQ](https://edstem.org/us/courses/16807/discussion/1352914) post will be maintained on Ed.

:::warning
This handout may look long, but that's because we're giving you a lot of notes and guidance on how to develop the required functions in Python. The amount of code you are writing is much less than the handout might suggest. Most of the time needed on this assignment will be in practicing notation and figuring out how to write list comprehensions.
:::

## Learning Objectives
This assignment mostly is to help you get practice writing Python functions using dictionaries and list comprehensions. You will not work with objects on this assignment (though you will in this week's lab).

## Assignment Link
Use this [GitHub Classroom link](https://classroom.github.com/a/IvAPmswS) to get the stencil code for this assignment.

## Setup

If you're having trouble with VSCode, check out our [First Time Setup Guide](https://hackmd.io/@csci0200/software-setup-s22) and these [folder setup instructions](https://edstem.org/us/courses/16807/discussion/1309739).

Please make sure to read [our Gradescope submission guide](https://hackmd.io/oTbF1IUuRs27VgVJF4z2rA) before handing in your assignment!

## Style Expectations
For all future assignments, including this one, we expect your code to conform to the [CS200 Python Style Guide](https://hackmd.io/@csci0200/rkeoOF22Y).

If you know how to do something in Java but need help translating it into Python, then you can reference our [Going to Python from Java Guide](https://hackmd.io/@csci0200/S15mJUUM9).

For more information on **testing** and **formatting tests** in Python, please see our [Python Testing Guide](https://hackmd.io/@csci0200/B1g1ZvvpF).

## Working with datasets as dictionaries

We have a dataset of information about the Billboard Hot 100. The data contain information about songs from 1950-2015, including analysis of the lyrics. (Feel free to read about the methodology in collecting the data [here](https://github.com/kevinschaich/billboard).)

The data are in a file called `songs.json`. The `.json` suffix indicates that the file is in a format called JSON, which has become a standard format (across programming languages) for representing structured data with strings (for ease of sharing between tools and over networks). More information about JSON is available at the end of the handout.

When we load `songs.json` into Python, we get a list of dictionaries. Each dictionary represents one song. The dictionary maps string keys to Python values and objects (such as lists and other dictionaries). Here is a concrete outline of many fields of the dictionary that you will get for one song (where `<value>` denotes a string, number, or boolean).

{
    "artist": <value>,         # Artist of the song.
    "num_lines": <value>,      # Number of lines in the song.
    "sentiment": {
        "neg": <value>,        # Negativity assoc. w/ lyrics. (between 0-1 inclusive, 1 being 100% negative).
        "neu": <value>,        # Neutrality assoc. w/ lyrics. (between 0-1 inclusive, 1 being 100% neutral).
        "pos": <value>,        # Positivity assoc. w/ lyrics. (between 0-1 inclusive, 1 being 100% positive).
        "compound": <value>    # A single summarizing value of sentiment
    },
    "tags": <list>,            # Genre tags associated with artist of the song.
    "f_k_grade": <value>,      # Flesch–Kincaid grade level of lyrics.
    "year": <value>,           # Release year of the song.
    "pos": <value>,            # Position of Billboard's Top 100 for year <year>.
    "title": <value>,          # Title of the song.
}

**Look at the JSON file for examples of what all of the entries look like!**

**Task 1:** Open `hw4.py` in the stencil. It contains the code needed to load `songs.json` into Python (as a list of dictionaries).

:::warning
For this assignment, you are writing Python functions. You are **not** writing a Python class. Your `hw4.py` file will simply consist of a sequence of function (and perhaps variable) definitions. We will deduct points for solutions that create classes.
:::

**Task 2:** In `hw4.py`, write a function called `most_positive` that takes in an artist name and returns their most positive song, according to the compound sentiment metric in the JSON. If there is a tie, return the song with the higher Flesch-Kincaid grade. You can assume the artist is in the JSON file.

Keep this simple: just write a `for` loop that goes through every song in `json_data`, keeping track of the one with the highest compound score.

***Note:** Given a song, you get the compound sentiment by writing `song['sentiment']['compound']`.*

***Note:** `None` is the Python term for `null`.*

## Built-in operations, list comprehensions, types, and exceptions

Now we will do some analysis of the songs that requires you to work with the data in more complicated ways.

**Task 3**: Start a function called `artist_average_pos` that takes an artist as input. Within the function, use a `for` loop to compute a list of all of the positions the artist has ever held on the Billboard Top 100 (including duplicates), in any order.

Example: the list for 'Moody Blues' should be `[50, 32, 91]`

This code should use the data from the JSON file as-is (i.e. should not perform any type conversions).

Assume that the input artist will be present in the JSON file. The function should not return anything at this point, but you can print the list or look at it in the interactive terminal for debugging.

***Note:** You create an empty list with `[]`.*

***Note:** The `append` method on lists adds an item to a list (with mutation).*

**Task 4**: Now we need to finish `average_artist_pos` by returning the average of the list you computed in task 3. Add a `return` statement that computes the average in a single expression using the standard `sum/length` formula. Use the [built-in functions](https://docs.python.org/3/library/functions.html) document to find the names of these functions in Python. Don't try anything fancy for computing the average (e.g, no additional loops, list comprehensions, lambdas, etc).

Now try running this function for the artist `Jay-Z`? (we do **NOT** expect this to run successfully). What happened? Could a similar problem have happened in Java? Why or why not? **Write your answer in a triple-quoted string immediately underneath the `artist_average_pos` function.**

**Task 5**: Revise the `return` statement so that it successfully computes the average position. The statement **should still be a single expression** but **should now use a *combination* of list comprehensions and built-ins**. This function should work for all artists. You can assume that the input artist will be present in `songs.json`.

***Note:** we are only using the list comprehension for the final comptuation of the average. Your code should still contain the `for` loop for extracting the `pos` values for the given artist.*

***Note:** There are several examples of list comprehensions at the bottom of the [Going to Python from Java Guide](https://hackmd.io/@csci0200/S15mJUUM9).*

***Note:** You convert a numeric string to an integer by using the function `int`, as in `int("5")`.*

**Task 6**: Make a copy of this function, but replace "pos" with "year" and change the function name to `artist_average_year`. What happens when you try to run this function for the artist `"Katy Perry"`? (don't turn in anything in for this question, just think about it.)

Revise your code to also work with songs whose year is given with `2k` as an abbreviation for the year 2000. If you search `songs.json` (in your favorite text editor or terminal tool), you'll find years such as `2k5` (for 2005), and `2k13` (for 2013). While you might handle this with an `if` statement, it is better to **do this using exceptions instead** (since the use of `2k` is an exceptional rather than a normal case).

As part of your solution, create a helper function called `convert_year` that takes a year-string and tries to convert it to an integer. If that conversion fails with a `ValueError`, replace the `k` in the year string with the needed number of zeros to make it a normal year, then convert the string to an `int` (the exception handling should be inside the `convert_year` function).

***Note:** you can use the following to replace `k` with the needed number of zeros in a string named `yr`:*

yr.replace('k', '0'*(5 - len(yr)))

**Task 7:** Extend `artist_average_year` once more, this time raising a `LookupError` if the artist is not in the JSON or the input is `None`. This check only needs to be done on `artist_average_year`, **NOT** also in `artist_average_pos`.

## Global variables, tuples, sets, and more comprehensions

Now we will try to compute the top non-pop genres in a small timespan. First, create an empty dictionary called `genre_counts_by_year` as a global variable. (`{}` is an empty dictionary.) This dictionary will be keyed on a year (`int`). Each year will map to a dictionary of genres (`string`) to counts (`int`).
- The count will be the number of times that the genre has appeared as a tag for a top song in that year.
- If a genre doesn't appear for some year, it will not appear in the dictionary for that year.
- Each year will also have an inner dictionary entry that maps from `"none"` to `0` (this will be useful for avoiding later errors in case some year only had pop songs -- we include it for ALL years by default, however).

**Task 8:** Use `for` loops (not list comprehensions or lambdas) to write a function `populate_genre_counts` that populates this dictionary based on the Billboard JSON data. The function takes no inputs and returns no outputs. It's only job is to set up the `genre_counts_by_year` dictionary for use in later functions.

Include a line in your file that runs this function after you define it so that `genre_counts_by_year` will be populated for use in other functions.

***Note:** Before you start coding, draw out a small example of what you expect the resulting dictionary to look like, say on the first 5 songs of the `songs.json` file.*

***Note:** Recall that you can use `in` to check whether a key exists in a dictionary. You can also use `not in` to check for the absence of a key.*

***Note:** Remember type conversions!*

***Hint:** This function provides a great opportunity to practice using the debugger: step through your code a bit and watch the dictionary get populated. This will help you learn the debugger even if your code seems to be running fine.*

**Task 9:** Write a function `top_non_pop_genre` that takes in a dictionary that maps genres to counts (i.e., one of the inner dictionaries) and returns the non-pop genre with the highest count. Assume there is at least one key in the dictionary that is not `"pop"`.

This function should use list comprehensions and built-in functions, so that the entire function is a single `return` statement with the expression to compute the answer (plus documentation as per the style guide).

***Note:** You can get a list of key, value tuples of a dictionary `d` by using `d.items()`. You can use this as you would any other list, including within a list comprehension.*

***Note:** You can access each element of a tuple using indexing, for example, if `t = ("key", "value")`, `t[0]` would return `"key"` and `t[1]` would return `"value"`.*

***Note:** To use `max()` on a list of tuples, you can use the optional named argument `key`, which takes in a function that extracts the value you want to sort based on. [Lambdas in Python](https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions) work similarly to how they do in Pyret and Racket. To find the maximum of a list of tuples based on the first value in each tuple, you would write:*

max(l, key=lambda x: x[0])

*(In our case, `x[0]` refers to the key in the inner dictionary.)*

<!--

**Task 9:** Write a function called `top_genres_around_year` that takes in a year and returns a list (in any order) that contains each of the top non-pop genres of every year in `genre_counts_by_year` that is within 3 years (inclusive) of the input year.

Assume a valid year will be given as an integer. If none of the years in the range are present in `genre_counts_by_year`, the output should be an empty list. The function should contain a single return statement and no other code, and should make use of the function you wrote above.

:::spoiler Hints
You can turn a list into a set and a set into a list by using the built-in functions. You can also do set comprehensions in the same way as list comprehensions, just replacing `[]` with `{}`:

{abs(x) for x in L} # set of absolute values of items in L

:::
-->

## Testing

Create a file named `test_hw4.py`.

Since many of the functions we have asked you to write here are hard-coded to use the Billboard data (which is large), it's difficult for us to ask you to develop tests that demonstrate your understanding of how to craft tests. Instead, we will use this assignment mainly to practice setting up tests in Python, with only one function graded for the set of tests that you came up with.

Refer to the [Python Testing Guide](https://hackmd.io/cLCJRBbXTSyOyEoNQq4N-Q) for guidance on setting up tests in Python.

We will **NOT** do wheat/chaff grading on this assignment.

**Task 10:** Here are several statements that should hold true once you have implemented your functions. Convert each of these to a pytest assertion (and make sure these hold true of your code!).

- For the artist named "Jive Bunny":
- the title of the most positive song is "Swing The Mood"
- the average positivity rating is 97.0
- the average year is 1990.0
- For the artist named "Coldplay":
- the title of the most positive song is "Speed Of Sound"
- the average positivity rating is 54.3333
- the average year is 2008.5
- For the artist named "Frank Sinatra":
- the title of the most positive song is "Strangers In The Night"
- the average positivity rating is 29.4
- the average year is 1957.8

**Task 11:** Develop a test function with assertions about `convert_year`. We will grade this on the thoroughness of your tests for inputs that are numeric strings and strings that would be numeric if `k` were replaced with three zeros.

**Task 12:** Develop a collection of smaller example dictionaries to use for testing `top_non_pop_genre` and write a series of assertions with those examples to test your function. We will grade this on
- whether you identified a good collection of scenarios to check (evidenced by your example dictionaries)
- whether your assertions explicitly check a variety of interesting data scenarios using those examples.

By "interesting", we are looking for whether you thought about meaningful different patterns within your data (e.g., how many genres, how genres split across years, etc).

## If you want to know more about JSON (Optional)

:::spoiler More Info on JSON (for those who are interested)
## Fun with JSON

`JSON` stands for JavaScript Object Notation. It is used all over the web to represent and package data.

A JSON object is a dictionary with strings as keys and values of the following data types:

- Number
- String
- Boolean
- Array (of any JSON data type), represented as a comma separated list surrounded by square brackets.
- Object, a collection of key-value pairs, where the key is a string and the value is any JSON data type, represented with curly brackets, colons, and commas.
- `null`

A JSON can be packaged up as any combination of these types, so the following are all valid JSONs!

Object:

{
    "title" : "JSONs are cool!"
}

Array:

[1, 2, 3]

Object with values of arrays and objects:

{
  "vegetables": ["carrot", "celery", "cucumber"],
  "colors": { "orange": ["carrot"], "green": ["celery", "carrot"] }
}

Array of objects:

[
  {
    "name": "asparagus",
    "color": "green"
  },
  {
    "name": "broccoli",
    "color": "green"
  },
  {
    "name": "eggplant",
    "color": "purple"
  }
]

The possibilities are endless!

They also reflect the structure of dictionaries and lists in Python, as JSON objects can be easily represented as a dictionary, and JSON arrays can easily be expressed as lists.

You can easily load JSONs into Python dictionaries and lists with the following code:

import json
with open("<file-name>.json") as file:
    json_data = json.load(file) # variable used for JSON data

:::

## Grading Expectations

In grading this assignment, we will be looking at:

- whether you have produced solutions that use the constructs indicated in each question. For example, if a question says "use a list comprehension" and you instead write the solution using a `for` loop, you will not earn many points. (This also covers creating classes, which you should not do on this assignment.)
- whether you have followed the Python Style Guide
- whether you have met the criteria listed in the testing tasks

## Handing In

To hand in your files, submit two files to **Homework 4** on Gradescope:
- `hw4.py`
- `test_hw4.py`

______________________________

*Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CSCI0200 document by filling out the [anonymous feedback form](https://docs.google.com/forms/d/e/1FAIpQLSdFzM6mpDD_1tj-vS0SMYAohPAtBZ-oZZH0-TbMKv-_Bw5HeA/viewform?usp=sf_link).*