# CS444 - Assignment 4

Due: Dec 8, 2020

**Update**: You are only required to complete dqn.ipynb to get full credit for this assignment. pg.ipynb is optional

## Instructions
### 1. Open the following notebooks in Google Colab

[dqn.ipynb](https://colab.research.google.com/drive/1Uri2kIKAA3BBzGKLBQv4aa-ham6YB1Vh?usp=sharing)

[pg.ipynb](https://colab.research.google.com/drive/1187p4Rqh0EVE1z3doyHcpI9FQPSkJvLe?usp=sharing) (Optional)

Please _do not_ share these links.

### 2. Save a copy to your Google Drive

In Google Colab, click "File" -> "Save a Copy in Drive". This will save a copy of the notebook to your account. By default the name will start with "Copy of ...". Rename the notebook to be the same as the original (in this case, "dqn.ipynb", "pg.ipynb").

### 3. Work on the assignment

Make sure you are working with your copy of the assignment. You can save your work ("File" -> "Save") and come back to it at a later time.

Colab collapses the cells by default so please make sure to expand the cells to make sure you do not miss any sections. You can expand all cells with "View" -> "Expand sections".

As indicated in the notebook, there are TODO blocks in the assignment that you will need to modify. **Do not edit any code outside of these blocks.** You may add cells for scratch work as you work through the assignment, but **please make sure to delete any extra cells before submitting.**

For this assignment, we will leverage GPUs for faster training. Please make sure to enable GPUs in Colab (Edit -> Notebook Settings).

**Make sure to run all cells before submitting**. You will only get credit for code that has been run.

Although this assignment requires you to write less code than the previous assignment, running DQN on the Pong environment (the last part) will take several hours (at least 4 from the reference solution with the provided hyperparameters), so please start early.

If you have questions or need clarifications, please email the TA ([e-mail]).

### 4. Download the ipynb files

Once you are finished with the notebooks, click "File" -> "Download .ipynb" to download your notebook.

**Make sure your downloaded files has the same name as the original notebook.**

### 5. Submit your notebook on Canvas

Upload your dqn.ipynb and pg.ipynb files to Canvas.

---

## Common Questions/Issues

- In the get_best_action method, you should turn off gradients when running the network. ie: wrap your code with this `with` statement:

        with torch.no_grad():
            # your code here

Otherwise, you may see errors like "Can't call numpy() on Tensor that requires grad..."

This is because we are just using our network to get the best action here and we do not need to do backpropagation through it, so we need to turn off the gradient calculations.

- Some students have run into memory issues with their notebooks on Colab for the Pong part of the assignment. To prevent this from happening, please change the config for the `buffer_size` when running Pong to 200000. You should still be able to achieve the optimal reward with this setting.

- After running the Pong part a few more times on Colab, I saw that the time taken varies quite a bit. On average, it seems to take almost 8 hours to completely finish (as opposed to the 4 hours mentioned in the assignment), but again there is quite a bit of variance.

You should start seeing improvements by 750,000 timesteps (the `t:` we print out).

Also, note that Colab has a timeout of 12 hours if you keep your browser open and 90 minutes if you close your browser, so make sure to keep your browser open while Pong is running.

- Don't forget to answer the question about the Test Environment near the top of the notebook. You need to determine the optimal reward for the test environment and briefly explain how you came up with that number.

- If you are seeing an error in the setup that says "Failed to fetch...", add a line "!apt-get update" before the other apt-get line. The revised cell will look like:

!pip install gym
!pip install "gym[atari]"
!pip install box2d-py
!pip install pyvirtualdisplay

!apt-get update
!apt-get install -y xvfb python-opengl ffmpeg