<style> h3 { border-bottom: 1px solid #ccc; } section { margin-bottom: 2rem; padding: 0em 1em; padding-bottom: 1em; border-radius: 4px; background-color: #f7f7f7; border: 1px solid #ccc; } center { text-align: center; } .alert details { background-color: transparent !important; } code { background-color: rgb(244,244,245) !important; color: #ff0000; } .alert-warning { background-color: #fcf8e3 !important; color: #8a6d3b !important; } .alert-success { background-color: #dff0d8 !important; color: #3c763d !important; } summary { font-weight: bolder; } summary:hover { text-decoration: underline; } </style> **[&laquo; Back to the main CSCI1660 website](https://brown-csci1660.github.io)** # Dropbox project setup guide This guide is designed to help you get started with setting up the development environment for the Dropbox project. In this project, you'll build a client for a simulated version of a secure storage system that uses end-to-end encryption. This project has significantly less infrastructure compared to other projects: you'll reuse our existing development container and build your implementation entirely in Python. This guide will describe how to set up your development container environment for this project and how to work with the stencil code for this project. ## Initial setup: Updating your development container > **New to docker?** If you do not have Docker installed from the last project, or you have not used the development container from Project 1, you can set up your environment using [this guide](https://hackmd.io/@cs1660/container-setup) and then skip this section. ### If you've had container issues in the past If you've had issues with the container environment in the past (slow performance, files not syncing, etc.), please take a look at the following notes for our most common fixes: :::info <details markdown="1"><summary markdown="span"><b><u>Updates if you had slow filesystem performance on MacOS</u></b></summary> If you experienced exceptionally slow filesystem performance on MacOS, see [here](https://hackmd.io/@cs1660/container-setup#Slow-filesystem-performance-on-Macs) for instructions on how to update your docker settings for a fix. If you still encounter poor filesystem performance, or you have issues after the fix, please let us know! </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Windows users: If you have memory issues or slowdowns/crashes</u></b></summary> It seems that recent versions of Docker and WSL have issues with memory usage. See [these instructions](https://hackmd.io/@cs1660/container-setup#Windows-specific-issues) for a full writeup of the problem and a list of workarounds that we think will help. If you encountered issues with Docker during project 1, we strongly recommend that you do the following steps from this guide now: 1. Update Docker to the latest version (if you installed it from a previous class) 2. [Set a memory limit on WSL](https://hackmd.io/@cs1660/container-setup#Workaround-1-Limit-WSL%E2%80%99s-memory-usage) (and thus your docker containers) The instructions also list some workarounds you can use at times to help reduce Docker/WSL's memory usage when they start to get problematic--skim these over and then keep them in mind in case you run into issues! </details> ::: ## Dropbox stencil setup ### Before you start: cleaning out previous containers If you've finished with Project 3, you will likely no longer need to use the handin container for this project. To save on disk space, you can remove the image for the Handin container as follows: 1. Open a new terminal on your **host machine** (or, on Windows, a **WSL terminal**). 3. `cd` to your Handin project stencil directory and run the following: ```shell! $ ./run-container clean-image ``` This will delete only the container image, freeing up disk space on your host machine. Any files you had in your `home` directory (ie, `/home/alice` in the container) will be preserved. > If you have not yet cleaned up your containers from the Flag project, see [these instructions](https://hackmd.io/@cs1660/handin-setup-guide#Before-you-start-cleaning-up-previous-containers) ### Cloning the stencil project :::warning :eyes: **Looking for a stencil link?** We will send you a link to create your stencil repo after teams have been assigned. We will post an announcement on Ed once team assignments have been sent, so you can be sure you didn't miss it. ::: We recommend working on this project using our development container, which is the container we used in Project 1. In the next few sections, we'll describe how to set up your container to develop for Dropbox and how to work with the stencil in VSCode. > **Don't want to use the container?** Since this project is in Python, you can technically work on it using any system with Python >= 3.9 that can load our virtual environment. However, our course staff may only be able to provide support if you use the development container as your base system. #### Recommended filesystem layout As with the other stencils, we recommend cloning the stencil so you can access it from your development container. To do this, we recommend cloning the stencil in your `DEV-ENVIRONMENT/home` directory, where `DEV-ENVIRONMENT` is the name of the folder you used when you cloned the development container. Here's an example of what your filesystem might look like (based on what you might have from Projects 1 and 2): ``` - ... |--DEV-ENVIRONMENT | |--docker/ | |--home/ | | |--p01-cryptography-<your GitHub username>/ | | |--p02-flag-<your GitHub username>/ | | |--pr03-handin-<your GitHub username>/ | | |--pr04-dropbox-<your team name>/ # <----------- Clone your stencil here! | |--cs1660-run-docker | |-- ... ... ``` :::success **Task:** Go clone the stencil repo! You should have received a link to clone the stencil in the email with your team assignment. If you didn't receive an email, please post on Ed or contact Nick. ::: ### Setting up your Dropbox environment #### Initial setup Now that your container environment is configured, we can set up our Dropbox environment for development. Our stencil contains a `Makefile` that automates the most important tasks like this. To use it: 1. If you haven't done so already, start a terminal inside the container. That is, in a terminal on your host machine (or a WSL terminal on Windows), run the comand `./run-container` 2. `cd` to your Dropbox stencil directory 3. Run the following command, which sets up our environment: ```shell1 cs1660-user@c808c3104e5a:~/dropbox-stencil $ make setup ``` After the setup finishes, your terminal output should end with `Successfully installed...`. If it does not, please let us know on Ed. #### Entering (or "activating") the environment Finally, we need to configure our terminal to run Python with our virtual environment. To do this, Python provides a script that sets various shell environment variables (including `$PATH`) to point to the version of python inside the virtual environment---this is called "activating" the environment. To do this: 1. From a terminal inside your container, `cd` to your Dropbox stencil directory (if you're not there already) 2. Run `source env/bin/activate` Once you do this, there won't be any output, but your terminal prompt should change, like this: ![](https://hackmd.io/_uploads/r1CxRkC-h.png) If your terminal prompt shows `(env)` it means your environment is loaded, yay! :::danger **Important note**: Every time you open a new terminal to run Python, you will need to activate your environment like this. As you work on the project, don't forget this step! ::: ## Working with the stencil code ### Running the tests Before we setup VSCode, it's important to make sure our test environment works. As you work on the project, you'll write unit tests to check the functionality of your implementation of your client. We've provided a starter test file that imports the stencil code and contains some initial tests in the file `test_client.py`. You can run these tests from any container terminal with the virtual environment setup, as follows: ```shell= $ python3 -m unittest -v test_client.py ``` If you haven't modified your stencil at all, you should see a bunch of output as each test runs, along with several failing tests: ``` (env) cs1660-user@1a50d835c745:~/csci1660/proj/dropbox-stencil$ python3 -m unittest -v test_create_user (test_client.ClientTests) Checks user creation. ... ERROR [... snipped ... ] support.util.DropboxError: Not Implemented ---------------------------------------------------------------------- Ran 6 tests in 0.002s FAILED (errors=5) ``` If your output looks like this, your environment is working properly! It's okay to see failing tests for the reason `support.util.DropboxError` at this point--these are TODOs for you to fill in. :wink: ### Test suites we provide Your stencil contains two files that contain tests: - `test_client.py`: contains some basic tests and a sufficient framework to add your own tests. We recommend running these tests first. - `test_functionality.py`: contains a more comprehensive test suite, with nearly all the tests run by the autograder. Once your implementation is more complete, we recommend running these tests as well. (*Why doesn't this include all the tests? Some tests require a more significant test harndes we can only provide on Gradescope--you can see the results of these tests after submittting to the autograder.*) :::info :fire: **Pro tip**: see the **[helpful commands for running tests](#Helpful-commands-for-running-tests)** for some useful ways to run specific tests and debug your tests more easily. Also see [this section](#Running-tests-in-VSCode) for instructions on how to run tests and debug your work in VSCode. ::: If you encounter issues with this process, make sure you're using the virtual environment and that it was set up correctly--otherwise, post on Ed and we can help! ## Working with the stencil in VSCode VSCode has some great features for developing with containers. While it's not required for this project, if you like using VSCode you can connect it to the container and use it to write your Python code for this project. To do this: 1. If you haven't done so already, install the Python plugin extension in VSCode. For instructions, see this section. 2. Open VSCode inside the container using its remote tools--see [this section for instructions](https://hackmd.io/@cs1660/container-setup#Connecting-VS-Code-to-the-container-optional-but-recommended) 3. Once VSCode is open from inside the container, go to **File > Open Folder...** and select your Dropbox stencil directory, which is likely under `/home/cs1660-user/...` :::warning **Warning**: When opening your Dropbox stencil directory, do NOT use the **Show Local** button in the open file dialog--for some reason, this will open the folder *outside* the container, which is not what we want! ::: 5. Open any python file, such as `client.py` ### Checking your VSCode Installation When VSCode starts, it should automatically find our Python virtual environment. To check, look in the lower-right corner of the VSCode window, which should look like this: ![](https://hackmd.io/_uploads/BJvb5xR-3.png) If your VSCode does not show the virtual environment, click on the Python version number (in the red box above) and select "env" in the prompt at the top. Your VSCode is now configured correctly! ### Running your client in VSCode To run your code from VSCode, we recommend using VSCode's builtin terminal. To use it: 1. In VSCode, open a new terminal from the menu using **Terminal > New Terminal** 2. VSCode should open a container terminal and automatically enter your virtual environment, which should look like this (if not, see the previous section): ![](https://hackmd.io/_uploads/ryPzslR-2.png) 3. You can use this terminal to run the tests or any Python commands. You're also welcome to use VSCode's built-in features for running unit tests, debugging, etc. For now, try to run the tests with `python3 -m unittest test_client.py`. For more tips on how to run tests, see [this section](#Helpful-commands-for-running-tests). If your test output looks the same as when you were running in the terminal, your VSCode is now properly configured for development, yay! If you encounter errors with modules, [check your interpreter settings](#Checking-your-VSCode-Installation) and feel free to post on Ed. ## Next steps: more resources and getting started At the start of the project, you should be focusing on understanding the specification and thinking about how you will implement each component. To get started, we recommend the following resources: - **[The Dropbox Wiki](https://brown-csci1660.github.io/dropbox-wiki/)** is the ultimate, comprehensive resource for the client you will implement. This will be your main resource on the project specifications. - **Gearup notes**: We'll have a gearup within the first week of the project's release, where we will explore the most important components of the project. Notes and a recording will be posted with the project resources, and in our Ed FAQ. - The **[reference client](#Using-the-reference-client)**: is a working client you can run to understand how the API works - **[Implementation tips](#Tips-and-implementation-resources)**: this section contains more tips and resources as you get into the details of your design and implementation (eg. how to store and serialize objects, how to test your work) ## Tips and implementation resources This section contains some useful tools and resources you may find useful as you build your design and get started on your implementation: - The **[serialization examples](#Storing-objects-in-the-dataserver)** demonstrate how to think about how to serialize and store various objects in the dataserver. You will find this helpful as you make decisions on how to to store datastructures in the dataserver - **[Looking up objects by string](#Using-strings-to-do-lookups-in-the-Dataserver)**: provides an example of how to generate memory locations from constant strings, allowing you to get objects by name, eg. `Get(alice@somefile)` - **[Test suites we provide](#Test-suites-we-provide)**: we provide a large set of tests you can run to help check your implementation--though you're also welcome to write your own. Look here for more info. - **[Helpful commands for running tests](#Helpful-commands-for-running-tests)**: Look here for some super-helpful ways to test your work faster and more effectively! - **[How to run/debug your tests in VSCode](#Running-tests-in-VSCode)**: contains instructions on how to configure VSCode to run your tests and use the VSCode debugger ### Storing objects in the dataserver While the dataserver allows you to store arbitrary data, it doesn't understand types--similar to writing to a raw file, you can only read and write `bytes()` objects to the Dataserver. To store any more complex data, you'll need to *serialize* it into bytes first. We've provided several utility methods and examples to help with this process: - Stencil functions `util.BytesToObject` and `util.ObjectToBytes` serialize some basic Python types - For more complex types like sets, custom classes, or dataclasses, we've provided some examples you can use For examples on how to use these, see [**this file**](https://github.com/brown-csci1660/dropbox-examples/blob/main/test_examples.py) in the Dropbox Examples repository. See the comments on each test for examples on how to use `util.BytesToObject` and `util.ObjectToBytes` to serialize different types of data. :::info <!-- **Demo**: For a video demo of this part with explanations, see the [Gearup II recording](https://brown.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=12a978f6-964d-45fd-a135-afe9017c5e3f). --> Note: We'll provide a demo of this part in Gearup II. ::: ### Using strings to do lookups in the Dataserver You can think of the dataserver as a `Map[Memloc,Data]` where `Memloc` is an arbitrary 16-byte value and `Data` is a `bytes` object of any size. Since 16-byte values are kind of unwieldy, you may want to store data in the Dataserver using a friendly name so you can easily look it up later without storing a random memory location. For example, you might want to store info about user Alice at the key "user_info:alice". To do this, we recommend hashing the name and then truncating the hash to 16-bytes, which produces a deterministic memory location from a friendly name. Here's an example of how you can do this, packaged as a convenient helper function: ```python def s_addr(s): return memloc.MakeFromBytes(crypto.Hash(s.encode("utf-8"))[:16]) ``` For examples on how to use this, see the [serialization examples](https://github.com/brown-csci1660/dropbox-examples/blob/main/test_examples.py) and the [Gearup II recording](https://brown.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=12a978f6-964d-45fd-a135-afe9017c5e3f). ### Using the reference client We've provided a reference version of the client you can use for testing. This client is provided as a compiled Python library, which you can import and test in a similar manner to your client. You can use this to get an idea of how the client API should work and how you can interact with it. We've provided a file `test_reference.py` that imports the reference library and runs runs similar tests to those in `test_client.py`. You can run these tests by running (from a terminal in your virtual environment): ```shell (env) cs1660-user@container:~/dropbox-stencil$ python3 -m unittest test_reference.py ``` Since this version is complete, you should see output with all the tests passing. :sunglasses: :::info **Note**: This binary-compiled version of the client is for testing purposes only--you cannot use it in your actual implementation. When we review your final design document, we will also manually review your code--so even if you do manage to circumvent our autograder we'll know about it. :wink: <details markdown="1"><summary markdown="span"><u>But wait, you just gave us the solution... can't I just reverse engineer the binary?</u></summary> In theory, yes, but this isn't likely to help you. The goal of this project is to get you to think about how to build a secure system--reversing the code will not be helpful toward building your own skills for thinking about system design. Remember that a large portion of your project grade is for the explanation of your design in your design document, so obtaining an implementation through reverse-engineering is not in your best interest. </details> ::: We recommend writing your own tests in `test_reference.py` to try stuff out and see how the implementation works. #### Playing around with the reference If you prefer to play around in a REPL, you can just run the file and type in code directly, like this: :::info **Note**: Once a user calls `create_user`, the user is also authenticated. You don't have to call `authenticate_user` after calling `create_user`. This listing is just an example of what example API calls look like. For more examples of how to use the client API, see `test_client.py`. ::: ``` (env) cs1660-user@container:~/dropbox-stencil$ python3 test_reference.py (env) cs1660-user@1a50d835c745:~/csci1660/proj/dropbox-stencil$ python3 test_reference.py Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> u = c.create_user("a", "hello") >>> u = c.authenticate_user("a", "hello") >>> some_data = "string encoded as bytes".encode("utf-8") >>> u.upload_file("filename", some_data) >>> recvd_data = u.download_file("filename") >>> some_data == recvd_data True ``` ### Helpful commands for running tests Python's `unittest` has a few useful options that can make testing faster. Here are some examples of some of our favorites: - `python3 -m unittest test_file.py -v`: Show tests as they run - `python3 -m unittest test_functionality.py -v test_functionality.TestUserCreation.test_bad_username`: Run a specific test - `python3 -m unittest test_file.py -f`: Stop on the first test failure - `python3 -m unittest --help`: Show more options ### Running tests in VSCode You can also run your tests directly in VSCode instead of the terminal. Here's how to set up VSCode to do this: 1. Make sure you have configured VSCode to use your virtual environment, as described in [this section](#Working-with-the-stencil-in-VSCode). 1. Open VSCode's testing panel by clicking on the beaker in the sidebar, which looks like this: ![](https://hackmd.io/_uploads/ByDFi6Q43.png) 2. VSCode will probaly ask you to configure testing, which tells VSCode how to find the tests. To do this: a. Click **Configure Python Tests**: ![](https://hackmd.io/_uploads/rJnWpTXN3.png) b. A menu will open at the top of the VSCode window like the picture below. In this menu, select **`unittest`**: ![](https://hackmd.io/_uploads/rytX6pQN2.png) c. Next, you will be prompted to select where the tests are located. Select **". Root Directory"**: ![](https://hackmd.io/_uploads/ry1rapX4h.png) d. Finally, you will be asked what types of files contain tests. Select **"test_*.py"**: ![](https://hackmd.io/_uploads/r1Wtp6QEn.png) 3. After you have configured the tests, your testing pane should look like the picture below. If you don't see any tests, see [this section](#Don%E2%80%99t-see-your-tests). ![](https://hackmd.io/_uploads/H1401RXV2.png) 4. To run the tests, expand the list and hover over any test (or set of tests) and clck **"Run Test"** or **"Debug test"**, as shown in the figure above, to run your tests! You should see the tests results show up in the sidebar, yay! Note that most tests will fail at the start of your project, since most methods haven't been implemented yet. If you want to test your work against a complete implementation, see [this section](#Using-the-reference-client) for how to use the our reference client (just run the tests from VSCode instead!). #### Don't see your tests? If you don't see your tests in the list, try the following: 1. **[Make sure your VSCode is in the virtual environment](#Checking-your-VSCode-Installation)** and opened directly to your Dropbox stencil directory 2. **Make sure your your Python files have no errors.** VSCode needs to run your .py files in order to find the tests--if there are any errors in your files, this process will fail and your tests won't show up! 3. If you're still unsure, try to **[run each test file from the terminal](#Running-your-client-in-VSCode)**--this might help find errors that VSCode isn't displaying for some reason. If you're still stuck, just continue running from a terminal instead or post on Ed. ## Final notes This guide has provided an overview of the most important mechanics you need to work with the environment for this project. If you have questions, please feel free to come to hours or post on Ed. Have fun, and happy hacking! ## FAQs/Common issues ### Windows line endings ("unexpected end of file") If you see errors along the lines of `bash: '\r': command not found` or `bash: /home/cs1660-user/.bash_profile: line 5: syntax error: unexpected end of file`, see [this section](https://hackmd.io/VvXjihLeQne4q84wq1SgWg?view#Windows-line-endings-%E2%80%9Cunexpected-end-of-file%E2%80%9D) for common fixes.