<style> .markdown-body h1, .markdown-body h2, .markdown-body h3, .markdown-body h4, .markdown-body h5, .markdown-body h6, .markdown-body { font-family: "Helvetica", "Helvetica Neue", sans-serif; } h3 { border-bottom: 1px solid #ccc; } section { margin-bottom: 2rem; padding: 0em 1em; padding-bottom: 1em; border-radius: 4px; background-color: #f7f7f7; border: 1px solid #ccc; } center { text-align: center; } .alert details { background-color: transparent !important; } code { background-color: rgb(244,244,245) !important; color: #ff0000; } .alert-warning { background-color: #fcf8e3 !important; color: #8a6d3b !important; } .alert-success { background-color: #dff0d8 !important; color: #3c763d !important; } summary { font-weight: bolder; } summary:hover { text-decoration: underline; } </style> <!-- <style> summary { text-decoration: underline; font-weight: bold; } details { padding: 1em; } /* details[open] { border: 1px solid #000000; border-radius: 5px; } */ </style> --> # Snowcast Setup Guide and Warmup <!-- **Due as part of milestone on September 16, 2024 at 11:59PM EDT** --- --> # Introduction This guide will provide some resources for getting started and building your implementation for the Snowcast project. The first part of this document is our "Warmup", which is designed to help you get oriented with the project and get started with socket programming. The remainder of this guide contains helpful resources for building your implementation and testing your work as you continue with the project. ## Getting Started **You may implement this project Go, C, C++, or Rust**. If you are unsure, we recommend using Go, even if it is new to you, as class examples this year will use it. We have curated a list of resources for each language [here](https://brown-csci1680.github.io/documents/). **If you want to use Rust**, note that only some members of the course staff can provide language support (though we are all happy to discuss conceptual questions!). For details, see the [course staff list](https://brown-csci1680.github.io/staff/). ## Essential Resources We have a number of documents and resources to help you get started with thinking about sockets and network programming: - Lectures 2 and 3 provide live code demos for getting started with sockets. We highly recommend watching these for an overview of how to think about network programming - The **[sockets demo code](https://github.com/brown-csci1680/lecture-examples/tree/main/golang-sockets)** provides a complete code example for a client-server application. Your code will follow a similar structure, but will do different things--we don't recommend copying from it directly (this would make your life harder), but it should be a good reference for the overall program structure and examples of boilerplate code - **If you are using C/C++**, you can use [this version](https://github.com/brown-csci1680/lecture-examples/tree/main/sockets-demo) instead. - **[Our favorite links for each programming language](https://brown-csci1680.github.io/documents/)** - As you get started, see [this section](#Resources-and-implementation-notes) for more on how to run our testing tools # Warmup The first part of this guide will lead you as you take your first steps in this assignment: you will implement a client and a server such that your client can successfully connect to your server, send a Hello command, and then wait for and print the Welcome reply. At the same time, we'll show you how to inspect your Snowcast traffic in Wireshark and run our built-in tests, to help with debugging. # Part 0: Initial Setup ### Development environment Before starting, please make sure you have set up our course container environment using [this guide](https://hackmd.io/@csci1680/container-setup). This will set up a development environment you can use to build your projects. If you are unable to get the container environment working on your system or are otherwise unable to run it, please see the container setup guide for instructions on how to contact us and let us know--we want to make sure that everyone can use this environment. The guide also explains how to run Wireshark within your container, which will be helpful for debugging your projects and understanding network traffic. In this guide, we will also help you get started using Wireshark for debugging. ### Cloning the stencil (and recommended filesystem layout) This guide is designed to be your starting point for the Snowcast project. Since you will likely end up using what you implement here in your project, we recommend starting work in your Snowcast repository. You can clone your repository using **[this github classroom link](https://classroom.github.com/a/jInmgNYe)**. To make development easier, we recommend cloning your stencil code for this project so you can access files in it from your development container. To do this, we recommend cloning the stencil in your `DEV-ENVIRONMENT/home` directory, where `DEV-ENVIRONMENT` is the name of the folder you used when you cloned the development container. Here's an example of what your filesystem might look like: ``` - ... |--DEV-ENVIRONMENT | |--docker/ | |--home/ | | |--snowcast-yourname/ # <------- Clone your stencil here! | |--run-container | |-- ... ... ``` # Part 1: Intro to Socket Programming ## Some definitions ### What is a socket? <!-- [Beej's Guide](https://beej.us/guide/bgnet/html/) is an excellent resource for understanding the basics of socket programming - although it uses C as the main language, many of the concepts presented in the guide are transferrable to other languages. Here is how Beej's Guide defines a socket: > You hear talk of “sockets” all the time, and perhaps you are wondering just what they are exactly. Well, they’re this: a way to speak to other programs using standard Unix file descriptors. > > What? > > Ok—you may have heard some Unix hacker state, “Jeez, everything in Unix is a file!” What that person may have been talking about is the fact that when Unix programs do any sort of I/O, they do it by reading or writing to a file descriptor. A file descriptor is simply an integer associated with an open file. But (and here’s the catch), that file can be a network connection, a FIFO, a pipe, a terminal, a real on-the-disk file, or just about anything else. Everything in Unix is a file! So when you want to communicate with another program over the Internet you’re gonna do it through a file descriptor, you’d better believe it. --> As we discussed in lecture 2, **sockets act just like file descriptors between two endpoints of a network connection**: you can read data from the socket (like you would read data from a file descriptor), and you can write data to the socket (like you would write data to a file descriptor). You can also think of a socket as a "pipe" between a client and the server that either endpoint can send data through. ### What is an IP address? What is a port? Roughly, - An **IP address** is the identifier for a network connection endpoint that the **network layer protocol** uses (e.g. IP) - A **port** is the identifier for a network connection endpoint that the **transport layer protocol** uses (e.g. TCP). A "network connection endpoint" here refers to one end of a network connection; for example, if your computer is connected to a remote server, then each of your computer and the remote server are both connection endpoints. [Beej's Guide](https://beej.us/guide/bgnet/html/) (a great network programming resource!) has a good analogy for how IP addresses and ports are used to deliver network packets: > Think of the IP address as the street address of a hotel, and the port number as the room number. In other words, the IP address is used for forwarding a packet to the destination endpoint, while the port is used for sending the packet to the *application using the port* at that endpoint. The kernel is responsible for knowing which applications map to which ports currently running on the system. ## Setting up a server-client connection with TCP The next few sections will guide you through building a client-server program and sending the first few messages of your Snowcast protocol. :::info **Note**: Much of the content here mirrors the sockets demo in Lectures 2 and 3, which implement a different protocol using the same mechanics described here. While you shouldn't just copy the lecture code into your repository, you're welcome to follow along with the lecture example and use it as a reference to get started! ::: ### Creating a listening socket Our first step is to create a socket for the server to “listen” for new connections. From there, clients can send your server a request to start a connection, and then your server can set up a connection to each client. #### Picking a port number First, we need to pick a port number on which the server will listen. In general, port numbers can be any 16-bit number (ie, 0-65535). Port numbers less than 1024 can only be used by root, so you the port you use must be higher. For designing a protocol in general, you're free to pick any port number you like, but, in practice, certain applications typically run on specific port numbers. **For the Snowcast project, your server should listen on port 16800** since it will let you test your work in Wireshark most easily (we'll see why shortly). For now, you can hard-code the port number you choose (i.e. define it as a constant in your server program), but later you will need to take in this value as an argument (per the [server specifications](https://hackmd.io/@csci1680/snowcast-spec#Server-Specification)). #### Listening for new connections The next step is to create the listening socket, which requires telling the kernel some information about the type of socket we want. The kernel has many types of sockets, not all of which involve IP, or even networking. Right now, since we want to establish TCP connections between clients and the server, we need to create TCP socket. The API to create sockets varies by language, but generally follows a similar structure. The diagram and sections below describe the most relevant functions you need in Go, C/C++, and Rust. ![](https://hackmd.io/_uploads/HyazzSTej.png) Regardless of the language you use, there are a few steps that neeed to happen to set up a listening socket. Expand the steps below for your language, which outlines what the process should look like. (Hint: Click the names of the functions for links to relevant documentation!) :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for Go</b></u></summary> In Go, you can accomplish this in two steps: 1. First, you can use `ResolveTCPAddr` from Go's `net` package ([`net.ResolveTCPAddr`](https://pkg.go.dev/net#ResolveTCPAddr)) to create the TCP socket and bind it to your chosen listen port. This returns a [`TCPAddr`](https://pkg.go.dev/net#TCPAddr) that represents the new socket. 2. Next, we need to tell the kernel that we want the socket to listen for new connections. You can do this with [`net.ListenTCP`](https://pkg.go.dev/net#ListenTCP), which takes in the [`TCPAddr`](https://pkg.go.dev/net#TCPAddr) from the previous call. </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for C/C++</b></u></summary> In C/C++, this process requires a few steps: 1. First, use [`getaddrinfo`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch) here to tell the kernel what kind of socket you want. Note that [`getaddrinfo`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch) essentially gives you back a struct you then need to use to make a socket. 2. Next, we need to create the socket itself. Here, you should pass in parameters from the struct returned from [`getaddrinfo`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch) into the system call [`socket`](https://beej.us/guide/bgnet/html/#socket). This returns the file descriptor for the listen socket. 3. Use [`bind`](https://beej.us/guide/bgnet/html/#bind) to bind the socket to the port you chose as the server's listening port. 4. Finally (for now), we need to tell the kernel that we want the socket to listen for new connections. For this part, we use [`listen`](https://beej.us/guide/bgnet/html/#listen); the second argument to `listen` specify the maximum number of waiting connections to clients your server can have at any given time (called the backlog, more on this later). > **Why are there so many steps for C/C++ compared to the other languages?!** The C API most directly mirrors the Linux kernel API. The other languages are all doing the same steps, they just hide some of them from you! In addition, since the C API predates all of the other languages, certain aspects are a bit more complex in order to maintain compatibility with older programs. </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for Rust</b></u></summary> Use [`TcpListener.bind`](https://doc.rust-lang.org/std/net/struct.TcpListener.html#method.bind). This function creates a new [`TcpListener`](https://doc.rust-lang.org/std/net/struct.TcpListener.html), which, as the name suggests, is a TCP socket that is set to listen for incoming connections on the specified address/port. </details> ::: <details markdown="1"><summary markdown="span">"I found some other examples or documentation that show different ways of creating TCP sockets, are they the same?"</summary> For any language, socket APIs can have many different features for convenience or historical reasons--all interact with the same low-level kernel API to perform essentially the same functions. We have done our best to outline a series of function calls that will get the job done, but there are certainly many ways to solve the same problem--you are free to use whichever version you like, so long as it meets the assignment requirements. If you have questions, feel free to ask! </details> :::success **Task:** Start writing your Snowcast server program! Use the functions mentioned so far in the language of your choice to set up a listening socket for your server (in your server program). ::: To use the above functions, you may need to provide some additional parameters to specify how the socket is created. For these, keep in mind the following guidelines, which are good "default values" to use for this lab, and the class in general (so just skim this list over and return here if you need it): - **IPv4 sockets only**: For our projects, we don't need to worry about IPv6 sockets, or dealing with IPv4/IPv6 compatibility, so we use IPv4 only. In Go, this means calling `ListenTCP` with `"tcp4"` as the protocol, or creating sockets with `AF_INET` in C/C++. - **Bind to any address**: Sockets can be bound to one or more *interfaces* by specifying a "bind address." Network interfaces are different network connections available to the system (eg. Ethernet, Wireless, etc.)--we'll learn more about these later. When you create sockets, they should be bound to *all* interfaces on your system, which usually equates to leaving the "bind address" blank. :::info **Hint:** A big part of this task, and systems programming in general, will be figuring out how to structure your programs well (since we don't give you stencil code, hehehe). To do this, reading up on function documentation and looking at some of the examples given in documentation can be really useful for creating a basic structure. Also, don't be afraid to come to office hours if you're struggling with the question "so where should I even be calling this function?" or similar ones! This is a skill we want you to learn, and it's okay to ask for help! ::: :::warning **Warning**: Don't forget error handling code! You should check if any of these functions (and any networking functions you use in general) return an error and handle them accordingly. For this part, if any functions encounter an error, you should just print out the error and exit the program. When the server is just starting up, you can usually do this in one-step with a function like `log.Fatal` in Go or `perror` in C/C++. When you start handling clients, you may need to be a bit more subtle to gracefully handle errors (see lecture 3 for details). Error handling is an essential practice in systems code, and it will save you a lot of debugging time to add this now! ::: Once you've written an implementation (even if you're not fully confident in it yet), continue onto the next step. After this, we'll show you a way you can test your program. ### Waiting for a client to connect If you run your program right now, it will just create the socket and then exit! To actually handle clients, we need to tell the server how to "accept" incoming connections. Each of Go, C/C++, and Rust have a single function call (named "accept", or something similar) to handle a client's connection request, as listed below in the diagram. ![](https://hackmd.io/_uploads/Sksmfragj.png) Either way, the principle of accepting new connections is the same: calling "accept" will normally *block* until a client connects. Once a client connects, "accepting" the connection creates a **new socket** specifically for the connection between the server and the new client, while the listen socket remains unchanged. This is important, because your server should now use the new socket file descriptor to communicate with the now-connected client. However, our server also needs to handle connections from multiple clients, so it should also continue listening on the original listening socket that you made in the previous steps. In other words, *your server should not stop listening for other clients once a new client connects*. Conceptually, you'll want your code to behave something like this (in pesudocode): ``` . . . listen_sock = create_listen_socket() while true { client_sock = do_accept(listen_sock, ...) // . . . Do something with client_sock (later) . . . } ``` To do this for real, consult the following instructions for your language: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for Go</b></u></summary> We recommend using [`net.TCPListener.AcceptTCP`](https://pkg.go.dev/net#TCPListener.AcceptTCP), which behaves similar to C/C++'s [`accept`](https://beej.us/guide/bgnet/html/#acceptthank-you-for-calling-port-3490.) in C/C++ has, except it is specific to incoming TCP connections. [`net.TCPListener.AcceptTCP`](https://pkg.go.dev/net#TCPListener.AcceptTCP) will *block* until the next incoming TCP connection request and return a new [`TCPConn`](https://pkg.go.dev/net#TCPConn), which represents the TCP socket for the connection between your server and the newly connected client. </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for C/C++</b></u></summary> In C/C++, you can use [`accept`](https://beej.us/guide/bgnet/html/#acceptthank-you-for-calling-port-3490.). A key thing to remember about [`accept`](https://beej.us/guide/bgnet/html/#acceptthank-you-for-calling-port-3490.) is that this function will *block* until a new client connects, and will return a *new socket descriptor* for the connection from your server to the client your server has just accepted. </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for Rust</b></u></summary> In Rust, [`TcpListener.incoming`](https://doc.rust-lang.org/std/net/struct.TcpListener.html#method.incoming) works a little bit differently. [`TcpListener.incoming`](https://doc.rust-lang.org/std/net/struct.TcpListener.html#method.incoming) "returns an iterator over the connections being received on this listener." In essence, this method calls [`TcpListener.accept`](https://doc.rust-lang.org/std/net/struct.TcpListener.html#method.accept) in a loop for you, so then all you have to handle is the [`TcpStream`](https://doc.rust-lang.org/std/net/struct.TcpStream.html) returned by [`TcpListener.accept`](https://doc.rust-lang.org/std/net/struct.TcpListener.html#method.accept). This [`TcpStream`](https://doc.rust-lang.org/std/net/struct.TcpStream.html) represents the new socket made for the new client connection, and you will later use it (as a stream) to communicate with the client. The example given in the documentation for [`TcpListener.incoming`](https://doc.rust-lang.org/std/net/struct.TcpListener.html#method.incoming) is pretty good and will help you understand how to use the method. </details> ::: :::success **Task:** Extend your Snowcast server program so that it can accept new client connections, while continuing to listen for other connection requests! ::: When you run your program now, it should just hang. This is good, it means the server is waiting for new connections! ##### Testing your server (so far) The main command you would use to do this is To see if your server is actually listening on the port you chose, we can use the command [`ss`](https://www.man7.org/linux/man-pages/man8/ss.8.html), which stands for "socket statistics." `ss` can view the kernel's table of open sockets to tell us if our server port is open. :::success While your server is running, open up another terminal and run the following: ``` ss -ln | grep <server port number> ``` You should see something like: ```sh tcp LISTEN 0 20 0.0.0.0:16800 0.0.0.0:* ``` This means that one of the sockets on your machine is listening on port 16800 for TCP connections! In other words, if you can see this output, then your server is running and listening for client connections! ::: :::info <details markdown="1"><summary markdown="span">Want to know more about listening sockets on your system?</summary> For information on other sockets that are listening, run: ```sh ss -lnp | grep LISTEN | less ``` You should be able to see a bunch of the sockets being used for various processes on your system. If you run it with `sudo`, `ss` will also output the names of the processes as well. </details> ::: Now that your server is listening for clients and can accept new connections, all that's left to set up is your client program! ### Starting up the client On start up, your client program should create a new TCP socket and connect to your server. To connect to the server, you'll need to provide its IP address and port so that the client knows how to reach it. Since our client and server are running on the same system, we will connect using Linux's *loopback interface*. This is a special, *virtual* network interface that exists only within the local system. This is useful so that programs on the same system can still communicate using networking protocols, even when there's no network! When making connections, the loopback interface always has the same IP address: `127.0.0.1`. We will use this IP when connecting to the server--for now, you can either pass it into your client program as an argument, or define it as a constant. :::info **Note**: We can also refer to this address using the *hostname* `localhost`. Hostnames like `localhost` (or `google.com` or `cs.brown.edu`) are translated into IP addresses by the OS (often using protocols like DNS, which we'll learn about later). If you've been following the functions in this lab, your code should be set up to also translate hostnames automatically, so you can also pass in the name `localhost` instead of `127.0.0.1`. ::: The server's port should be the one you originally chose for your server, port 16800. To build your client, you will create a socket in a similar manner to creating the server. However, instead of creating a *listening* socket, we will be creating one to connect to a specific destination, the server. To do this, the relevant function calls are outlined in the diagram below and in the following sections: ![](https://hackmd.io/_uploads/BkKVMHags.png) :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for Go</b></u></summary> In Go, use [`net.Dial`](https://pkg.go.dev/net#Dial), which takes in a network type (i.e. TCP) and an IP address/port number, and attempts to connect to the specified address. The address/port string should be of the form `address:port`---ie. `127.0.0.1:16800` or `localhost:16800`, to connect to your system's loopback address on port 16800. </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for C/C++</b></u></summary> For C/C++, you again need to do a few steps: First, call [`getaddrinfo`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch) once again to tell the kernel what type of socket you want. Here however, you will need to pass both the _server_’s address and port in as part of the specifications for the socket (along with TCP, IPv4 etc). `getaddrinfo` actually returns a list of possible socket options for making the connection. You will need to iterate over the results to find one that allows you to connect--take a look at [`Beej's guide`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch) or [this old class example](https://github.com/brown-csci1680/lecture-examples/blob/main/sockets-demo/guessing-game/client.c) for details. In the loop over [`getaddrinfo`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch)'s results, you should attempt to create a socket using the current result (by calling [`socket`](https://beej.us/guide/bgnet/html/#socket)) and then attempt to connect to the server using the new socket. To connect to the server, you will need to use the system call [`connect`](https://beej.us/guide/bgnet/html/#connect), which takes in a socket file descriptor (i.e. the one returned from [`socket`](https://beej.us/guide/bgnet/html/#socket)) along with the result from [`getaddrinfo`](https://beej.us/guide/bgnet/html/#getaddrinfoprepare-to-launch). Once [`connect`](https://beej.us/guide/bgnet/html/#connect) successfully returns, you have connected to the server! You can then exit out of your loop, and begin sending data to the server. </details> ::: :::info <details markdown="1"><summary markdown="span"><b><u>Instructions for Rust</b></u></summary> In Rust, use [`TcpStream.connect`](https://doc.rust-lang.org/std/net/struct.TcpStream.html#method.connect), which takes in an address (see the examples in the documentation for the structure of the input address) and attempts to connect to it. </details> ::: :::success **Task:** Start writing your Snowcast client program. Use the functions mentioned in the language of your choice to create a TCP socket for your client control program and connect the new socket to the server using the server's IP address and port number. Make sure your code compiles--we'll test it in the next section. ::: ## Viewing the connection in Wireshark To test your code so far, we can watch the connection attempt happen in wireshark. Wireshark is a great debugging tool to see packets sent over the network as you test your code. Any packets that are sent, Wireshark can see! As part of your warmup, we'll guide you through how to set up Wireshark to view snowcast traffic to demonstrate how you can use it for debugging. :::info This guide assumes that you've already set up and are able to run wireshark per [these instructions](https://hackmd.io/@csci1680/container-setup#Running-Wireshark) from the container setup guide. <details><summary>If you can't run wireshark</summary> If you are unable to run Wireshark, you can skip this part for now--please make sure the course staff is aware that using Wireshark is problematic for you. In lieu of using wireshark, you can debug using print statements--this is less efficient and more error-prone, but it works in a pinch. At this point in the setup, print out something when the connection is established (ie, after `Dial`/`connect` on the client, and after `accept` on the server). If you see your message printed (and nothing returned with an error), your connection was established! </details> ::: ### First time setup: Snowcast dissector Wireshark has many *dissectors* that can decode packets in various protocols. Since we created Snowcast, we have written a custom disssector that can decode (or, dissect) Snowcast packets for you. You can use this to tell if your data is formatted properly! To install the dissector: 1. Open a terminal inside the container. That is, in a terminal on your host machine (or a WSL terminal on Windows), run the command `./run-container`. 2. `cd` to the directory where you cloned the stencil 3. Run the following command ```shell cs1680-user@c808c3104e5a:~/snowcast-stencil $ util/snowcast-dissector/install.sh ``` After you've run this command, your terminal should show a file being copied successfully, like this: ![](https://hackmd.io/_uploads/HyqFOAykp.png) Your Snowcast dissector should now be successfully installed! If you get any errors, please let us know on Ed. ### Starting up Wireshark Next, start up Wireshark (inside the container as described in the [setup guide](https://hackmd.io/@csci1680/container-setup#Running-Wireshark). You can use either the primary method or the backup method, whichever works. Once you have started up Wireshark, you should be presented with a window like the one below: ![](https://hackmd.io/_uploads/ByB2oHRJs.png) ### Starting a capture Packet captures are performed by attaching to one of the system's network interfaces. In this case, we will be capturing on one of the virtual network interfaces inside the container, so we will only see network traffic from container processes. To start a capture, select **Capture** > **Options** from the menus at the top. Each of these options represents an interface on which packets can be received. In general, there are two interfaces to care about: - `lo`, the loopback interface: This contains traffic to and from `localhost`--it lives exclusively inside the container. Your own system has its own loopback network, which is local to your own machine. **You should capture on this interface if you are debugging a program that connects to `localhost`, like your assignments.** - `eth0`: This is the container's link to the outside world (i.e., your host system, and the Internet). **You should capture on this interface to see traffic that leaves the container.** > (Not all machines will have the same naming for these devices, and in some cases they may not all be available. For example, your host system may show a wired and wireless interface.) Here's how to set up a capture for Snowcast traffic: 1. Select the **loopback interface** (since we will only use the localhost network) 2. In the box at the bottom, **enter the capture filter `tcp port 16800 or udp`**. This tells Wireshark that we're only interested in TCP traffic on port 16800 (for the control protocol) or UDP traffic (for the data protocol), so it can ignore everything else Ultimately, your capture window should look like this: ![](https://hackmd.io/_uploads/HJKoqIC0n.png) 4. Press **OK** to start the capture Wireshark should now switch to its main capture window, but you shouldn't see any packets, because we haven't sent any yet! Let's fix that. ### Viewing the connection To watch the connection happen, we need to start both our client and server while Wireshark is running. To do this, you may need to open a few terminals inside the container--if you're using [VSCode inside the container](https://hackmd.io/@csci1680/container-setup#Connecting-VS-Code-to-the-container-highly-recommended), we recommend using VSCode's terminals, which are easy to open and split into multiple panes. To start your client and server: 1. Make sure Wireshark is open and has a capture running, as in the previous section 2. In one container terminal, run your server program. You should see your server start and hang while it waits for a new connection--this is expected. 3. Start your client program If the connection was established, you should see some packets in Wireshark like this: ![](https://hackmd.io/_uploads/SkQWaLR03.png) **"But wait, I didn't send anything yet!"** You're correct, *you* (the developer) didn't. These three packets are part of how represent the initial TCP packets that should be sent between your client and server when your client attempts to connect to your server--we call them the "TCP handshake." Don't worry too much about understanding these just yet! We will cover what these packets mean closer to the TCP project. For now, this just means our connection is set up properly, yay! We can represent these packets on our flow diagram below: ![](https://hackmd.io/_uploads/B1UBGS6xj.png) ## Handling a client (server side) Back to the implementation: at this point, your server program can accept a new client connection, but we haven't sent any data yet! In this section, you will add to your server's current functionality so that it can handle communication with multiple clients concurrently. You could implement this in many ways, but we recommend you start by making a thread per new client connection, which can then do any client-specific tasks. The important thing here is to make sure you store enough data about the client somewhere, perhaps by making a new data structure. When you create the new thread, you will need to pass this information to the thread so that it can communicate with the client. For now, the most important information to pass to the thread is the client's socket. For this, see the instructions based on your language: :::info <details markdown="1"><summary markdown="span">Instructions for Go</summary> All of the information about your client's socket is contained in the `net.Conn` object returned by `net.AcceptTCP`. This object has various helper methods for sending/receiving on the socket, getting the client's IP address, and more. Consider creating a goroutine for each new client, which takes the client's `net.Conn` as an argument. Later, you may wish to pass a custom `struct` instead that also includes more information. </details> ::: :::info <details markdown="1"><summary markdown="span"> Instructions for C/C++</summary> Right now, you should probably store: - The socket file descriptor the server uses to communicate with the client - The IP address of the client's socket (returned in the argument from `accept`) - The thread ID of the client thread We recommend making a `struct` to hold these and passing them to the client thread. Take a look at [this old class example](https://github.com/brown-csci1680/lecture-examples/blob/main/sockets-demo/guessing-game/server.c) for details. </details> ::: :::info <details markdown="1"><summary markdown="span">Instructions for Rust</summary> All of the information about your client's socket is contained in the `TcpListener` returned when accepting the client's connection This object has various helper methods for sending/receiving on the socket, getting the client's IP address, and more. </details> ::: As you build your snowcast project, it will be important to consider what *state* you need to keep track of for each client. For example, when you start sending data from radio stations to client listener programs, you may also want to store the UDP port of the client's listener. Keep this in mind as you think about how you'll implement the protocol. :::success **Task:** In your server program, for each new client connection, store the new client's data and initialize a thread to handle communication with the new client. For now, just make your client thread print out something like "Client connected!" and then exit. If you see your message being printed when a client connects, your threads are being created! ::: ## `Hello` and `Welcome` Now that your server can handle communication with multiple clients concurrently, it's time to send something useful over the connection. This is where we start needing the [Snowcast Specification](https://hackmd.io/@csci1680/snowcast-spec), specifically the [Protocol Specification](https://hackmd.io/@csci1680/snowcast-spec#Protocol-Specifications), which defines the set of rules and message formats for how applications implementing Snowcast should communicate. As seen in the diagram below, the protocol you are asked to implement in Snowcast requires your client to first send a `Hello` message, and once your server receive's the client's `Hello`, it should respond with a `Welcome`Other message types (such as `Announce`, `SetStation`, and `InvalidCommand`) should be sent for other parts of the protocol, but you don't need to worry about them for this warmup. ![](https://hackmd.io/_uploads/ryW8fBTlj.png) As a reminder, a client's `Hello` message must be 3 bytes long and has this format: ```go Hello: uint8 commandType = 0; uint16 udpPort; ``` And this is what a server's `Welcome` message must by 3 bytes long and has this format: ```go Welcome: uint8 replyType = 2; uint16 numStations; ``` Therefore, to send a message, you should send a series of **bytes** corresponding to one of these messages. Each of Go, C/C++, and Rust have function calls for sending and receiving data over a socket, as summarised in the table below: | Function | C/C++ | Go | Rust | | -------- | -------- | -------- | -------- | | Send | [`send`](https://beej.us/guide/bgnet/html/#sendrecv) | [`(net.Conn).Write`](https://pkg.go.dev/net#Conn) | [`TcpStream.write`](https://doc.rust-lang.org/std/net/struct.TcpStream.html) | | Receive | [`recv`](https://beej.us/guide/bgnet/html/#sendrecv) | [`(net.Conn).Read`](https://pkg.go.dev/net#Conn) | [`TcpStream.read`](https://doc.rust-lang.org/std/net/struct.TcpStream.html) | There are several ways to "compose" the data to send: for example, you can create a data-type (such as a `struct`) to represent the message, and then "serialize" or "marshal" it into a byte array. Alternatively, you can call your language's `send` function once for each message field, after converting each field into the appropriate size. There are many ways to do these operations in each language. For examples, we recommend starting by taking a look at the notes for Lectures 2 and 3, and then consulting your language's resources for the method you like best. :::success **Task**: Now that you've started building and parsing messages, update your client and server to do the following: 1. Once your control client has successfully connected to your server, send a `Hello` message to the server and wait for a `Welcome` in response. Be sure to send your messages as bytes in the format described by the Specification. When sending the `Hello`, you can start by hard-coding any value for `udpPort` (but you'll need to change this later.) 2. In your server program, once your server receives a client `Hello`, send a `Welcome` response back to the client. After you send your first packet, take a look at the output in Wireshark to make sure it looks correct. Whens ending the `Welcome`, you can start by hard-coding any value for `numStations` (but you'll need to change it later). 3. When your control client receives a Welcome message, you should print a message **to stdout** in **this exact format** (followed by a newline): ``` Welcome to Snowcast! The server has N stations ``` where `N` is the `NumStations` received in the Welcome message. This will ensure that your output matches the format of the spec, and our autograder (more info on this in the [protocol specification](https://hackmd.io/@csci1680/snowcast-spec#TCP-Client-snowcast_control). ::: ### Checking your work After you send each packet, check Wireshark's window. It should look something like this: ![](https://hackmd.io/_uploads/By7XJD0R2.png) Let's break down what we're seeing: 1. In the "protocol" column of Wireshark's packet view, you should see one packet labeled `CS168SNOWCAST`, which means that Wireshark correctly ran the dissector and tried to decode your traffic. **Click on this packet**. <details><summary>If you don't see this</summary> If you are sure you sent a packet but don't see `CS168SNOWCAST`: 1. Make sure your server is using port 16800 2. Make sure you have installed the [Wireshark dissector](#First-time-setup-Snowcast-dissector) If you still have issues getting Wireshark to show your traffic, please post on Ed or let us know in hours. </details> 2. In Wireshark's lower panes, you should see a breakdown of the packet's headers and data. At the bottom of the protocol stack, expand the part labeled **"Snowcast Protocol Data"** 3. Take a look at the data. Does it match what you intended to send? You can click on each field to look at the raw bytes that were sent. :::warning **If the decoded data doesn't match what you wanted to send**: Check how you are encoding the packet. Specifically, make sure you are using [network byte order](https://beej.us/guide/bgnet/html/#byte-order) to send packets, and that you are converting data back to host byte order once you have received a packet. ::: Using wireshark to debug, keep iterating until you can send a hello and welcome message that are properly formatted--you'll know they're correct when what you see in Wireshark matches that you intended to send. Congratulations, you have your Snowcast server and client programs exchanging messages! ## Preparing your code for Snowcast The client and server programs you created here can serve as your starting point for Snowcast: your TCP client is the start of the Snowcast control client (ie `snowcast_control`), and your server implements (you guessed it!) `snowcast_server`! Before we can check off your work for the milestone with our autograder, you'll need to refactor your code a bit so that your programs have the same names and command-line arguments as the Snowcast programs. While this may seem annoying now, we want to make sure that you can interact with the autograder so that you can test your work--and so we can test our autograder! :::info More concretely, you should make sure that your code adheres to the following requirements (this list sourced from the [submission requirements](https://hackmd.io/@csci1680/snowcast-f24#Submitting-your-work) and [implementation specification](https://hackmd.io/@csci1680/snowcast-f24#Implementation-specifications): 1. As described in the submission instructions, create a `Makefile` such that `make` compiles your client and server and gives them the names `snowcast_control` and `snowcast_server`, respectively. 2. Update your client and server program to accept the same command-line arguments as `snowcast_control` and `snowcast_server`. Specifically: For example, your server should be able to run as: ``` $ ./snowcast_server <listen port> <file0> [file 1] [file 2] ... ``` Your client program should run as: ``` $ ./snowcast_control <server IP> <server port> <listener port> ``` <details><summary><i>What do the different brackets (&lt;&gt;, `[]`) mean? </i> </summary> This is a convention from man pages for specifying arguments: - Arguments in &lt;angle brackets&gt; are **required** - Arguments in `[square brackets]` are **optional**: you can leave them out Thus, `snowcast_server` takes in a minimum of 2 arguments (listen port, one file), but can also accept any larger number of arguments (for additional files). </details> 3. (If you have not done so already) Update your programs to take in the client and server's address and port numbers as arguments, as shown in the spec above. 4. (If you have not done so already) Make sure your client is printing out the Welcome message in the format specified [here](#Hello-and-Welcome) 4. Update your server's Welcome message to list the **the number of files passed in on the command line** when running `snowcast_server`. You don't need to handle opening/reading from files right now, but make sure that the number of stations aligns with the command-line arguments (i.e. `NumStations == len(args[2:])`). 5. In `snowcast_control`, update your client's Hello message such that **value for `udpPort` is the value passed as `<listener port>` on the command line**. For the final version, the value you will pass for `<listener port>` will be the port on which you run the listener client--for now, you can just pass in any unsigned 16-bit integer. ::: ### Testing your milestone implementation To test your milestone work, you should do the following: 1. Run your client and server binaries and look at the packets in Wireshark: you should see `Hello` and `Welcome` packets that contain the data you intended (as described in [Checking your work](#Checking-your-work)) 2. Double-check that your client and server binaries follow our requirements on command-line arguments (as in the previous section) 3. Once you've verified (1) and (2), you can test your work using our built-in tester available in your stencil repo. See [this section](#Running-the-built-in-tester) for instructions--for this part, you'll want to run the **milestone tests only**. ### Final step: running the reference version Before we're done, you should also try running our **reference implementation**: this is a complete version of the Snowcast client, server, and listener that meets our requirements, which is located in the `reference` directory of your repository. You should run the reference version now to make sure you get a sense of how things should work, which will also demonstrate how your program should operate. To do this: 1. If you already have a wireshark capture open to look for snowcast traffic, stop and restart it so you have a clear view. If you don't have a capture open, start one now. 1. Open 3 terminals inside your container (either standalone terminals, or just split your VSCode terminal in 3 parts). 2. Check the working directory for each terminal to make sure each one is located in your stencil directory (eg. `snowcast-youruser`). If not, `cd` into your stencil directory. 3. In one terminal, run the server, listening on port 16800, and pass it all of the mp3 files in the `mp3` directory, like the example below. Note that we provide two versions of each binary (for Intel or Apple Silicon systems)--be sure to run the version for your architecture, like this: ```shell ## Run the reference server (x86-64 systems) reference/snowcast_server 16800 mp3/* ## Run the reference server (M1/M2/M3 macs) reference/arm64/snowcast_server 16800 mp3/* ``` 3. In another terminal, run the control client and pick a random port number to use for the listener (eg. 1234): ```shell ## (If you have an M1/M2/M3 mac, use ## reference/arm64/snowcast_control instead) reference/snowcast_control 16800 1234 ``` Once you start your control client, you should see the Hello/Welcome handshake process in Wireshark, similar to your version! 4. In another terminal, run the listener client. To measure the streaming rate, we can pipe the output to a program called `pv`, which measures the rate at which data is written to it, like this: ```shell ## (If you have an M1/M2/M3 mac, use ###reference/arm64/snowcast_listener instead) reference/snowcast_listener 1234 | pv > /dev/null ``` 5. In your control client, **subscribe to any station** by typing in the station number. You should see a `SetStation` and `Announce` message appear in Wireshark, and, in your listener terminal, you should now be seeing `pv` measure some data at 16KiB/s! Here's an example of what it should all look like (if it's too small to see, right click and do "open image in new tab"): ![](https://hackmd.io/_uploads/rJo-uZoyT.png) You've now run the reference version, yay! This should give you a good baseline for how things should look as you run your implementation. **As you write your own implementation, remember this workflow--this is the best way to test your work early-on to make sure things are working.** We know it's tempting to just run the autograder/built-in tester, but it's you'll find it's a **LOT** easier to interpret the output from this view! ### Submitting your warmup <!-- :::warning We're still setting up our autograder and will release details on how to upload your work to Gradescope soon! **If you can pass the milestone tests using the built-in-tester, you're set for the implementation part!** ::: --> Congrats on taking the first steps toward your Snowcast implementation! To submit your work, you should push all of the following to your Github repository and submit it as described [here](https://hackmd.io/@csci1680/snowcast-f24#Milestone---20): - All of your code for your client and server - A `Makefile` that compiles your `snowcast_control` and `snowcast_server` binaries when running `make` (as described [here](https://hackmd.io/@csci1680/snowcast-f24#Autograder-setup-how-we-will-build-your-code)) - A screenshot of Wireshark while you run the **reference version**: your screenshot should show a `SetStation` and an `Announce` command (you can enter the filter `cs168snowcast` to get rid of the UDP traffic) - Your design document with a plan for the rest of your implementation (details [here](https://hackmd.io/@csci1680/snowcast-f24#Milestone-part-2-Design-Document)) ## Closing thoughts This concludes the warmup portion of this guide. We hope this provides some first steps to get started. See the rest of this document for implementation notes and guides on testing, debugging, and using other resources. If you have questions on anything, please don't hesitate to ask us on Ed or in hours! # Essential implementation resources The following sections provide guidelines on helpful resources, as well as how to debug and test your code. We'll update this section with more resources and tutorials as the project is out and everyone has questions. If you aren't sure how to do something, just ask! Here are our most important resources: - We provide a complete [reference implementation](#Reference-Implementations) of the assignment you can run to see how it works! - Once you can verify your implementation works manually, you can [run many of our autograder tests yourself](#Running-the-built-in-tester) to help debug. - The **[sockets demo code](https://github.com/brown-csci1680/lecture-examples/tree/main/golang-sockets)** provides a complete code example for a client-server application. Your code will follow a similar structure, but will do different things--we don't recommend copying from it directly, but it should be a good reference for the overall program structure and examples of boilerplate code ## Reference Implementations We have provided reference versions of the client and the server that follow the protocol and meet all the requirements, located in the `reference` directory of your repository. :::info **Apple-silicon (M1/M2/etc) mac users**: You should use the versions of these binaries in the `reference/arm64` directory instead--these binaries are compiled for your CPU architecture. :::: We highly recommend using the reference versions to help make sure you understand the protocol, and make sure that your programs interoperate with the reference. You can test your adherence to the protocol based on how well your programs interact with them, ie. run your client with reference server, and vice versa. For instructions on how to test with the reference, see [here](#Testing-streaming-rate)--just substitute the reference version for any program. (The linked instructions are about streaming, but apply generally--you can skip the parts about `snowcast_listener` and `pv` unless you are explicitly testing streaming). For a visual guide on how to run the reference versions, see the [gearup recording](https://brown.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=f6e1aba0-a2e1-4918-a68d-b07d01764600). ## Running the built-in tester Our stencil repo includes tester programs for your server and control clients. **These testers run almost all of the tests in the autograder**, so you can check your work on your own before submitting! Even better, you can see the test traffic in Wireshark, which will help you debug! To run the tests, we provide a script `run_tests` in the `util` directory of your stencil repo. You can run it like this: 1. Open a terminal inside the container environment 2. `cd` to the directory that holds your stencil 3. Build your code--you should make a `Makefile` to do this, if you have not done so already. The result should be that binaries `snowcast_server` and `snowcast_control` are located in this directory 4. Run the following: ```shell # Make sure the snowcast_* binaries are in this directory cs1680-user@c808c3104e5a:~/snowcast-stencil:$ ls mp3 reference snowcast_control snowcast_server stations util [. . .] # Run the tests (here, just the milestone tests--see below for more options) cs1680-user@c808c3104e5a:~/snowcast-stencil:$ util/run_tests milestone ``` From here, the tester will look for your `snowcast_control` and `snowcast_server` binaries in this directory and run the specified set of tests. For the milestone tests, the output should look like this: ![](https://hackmd.io/_uploads/rJGZMdAA2.png) If your output looks like this, your milestone tests passed, yay! :sunglasses: If any tests fail, you'll see a bunch of log output that contains some info about what failed that may help with debugging. See [this section](#If-you-have-failing-tests) for guidance on how to think about debugging tests, and for some common things to check. As you continue with the project, you can run the full suite or tests, or single tests just for debugging. See the next section for details. ### Our favorite testing commands Here's a list of our favorite testing commands (run in the same way as the steps in the previous section): - Run the full test suite: `util/run_tests all` - Run the test suite on the reference version: `util/run_tests --bin-dir reference all` - Stop on the first test failure (**great for debugging!**): `util/run_tests --fail-fast all` - **Run a single server test**, with output decode-able in Wireshark (**great for debugging!**): ``` SERVER_PORT=16800 util/run_tests server -test.run="TestServer/TestCompletesHandshake" ``` - Show more options: `util/run_tests --help` For example, to make sure all the test are working on your system, you can run the tests on our reference implementation (located in the `reference` directory, or `reference/arm64` on for Apple Silicon macs) like this: ``` # Run the full test suite on the *reference* version cs1680-user@c808c3104e5a:~/snowcast-stencil:$ util/run_tests --fail-fast --bin-dir reference all ``` This should take a couple of minutes to run. The end result should look like this: ``` === RUN TestServer === RUN TestServer/TestAcceptsClientConnection === RUN TestServer/TestAcceptsMultipleConnections === RUN TestServer/TestClosesConnectionOnInvalidCommandType === RUN TestServer/TestCompletesHandshake === RUN TestServer/TestInvalidStationFails === RUN TestServer/TestMultipleHellosFails <...> --- PASS: TestServer (23.99s) --- PASS: TestServer/TestAcceptsClientConnection (0.14s) --- PASS: TestServer/TestAcceptsMultipleConnections (0.25s) --- PASS: TestServer/TestClosesConnectionOnInvalidCommandType (0.23s) --- PASS: TestServer/TestCompletesHandshake (0.13s) --- PASS: TestServer/TestInvalidStationFails (0.15s) --- PASS: TestServer/TestMultipleHellosFails (0.15s) --- PASS: TestServer/TestNoStationsFails (0.54s) --- PASS: TestServer/TestPartialHelloTimeoutFails (0.39s) <...> PASS ``` ## If you have failing tests If you have test failures, please check the following first: 1. **If all your tests fail immediately**, make sure that the directory where you run the tests contains your `snowcast_*` binaries and that your programs follow [our specification on command-line arguments](https://hackmd.io/@csci1680/snowcast-spec#Implementation-specifications). If that doesn't help, keep reading . 3. **Run your programs *manually* (ie, no tester)**: Sometimes the tester isn't able to give feedback when your program crashes in certain ways (we're working on improving this). You can often get really good feedback by opening up 3 terminals and running your server, control client and (if applicable) listener manually: start up your server, connect your client, and then try and [replicate the what the test does](https://hackmd.io/@csci1680/snowcast-autograder-tests). If you're unsure how how to run the programs manually, see the [here](#Testing-manually), or the gearup video (Linked in the Ed FAQ) for a demo. To test interoperability, you can try out our [reference server](#Reference-Implementations) with your clients (and vice versa) to make sure you're implementing the protocol correctly. 5. **Check the details of how the test runs**: We have documented broadly [what each test does](https://hackmd.io/@csci1680/snowcast-autograder-tests) and what features it requires from your program. Note that some tests require you to have implemnted certain REPL commands (eg. the `p` or `q` command) so that we can get information we need in the test--see the list for details. 4. **Pay attention to what you print (and where)**: Some control client tests read your program's `stdout` (ie, stuff you print) and check it against [the specification](https://hackmd.io/@csci1680/snowcast-spec#Implementation-specifications). Make sure that you don't have any extra print statements, or those that you do have print to **`stderr`, NOT `stdout`**. Also make sure messages are printed with a newline (`\n`) after them. If you use `fmt.Println`, a newline is added automatically. 5. **Check how you read from TCP sockets**: Remember that reading from a TCP socket (eg. `conn.Read`) may not always return the same number of bytes you expect. To handle this, be sure that you are always reading the correct number of bytes, e.g. with `io.ReadFull`. For more info, see [this FAQ](#Reading-on-a-TCP-socket-isnt-returning-the-correct-number-of-bytes-whats-going-on). 6. **Run the failing test individually** ([example](#Our-favorite-testing-commands)) and try to understand what each test is doing based on 1) the test output, and 2) wireshark. To see what's happening in wireshark, follow [these steps](#Viewing-the-connection) as you run the test and take a look at the snowcast messages you see. 7. **Make sure you are properly closing your server's <u>listen socket</u>**. Our tester needs to bind to this socket on the same port, so it may fail if your server does not clean up the socket. See here for more info. 8. **Please check Ed**, especially the **Snowcast reading list/FAQ thread**. We have a section for **known tester issues**, and otherwise there may be other posts relevant to your situation. We try hard to keep this updated! 9. To help you debug further, see our **[Snowcast Debugging Guide](https://docs.google.com/document/d/1rE2hxkI6bjCfQBis-oAE2YmthOkAD_6BRoL4xys_SGo/edit?usp=sharing)**, which outlines a general procedure for thinking about debugging and describes some more detailed issues. **If you are convinced that your program works but your tests are still failing**, please don't panic--just document this in your readme when you submit. When grading, we will manually review your autograder results and will make adjustments as necessary. Autograders exist to help reduce our grading workload, but rest assured that your grade is ultimately determined by a human. <!-- ### `run_test` reference `run_tests` has a lot of options to help you debug. Here's a command full command reference: ``` run_tests [OPTIONS] <command> [test args] ``` `<command>` controls which test suit is run. Possible values are: - `milestone`: Tests for the milestone part - `server`: All `snowcast_server` tests - `control`: All `snowcast_control` tests - `all`: The full test suite Useful options are: - `--bin-dir <dir>`: Specify where the `snowcast_*` binaries are located - `--fail-fast`: Stop on the first test failure (**great for debugging!**) --> ## Testing manually See [this section](#Final-step-running-the-reference-version) on how to run the reference version for some instructions on how to run your programs manually--to run your program, just run your versions of the snowcast binaries instead of the reference binaries. The gearup recording (linked in the Ed FAQ) also has a demo of this. Some things to keep in mind when testing: - To try and isolate a problem, you can run a combination of your programs and the reference versions. Not sure if your bug is in the client or the server? Run the reference client, and your server, and vice versa! - In general, you can skip the section about `snowcast_listener` and `pv` unless you are testing streaming. ## Testing streaming rate :::info **Note**: This section is about how to test streaming. For a conceptual overview of how to stream at a precise rate, see [this FAQ](#How-to-think-about-streaming). ::: If you want to test your program's streaming rate independent of the tester, you can use a a tool like `pv`. `pv` is a built-in Linux tool that passes input from stdin to stdout, and prints statistics about the rate at which it is receiving data to stderr. We'll be testing to see that your rate is consistently 16 KiB/s. You can run it as follows: 1. Open 3 terminals inside your container 2. In one terminal, start your server, eg. `./snowcast_server 16800 mp3/*` 3. In another terminal, start your control client. Pick a random port number to use for the listener (eg. 1234): `./snowcast_control localhost 16800 1234` 4. Subscribe to any station using your control client. 5. Run your listener client using the same listener port number you provided in the previous step, and pipe the output to `pv`, like this: ``` ./snowcast_listener 1234 | pv > /dev/null ``` Here's an example of what it should all look like (if it's too small to see, right click and do "open image in new tab"): ![](https://hackmd.io/_uploads/rJo-uZoyT.png) This sends all the data to the `pv` program and then discards it. `pv` should print the average rate of data output by the listener (press `Ctrl+C` to quit. We will test that your average streaming rate is roughly `16KiB/s`. :::info **Streaming rate not what you expect?** To start debugging, the first thing we recommend is to **use the reference version of the listener with your server** to help rule out if the problem is with your server or your listener. Once you know your server works (the most important part), try using your own listener client. The reference versions of each program are located in the `reference` directory of your stencil or `reference/arm64` on M1. See [here](#Reference-Implementations) for details. ::: ## Stencil updates In the event of an update to the tester or other helper scripts, we may release an update to your starter repo. If this happens, you will receive a Pull Request (PR) from Github Classroom to merge our changes into your repository. To do this: 1. Open your repo on Github and go to the "Pull Requests" tab. You should see one PR from Github Classroom labeled "Sync assignment", like this: ![github-pr-list-annotated](https://hackmd.io/_uploads/SyvSHsTaC.png) 2. Open up the PR, scroll down to the bottom and click, "**Merge the pull request"** and then **"Confirm merge"**, like in the figure. This should merge our changes into your repo! ![github-pr-accept](https://hackmd.io/_uploads/Bk76LiTaC.png) 3. In a terminal where you use git, pull our latest changes however you normally pull code from git (eg. `git pull origin main`). You should see show some new changes, like this: ![github-pr-applied](https://hackmd.io/_uploads/ByW2LsaT0.png) # Frequently Asked Questions ## My autograder tests aren't working! See [here](#If-you-have-failing-tests). ## What is network byte order? Byte order refers to the order in which data of a multi-byte value is sent over the network. So for example, a 16-bit integer `0xAABB` could be sent with the most significant byte first, ie. `{0xAA, 0xBB}` (called "big endian"); or with the least significant byte first `{0xBB, 0xAA}` (called "little endian"). Network byte order uses big endian. Go provides a [binary](https://pkg.go.dev/encoding/binary) package, and Rust provides a [byteorder](https://docs.rs/byteorder/latest/byteorder/) crate. In C/C++, you may find utilities similar to `ntohs`, `htons`, `ntohl`, and `htonl` from the `<arpa/inet.h>` library useful. ## How should data be sent/received? ### C The primary method of I/O through network sockets is [`send()`](https://linux.die.net/man/2/send)/[`recv()`](https://linux.die.net/man/2/recv). One can then pass in a pointer to a struct: ```c struct_t s = {1, 2, 3}; send(sockfd, &s, sizeof(s)); ``` ### Go In messages that are of a fixed size, one option is to create a struct that contains the message data and write that struct into a buffer directly using `binary.Write()` in the [binary package](https://pkg.go.dev/encoding/binary). For example, ```go buf := new(bytes.Buffer) var pi float64 = math.Pi err := binary.Write(buf, binary.BigEndian, data) ``` Data from a buffer can be read into a struct using `binary.Read()`, or any of the utilities within `io`/`bufio`. ### Rust A byte slice or vector can hold the bytes of each field within a struct: ```rust // If we have a `let s = Struct { field1: u8, field2: u16}`: let mut buf = vec![0; 3];// or Vec::with_capacity(mem::size_of::<Struct>()); let num = s.field2.to_be_bytes(); buf[0] = s.field1; buf[1..].copy_from_slice(&num); ``` This can then be written to a `TCPStream` or `UDPSocket`. ## How does the server know how where to send UDP packets to the listener? In our protocol, we assume that the control client and listener client run on the same machine and have the same IP address. You can find the client's IP address based on the socket information returned when accepting a new connection. (In Go, this is part of the client's `Conn`, in C, this is returned in one of the arguments to `accept`.) For the listener's port number, you should use the UDP port number from the `Hello` message. For more information, see the [Song Data Protocol specification](https://hackmd.io/@csci1680/snowcast-f24#Song-data-protocol). ## How can we time out a socket operation? In Go/Rust, sockets have a `SetReadDeadline`/`SetWriteDeadline` function that can be used to specify a deadline on I/O operations. In C/C++, you can use `setsockopt` with `SO_RCVTIMEO`/`SO_SNDTIMEO` and a `struct timeval` to configure a socket before executing I/O operations. ## Which IP version should the server and client support? To ensure compatibility, we recommend creating all of your sockets as **IPv4 only** sockets. Your implementation does not need to support IPv6. ## Reading on a TCP socket isn't returning the correct number of bytes, what's going on? This is actually the expected behavior of reading on a TCP socket, since TCP is designed to provide a continuous *stream* of bytes, rather than discrete messages. For more info on this, see the start of Lecture 4, and/or expand the box below: <details><summary><b>Why does TCP do this?</b></summary> Let's say we wanted to read 5 bytes on a socket into a buffer, like this: ```go buffer := make([]byte, 2) bytesRead, err := conn.Read(buffer) // ... error handling, etc ... fmt.Printf("Read %d bytes\n") ``` When reading from a TCP socket (here, `conn.Read`, the underlying kernel system call (`read`) unblocks whenever **any** amount of data is available, so the number of bytes returned (`bytesRead`, in this example) may be less than the size of the buffer. **Why?** it's possible that not all 5 bytes have been received by the OS yet--maybe they were sent at different times, or maybe there was delay or packet loss in the network and not all of the bytes have arrived yet. Alternately, many protocols have messages of different sizes, so it may be necessary to read a message into a larger buffer and then use `bytesRead` to determine the size. From a conceptual perspective, the kernel considers data sent by TCP as a continuous stream of bytes, rather than discrete messages: the kernel has no idea where our "messages" start and end--so it's doing the best it can by providing what data is has right now. </details> <br /> **How to deal with it**: To read exactly N bytes, you should read on the socket repeatedly (say, in a loop) until N total bytes have been received (or an error occurs). If you are using Go, we recommend checking out `io.ReadFull` ([doc, with example](https://pkg.go.dev/io#ReadFull)), which is designed for this purpose. ## How to think about streaming? Here's a way to start breaking down this problem. Based on the protocol specifications, there are two constraints for how you need to send song data: 1. The goal is to send out song data to all listeners on a station at an **average rate of 16KiB/s, which is 16384 bytes every 1 second** 2. Per the [song data protocol](https://hackmd.io/@csci1680/snowcast-f24#Song-data-protocol), our **UDP data packets must be no larger than 1500 bytes**, which is a typical maximum packet size for packets sent over the Internet. To stream data at a periodic rate, you should send "chunks" of size <= 1500 bytes at a regular interval, say `t_chunk`, such that the average adds up to 16KiB's. <details><summary>How to find <code>t_chunk</code>? (Click to expand)</summary> Let's say we use a chunk size of 1024 bytes. That means that every second we should send `(16384 total bytes/sec)/(1024 bytes/chunk)= 16 chunks`. To send data at an even rate of 16KiB/s, we should space out these chunks at even time intervals, such that: `t_chunk = 1 sec / 16 chunks = 0.0625s/chunk = 62.5ms/chunk` </details> At every `t_chunk` time interval, you'd want to send out a chunk of song data to every connected listener, ie something like this: ``` while(1) { // Read a chunk from the file for all connected listeners { // Send this chunk to the IP:port of that client's listener } // Sleep for t_chunk } ``` Each time this loop runs, you'd read the next chunk from the station's file and continue the process. When you reach the end of the file, you should continue reading again from the beginning (and send an `Announce` message). To measure your streaming rate, see [Testing streaming rate](#Testing-streaming-rate). Once you are confident you are close, try the `TestStreamRate` tests in the server's test suite. #### How to stream more precisely? In general, you don't need to be incredibly precise about sending a message every `t_chunk`, so long as you get a rate close to 16KiB/s. If you have issues, one way to be a bit more precise is to measure the time it takes to send out the data, and subtract this from the sleep time, ie: ``` while(1) { // Read a chunk from the file t1 = current_time() for all connected listeners { // Send this chunk to the IP:port of that client's listener } t2 = current_time() diff = t2 - t1 sleep(t_chunk - diff) } ``` ## What if the streaming rate is not exactly `16KiB/s`? We will be lenient when testing the streaming rate, but you should expect your average streaming rate to be around `16KiB/s`. If you pass the `TestStreamRate` tests, you're fine. To measure your streaming rate manually, see [here](#Testing-streaming-rate). For info on how to think about streaming conceptually (if your rate is incorrect and you don't know why), see [this FAQ](#How-to-think-about-streaming). ## How do I play sound? **Your implementation is **not** required to play music--we will only test that you send UDP data at a constant rate of `16KiB/s`.** Unfortunately, our container environment cannot play sound at present--we are working on a solution to this and should have an update soon, but this is not (and will not become) a requirement. However, if you are developing locally, you can pipe the output of your listen to an mp3 player to hear sound: `./snowcast_listener port | mpg123 -` If the output is stuttery consider trying `mplayer` instead of `mpg123`. If you are able to play sound, you can also pipe the output of `pv` into `mpg123` or `mplayer` to play the sound while measuring the bitrate. ## Tester issue: `bind: address alerady in use` This issue can occur if your server does not close its **listen socket**. Listen sockets are "bound" to a particular port in the kernel--if the socket is not closed, the kernel may still think the socket is bound to your process when it exits, preventing you (or our tester) from binding to it again! To avoid this, you should **make sure that your code correctly closes your listen socket** when your server exits, or if it encounters an error. :::warning **Note**: In Go, we recommend closing sockets using [`defer`](https://gobyexample.com/defer), which is usually a good way to make sure resources are cleaned up. However, note that deferred statements **will not run** if your program exits with `log.Fatal` or `os.Exit`, since these functions force your program to terminate immediately. ::: Usually, the kernel will figure out your process has exited and remove the old binding quickly. If it doesn't go away fast enough, you should be able to clear it by [stopping the container](https://hackmd.io/@csci1680/container-setup#Stopping-the-container).