Snowcast Setup Guide and Warmup

Introduction

This guide will provide some resources for getting started and building your implementation for the Snowcast project. The first part of this document is our "Warmup", which is designed to help you get oriented with the project and get started with socket programming. The remainder of this guide contains helpful resources for building your implementation and testing your work as you continue with the project.

Getting Started

You may implement this project Go, C, C++, or Rust. If you are unsure, we recommend using Go, even if it is new to you, as class examples this year will use it. We have curated a list of resources for each language here.

If you want to use Rust, note that only some members of the course staff can provide language support (though we are all happy to discuss conceptual questions!). For details, see the course staff list.

Essential Resources

We have a number of documents and resources to help you get started with thinking about sockets and network programming:

• Lectures 2 and 3 provide live code demos for getting started with sockets. We highly recommend watching these for an overview of how to think about network programming
• The sockets demo code provides a complete code example for a client-server application. Your code will follow a similar structure, but will do different things–we don't recommend copying from it directly (this would make your life harder), but it should be a good reference for the overall program structure and examples of boilerplate code
  • If you are using C/C++, you can use this version instead.
• Our favorite links for each programming language
• As you get started, see this section for more on how to run our testing tools

Warmup

The first part of this guide will lead you as you take your first steps in this assignment: you will implement a client and a server such that your client can successfully connect to your server, send a Hello command, and then wait for and print the Welcome reply. At the same time, we'll show you how to inspect your Snowcast traffic in Wireshark and run our built-in tests, to help with debugging.

Part 0: Initial Setup

Development environment

Before starting, please make sure you have set up our course container environment using this guide. This will set up a development environment you can use to build your projects. If you are unable to get the container environment working on your system or are otherwise unable to run it, please see the container setup guide for instructions on how to contact us and let us know–we want to make sure that everyone can use this environment.

The guide also explains how to run Wireshark within your container, which will be helpful for debugging your projects and understanding network traffic. In this guide, we will also help you get started using Wireshark for debugging.

Cloning the stencil (and recommended filesystem layout)

This guide is designed to be your starting point for the Snowcast project. Since you will likely end up using what you implement here in your project, we recommend starting work in your Snowcast repository. You can clone your repository using this github classroom link.

To make development easier, we recommend cloning your stencil code for this project so you can access files in it from your development container. To do this, we recommend cloning the stencil in your DEV-ENVIRONMENT/home directory, where DEV-ENVIRONMENT is the name of the folder you used when you cloned the development container.

Here's an example of what your filesystem might look like:

 - ...
 |--DEV-ENVIRONMENT
 | |--docker/ 
 | |--home/
 | | |--snowcast-yourname/    # <------- Clone your stencil here!
 | |--run-container
 | |-- ...
 ...

Part 1: Intro to Socket Programming

Some definitions

What is a socket?

As we discussed in lecture 2, sockets act just like file descriptors between two endpoints of a network connection: you can read data from the socket (like you would read data from a file descriptor), and you can write data to the socket (like you would write data to a file descriptor). You can also think of a socket as a "pipe" between a client and the server that either endpoint can send data through.

What is an IP address? What is a port?

Roughly,

• An IP address is the identifier for a network connection endpoint that the network layer protocol uses (e.g. IP)
• A port is the identifier for a network connection endpoint that the transport layer protocol uses (e.g. TCP).

A "network connection endpoint" here refers to one end of a network connection; for example, if your computer is connected to a remote server, then each of your computer and the remote server are both connection endpoints.

Beej's Guide (a great network programming resource!) has a good analogy for how IP addresses and ports are used to deliver network packets:

Think of the IP address as the street address of a hotel, and the port number as the room number.

In other words, the IP address is used for forwarding a packet to the destination endpoint, while the port is used for sending the packet to the application using the port at that endpoint. The kernel is responsible for knowing which applications map to which ports currently running on the system.

Setting up a server-client connection with TCP

The next few sections will guide you through building a client-server program and sending the first few messages of your Snowcast protocol.

Note: Much of the content here mirrors the sockets demo in Lectures 2 and 3, which implement a different protocol using the same mechanics described here. While you shouldn't just copy the lecture code into your repository, you're welcome to follow along with the lecture example and use it as a reference to get started!

Creating a listening socket

Our first step is to create a socket for the server to “listen” for new connections. From there, clients can send your server a request to start a connection, and then your server can set up a connection to each client.

Picking a port number

First, we need to pick a port number on which the server will listen. In general, port numbers can be any 16-bit number (ie, 0-65535). Port numbers less than 1024 can only be used by root, so you the port you use must be higher. For designing a protocol in general, you're free to pick any port number you like, but, in practice, certain applications typically run on specific port numbers.

For the Snowcast project, your server should listen on port 16800 since it will let you test your work in Wireshark most easily (we'll see why shortly).

For now, you can hard-code the port number you choose (i.e. define it as a constant in your server program), but later you will need to take in this value as an argument (per the server specifications).

Listening for new connections

The next step is to create the listening socket, which requires telling the kernel some information about the type of socket we want. The kernel has many types of sockets, not all of which involve IP, or even networking. Right now, since we want to establish TCP connections between clients and the server, we need to create TCP socket.

The API to create sockets varies by language, but generally follows a similar structure. The diagram and sections below describe the most relevant functions you need in Go, C/C++, and Rust.

Regardless of the language you use, there are a few steps that neeed to happen to set up a listening socket. Expand the steps below for your language, which outlines what the process should look like. (Hint: Click the names of the functions for links to relevant documentation!)

Instructions for Go

In Go, you can accomplish this in two steps:

1. First, you can use ResolveTCPAddr from Go's net package (net.ResolveTCPAddr) to create the TCP socket and bind it to your chosen listen port. This returns a TCPAddr that represents the new socket.

2. Next, we need to tell the kernel that we want the socket to listen for new connections. You can do this with net.ListenTCP, which takes in the TCPAddr from the previous call.

Instructions for C/C++

In C/C++, this process requires a few steps:

1. First, use getaddrinfo here to tell the kernel what kind of socket you want. Note that getaddrinfo essentially gives you back a struct you then need to use to make a socket.

2. Next, we need to create the socket itself. Here, you should pass in parameters from the struct returned from getaddrinfo into the system call socket. This returns the file descriptor for the listen socket.

3. Use bind to bind the socket to the port you chose as the server's listening port.

4. Finally (for now), we need to tell the kernel that we want the socket to listen for new connections. For this part, we use listen; the second argument to listen specify the maximum number of waiting connections to clients your server can have at any given time (called the backlog, more on this later).

Why are there so many steps for C/C++ compared to the other languages?! The C API most directly mirrors the Linux kernel API. The other languages are all doing the same steps, they just hide some of them from you!
In addition, since the C API predates all of the other languages, certain aspects are a bit more complex in order to maintain compatibility with older programs.

Instructions for Rust

Use TcpListener.bind. This function creates a new TcpListener, which, as the name suggests, is a TCP socket that is set to listen for incoming connections on the specified address/port.

"I found some other examples or documentation that show different ways of creating TCP sockets, are they the same?"

For any language, socket APIs can have many different features for convenience or historical reasons–all interact with the same low-level kernel API to perform essentially the same functions. We have done our best to outline a series of function calls that will get the job done, but there are certainly many ways to solve the same problem–you are free to use whichever version you like, so long as it meets the assignment requirements. If you have questions, feel free to ask!

Task: Start writing your Snowcast server program! Use the functions mentioned so far in the language of your choice to set up a listening socket for your server (in your server program).

To use the above functions, you may need to provide some additional parameters to specify how the socket is created. For these, keep in mind the following guidelines, which are good "default values" to use for this lab, and the class in general (so just skim this list over and return here if you need it):

• IPv4 sockets only: For our projects, we don't need to worry about IPv6 sockets, or dealing with IPv4/IPv6 compatibility, so we use IPv4 only. In Go, this means calling ListenTCP with "tcp4" as the protocol, or creating sockets with AF_INET in C/C++.
• Bind to any address: Sockets can be bound to one or more interfaces by specifying a "bind address." Network interfaces are different network connections available to the system (eg. Ethernet, Wireless, etc.)–we'll learn more about these later. When you create sockets, they should be bound to all interfaces on your system, which usually equates to leaving the "bind address" blank.

Hint: A big part of this task, and systems programming in general, will be figuring out how to structure your programs well (since we don't give you stencil code, hehehe). To do this, reading up on function documentation and looking at some of the examples given in documentation can be really useful for creating a basic structure.

Also, don't be afraid to come to office hours if you're struggling with the question "so where should I even be calling this function?" or similar ones!

This is a skill we want you to learn, and it's okay to ask for help!

Warning: Don't forget error handling code! You should check if any of these functions (and any networking functions you use in general) return an error and handle them accordingly. For this part, if any functions encounter an error, you should just print out the error and exit the program.

When the server is just starting up, you can usually do this in one-step with a function like log.Fatal in Go or perror in C/C++. When you start handling clients, you may need to be a bit more subtle to gracefully handle errors (see lecture 3 for details).

Error handling is an essential practice in systems code, and it will save you a lot of debugging time to add this now!

Once you've written an implementation (even if you're not fully confident in it yet), continue onto the next step. After this, we'll show you a way you can test your program.

Waiting for a client to connect

If you run your program right now, it will just create the socket and then exit! To actually handle clients, we need to tell the server how to "accept" incoming connections.

Each of Go, C/C++, and Rust have a single function call (named "accept", or something similar) to handle a client's connection request, as listed below in the diagram.

Either way, the principle of accepting new connections is the same: calling "accept" will normally block until a client connects. Once a client connects, "accepting" the connection creates a new socket specifically for the connection between the server and the new client, while the listen socket remains unchanged. This is important, because your server should now use the new socket file descriptor to communicate with the now-connected client.

However, our server also needs to handle connections from multiple clients, so it should also continue listening on the original listening socket that you made in the previous steps. In other words, your server should not stop listening for other clients once a new client connects.

Conceptually, you'll want your code to behave something like this (in pesudocode):

. . .
listen_sock = create_listen_socket()

while true {
    client_sock = do_accept(listen_sock, ...)
    
    // . . . Do something with client_sock (later) . . .
}

To do this for real, consult the following instructions for your language:

Instructions for Go

We recommend using net.TCPListener.AcceptTCP, which behaves similar to C/C++'s accept in C/C++ has, except it is specific to incoming TCP connections.

net.TCPListener.AcceptTCP will block until the next incoming TCP connection request and return a new TCPConn, which represents the TCP socket for the connection between your server and the newly connected client.

Instructions for C/C++

In C/C++, you can use accept. A key thing to remember about accept is that this function will block until a new client connects, and will return a new socket descriptor for the connection from your server to the client your server has just accepted.

Instructions for Rust

In Rust, TcpListener.incoming works a little bit differently.

TcpListener.incoming "returns an iterator over the connections being received on this listener." In essence, this method calls TcpListener.accept in a loop for you, so then all you have to handle is the TcpStream returned by TcpListener.accept. This TcpStream represents the new socket made for the new client connection, and you will later use it (as a stream) to communicate with the client.

The example given in the documentation for TcpListener.incoming is pretty good and will help you understand how to use the method.

Task: Extend your Snowcast server program so that it can accept new client connections, while continuing to listen for other connection requests!

When you run your program now, it should just hang. This is good, it means the server is waiting for new connections!

Testing your server (so far)

The main command you would use to do this is To see if your server is actually listening on the port you chose, we can use the command ss, which stands for "socket statistics." ss can view the kernel's table of open sockets to tell us if our server port is open.

While your server is running, open up another terminal and run the following:

ss -ln | grep <server port number>

You should see something like:

tcp     LISTEN  0   20      0.0.0.0:16800        0.0.0.0:*   

This means that one of the sockets on your machine is listening on port 16800 for TCP connections! In other words, if you can see this output, then your server is running and listening for client connections!

Want to know more about listening sockets on your system?

For information on other sockets that are listening, run:

ss -lnp | grep LISTEN | less

You should be able to see a bunch of the sockets being used for various processes on your system. If you run it with sudo, ss will also output the names of the processes as well.

Now that your server is listening for clients and can accept new connections, all that's left to set up is your client program!

Starting up the client

On start up, your client program should create a new TCP socket and connect to your server.

To connect to the server, you'll need to provide its IP address and port so that the client knows how to reach it. Since our client and server are running on the same system, we will connect using Linux's loopback interface. This is a special, virtual network interface that exists only within the local system. This is useful so that programs on the same system can still communicate using networking protocols, even when there's no network!

When making connections, the loopback interface always has the same IP address: 127.0.0.1. We will use this IP when connecting to the server–for now, you can either pass it into your client program as an argument, or define it as a constant.

Note: We can also refer to this address using the hostname localhost. Hostnames like localhost (or google.com or cs.brown.edu) are translated into IP addresses by the OS (often using protocols like DNS, which we'll learn about later).

If you've been following the functions in this lab, your code should be set up to also translate hostnames automatically, so you can also pass in the name localhost instead of 127.0.0.1.

The server's port should be the one you originally chose for your server, port 16800.

To build your client, you will create a socket in a similar manner to creating the server. However, instead of creating a listening socket, we will be creating one to connect to a specific destination, the server.

To do this, the relevant function calls are outlined in the diagram below and in the following sections:

Instructions for Go

In Go, use net.Dial, which takes in a network type (i.e. TCP) and an IP address/port number, and attempts to connect to the specified address. The address/port string should be of the form address:port–-ie. 127.0.0.1:16800 or localhost:16800, to connect to your system's loopback address on port 16800.

Instructions for C/C++

For C/C++, you again need to do a few steps:

First, call getaddrinfo once again to tell the kernel what type of socket you want. Here however, you will need to pass both the server’s address and port in as part of the specifications for the socket (along with TCP, IPv4 etc).

getaddrinfo actually returns a list of possible socket options for making the connection. You will need to iterate over the results to find one that allows you to connect–take a look at Beej's guide or this old class example for details.

In the loop over getaddrinfo's results, you should attempt to create a socket using the current result (by calling socket) and then attempt to connect to the server using the new socket. To connect to the server, you will need to use the system call connect, which takes in a socket file descriptor (i.e. the one returned from socket) along with the result from getaddrinfo. Once connect successfully returns, you have connected to the server! You can then exit out of your loop, and begin sending data to the server.

Instructions for Rust

In Rust, use TcpStream.connect, which takes in an address (see the examples in the documentation for the structure of the input address) and attempts to connect to it.

Task: Start writing your Snowcast client program. Use the functions mentioned in the language of your choice to create a TCP socket for your client control program and connect the new socket to the server using the server's IP address and port number. Make sure your code compiles–we'll test it in the next section.

Viewing the connection in Wireshark

To test your code so far, we can watch the connection attempt happen in wireshark. Wireshark is a great debugging tool to see packets sent over the network as you test your code. Any packets that are sent, Wireshark can see!

As part of your warmup, we'll guide you through how to set up Wireshark to view snowcast traffic to demonstrate how you can use it for debugging.

This guide assumes that you've already set up and are able to run wireshark per these instructions from the container setup guide.

If you can't run wireshark

If you are unable to run Wireshark, you can skip this part for now–please make sure the course staff is aware that using Wireshark is problematic for you.

In lieu of using wireshark, you can debug using print statements–this is less efficient and more error-prone, but it works in a pinch.

At this point in the setup, print out something when the connection is established (ie, after Dial/connect on the client, and after accept on the server). If you see your message printed (and nothing returned with an error), your connection was established!

First time setup: Snowcast dissector

Wireshark has many dissectors that can decode packets in various protocols. Since we created Snowcast, we have written a custom disssector that can decode (or, dissect) Snowcast packets for you. You can use this to tell if your data is formatted properly!

To install the dissector:

1. Open a terminal inside the container. That is, in a terminal on your host machine (or a WSL terminal on Windows), run the command ./run-container.
2. cd to the directory where you cloned the stencil
3. Run the following command

cs1680-user@c808c3104e5a:~/snowcast-stencil $ util/snowcast-dissector/install.sh

After you've run this command, your terminal should show a file being copied successfully, like this:

Your Snowcast dissector should now be successfully installed! If you get any errors, please let us know on Ed.

Starting up Wireshark

Next, start up Wireshark (inside the container as described in the setup guide. You can use either the primary method or the backup method, whichever works.

Once you have started up Wireshark, you should be presented with a window like the one below:

Starting a capture

Packet captures are performed by attaching to one of the system's network interfaces. In this case, we will be capturing on one of the virtual network interfaces inside the container, so we will only see network traffic from container processes.

To start a capture, select Capture > Options from the menus at the top. Each of these options represents an interface on which packets can be received.

In general, there are two interfaces to care about:

• lo, the loopback interface: This contains traffic to and from localhost–it lives exclusively inside the container. Your own system has its own loopback network, which is local to your own machine. You should capture on this interface if you are debugging a program that connects to localhost, like your assignments.
• eth0: This is the container's link to the outside world (i.e., your host system, and the Internet). You should capture on this interface to see traffic that leaves the container.

(Not all machines will have the same naming for these devices, and in some cases they may not all be available. For example, your host system may show a wired and wireless interface.)

Here's how to set up a capture for Snowcast traffic:

1. Select the loopback interface (since we will only use the localhost network)
2. In the box at the bottom, enter the capture filter tcp port 16800 or udp. This tells Wireshark that we're only interested in TCP traffic on port 16800 (for the control protocol) or UDP traffic (for the data protocol), so it can ignore everything else

Ultimately, your capture window should look like this:

4. Press OK to start the capture

Wireshark should now switch to its main capture window, but you shouldn't see any packets, because we haven't sent any yet! Let's fix that.

Viewing the connection

To watch the connection happen, we need to start both our client and server while Wireshark is running. To do this, you may need to open a few terminals inside the container–if you're using VSCode inside the container, we recommend using VSCode's terminals, which are easy to open and split into multiple panes.

To start your client and server:

1. Make sure Wireshark is open and has a capture running, as in the previous section
2. In one container terminal, run your server program. You should see your server start and hang while it waits for a new connection–this is expected.
3. Start your client program

If the connection was established, you should see some packets in Wireshark like this:

"But wait, I didn't send anything yet!" You're correct, you (the developer) didn't. These three packets are part of how represent the initial TCP packets that should be sent between your client and server when your client attempts to connect to your server–we call them the "TCP handshake." Don't worry too much about understanding these just yet! We will cover what these packets mean closer to the TCP project.

For now, this just means our connection is set up properly, yay!

We can represent these packets on our flow diagram below:

Handling a client (server side)

Back to the implementation: at this point, your server program can accept a new client connection, but we haven't sent any data yet! In this section, you will add to your server's current functionality so that it can handle communication with multiple clients concurrently.

You could implement this in many ways, but we recommend you start by making a thread per new client connection, which can then do any client-specific tasks.

The important thing here is to make sure you store enough data about the client somewhere, perhaps by making a new data structure. When you create the new thread, you will need to pass this information to the thread so that it can communicate with the client.

For now, the most important information to pass to the thread is the client's socket. For this, see the instructions based on your language:

Instructions for Go

All of the information about your client's socket is contained in the net.Conn object returned by net.AcceptTCP. This object has various helper methods for sending/receiving on the socket, getting the client's IP address, and more.

Consider creating a goroutine for each new client, which takes the client's net.Conn as an argument. Later, you may wish to pass a custom struct instead that also includes more information.

Instructions for C/C++

Right now, you should probably store:

• The socket file descriptor the server uses to communicate with the client
• The IP address of the client's socket (returned in the argument from accept)
• The thread ID of the client thread

We recommend making a struct to hold these and passing them to the client thread. Take a look at this old class example for details.

Instructions for Rust

All of the information about your client's socket is contained in the TcpListener returned when accepting the client's connection This object has various helper methods for sending/receiving on the socket, getting the client's IP address, and more.

As you build your snowcast project, it will be important to consider what state you need to keep track of for each client. For example, when you start sending data from radio stations to client listener programs, you may also want to store the UDP port of the client's listener. Keep this in mind as you think about how you'll implement the protocol.

Task: In your server program, for each new client connection, store the new client's data and initialize a thread to handle communication with the new client.

For now, just make your client thread print out something like "Client connected!" and then exit. If you see your message being printed when a client connects, your threads are being created!

Hello and Welcome

Now that your server can handle communication with multiple clients concurrently, it's time to send something useful over the connection. This is where we start needing the Snowcast Specification, specifically the Protocol Specification, which defines the set of rules and message formats for how applications implementing Snowcast should communicate.

As seen in the diagram below, the protocol you are asked to implement in Snowcast requires your client to first send a Hello message, and once your server receive's the client's Hello, it should respond with a WelcomeOther message types (such as Announce, SetStation, and InvalidCommand) should be sent for other parts of the protocol, but you don't need to worry about them for this warmup.

As a reminder, a client's Hello message must be 3 bytes long and has this format:

Hello:
    uint8 commandType = 0;
    uint16 udpPort;

And this is what a server's Welcome message must by 3 bytes long and has this format:

Welcome:
    uint8 replyType = 2;
    uint16 numStations;

Therefore, to send a message, you should send a series of bytes corresponding to one of these messages.

Each of Go, C/C++, and Rust have function calls for sending and receiving data over a socket, as summarised in the table below:

Function	C/C++	Go	Rust
Send	send	(net.Conn).Write	TcpStream.write
Receive	recv	(net.Conn).Read	TcpStream.read

There are several ways to "compose" the data to send: for example, you can create a data-type (such as a struct) to represent the message, and then "serialize" or "marshal" it into a byte array. Alternatively, you can call your language's send function once for each message field, after converting each field into the appropriate size.

There are many ways to do these operations in each language. For examples, we recommend starting by taking a look at the notes for Lectures 2 and 3, and then consulting your language's resources for the method you like best.

Task: Now that you've started building and parsing messages, update your client and server to do the following:

1. Once your control client has successfully connected to your server, send a Hello message to the server and wait for a Welcome in response. Be sure to send your messages as bytes in the format described by the Specification. When sending the Hello, you can start by hard-coding any value for udpPort (but you'll need to change this later.)
2. In your server program, once your server receives a client Hello, send a Welcome response back to the client. After you send your first packet, take a look at the output in Wireshark to make sure it looks correct. Whens ending the Welcome, you can start by hard-coding any value for numStations (but you'll need to change it later).
3. When your control client receives a Welcome message, you should print a message to stdout in this exact format (followed by a newline):

Welcome to Snowcast! The server has N stations

where N is the NumStations received in the Welcome message. This will ensure that your output matches the format of the spec, and our autograder (more info on this in the protocol specification.

Checking your work

After you send each packet, check Wireshark's window. It should look something like this:

Let's break down what we're seeing:

1. In the "protocol" column of Wireshark's packet view, you should see one packet labeled CS168SNOWCAST, which means that Wireshark correctly ran the dissector and tried to decode your traffic. Click on this packet.

If you don't see this

If you are sure you sent a packet but don't see CS168SNOWCAST:

1. Make sure your server is using port 16800
2. Make sure you have installed the Wireshark dissector

If you still have issues getting Wireshark to show your traffic, please post on Ed or let us know in hours.

2. In Wireshark's lower panes, you should see a breakdown of the packet's headers and data. At the bottom of the protocol stack, expand the part labeled "Snowcast Protocol Data"
3. Take a look at the data. Does it match what you intended to send? You can click on each field to look at the raw bytes that were sent.

If the decoded data doesn't match what you wanted to send: Check how you are encoding the packet. Specifically, make sure you are using network byte order to send packets, and that you are converting data back to host byte order once you have received a packet.

Using wireshark to debug, keep iterating until you can send a hello and welcome message that are properly formatted–you'll know they're correct when what you see in Wireshark matches that you intended to send.

Congratulations, you have your Snowcast server and client programs exchanging messages!

Preparing your code for Snowcast

The client and server programs you created here can serve as your starting point for Snowcast: your TCP client is the start of the Snowcast control client (ie snowcast_control), and your server implements (you guessed it!) snowcast_server!

Before we can check off your work for the milestone with our autograder, you'll need to refactor your code a bit so that your programs have the same names and command-line arguments as the Snowcast programs.

While this may seem annoying now, we want to make sure that you can interact with the autograder so that you can test your work–and so we can test our autograder!

More concretely, you should make sure that your code adheres to the following requirements (this list sourced from the submission requirements and implementation specification:

1. As described in the submission instructions, create a Makefile such that make compiles your client and server and gives them the names snowcast_control and snowcast_server, respectively.
2. Update your client and server program to accept the same command-line arguments as snowcast_control and snowcast_server. Specifically:
   For example, your server should be able to run as:
   ​​​​$ ./snowcast_server <listen port> <file0> [file 1] [file 2] ...
   

   Your client program should run as:
   ​​​​$ ./snowcast_control <server IP> <server port> <listener port>
   

What do the different brackets (<>, `[]`) mean?

This is a convention from man pages for specifying arguments:

• Arguments in <angle brackets> are required
• Arguments in [square brackets] are optional: you can leave them out

Thus, snowcast_server takes in a minimum of 2 arguments (listen port, one file), but can also accept any larger number of arguments (for additional files).

3. (If you have not done so already) Update your programs to take in the client and server's address and port numbers as arguments, as shown in the spec above.
4. (If you have not done so already) Make sure your client is printing out the Welcome message in the format specified here
5. Update your server's Welcome message to list the the number of files passed in on the command line when running snowcast_server. You don't need to handle opening/reading from files right now, but make sure that the number of stations aligns with the command-line arguments (i.e. NumStations == len(args[2:])).
6. In snowcast_control, update your client's Hello message such that value for udpPort is the value passed as <listener port> on the command line. For the final version, the value you will pass for <listener port> will be the port on which you run the listener client–for now, you can just pass in any unsigned 16-bit integer.

Testing your milestone implementation

To test your milestone work, you should do the following:

1. Run your client and server binaries and look at the packets in Wireshark: you should see Hello and Welcome packets that contain the data you intended (as described in Checking your work)
2. Double-check that your client and server binaries follow our requirements on command-line arguments (as in the previous section)
3. Once you've verified (1) and (2), you can test your work using our built-in tester available in your stencil repo. See this section for instructions–for this part, you'll want to run the milestone tests only.

Final step: running the reference version

Before we're done, you should also try running our reference implementation: this is a complete version of the Snowcast client, server, and listener that meets our requirements, which is located in the reference directory of your repository.

You should run the reference version now to make sure you get a sense of how things should work, which will also demonstrate how your program should operate. To do this:

1. If you already have a wireshark capture open to look for snowcast traffic, stop and restart it so you have a clear view. If you don't have a capture open, start one now.
2. Open 3 terminals inside your container (either standalone terminals, or just split your VSCode terminal in 3 parts).
3. Check the working directory for each terminal to make sure each one is located in your stencil directory (eg. snowcast-youruser). If not, cd into your stencil directory.
4. In one terminal, run the server, listening on port 16800, and pass it all of the mp3 files in the mp3 directory, like the example below. Note that we provide two versions of each binary (for Intel or Apple Silicon systems)–be sure to run the version for your architecture, like this:

## Run the reference server (x86-64 systems)
reference/snowcast_server 16800 mp3/*

## Run the reference server (M1/M2/M3 macs)
reference/arm64/snowcast_server 16800 mp3/*

3. In another terminal, run the control client and pick a random port number to use for the listener (eg. 1234):

## (If you have an M1/M2/M3 mac, use 
## reference/arm64/snowcast_control instead)
reference/snowcast_control 16800 1234

Once you start your control client, you should see the Hello/Welcome handshake process in Wireshark, similar to your version!

4. In another terminal, run the listener client. To measure the streaming rate, we can pipe the output to a program called pv, which measures the rate at which data is written to it, like this:

## (If you have an M1/M2/M3 mac, use 
###reference/arm64/snowcast_listener instead)
reference/snowcast_listener 1234 | pv > /dev/null

5. In your control client, subscribe to any station by typing in the station number. You should see a SetStation and Announce message appear in Wireshark, and, in your listener terminal, you should now be seeing pv measure some data at 16KiB/s!

Here's an example of what it should all look like (if it's too small to see, right click and do "open image in new tab"):

You've now run the reference version, yay!

This should give you a good baseline for how things should look as you run your implementation. As you write your own implementation, remember this workflow–this is the best way to test your work early-on to make sure things are working. We know it's tempting to just run the autograder/built-in tester, but it's you'll find it's a LOT easier to interpret the output from this view!

Submitting your warmup

Congrats on taking the first steps toward your Snowcast implementation!

To submit your work, you should push all of the following to your Github repository and submit it as described here:

• All of your code for your client and server
• A Makefile that compiles your snowcast_control and snowcast_server binaries when running make (as described here)
• A screenshot of Wireshark while you run the reference version: your screenshot should show a SetStation and an Announce command (you can enter the filter cs168snowcast to get rid of the UDP traffic)
• Your design document with a plan for the rest of your implementation (details here)

Closing thoughts

This concludes the warmup portion of this guide. We hope this provides some first steps to get started. See the rest of this document for implementation notes and guides on testing, debugging, and using other resources. If you have questions on anything, please don't hesitate to ask us on Ed or in hours!

Essential implementation resources

The following sections provide guidelines on helpful resources, as well as how to debug and test your code. We'll update this section with more resources and tutorials as the project is out and everyone has questions. If you aren't sure how to do something, just ask!

Here are our most important resources:

• We provide a complete reference implementation of the assignment you can run to see how it works!
• Once you can verify your implementation works manually, you can run many of our autograder tests yourself to help debug.
• The sockets demo code provides a complete code example for a client-server application. Your code will follow a similar structure, but will do different things–we don't recommend copying from it directly, but it should be a good reference for the overall program structure and examples of boilerplate code

Reference Implementations

We have provided reference versions of the client and the server that follow the protocol and meet all the requirements, located in the reference directory of your repository.

Apple-silicon (M1/M2/etc) mac users: You should use the versions of these binaries in the reference/arm64 directory instead–these binaries are compiled for your CPU architecture.

We highly recommend using the reference versions to help make sure you understand the protocol, and make sure that your programs interoperate with the reference. You can test your adherence to the protocol based on how well your programs interact with them, ie. run your client with reference server, and vice versa.

For instructions on how to test with the reference, see here–just substitute the reference version for any program. (The linked instructions are about streaming, but apply generally–you can skip the parts about snowcast_listener and pv unless you are explicitly testing streaming).

For a visual guide on how to run the reference versions, see the gearup recording.

Running the built-in tester

Our stencil repo includes tester programs for your server and control clients. These testers run almost all of the tests in the autograder, so you can check your work on your own before submitting! Even better, you can see the test traffic in Wireshark, which will help you debug!

To run the tests, we provide a script run_tests in the util directory of your stencil repo.

You can run it like this:

1. Open a terminal inside the container environment
2. cd to the directory that holds your stencil
3. Build your code–you should make a Makefile to do this, if you have not done so already. The result should be that binaries snowcast_server and snowcast_control are located in this directory
4. Run the following:

# Make sure the snowcast_* binaries are in this directory
cs1680-user@c808c3104e5a:~/snowcast-stencil:$ ls
mp3  reference  snowcast_control   snowcast_server  stations  util  [. . .]


# Run the tests (here, just the milestone tests--see below for more options)
cs1680-user@c808c3104e5a:~/snowcast-stencil:$ util/run_tests milestone

From here, the tester will look for your snowcast_control and snowcast_server binaries in this directory and run the specified set of tests. For the milestone tests, the output should look like this:

If your output looks like this, your milestone tests passed, yay!

If any tests fail, you'll see a bunch of log output that contains some info about what failed that may help with debugging. See this section for guidance on how to think about debugging tests, and for some common things to check.

As you continue with the project, you can run the full suite or tests, or single tests just for debugging. See the next section for details.

Our favorite testing commands

Here's a list of our favorite testing commands (run in the same way as the steps in the previous section):

• Run the full test suite: util/run_tests all
• Run the test suite on the reference version: util/run_tests --bin-dir reference all
• Stop on the first test failure (great for debugging!): util/run_tests --fail-fast all
• Run a single server test, with output decode-able in Wireshark (great for debugging!):
  ​​​​SERVER_PORT=16800 util/run_tests server -test.run="TestServer/TestCompletesHandshake"
  

• Show more options: util/run_tests --help

For example, to make sure all the test are working on your system, you can run the tests on our reference implementation (located in the reference directory, or reference/arm64 on for Apple Silicon macs) like this:

# Run the full test suite on the *reference* version
cs1680-user@c808c3104e5a:~/snowcast-stencil:$ util/run_tests --fail-fast --bin-dir reference all

This should take a couple of minutes to run. The end result should look like this:

=== RUN   TestServer
=== RUN   TestServer/TestAcceptsClientConnection
=== RUN   TestServer/TestAcceptsMultipleConnections
=== RUN   TestServer/TestClosesConnectionOnInvalidCommandType
=== RUN   TestServer/TestCompletesHandshake
=== RUN   TestServer/TestInvalidStationFails
=== RUN   TestServer/TestMultipleHellosFails
<...>

--- PASS: TestServer (23.99s)
    --- PASS: TestServer/TestAcceptsClientConnection (0.14s)
    --- PASS: TestServer/TestAcceptsMultipleConnections (0.25s)
    --- PASS: TestServer/TestClosesConnectionOnInvalidCommandType (0.23s)
    --- PASS: TestServer/TestCompletesHandshake (0.13s)
    --- PASS: TestServer/TestInvalidStationFails (0.15s)
    --- PASS: TestServer/TestMultipleHellosFails (0.15s)
    --- PASS: TestServer/TestNoStationsFails (0.54s)
    --- PASS: TestServer/TestPartialHelloTimeoutFails (0.39s)
    <...>
PASS

If you have failing tests

If you have test failures, please check the following first:

1. If all your tests fail immediately, make sure that the directory where you run the tests contains your snowcast_* binaries and that your programs follow our specification on command-line arguments. If that doesn't help, keep reading .
2. Run your programs manually (ie, no tester): Sometimes the tester isn't able to give feedback when your program crashes in certain ways (we're working on improving this). You can often get really good feedback by opening up 3 terminals and running your server, control client and (if applicable) listener manually: start up your server, connect your client, and then try and replicate the what the test does. If you're unsure how how to run the programs manually, see the here, or the gearup video (Linked in the Ed FAQ) for a demo.
   To test interoperability, you can try out our reference server with your clients (and vice versa) to make sure you're implementing the protocol correctly.
3. Check the details of how the test runs: We have documented broadly what each test does and what features it requires from your program. Note that some tests require you to have implemnted certain REPL commands (eg. the p or q command) so that we can get information we need in the test–see the list for details.
4. Pay attention to what you print (and where): Some control client tests read your program's stdout (ie, stuff you print) and check it against the specification. Make sure that you don't have any extra print statements, or those that you do have print to stderr, NOT stdout. Also make sure messages are printed with a newline (\n) after them. If you use fmt.Println, a newline is added automatically.
5. Check how you read from TCP sockets: Remember that reading from a TCP socket (eg. conn.Read) may not always return the same number of bytes you expect. To handle this, be sure that you are always reading the correct number of bytes, e.g. with io.ReadFull. For more info, see this FAQ.
6. Run the failing test individually (example) and try to understand what each test is doing based on 1) the test output, and 2) wireshark. To see what's happening in wireshark, follow these steps as you run the test and take a look at the snowcast messages you see.
7. Make sure you are properly closing your server's listen socket. Our tester needs to bind to this socket on the same port, so it may fail if your server does not clean up the socket. See here for more info.
8. Please check Ed, especially the Snowcast reading list/FAQ thread. We have a section for known tester issues, and otherwise there may be other posts relevant to your situation. We try hard to keep this updated!
9. To help you debug further, see our Snowcast Debugging Guide, which outlines a general procedure for thinking about debugging and describes some more detailed issues.

If you are convinced that your program works but your tests are still failing, please don't panic–just document this in your readme when you submit. When grading, we will manually review your autograder results and will make adjustments as necessary. Autograders exist to help reduce our grading workload, but rest assured that your grade is ultimately determined by a human.

Testing manually

See this section on how to run the reference version for some instructions on how to run your programs manually–to run your program, just run your versions of the snowcast binaries instead of the reference binaries. The gearup recording (linked in the Ed FAQ) also has a demo of this.

Some things to keep in mind when testing:

• To try and isolate a problem, you can run a combination of your programs and the reference versions. Not sure if your bug is in the client or the server? Run the reference client, and your server, and vice versa!
• In general, you can skip the section about snowcast_listener and pv unless you are testing streaming.

Testing streaming rate

Note: This section is about how to test streaming. For a conceptual overview of how to stream at a precise rate, see this FAQ.

If you want to test your program's streaming rate independent of the tester, you can use a a tool like pv. pv is a built-in Linux tool that passes input from stdin to stdout, and prints statistics about the rate at which it is receiving data to stderr. We'll be testing to see that your rate is consistently 16 KiB/s. You can run it as follows:

1. Open 3 terminals inside your container
2. In one terminal, start your server, eg. ./snowcast_server 16800 mp3/*
3. In another terminal, start your control client. Pick a random port number to use for the listener (eg. 1234): ./snowcast_control localhost 16800 1234
4. Subscribe to any station using your control client.
5. Run your listener client using the same listener port number you provided in the previous step, and pipe the output to pv, like this:

./snowcast_listener 1234 | pv > /dev/null

Here's an example of what it should all look like (if it's too small to see, right click and do "open image in new tab"):

This sends all the data to the pv program and then discards it. pv should print the average rate of data output by the listener (press Ctrl+C to quit. We will test that your average streaming rate is roughly 16KiB/s.

Streaming rate not what you expect? To start debugging, the first thing we recommend is to use the reference version of the listener with your server to help rule out if the problem is with your server or your listener. Once you know your server works (the most important part), try using your own listener client.

The reference versions of each program are located in the reference directory of your stencil or reference/arm64 on M1. See here for details.

Stencil updates

In the event of an update to the tester or other helper scripts, we may release an update to your starter repo. If this happens, you will receive a Pull Request (PR) from Github Classroom to merge our changes into your repository. To do this:

1. Open your repo on Github and go to the "Pull Requests" tab. You should see one PR from Github Classroom labeled "Sync assignment", like this:

2. Open up the PR, scroll down to the bottom and click, "Merge the pull request" and then "Confirm merge", like in the figure. This should merge our changes into your repo!

3. In a terminal where you use git, pull our latest changes however you normally pull code from git (eg. git pull origin main). You should see show some new changes, like this:

Frequently Asked Questions

My autograder tests aren't working!

See here.

What is network byte order?

Byte order refers to the order in which data of a multi-byte value is sent over the network. So for example, a 16-bit integer 0xAABB could be sent with the most significant byte first, ie. {0xAA, 0xBB} (called "big endian"); or with the least significant byte first {0xBB, 0xAA} (called "little endian").

Network byte order uses big endian. Go provides a binary package, and Rust provides a byteorder crate. In C/C++, you may find utilities similar to ntohs, htons, ntohl, and htonl from the <arpa/inet.h> library useful.

How should data be sent/received?

C

The primary method of I/O through network sockets is send()/recv(). One can then pass in a pointer to a struct:

struct_t s = {1, 2, 3};
send(sockfd, &s, sizeof(s));

Go

In messages that are of a fixed size, one option is to create a struct that contains the message data and write that struct into a buffer directly using binary.Write() in the binary package. For example,

buf := new(bytes.Buffer)
var pi float64 = math.Pi
err := binary.Write(buf, binary.BigEndian, data)

Data from a buffer can be read into a struct using binary.Read(), or any of the utilities within io/bufio.

Rust

A byte slice or vector can hold the bytes of each field within a struct:

// If we have a `let s = Struct { field1: u8, field2: u16}`:
let mut buf = vec![0; 3];// or Vec::with_capacity(mem::size_of::<Struct>());
let num = s.field2.to_be_bytes();
buf[0] = s.field1;
buf[1..].copy_from_slice(&num);

This can then be written to a TCPStream or UDPSocket.

How does the server know how where to send UDP packets to the listener?

In our protocol, we assume that the control client and listener client run on the same machine and have the same IP address. You can find the client's IP address based on the socket information returned when accepting a new connection. (In Go, this is part of the client's Conn, in C, this is returned in one of the arguments to accept.)

For the listener's port number, you should use the UDP port number from the Hello message.

For more information, see the Song Data Protocol specification.

How can we time out a socket operation?

In Go/Rust, sockets have a SetReadDeadline/SetWriteDeadline function that can be used to specify a deadline on I/O operations.

In C/C++, you can use setsockopt with SO_RCVTIMEO/SO_SNDTIMEO and a struct timeval to configure a socket before executing I/O operations.

Which IP version should the server and client support?

To ensure compatibility, we recommend creating all of your sockets as IPv4 only sockets. Your implementation does not need to support IPv6.

Reading on a TCP socket isn't returning the correct number of bytes, what's going on?

This is actually the expected behavior of reading on a TCP socket, since TCP is designed to provide a continuous stream of bytes, rather than discrete messages. For more info on this, see the start of Lecture 4, and/or expand the box below:

Why does TCP do this?

Let's say we wanted to read 5 bytes on a socket into a buffer, like this:

buffer := make([]byte, 2)
bytesRead, err := conn.Read(buffer)
// ... error handling, etc ...
fmt.Printf("Read %d bytes\n")

When reading from a TCP socket (here, conn.Read, the underlying kernel system call (read) unblocks whenever any amount of data is available, so the number of bytes returned (bytesRead, in this example) may be less than the size of the buffer.

Why? it's possible that not all 5 bytes have been received by the OS yet–maybe they were sent at different times, or maybe there was delay or packet loss in the network and not all of the bytes have arrived yet. Alternately, many protocols have messages of different sizes, so it may be necessary to read a message into a larger buffer and then use bytesRead to determine the size.

From a conceptual perspective, the kernel considers data sent by TCP as a continuous stream of bytes, rather than discrete messages: the kernel has no idea where our "messages" start and end–so it's doing the best it can by providing what data is has right now.

How to deal with it: To read exactly N bytes, you should read on the socket repeatedly (say, in a loop) until N total bytes have been received (or an error occurs). If you are using Go, we recommend checking out io.ReadFull (doc, with example), which is designed for this purpose.

How to think about streaming?

Here's a way to start breaking down this problem. Based on the protocol specifications, there are two constraints for how you need to send song data:

1. The goal is to send out song data to all listeners on a station at an average rate of 16KiB/s, which is 16384 bytes every 1 second
2. Per the song data protocol, our UDP data packets must be no larger than 1500 bytes, which is a typical maximum packet size for packets sent over the Internet.

To stream data at a periodic rate, you should send "chunks" of size <= 1500 bytes at a regular interval, say t_chunk, such that the average adds up to 16KiB's.

How to find t_chunk? (Click to expand)

Let's say we use a chunk size of 1024 bytes. That means that every second we should send (16384 total bytes/sec)/(1024 bytes/chunk)= 16 chunks.

To send data at an even rate of 16KiB/s, we should space out these chunks at even time intervals, such that:
t_chunk = 1 sec / 16 chunks = 0.0625s/chunk = 62.5ms/chunk

At every t_chunk time interval, you'd want to send out a chunk of song data to every connected listener, ie something like this:

while(1) {
    // Read a chunk from the file
    for all connected listeners {
       // Send this chunk to the IP:port of that client's listener
    }
    // Sleep for t_chunk
}

Each time this loop runs, you'd read the next chunk from the station's file and continue the process. When you reach the end of the file, you should continue reading again from the beginning (and send an Announce message).

To measure your streaming rate, see Testing streaming rate. Once you are confident you are close, try the TestStreamRate tests in the server's test suite.

How to stream more precisely?

In general, you don't need to be incredibly precise about sending a message every t_chunk, so long as you get a rate close to 16KiB/s. If you have issues, one way to be a bit more precise is to measure the time it takes to send out the data, and subtract this from the sleep time, ie:

while(1) {
    // Read a chunk from the file
    t1 = current_time()
    for all connected listeners {
       // Send this chunk to the IP:port of that client's listener
    }
    t2 = current_time()

    diff = t2 - t1
    sleep(t_chunk - diff)
}

What if the streaming rate is not exactly 16KiB/s?

We will be lenient when testing the streaming rate, but you should expect your average streaming rate to be around 16KiB/s. If you pass the TestStreamRate tests, you're fine.

To measure your streaming rate manually, see here. For info on how to think about streaming conceptually (if your rate is incorrect and you don't know why), see this FAQ.

How do I play sound?

Your implementation is not required to play music–we will only test that you send UDP data at a constant rate of 16KiB/s. Unfortunately, our container environment cannot play sound at present–we are working on a solution to this and should have an update soon, but this is not (and will not become) a requirement. However, if you are developing locally, you can pipe the output of your listen to an mp3 player to hear sound:

./snowcast_listener port | mpg123 -

If the output is stuttery consider trying mplayer instead of mpg123.

If you are able to play sound, you can also pipe the output of pv into mpg123 or mplayer to play the sound while measuring the bitrate.

Tester issue: bind: address alerady in use

This issue can occur if your server does not close its listen socket. Listen sockets are "bound" to a particular port in the kernel–if the socket is not closed, the kernel may still think the socket is bound to your process when it exits, preventing you (or our tester) from binding to it again!

To avoid this, you should make sure that your code correctly closes your listen socket when your server exits, or if it encounters an error.

Note: In Go, we recommend closing sockets using defer, which is usually a good way to make sure resources are cleaned up.

However, note that deferred statements will not run if your program exits with log.Fatal or os.Exit, since these functions force your program to terminate immediately.

Usually, the kernel will figure out your process has exited and remove the old binding quickly. If it doesn't go away fast enough, you should be able to clear it by stopping the container.