« Back to the main CSCI1680 website

Project 1: Snowcast (F23) :snowflake:

Milestone due: September 19, 2023 at 11:59pm EDT

Final submission due: September 25, 2023 at 11:59PM EDT

---

Introduction

In this project, you will implement an Internet radio station. This assignment is designed to introduce you to socket programming, get you used to thinking about network protocols, and re-familiarize you with systems programming and building concurrent applications.

You will develop and submit your work using Github Classroom, which will create a repository you can use to develop your code.

Overview

Internet radio

What's an Internet radio station, anyway? Internet radio is a somewhat older method of listening to streaming music over the network, similar to an AM/FM radio station you might use use in person. Basically, a server is continuously playing music on a discrete number of channels (or "stations"), and clients can join any station whenever they want and listen in.

Unlike modern, on-demand streaming audio platforms (Spotify, Youtube, podcasts, etc.), the server is always sending out the same data to all connected clients at the same time–so everyone should always be listening to (roughly) the same thing. If you've ever used Pandora or iHeartRadio, these are probably the closest modern analogs.

Our version: Snowcast

In Snowcast, you'll build a simplified version of an Internet radio station that includes a server and client. The server is responsible for keeping track of subscribed clients and sending them streaming data at a precise rate (16KiB/s). Clients provide the user's interface and are responsible for joining/switching/leaving stations, and "playing" the music.

In our version, we don't need to be concerned with streaming music and listening to it. Since the goal of this project is get you familiar with network protocols and implementing them, we'll just think about streaming arbitrary data to clients, which will write the data they receive to stdout. To check your work, we'll simply test that your implementation streams the correct data at a rate of 16KiB/s. (In other words, you don't need to worry about anything related to processing/encoding actual audio data, which would actually be a challenging problem!)

What you will build: the big picture

You will implement Snowcast by building three programs: one for the server, and two programs that make up the client. To make these programs communicate, you'll implement two different protocols (one TCP, one UDP) that handle different parts of the application. Here's an overview of how these components will interact (with details below):

The most important parts are:

• The server (snowcast_server) is responsible for reading a set of "song" files. Each song becomes one "station". Clients connect to the server and select which station they want to hear. After a client selects a station, the server begins sending it song data at a rate of 16KiB/s. The server uses two different protocols to communicate with the client: a control protocol to handle selecting stations and metadata, and a song data protocol for sending the actual song data.
• The client is composed of two programs. All clients run both programs at the same time:
  • The control client (snowcast_control): Communicates with the server using the control protocol. This program takes in input from the user to set the station, and receives announcements from the server.
  • The listener client (snowcast_listener): Receives song data from the server using the data protocol and just writes it to stdout (making it easy to check/measure with another program).

As part of this assignment, we want you to learn about how network protocols are specified and what it's like to implement them. You will build your client and server according to our specifications–-these cover how the control and data protocols work, as well as how to run and interact with each program. A big part of networking is building systems that interoperate well with other systems that "speak" the same protocol–you'll be graded based on how well you conform to the specification.

How to get started

A big part of building network systems, and systems programming in general, is learning how to design programs to build a larger system. In this project, and future projects, you will build your work from the ground up (ie, no stencil code) to give you the opportunity to design and structure your own program.

While this may seem like a lot at first, we have structured the project (and the course) to help with this process and give you feedback. Before the final deadline, you will submit two things that serve as a "milestone" for this project:

• The snowcast warmup will provide a starting point for socket programming and an introduction to debugging with Wireshark. Look here for instructions on how to clone the assignment, and a link to the stencil code.
• In addition, you will submit a design document that describes a plan for the rest of your implementation, which we will review and provide feedback.

The milestone portion of this project is due September 19 by 11:59pm EDT. For more details, see here.

We also have a lot of our resources to help you get started. In Lectures 2–3, we will introduce socket programming in detail by building some programs during class. From there, please see the The warmup and Implementation guide for helpful tutorials and implementation-level details (like how to run the tests).

The remainder of this document is structured as follows:

• Assignment details: logistics on how you'll implement Snowcast and how to submit your work
• Grading: How we'll grade it
• Specifications (skim now, return as you need them): The full details on our requirements for how the protocols work and how we expect you should implement the server and client

Assignment details

Languages

You may implement Snowcast in Go, C, C++, or Rust. If you are unsure, we recommend using Go, even if it is new to you, as class examples this year will use it. We have curated a list of resources for each language here.

If you want to use Rust, note that only some members of the course staff can provide language support (though we are all happy to discuss conceptual questions!). For details, see the course staff list.

You may implement this project in C, C++, Go, or Rust. Generally, you should be working with your language's socket API, eg. the Berkeley socket API from sys/socket.h in C/C++, net in Go, or std::net in Rust. You may not use RPC libraries or libraries for mashalling/unmarshalling structures to/from bytes such as protobuf. If you want to use any other libraries related to networking or byte-packing, please ask on Edstem first.

If you are using C, we provide a small library that with a linked list and hash table implementation in this repository. You do not need to use this, but it is available if you want–feel free to just copy the code directly into your repository. For examples on how to use the library, see the code demos in the examples directory.

Testing

We've provided multiple ways to help test your code (see links for details on how to run each one):

• Reference implementation: We've included completed versions of all Snowcast programs that implement all of the requirements in your stencil repo. You can use this to get a feel for how the program should work, and to test how it interoperates with your program
• Built-in tester: we've also provided a test suite you can use to check your server and control client's functionality.

Warning: When using the built-in tester, please take care to make sure your programs exactly follow our specifications on command line arguments and printing to stdout. Extra print/logging statements can break the tests if you write these to stdout.

For more details and recommendations on how to make sure your code follows the requirements, see here.

Submitting your work

When you are ready to submit, please push your code to GitHub and upload it to the appropriate Gradescope submission using the "GitHub" method. If the GitHub method doesn't work, you may also upload a zip file of your work instead, but we highly recommend submitting via GitHub.

Autograder setup: how we will build your code

Regardless of what language you use, please include a Makefile such that running make will compile all of your programs and place them in the base directory, and that make clean will clean up any compiled programs and object files.

Specifically, make should produce three executables, snowcast_server, snowcast_control, and snowcast_listener, in the main directory of your repository.

If you are using Go or Rust and your program requires external dependencies, make sure that make also installs these. If you are using C/C++ and are considering external dependencies, please contact the course staff first to check if we can accommodate them.

If you have any questions about how your project should be packaged for grading, please make a post on Edstem.

Grading

Milestone - 30%

To make sure you're on the right track, you will submit two things before the final deadline that serve as a "milestone" point. Both components are due on September 20, by 11:59pm EDT.

You should submit your work for each part by uploading your repository to the Snowast (Milestone) submission on Gradescope.

Warmup — 15%

The warmup will help provide an introduction to socket programming to help kickoff your implementation and ensure your programs can correctly interface with our autograder environment.
Please see the warmup document for details on what you need to implement and how to submit your work for this part.

Design Document — 15%

In addition to the warmup, you will also submit a design document describing your planned server design. Your design should answer at least the following questions:

• How will your server use threads/goroutines? In other words, when are they created, and for what purpose?
• What data does your server need to store for each client?
• What data does your server need to store for each station?
• What synchronization primitives (channels, mutexes, condition variables, etc.) will you need to have threads/goroutines communicate with each other and protect shared data?
• What happens on the server (i.e., how does the stored server state change) when a client changes stations? (Thinking about this may help you answer the previous question.)

If you're having trouble with the design, please come to our hours, or post on Ed. If you would like to talk in real-time (in person or remote) and are unable to attend any scheduled TA hours, please let us know and we will do our best to accommodate you.

When you submit your milestone work, please submit your design document as a plain text file (milestone.md or milestone.txt) or PDF (milestone.pdf) in the top-level directory of your repository.

Implementation (autograded) — 60%

Most of your grade will be based on how well your program conforms to the protocol and specification, including how well it interacts with our automated tests and the reference implementations. We'll release more details on weightings for each autograder test shortly.

When grading, we will review your code and test output to verify that your work meets the requirements for each test and may apply partial credit as applicable.

For our later projects, we will do most of our grading interactively, ie, by meeting with you and talking about your work (more on this later). Automated testing is designed to help you test your code out before the deadline and check for common issues, in addition to reducing some of the grading workload.

README and Design — 10%

Please include a README file with your program named README.md or README.txt. In your readme, you should describe your major design decisions and how your implementation meets the design requirements, such as how your server is structured in terms of concurrency, how it handles announces, how it handles multiple clients, etc.

If you know any bugs or limitations in your design that do not meet the requirements, please list them and suggested how you would correct them if you had the time. We will deduct fewer points for any bugs or design issues that you have documented.

Extra Credit — Up to 15%

The protocol we've defined is quite limited. We will consider any addition to the protocol for extra credit. You can also augment the server or client in a non-trivial way. Here are some ideas:

Click to expand

• Add a command which requests a listing of what each of the stations is currently playing (it is acceptable for the TA binary to respond to this with InvalidCommand).
• Add support for multiple songs per station.
• Add a command to retrieve a station's playlist (maybe the next 5 items or so).
• Add support for adding and removing stations while the server is running through the command line interface. If you remove a station while a client is listening to it, send a StationShutdown packet, or something along those lines, to inform them. If a new station is added, you could send a NewStation packet to all currently connected clients to inform them.

Feel free to ask what we think about your addition.

When implementing extra credit, your implementation MUST still adhere to the protocol interoperate with the reference implementation. It is acceptable for the reference implementation to return an InvalidCommand in response to your added commands/replies, but the existing commands/replies must still work as specified by the protocol.

Snowcast Specifications

The following specifications describe our requirements for your implementation. There are two parts:

• The Implementation specification describes the functional requirements for your client and server binaries (eg. command line arguments, how to handle user input, etc.)
• The Protocol specification describes the requirements for the control and song data protocols (ie, what messages should be sent and how they should be formatted)

What to expect: If you're reading this for the first time, please skim over this part get a sense of: 1) what you need to build, and 2) the kind of things we are specifying before you start writing code.

You don't need to internalize all the details yet, we just want you to get a sense of the structure of the specifications so you know how to go back to them.

We give you broad freedom on how you design your program and write your code, but we need to be very specific on what you're implementing. This is important in order to make sure that your program interoperates with other programs (a critical part of building network programs!) and to make sure that your program works with our autograder.

Implementation specifications

This section specifies how the server (snowcast_server) and client (snowcast_control and snowcast_listener) should operate.

Client Specification

You will write two separate programs that act as the "client".

UDP Client (snowcast_listener)

The listener client receives song data as UDP packets and writes the data to stdout. The executable must be called snowcast_listener. Its command line must be:

./snowcast_listener <udp_port>

The UDP client must print all data received on the specified UDP port to stdout.

TCP Client (snowcast_control)

The TCP client handles the control data. The executable must be called snowcast_control. Its command line must be:

./snowcast_control <server_name> <server_port> <udp_port>

<server_name> represents the IP address (e.g. 128.148.38.158) or hostname (e.g. localhost, cslab6c) to which the control client should connect, and <serverport> is the port to connect to. <udpport> is the port on which the local UDP client is watching for song data.

The control client MUST connect to the server and communicate with it according to the protocol.

After the handshake, it should show a prompt and wait for input from stdin.

• If the user types in q followed by a newline, the client should quit.
• If the user types in a number followed by a newline, the control should send a SetStation command with the user-provided station number, unless that station number is outside the range given by the server; you may choose how to handle this situation.

If the client gets an invalid reply from the server (one whose replyType is not 2, 3, or 4), then it MUST close the connection and exit.

The client MUST print whatever information the server sends it (i.e. the numStations in a Welcome) in real time. The printed lines MUST adhere to the following formats:

• In response to a Welcome reply, the client MUST print to stdout the following:

Welcome to Snowcast! The server has N stations.

where N is the value in the Welcome reply.

• In response to an Announce reply, the client MUST print to stdout the following:

New song announced: <song name>

where <song name> is the value in the Announce reply.

Handling user input

Your client MUST attempt to sanitize user input to prevent the user from sending a message not allowed by the protocol. You can decide what happens if the user does something wrong–eg. you MAY exit the program, or you MAY display error. However, you MUST NOT allow user input to send any packets that would otherwise violate the protocol specification.

Server Specification

The server executable must be called snowcast_server. Its command line MUST be

./snowcast_server <tcpport> <file0> [file 1] [file 2] ...

That is, a port number on which the server will listen, followed by a list of files; the list MUST be at least one file long. Each station contains only one song. Station 0 should play file0, Station 1 should play file1, etc. Each station MUST loop its song indefinitely. Additionally, the multiple stations can play the same song. Station 0 and Station 1 can both play file0 if desired.

When the server starts, it MUST begin listening for connections. When a client connects, it MUST interact with it as specified by the protocol. Additionally, it MUST send an Announce whenever a song repeats.

Streaming Music

To stream music, you should send data to each client at a fixed rate, rather than sending the song data as fast as possible. To do this, assume that all mp3 files are 128kbps, i.e. the server MUST send data at a rate of 128Kibps, or 16KiB/s.^[1]

If multiple clients are connected to one station, they MUST all be listening to the same part of the song, even if they connected at different times. If no clients are connected to a station, the current position in the song MUST still progress, without sending any data. In other words, the radio doesn't stop when no one is listening.

Finally, the server MUST NOT read the entire song file into memory at once. It MAY read the entire file in for some sizes, but there must be a size beyond which it will read data in chunks. For example, you should be able to have /dev/urandom (which is an infinite stream of random bytes) as a station.

Server CLI

When the server receives a command, it MUST print out a log of the command receives and any replies it sends to stdout.

It will also have a simple command-line interface:

• p followed by a newline MUST cause the server to print to stdout a list of its stations along with the listeners that are connected to each one.
• p <filename> (eg. p stations.txt) followed by a newline MUST cause the server to write the list of stations to the specified file. Any existing contents of the file MUST be overwritten.
• q followed by a newline MUST cause the server to close all connections and exit gracefully. An exit will be considered graceful if the server process returns 0.

The printed stations MUST follow the following format:

id,station_name[,connection1,connection2,...]

where connectionN has the form <IPAddr>:<UDPPort>.

Example output

If the server is run with

./snowcast_server <port> mp3/Beethoven-SymphonyNo5.mp3 ../DukeEllington-Caravan.mp3 VanillaIce-IceIceBaby.mp3

and the server stations have the following client connections (i.e. listeners):

• Station 0 playing mp3/Beethoven-SymphonyNo5.mp3 with listeners 127.0.0.1:9000 and 127.0.0.1:9001
• Station 1 playing ../DukeEllington-Caravan.mp3 with no listeners
• Station 2 playing VanillaIce-IceIceBaby.mp3 with listeners 127.0.0.1:10000 and 127.0.0.1:10001

the output should be:

0,mp3/Beethoven-SymphonyNo5.mp3,127.0.0.1:9000,127.0.0.1:9001
1,../DukeEllington-Caravan.mp3
2,VanillaIce-IceIceBaby.mp3,127.0.0.1:10000,127.0.0.1:10001

Design requirements

Part of this project is thinking about how to build a robust server. As you implement your server, you should pay attention to the following design requirements:

• The server MUST support multiple clients simultaneously.
• There MUST be no hard-coded limit to the number of stations your server can support or to the number of clients connected to a station.
• Invalid control messages received from clients should be handled by sending an InvalidCommand message (see Invalid Conditions for the specification).
• In other words, in the event of an error handling a client, the server MUST gracefully handle the error by terminating the connection to that client, rather than crashing the whole server.
• When your server exits (ie, with the q command, or on some unrecoverable error), your server MUST close any open TCP sockets. You do not need any special handling for UDP sockets.

Protocol Specifications

There are two kinds of data being sent between the server and the client that use two different protocols:

• The control protocol is used by the client to issue commands (e.g. specifying which station to listen to), and the server uses it to reply to commands (e.g. giving the client song information). This protocol uses TCP.
• The song data protocol is used by the server to send data to the listener client. This protocol uses UDP.

How we specify protocols

The following sections describe the control and data protocols the client and server use to communicate. We've written these specifications in the style of real-world networking standards (based on the IETF format from RFC2119). In later projects, you'll be reading real standards using this language, so we want you to practice with it now.

A key part of specifying network standards is identifying which features are required and which are optional and perhaps helpful or recommended. To make this clear, the standards use some specific keywords for this. Here are ours:

• MUST: This is a required feature to make the protocol work and interoperate with our reference version and our autograder. For this assignment, you'll lose points if you don't have it. (eg. "your server MUST use a certain format for command-line arguments")
• MUST NOT: This is the opposite of MUST. If you do this thing, you will lose points. (ie. Your program MUST NOT segfault during normal operation.)
• SHOULD: This feature is probably a good idea. We use this if we think that doing something a certain way will make your life easier, but we don't require it.
• SHOULD NOT: Opposite of SHOULD. We don't recommend this–doing something this way may make things harder.
• MAY: This feature is optional. You can do it if you want, but it's not required.

We're not shouting!: The convention is to write these keywords in CAPS to signal that they are keywords. Please don't interpret this as us using a negative tone–it's simply the convention, from a time when there wasn't a good way to do text formatting.

Control protocol

The control protocol is a TCP protocol between the control client and the server.

The server will typically listen for connections on port 16800 (though this is configurable on server startup).

Client to Server Commands

The client sends the server messages called commands. There are two commands the client can send the server, in the following format:

Hello:
    uint8 commandType = 0;
    uint16 udpPort;

SetStation:
    uint8 commandType = 1;
    uint16 stationNumber;

A uint8 is an unsigned 8-bit integer; a uint16 is an unsigned 16-bit integer. Your programs MUST use network byte order. So, to send a Hello command, your client would send exactly three bytes to the server: one for the command type and two for the port.

The Hello command MUST be sent as soon as the client connects to the server. The field udpPort gives the port number on which the client's listener program (ie snowcast_listener is listening. The server will use this when sending song data.

The SetStation command is sent to pick an initial station or to change stations. stationNumber identifies the station.

Server to Client Replies

There are three possible messages called replies the server may send to the client:

Welcome:
    uint8 replyType = 2;
    uint16 numStations;

Announce:
    uint8 replyType = 3;
    uint8 songnameSize;
    char  songname[songnameSize];

InvalidCommand:
    uint8 replyType = 4;
    uint8 replyStringSize;
    char  replyString[replyStringSize];

A Welcome reply MUST be sent in response to a Hello command. Stations are numbered sequentially from 0, so a numStations of 30 means 0 through 29 are valid. A Hello command, followed by a Welcome reply, is called a handshake.

An Announce reply MUST be sent on two occasions: after a client sends a SetStation command, or when the station restarts playing from the beginning of the file. songnameSize represents the length, in bytes, of the filename, while songname contains the filename itself. The string must be formatted in ASCII and MUST NOT be null-terminated. So, to announce a song called Gimme Shelter, your client must send the replyType byte, followed by a byte whose value is 13, followed by the 13 bytes whose values are the ASCII character values of Gimme Shelter.

Invalid Conditions

Since neither the client nor the server may assume that the program with which it is communicating is compliant with this specification, they must both be able to behave correctly when the protocol is used incorrectly.

On the Server

On the server side, an InvalidCommand reply is sent in response to any invalid command. replyString should contain a brief error message explaining what went wrong.

Where possible, you should provide helpful strings stating the reason for failure. If a SetStation command was sent with 1729 as the stationNumber, a bad replyString is Error, closing connection., while a good one is Station 1729 does not exist. We won't test this, but it will be useful for you when debugging.

To simplify the protocol, whenever the server receives an invalid command, it MUST reply with an InvalidCommand and then close the connection to the client that sent it. Invalid commands happen in the following situations:

Click to view

• SetStation
  • The station given does not exist.
  • The command was sent by a client before they've sent a Hello command. Clients must send a Hello command before sending any other commands.
  • If the command was sent by a client before the server responded to a previous SetStation from the client with an Announce reply, then your server MAY reply to this with an InvalidCommand. This means your clients should be careful and wait for an Announce before sending another SetStation, but your server can be lax about this.
• Hello
  • More than one Hello command was sent by the same client. Only one should be sent per client, at the very beginning of their connection to the server.
• An unknown command was sent (one whose commandType was not 0 or 1).

On the Client

On the client side, invalid uses of the protocol MUST be handled simply by disconnecting. This happens in the following situations:

Click to view

• Announce
  • The server sends an Announce before the client has sent a SetStation
• Welcome
  • The server sends a Welcome before the client has sent a Hello
  • The server sends more than one Welcome at any point.
• InvalidCommand
  • The server sends an InvalidCommand. This may indicate that the client itself is incorrect, or the server may have sent it out of error. In either case, the client MUST disconnect.
• An unknown response was sent (one whose replyType was not 2, 3, or 4).

Timeouts

Sometimes, a host you’re connected to may misbehave in such a way that it simply doesn’t send any data. In such cases, it’s imperative that you are able to detect such errors and reclaim the resources consumed by that connection. In light of this, there are a few cases in which you will be required to time out a connection if data isn’t received after a certain amount of time.

These timeouts should be treated as errors just like any other I/O error you might have, and handled accordingly. In particular, on the server, a timeout must only affect the connection in question, and not unrelated connections. If the client encounters an error, it should close its connection to the server and exit.

You are required to handle three timeout conditions:

Click to view

1. If a client or server receives some of the bytes of a message within 100ms, if it does not receive all of the remaining bytes of the message within the next 100 milliseconds, the client or server MUST time out that connection. See the FAQs section for details.
   • For instance, if the server receives one byte of a SetStation command, it MUST receive the remaining bytes within the next 100ms, or else close the connection.
2. If a client connects to a server, and the server does not receive a Hello command within 100ms, the server MUST time out that connection. If the server receives some, but not all, bytes of the Hello message, the timeout is reset.
   • Thus, if the server receives 1 byte of a Hello message within 100ms, the client has another 100ms to send the remaining bytes. If the remaining bytes are not received within 100ms following the first byte, the server must close the connection; the timeout will NOT reset again.
3. If a client connects to a server and sends a Hello command, and the server does not respond with a Welcome reply within 100ms, the client MUST time out that connection. If the client receives some, but not all, bytes of the Hello message, the timeout is reset.

There is another condition that MAY trigger a timeout. Handling this is optional:

4. If a client has completed a handshake with a server, and has sent a SetStation command, and the server does not respond with an Announce reply within some preset amount of time, the client MAY time out that connection. If this happens, the timeout MUST NOT be less than 100 milliseconds.

A timeout MUST NOT occur in any circumstance not listed above.

Note: while we specify precise times for these timeouts, we don’t expect your program to behave with absolute precision. Processing delays and constraints of running in a multi-threaded environment make exact guarantees impossible.

Song data protocol

The song data protocol is much less structured than the control protocol. When a client is subscribed to a station, the server MUST send song data to the the UDP port specified in the client's Hello message.

In our protocol, we assume that the control client and listener client run on the same machine and have the same IP address. Thus, if a control client connects from IP 1.2.3.4 and sends a Hello listing its UDP port as 5445, song data packets should be sent to 1.2.3.4:5445.

Each individual song data packet MUST NOT exceed 1500 bytes of song data. This is a standard limit for a size of single packets forwarded over the Internet. Otherwise, you may choose any packet size that best fits how you stream data. Each packet SHOULD be roughly the same size, though it's okay if this isn't always the case (for example, when the song loops).

Frequently Asked Questions

See here.

---

1. See the Conventions for more information on data sizes. ↩︎