Snowcast Autograder Test Documentation

This document provides a description of what each test performs.

Important notes

For each test, we try to note the following requirements for each test to run:

• Stdin: What commands the tester will send to your program on stdin (as if the user entered a key on the keyboard). (For example, the q command.)
• Stdout: What output the tester expects you to print on stdout (eg. The server has N stations)
• What Snowcast packets the server will send to your program, and what packets the tester expects to receive

In all cases, the formats for what the tester expects are defined by the Snowcast specification.

Control Tests

Startup And Connecting

TestConnectsToServer

Checks that the control can connect to the server and send a HELLO message to
the server.

TestCompletesHandshake

Stdin: None
Stdout: Reads The server has N stations on stdout

Checks that the the control receives and handles the WELCOME reply from the
server after the control sends a HELLO message. This test passes if the
control is able to print out the number of stations.

TestMultipleWelcomesFails

Stdin: None
Stdout: None

Checks that the control disconnects from the server if the server sends
multiple WELCOME replies.

TestNoWelcomeFails

Stdin: None
Stdout: None

Checks that the control closes the connection to the server if the server
does not respond with a WELCOME 100ms after the control sends a HELLO.

Handling Partial Messages

TestPartialWelcomeTimeoutFails

Stdin: None
Stdout: None

Checks that control terminates its connection to the server if the server
does not receive a complete WELCOME message.

This test waits for the control to send a HELLO, and then responds with
only the first few bytes a of a WELCOME reply. The control is expected to
to terminate its connection to server after 100ms and not receiving the
complete WELCOME.

TestSplitWelcomeSucceeds1

Stdin: None
Stdout: Expects The server has N stations

Checks that the control can successfully receive a WELCOME message from the
server that is sent across two packets.

This test waits for the control to send a HELLO, and then the test server
responds with the first half of the WELCOME message, waits some time, and
then sends the final half of the WELCOME message to the control.

TestSplitWelcomeSucceeds2

Stdin: None
Stdout: Expects The server has N stations

Checks that the control can successfully receive a WELCOME message from the
server that is sent across multiple packets.

This test waits for the control to send a HELLO, and then the test server
responds with packets each containing one byte of the WELCOME reply, with
some delay between each packet.

TestPartialAnnounceTimeoutFails

Stdin: Sets a station
Stdout: None

Checks that the control closes its connection to the server if the control
receives part of the ANNOUNCE message and does not receive the remainder of
the ANNOUNCE message within 100ms of receiving the first part.

TestSplitAnnounceSucceeds1

Stdin: Sets a station
Stdout: Expects New song announced: <song name>

Checks that the control can successfully receive an ANNOUNCE message from the
server that is sent across two packets.

The test server responds with the first half of the ANNOUNCE message,
waits some time, and then sends the final half of the ANNOUNCE message to
the control.

TestSplitAnnounceSucceeds2

Stdin: Sets a station
Stdout: Expects New song announced: <song name>

Checks that the control can successfully receive a ANNOUNCE message from the
server that is sent across multiple packets.

The test server responds with packets each containing one byte of the
ANNOUNCE reply, with some delay between each packet.

Message Formatting

TestSetStationFormat

Stdin: Sets a station
Stdout: None

Checks that the control sends SET_STATION commands in the correct format.

TestExitsOnInvalidCommand

Stdin: None
Stdout: Expects The server has N stations on to verify startup

Checks that the control terminates if the server sends an invalid command
message to the control.

TestExitsOnInvalidReplyType

Stdin: None
Stdout: None

Checks that the control terminates its connection with the server if the
server responds with an invalid reply type to the control (i.e some unknown
reply type).

Miscellaneous Functionality

TestPrintsAnnounce1

Stdin: Sets a station
Stdout: Expects New song announced: <song name>

Checks that the control can print a single ANNOUNCE message received from the
server.

TestPrintsAnnounce2

Stdin: Sets a station
Stdout: Expects New song announced: <song name>

Checks that the control can continuously print a ANNOUNCE messages received
from the server. This test sends 10 ANNOUNCE messages to the control and
checks that the control can print all 10 ANNOUNCE messages.

TestAnnounceBeforeSetStationFails

Stdin: None
Stdout: None

Checks that the control terminates its connection with the server if the
control receives an ANNOUNCE message from the server before sending a
SET_STATION message to the server.

TestSetStationInvalidCommandResponseFails

Stdin: Sets station
Stdout: None

Checks that the control terminates its connection to the server if the server
responds with the incorrect response type (i.e responding with a WELCOME
instead of an ANNOUNCE)

TestQuitsCleanly

Stdin: q command
Stdout: None

Checks that the control can exit cleanly when the user inputs the quit command to the control CLI.

Server Tests

TestNoStationsFails

Stdin: None
Stdout: None

Checks that the server fails to start if no stations were provided

TestAcceptsClientConnection

Stdin: None
Stdout: None

Checks that the server does not fail when accepting a single connection.

TestTimesOutConnectionWithNoHello

Stdin: None
Stdout: None

Checks that the server closes a client connection when the client does not
send a HELLO message within 100ms of connecting to the server.

TestCompletesHandshake

Stdin: None
Stdout: None

Checks that the server responds with a WELCOME response, completing the
handshake, when the client sends a HELLO message.

TestMultipleHellosFails

Stdin: None
Stdout: None

Checks that the server terminate a client connection if the client sends
multiple HELLO messages.

TestServerSurvivesClientDisconnect

Stdin: p <file> command
Stdout: None, but must read station list from file

Checks that the server does not terminate if a client disconnects.

TestPartialHelloTimeoutFails

Stdin: None
Stdout: None

Checks that server terminates a client connection if the server does not receive a complete HELLO message.

This test sends only the first few bytes of the HELLO message to the server and the server is expected to terminate the client after not receiving the full HELLO message with 100ms to receiving the first few bytes.

TestSplitHelloSucceeds1

Stdin: None
Stdout: None

Checks that the server can successfully accept a HELLO message from the client that is sent across multiple packets.

This test sends the first few bytes of the HELLO message to the server, waits some time, and then sends the final bytes of the HELLO message to the server.

TestSplitHelloSucceeds2

Stdin: None
Stdout: None

Checks that the server can successfully accept a HELLO message from the client that is sent across multiple packets.

This test sends each byte of the HELLO message to the server in a separate
packet, with some delay between each of the sends.

TestPartialSetStationTimeoutFails

Stdin: None
Stdout: None

Checks that server terminates a client connection if the server does not
receive a complete SET_STATION message.

This test sends only the first few bytes of the SET_STATION message to the
server and the server is expected to terminate the client after not
receiving the full SET_STATION message with 100ms to receiving the first few
bytes.

TestSplitSetStationSucceeds1

Stdin: None
Stdout: None

Checks that the server can successfully accept a SET_STATION message from the
client that is sent across multiple packets.

This test sends the first few bytes of the SET_STATION message to the server,
waits some time, and then sends the final bytes of the SET_STATION message
to the server.

TestSplitSetStationSucceeds2

Stdin: None
Stdout: None

Checks that the server can successfully accept a SET_STATION message from
the client that is sent across multiple packets.

This test sends each byte of the SET_STATION message to the server in a
separate packet, with some delay between each of the sends.

TestSetStationBeforeHelloFails

Stdin: None
Stdout: None

Checks that the server terminates a client connection if the client sends a SET_STATION message before a HELLO message.

TestSetStationSucceeds

Stdin: None
Stdout: None

Sends SetStation message to the server and checks to make sure the correct Announce is received.

TestSupportsMultipleStations

Stdin: p <file> command
Stdout: None, but must read station list from file

Checks that the server can support multiple stations. This test sets up the
server with multiple stations and then iterates over each file provided for a
station. For each file, the test does the following:

• send a SET_STATION message to the server for each station and assert that
  the server responds with a corresponding ANNOUNCE.
• checks that the test client is placed with the correct station in the
  server.

TestStreamsWithoutReadingEntireFile

Stdin: p <file> command
Stdout: None, but must read station list from file

Checks that file data is streamed by reading a fixed number of bytes at a time instead of reading the entire file into memory at once.

To ensure this behavior, this test uses /dev/urandom and /dev/zero as files for the stations.

TestInvalidStationFails

Stdin: None
Stdout: None

Checks that the server terminates a client connection if the client sends a SET_STATION message to the server that contains an invalid station number.

TestAcceptsMultipleConnections

Stdin: p <file> command
Stdout: None, but must read station list from file

Checks that the server can handle multiple connections with each connection
have different stations. This test makes several connections from different client, prints the station list to a file, and verifies that each client is listed on the correct station.

TestAcceptsMultipleConnectionsRegardlessOfStation

Stdin: None
Stdout: None

Checks that the server can handle multiple connections with each connection
have different stations. This test performs the same check as TestAcceptsMultipleConnections, except the connections are verified only by looking at the set of sockets used by the snowcast_server process–there is no verification that clients are joined to the correct station.

Note: This test is designed to test a strict subset of the functionality in TestAcceptsMultipleConnections. If you are failing this test but passing TestAcceptsMultipleConnections, don't worry–we'll adjust manually.

TestStreamRateSingle

Stdin: None
Stdout: None

Checks that the server's stream rate is 16KiB/s for a single connection listening to one station.

TestStreamRateMultipleOneStation

Stdin: None
Stdout: None

Checks that the server's stream rate is 16KiB/s for a multiple connections listening to one station.

TestStreamRateMultipleStations

Stdin: None
Stdout: None

Checks that the server's stream rate is 16KiB/s for a multiple connections listening to different stations.

TestStreamsWithNoListeners

Stdin: None
Stdout: None

This test attempts to verify that the station continues playing music when no clients are joined to the station.

To do this, the test starts the server and waits >1s before joining a client. When the client first receives listener data, the data must not match the beginning of the file.

Note: The functionality tested here is generally straightforward manually, but quite difficult to test programmatically. We will manually review test failures to check for false negatives, and adjust accordingly. If you are failing this test, ask yourself: "Is the station still reading the file when no clients are connected?" If your design for stations does this, you will get credit.

TestServerClosesConnectionOnInvalidCommandType

Stdin: None
Stdout: None

Checks that the server terminates a client connection if the client sends a
command with an unrecognized command type.