HackMD
Writing Examples and Tests
Examples and tests play an important role in the lifecycle of programs.
- People write code to solve problems: both examples and tests help insure that the solution actual does what it needs to do
- People sometimes need to look at code after it's been written (to find out or remember what it does): examples are a form of documentation.
- Programs often get modified over time as new features get added or analysis needs get refined: tests help check that the additions don't break needed functionality.
You may think that the programs you develop in 111 are too small for these practices to matter. This is less true on later assignments and projects. More importantly, 111 is teaching you the practice of responsible programming. That includes developing robust programming habits. If you head out from 111 to help a professor elsewhere at Brown with their research, the accuracy and lifecycle of your work could impact the validity of their results. These habits matter.
The rest of this document describes our expectations and offers guidelines for writing examples and tests.
What is the structure of an example or test?
An example or test has two critical parts: the expression that you want to run and the answer that you expect that expression to produce.
double(10) is 20
double(15) is 15 * 2 # the answer can be an expression
Examples and tests should be actual code that can be run, not just notes in a comment. People ignore most notes in comments. For examples and tests to help us develop robust programs, they need to be run against our code.
When to write examples and tests
Generally speaking, write examples before you write your code. This helps make sure you are clear in your own mind about what you are trying to write.
Write tests before, during, and after you write your code. Tests capture more nuance, deeper examples, and more fine details than examples. They'll occur to you at different times. Write them down as you go.
Examples (in where blocks)
- All functions, including helper functions, require examples. The only exceptions are functions that return
Imageandlamfunctions (which we will learn later in the course). - Examples should illustrate interesting variations in either the inputs or the possible function behavior. You want some variation here (remember: examples are both documentation and a way for you to think out the cases your code needs to handle), but you don't have to cover everything.
Tests (in check blocks)
- Tests should be written for more complicated functions or in situations where we tell you to put your tests in a separate file.
- Tests should cover the space of possible inputs pretty thoroughly. You can't test every possible input value, but you can test different kinds of inputs as matter to your function. The guidelines below help you think through this.
Guidelines for developing examples and tests
Cover Edge Cases
Edge cases are inputs at the boundaries within the space of inputs. For instance, a function that checks for "numbers between 4 and 8" or "strings with at least 3 characters" has boundaries.
If the problem has boundaries, test values both at and on either side of each boundaries. For example, if a function looks at values "greater than or equal to 5", you should have examples that use inputs such as 4.5, 5, and 6 (this would catch typos in your use of > vs >=).
A more detailed example
As an example, let's consider a function compare that takes two numerical inputs x and y and returns a string. It should return "way less" if x is smaller than y - 2, "about the same" if x is between y - 2 and y + 2 (inclusive) and "way more" if x is greater than y + 2. Here's a set of good examples for this function.
fun compare(x :: Number, y :: Number) -> String:
doc: "decribe the relationship between input numbers x and y"
... # function body elided
where:
compare(1, 40) is "way less"
compare(0, 3) is "way less"
compare(8, 10) is "about the same"
compare(1, 1) is "about the same"
compare(0, 1) is "about the same"
compare(17, 20) is "way more"
end
Notice that these examples cover each interesting "case" of the program. They also include edge cases testing the boundaries–for instance, compare(8, 10) is an edge case, since it's at the boundary between the "way less" and "about the same" cases.
Think about meaningful variations in the input based on what it represents
Consider two pieces of information that you might represent with a string: names and passwords. Names might have spaces but likely not symbols like #, while passwords won't have spaces but often have symbols. When coming up with inputs to test, think about the information your input represents and what might be meaningful.
For example, if you are writing a program that processes passwords, your input might have numbers, letters (lower and uppercase) and symbols. The symbols could be anywhere in the string (beginning, middle, end). There's likely a minimum length and a practical maximum. All of these scenarios should show up in your examples and/or tests.
Think about variations in the structure of the type of input data
Some types of data have structure to their variations. A list can be empty or non-empty. A traffic light color can be red, yellow, or green. When your data has structural variation (as opposed to representational variation, as in the previous point), you should have an example for each variation.
Ignore out-of-scope inputs unless you're told otherwise
If we tell you that we are writing a program that takes a positive integer as input, use positive integers in all of your examples and tests. You do not have to test 0, -1, or some other input that is outside the scope of the problem.
Why? Error handling is a complex topic on its own. We'll touch it very lightly in 111; you'd see more about it in later CS courses, where you have the tools and skills that you need to actually deal with such situations. For now, our goal is to get you writing programs that are robust on valid inputs.
If we tell you to have a program report an error, test that behavior
There is a difference between a program being called on an invalid input and a program needing to report that it can't produce a valid answer in a particular situation. If a problem statement tells you to report an error, you should have examples and/or tests that cover the error-inducing behavior.
How we grade examples and tests
We use the criteria that we just gave you: edge cases, representational variation, structural variation, and error checking (if errors are part of the problem statement).
Sometimes, we grade your tests for their correctness (do they accurately capture the problem statement) and thoroughness (did they do a good enough job with the criteria that they can distinguish working from broken solutions to a problem). We'll introduce this in more detail as we go.
