[BASIC VERSION] Lab 3a: Testing and Debugging

Setup

Before beginning this assignment, clone the source code from GitHub to make your personal repository for this lab. Here is the GitHub Classroom link.

Learning Objectives

Use concrete examples and debugging to explore or explain how a program works
Develop a collection of examples and tests that exercise key behavioral requirements aspects of a problem
Practice using debugging diagrams in tandem with the IntelliJ debugger to find and fix bugs in code

Testing - I Write Tests Not Tragedies

One of the most important aspects of coding is being able to reliably document that you know exactly how your code works. The process of doing this is called testing, during which you will use functions that you've written and predict what the output should be. If your expected output matches your actual output, your test will pass. If the two are different, your test will fail. Importantly, testing is often most effective for you to do before you code. It gives you a chance to define your problem and write code that solves the problem you've set out to do.

General Cases

Testing is often broken into two parts, a general case and an edge case. The general case is often the case that drove you to try to solve the problem in the first place. For example, if you were given a list of integers and wanted to add one to each of them, the general case might be something where you input something like [1, 2, 3, 4, 5], which you would expect to return [2, 3, 4, 5, 6]. Think of the general case as a problem motivator - you should write tests on these sorts of cases because they communicate that that your program generally works.

Edge Cases

Edge cases are often what makes testing tricky. These are the cases that probably are not motivating your need to solve a problem, but are cases that will arise nonetheless. From the previous example of adding one to every element in a list of integers, an edge case might be inputting the empty list. It's not immediately clear from how we defined the problem how this case should be handled, and therefore is an edge case of this program. In this case, let's say that inputting an empty list should return an empty list. You should include a test like this in your test suite, as it is a feature of your program and you should define behavior for the possible cases you could be given.

Testing `void` methods

Sometimes, in Java, methods will return void, which means that they return nothing. The main method that runs your programs is one such example of a void method. Although you should never write tests for main, any other void method you write should be tested. However, it's not immediately clear how you should do this, since the method does not output anything at all.

The strategy you should take is to write tests to check out the side effects, or changes that the void method is making. These methods do perform operations - they wouldn't exist if they didn't - but they perform operations on other pieces of data. In this class, your void methods will almost always be modifying the fields of classes. It is, therefore, your responsibility to check that these modifications are registered in your object. To do this, you should call the void method, and then in your test case, make sure that the proper field has been modified in the approriate way.

For example, our addFirst method on the ILists from lecture is a void method. If you were to be the one implementing this method and wanted to check that it had worked, you should make sure that first element of the list is the element that you added. Note that this means that the call to the method is not what would appear in the JUnit test. Your test case would look something like this:



IList lst = new Empty();
lst.addFirst(3);
Assert.assertEquals(3, lst.firstVal());

There are more advanced ways that void methods can work - if you go on to take CSCI0330, in particular, you will learn about some interesting things void methods can achieve.

Testing Scope

Another important aspect of testing is to document exactly how far your program changes pieces of data. Say you are building your own LinkedList class and you want it to have a field for length. Let's suppose that the addFirst method that you've written does indeed add the element to your list. Is it a given that the size has been updated too? No! These changes should also be documented in your code. If a method affects a variable that exists outside of the scope of the method (i.e. a field, not declared within the method), you should check to make sure that it is altered as you expect it to be. Thus, the test block is not sufficient to test that something was added correctly. We should have something that looks more like this:




IList lst = new Empty();
lst.addFirst(3);
Assert.assertEquals(3, lst.firstVal());
Assert.assertEquals(1, lst.length());

Testing in Java

If you were in the CSCI0170+ track, you will know how we will be testing in Java and can skip reading this section!

In order to use JUnit (the framework you will be using to write tests in this class), you will need to add both the hamcrest-core-1.3.jar and junit-4.13.2.jar files as dependencies to your IntelliJ project. When opening this project in IntelliJ for the first time, they should be added automatically. If they don't work out of the box, you'll need to add them manually. If you're not sure how to do this, check out the Adding Jars/Dependencies section of the IntelliJ Setup Guide.

We will provide you with starter files for labs and homeworks, which will look something like [ClassOrAssignmentName]Test.java.

How do I test methods?

The format for testing a method is as follows:




@Test
public void <test-name>() {
    // code that tests the specific method 
}

In practice, it looks something like this:





@Test
public void testOne() {
    Dillo deadDillo = new Dillo(15, true);
    Assert.assertEquals(15, deadDillo.length());
}

Here we're first checking if the length of the new Dillo equals 15.

How do I test Exceptions?

The format for checking Exceptions is as follows:




@Test(expected = <exception-type>.class)
public void <test-name>() {
    // code that results in exception
}

In practice, it looks something like this:




@Test(expected = IllegalArgumentException.class)
public void courseIllegalArgumentException() {
    Dillo preciseDillo = new Dillo(20.0, false);
}

Where creating a Dillo with length 20.0 (a double, where an int is expected) throws an IllegalArgumentException.

How do I use @Before?

You may sometimes find it helpful to set up your testing data using a @Before method, as follows:








Zoo myZoo;
  
@Before
public void setupData() {
    // set up a list of animals called aniList
    this.myZoo = new Zoo(aniList);
    // any other operations needed to set up myZoo
}

JUnit will automatically run setupData before it runs every @Test method. This lets you write the code to build your testing data once, then have it reset to a clean value at the start of each test. This way, you don't have to worry about whether one test added or removed elements, which could break the expected answers in other tests.

Testing Task

For this task, you will be working to test a sample program.

In this section, do not modify the program – in fact, we don't even want you to read it. You should write your tests based on the following specification, in order to find out if the program we gave you has bugs.

Program specification

We have given you a stack, which is a last-in, first-out data structure. That is, you only interact with the top of the stack, by adding and removing items, and the last item you put on is the first item you can take off. Think of this like a stack of plates on a counter – you can put clean plates on top of the stack, and you can remove plates starting with the most recent one you put on.

You can instantiate a new empty stack by using our LabStack class:

LabStack myStack = new LabStack();

The methods for the stack are:

void push(int i): adds i to the top of the stack.

int pop(): removes the top item from the stack and returns it. Throws an IllegalStateException if called on an empty stack.

int peek(): "peeks" at the top item from the stack by returning it but not removing it. Throws an IllegalStateException if called on an empty stack.

int size(): returns the number of items in the stack.

void clear(): resets the stack to empty.

Write the tests

Task 1: Write down some things that should be true for this program. One example is "popping from a non-empty stack should decrease its size by 1." What other truths (properties) can you come up with? Come up with at least one for every method listed above, and more if appropriate. Remember to think about edge cases!

Task 2: Now that you know what you should test, we want you to create a test suite for your program. Write these out in LabStackTest.java. Be ready to tell a TA about what you chose to test and what edge cases you cover.

Note: If you run your test suite using TestRunner.java, you will notice that not every test will pass. That is expected!

Checkpoint! Call over a TA once you have written out your complete test suites.

Testing and Domain Knowledge

Testing, edge cases, and debugging go beyond ensuring your code runs and functions as expected when passed a technical edge case like an empty list or a negative number. The context in which your code is deployed and types of users who interact with your program can alter its expected functionality and introduce vulnerability to context-dependent bugs. We call the specific knowledge of the rules, logic, and expectations of this context domain knowledge.

For example: Imagine you wrote a program to handle Brown student registrations for Spring Weekend. Everyone is excited to see the performers, so the event is expecting a huge turn out! You decided the simplest approach for registrations will be to create Student objects and add them to a list.

The registration form looks like this:

Your plan is to take input from this form and use it to create Student objects, which look like this:






class Student {
    String firstName;
    String lastName;
    String phoneNumber;
}

You plan to use the names of the students to create a simple to navigate registration list for check-ins. This list will be ordered by last name and printed out so security can confirm registrations efficiently to keep lines moving.

You will use the phone numbers to send out reminder text messages for performances.

Because the form takes in a full name as a String, you decide that you need two functions, extractFirstName and extractLastName, to convert the full name to first and last name fields. Here are the documented functions:






















/*
Looks for the first instance of a space in the fullName string
and returns the contents of the String before the space! 
If there is no space, it returns the input.
Ex: “John Smith” -> “John”
Ex: “My Name” -> “My”
Ex: “NoSpace” -> “NoSpace”
Ex: “” -> “”
*/
public String extractFirstName(String fullName) {...}
 
/*
Looks for the first instance of a space in the fullName string
and returns the contents of the String after the space! 
If there is no space, it returns the input.
Ex: “John Smith” -> “Smith”
Ex: “This Is My Name” -> “Is My Name”
Ex: “NoSpace” -> “NoSpace”
Ex: “” -> “”
*/
public String extractLastName(String fullName) {…}

A user persona is a fictional archetypical user whose goals and characteristics represent the needs of a larger group of users. Creating personas is a common industry practice that allows designers and engineers to ask “who are we building for?” and ensure the software they create meets the user's needs.

For each of the following user personas, go through the registration process using the code above.

Task 3: For each of the user personas above:

What would the program store for their first name and last name? Does this seem correct?
What assumptions were made by the programmer that led to these results?

Task 4: Now that you have interacted with this system, each partner should come up with a proposed change to fix the system. You don't need to write any code, just think about how you might solve some of the problems you encountered.

Here the system has three main components: the form, the student object fields, and the name extraction methods. If you opt to alter the form as part of your proposed change, make sure you still collect full names and phone numbers in some capacity!

Task 5: Now think of a persona that would break your partner’s solution!

(Hint: If you can’t think of one, take another look at the user personas above and think internationally. Are we making any assumptions about ordering? Check out Falsehoods Programmers Believe About Names for some common assumptions.)

Reflect on how thinking through user personas is similar to or different to the type of testing you did in Part 1 of this lab. It may help to think about the ways you think about coming up with JUnit assertions vs. how you came up with a persona to break your partner’s solution. In your opinion, how do you define the purpose of testing, keeping in mind both types of testing you’ve encountered in this lab?

Checkpoint! Call a TA over to talk through your answers.

Debugging - Debug-A-Boo

Debugging is the process through which we discover where our program does not work like we think it should. Good news: if you've written enough tests, you should be able to narrow down the scope of your search for bugs! In fact, the hardest part of debugging is finding out where the bug may be, which is why narrowing down the places that you need to look by defining your problem is so important. Once you have an idea of where things are going pear shaped, you can start to use some very useful tools.

A debugger is a program that allows you to follow the execution of your own program and examine the values (environment/heap) you are working with. IntelliJ has a fantastic debugger that lays out everything you might need to know rather well. You can run the debugger by clicking the little bug icon in the top corner.

This will run the program as if you hit the run button, but within the scope of the debugger.

Breakpoints

Before starting debugging, you need to set one or more breakpoints, or places in the program where you want the debugger to stop so that you can examine the code.

To insert a breakpoint, move your mouse to the left of the line you want to break on (1 in the diagram above). If you have line numbers enabled, it should be to the right of the line numbers. Then, single-click, and a red dot should appear. To remove it, single-click on the breakpoint.

After setting a breakpoint and running the debugger, IntelliJ provides you with a debugging window, giving you access to all of its features.

Other Debugging Commands

Breakpoint - When running the debugger, your program pauses each time it encounters a breakpoint. Note that your program pauses before executing the line it highlights.
Debugging Window - This is where you will see everything you’ll need to debug your code such as all the useful debugging buttons and variables related to your program.
Resume - The resume function continues running your program until it reaches another breakpoint or terminates.
Step Over - The step over function does not enter a function but instead jumps over to the next line of code. It still executes that function, however.
Step Into - If a function is encountered or you had a breakpoint at a function call, step into will stop program execution at each line in the function and show you how each line gets processed
Force Step Into - We won’t be needing or using this action, but it lets you debug methods defined in the APIs or Libraries.
Step Out - This will let you skip stepping through code in a method line by line and return to the calling method.
Variables Panel - This is where you will see all the variables you have defined in the method you are debugging. By clicking on each variable you can see the values associated with them which can help you determine whether your variables are storing the values you expected them to store. Stepping into your code and monitoring your variables to make sure they are storing reasonable values is what will help you locate your issue as quickly as possible!
Stop Button - If you want to stop debugging mode, you can click on the red square stop button in the top right corner next to the debugging button or on the left hand column of the debugging window.

Actions to Take During Debugging

Always narrow down where you think the problem is. Do this by writing tests and seeing which cases are failing that shouldn't be. Doing this will make the whole debugging process significantly easier.
Set a breakpoint on the method that is causing the error.
Step into/over each line, keeping track of what your code is actually doing in memory. It is helpful to have some pencil and paper handy. This will also provide a helpful record that you can look back on later, in case you need to try debugging again.
Have patience! It is normal to be unable to locate a bug immediately, but persistence is key. The length of the debugging process is why we always suggest starting early.
Make the changes you need to and run your program to ensure that you've found the bugs!

Using Testing to Debug

The underlying data structure of LabStack is a linked list implementation, where the front Node of the list is the top element of the stack. The end of the stack is denoted using a null. LabStack also has a size field that changes as elements are added/removed from the stack.

Task 6: Draw out some examples with different cases in mind for how the methods of LabStack should output or alter the underlying data (underlying linked list and size field). You should not change/add elements to the LabStack you have already drawn out, but instead redraw the entire LabStack with the change that you expect to see. We call these "Before and After" diagrams. You should draw these diagrams for every method in LabStack, including the constructor.

Using failing tests as breakpoints

Failing tests are a great starting point for creating breakpoints. For example, say that the following test was failing:





@Test
public void testOne() {
    Dillo deadDillo = new Dillo(15, true);
    Assert.assertEquals(deadDillo.length, 15);
}

Before debugging this failure, you would have drawn an example of what you expect the result of the method under test (in this case, the constructor) would be. It would probably look like:

To debug, you would place a breakpoint at the place in your test where you were calling that method. For this example, the breakpoint would be at line 3. Then you would step into the code and examine whether the environment/heap match what you expect. If some step causes a discrepancy between what you expect and what you see, you know the code for the step is what is causing the bug!

For example, if you entered the Dillo constructor and saw that it set this.length to 14 in the debugger instead of the desired 15, you would know that is where the bug comes from:

Task 7: Using the test suite you create in the last task and IntelliJ's built-in debugger, go through the code that we have given you in LabStack and find what is causing the bugs. Make the fixes you need to until your whole test suite passes. Even if you see the problem in the code before you start the debugger, we want you to actually use the debugger to examine the program state. Your TA will ask you for examples of how you used your diagram in conjunction with the debugger to find the bugs in the code.

Checkoff! Congratulations! Call over a TA to get credit for completing the lab!

Just for fun: Debug a restaurant management system! - Kathi's Diner

If you have time, check out the restaurant management debugging task in the Advanced lab! This task has you work with more complicated code that somebody wrote, and deal with debugging code that has randomness.

Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CSCI0200 document by filling out the anonymous feedback form.