# Testing

[Coverage](https://www.bullseye.com/coverage.html)

# Lecture 1

## What is testing


## Testing benefits


## Validation

Har vi rätt krav? Bygger vi rätt produkt? Löser vi kundens problem?


## Verification

Har vi byggt produkt som uppfyllar kraven?



## Omissions and Suprises


## Testing vs Checking


## Fault


## Error


## To find faults with tests

## Reachabilitiy, infection and Propagation


## Failure


## Goals

## Controllability - Observability => Testability

+ smallness


## Accuracy - Traceability


## Conclusion


## Test-case components


## Oracle


## Flaky Tests

Tests that flip between being successful and failures are called **Flaky Tests**.


## Reproducibility


## Input Domain - Reliability - Correctness


## Exhaustive testing


## Test-levels


## Testing Types


## Black box testing


## White box testing


## Smoke and Random Testing

random testing is bit flaky testing

both smoke and random testing has low accuracy because we don't know where the bug exists




# Lecture 2

## X-Unit frameworks parts


## Test automation benefits


## Hard to test code

- Error handling : med mocking kan vi felen att hända


## Dependency injection


## Mocking

test behaviour: like test adding user to DB

testbeteendet med hjälp av mockobjekt



## Unit testing can be done


## Quality of tests


## TTD - Better?


## TTD Basics


## Adequate test


## Types of Coverage

Block är kod som alltid exekveras tillsammans dvs att det inte har nån if sats eller sånt


## Unreachable Coverage


## Hard to test code / hard to show coverage


## Private method

lösningen är att försöka ersätta privata metoder mot klasser så vi kan testa dess public interface ist

## Complementary Negative tests


## Mutation Score

Vi trycker in defekter i programkoden vi skrivit och sen ser vi om test suites som vi skrivit till detta program hittar defekterna


example

skapade mutant genom att byta operator


example på equivalent implementations

det är samma implementation for metoden


## Checked Coverage

Visar tydligt vilka rader som man inte testat och borde ha eget test fall för



## Requirements pyramid


## Scenarios - path coverage



## Test Traceability


## Non functional requirements


## Test Strategy



## Test Strategy - budget


## Test Plan




## Judge test-result


## Who is responsible


## Test plan, the risks


## Clarity



artifcat = test report


# Code Coverage Analysis

## Code coverage analysis is the process of

You use coverage analysis to assure quality of your set of tests, not the quality of the actual product


## Structural Testing and Functional Testing

Code coverage analysis is a structural testing technique. Structural testing compares test program behavior against the apparent intention of the source code. This contrasts with functional testing which compares test program behavior against a requirements specification

## Statement Coverage

This metric reports whether each executable statement is encountered. Control-flow statements, such as if, for, and switch are covered if the expression controlling the flow is covered as well as all the contained statements. Implicit statements, such as an omitted return, are not subject to statement coverage.

Also known as: line coverage, segment coverage, c1 and basic block coverage

The chief advantage of this metric is that it can be applied directly to object code (The product of compiler) and does not require processing source code

The chief disadvantage of statement coverage is that it is insensitive to some control structures.


Statement coverage does not report whether loops reach their termination condition - only whether the loop body was executed. this limitation affects loops that contain break statements

Since do-while loops always execute at least once, statement coverage considers them the same rank as non-branching statements

Statement coverage is completely insensitive to the logical operators (|| and &&).

Statement coverage cannot distinguish consecutive switch labels.

You probably would not have 10 separate test cases for a sequence of 10 non-branching statements; you would have only one test case. For example, consider an if-else statement containing one statement in the then-clause and 99 statements in the else-clause. After exercising one of the two possible paths, statement coverage gives extreme results: either 1% or 99% coverage. Basic block coverage eliminates this problem

One argument in favor of statement coverage over other metrics is that bugs are evenly distributed through code; therefore the percentage of executable statements covered reflects the percentage of faults discovered. However, one of our fundamental assumptions is that faults are related to control flow, not computations. Additionally, we could reasonably expect that programmers strive for a relatively constant ratio of branches to statements.

In summary, this metric is affected more by computational statements than by decisions.

## Decision Coverage

This metric reports whether Boolean expressions tested in control structures (such as the if-statement and while-statement) evaluated to both true and false. The entire Boolean expression is considered one true-or-false predicate regardless of whether it contains logical-and or logical-or operators. Additionally, this metric includes coverage of switch-statement cases, exception handlers, and all points of entry and exit. Constant expressions controlling the flow are ignored.

Also known as: branch coverage, all-edges coverage, C2 decision-decision-path testing

the FAA definition of decision coverage requires all Boolean expressions to evaluate to both true and false, even those that do not affect control flow

This metric has the advantage of simplicity without the problems of statement coverage.

This metric has the advantage of simplicity without the problems of statement coverage.


This metric could consider the control structure completely exercised without a call to function1. The test expression is true when condition1 is true and condition2 is true, and the test expression is false when condition1 is false. In this instance, the short-circuit operators preclude a call to function1.

## Condition Coverage

Condition coverage reports the true or false outcome of each condition. A condition is an operand of a logical operator that does not contain logical operators. Condition coverage measures the conditions independently of each other.

This metric is similar to decision coverage but has better sensitivity to the control flow.

However, full condition coverage does not guarantee full decision coverage.

## Multiple Condition Coverage

Multiple condition coverage reports whether every possible combination of conditions occurs.

A disadvantage of this metric is that it can be tedious to determine the minimum set of test cases required, especially for very complex Boolean expressions. An additional disadvantage of this metric is that the number of test cases required could vary substantially among conditions that have similar complexity

As with condition coverage, multiple condition coverage does not include decision coverage.

## Condition/Decision Coverage

Condition/Decision Coverage is a hybrid metric composed by the union of condition coverage and decision coverage.

It has the advantage of simplicity but without the shortcomings of its component metrics.

## Modified Condition/Decision Coverage

The formal definition of modified condition/decision coverage is:

Every point of entry and exit in the program has been invoked at least once, every condition in a decision has taken all possible outcomes at least once, every decision in the program has taken all possible outcomes at least once, and each condition in a decision has been shown to independently affect that decision's outcome. A condition is shown to independently affect a decision's outcome by: (1) varying just that condition while holding fixed all other possible conditions, or (2) varying just that condition while holding fixed all other possible conditions that could affect the outcome.

**This metric is stronger than condition/decision coverage, requiring more test cases for full coverage.**

Two difficult issues with MCDC are:

* short circuit operators
* multiple occurrences of a condition

## Path Coverage

This metric reports whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit.

Also known as predicate coverage.

Since loops introduce an unbounded number of paths, this metric considers only a limited number of looping possibilities. A large number of variations of this metric exist to cope with loops. Boundary-interior path testing considers two possibilities for loops: zero repetitions and more than zero repetitions [Ntafos1988]. For do-while loops, the two possibilities are one iteration and more than one iteration.

Path coverage has the advantage of requiring very thorough testing. Path coverage has two severe disadvantages. The first is that the number of paths is exponential to the number of branches. For example, a function containing 10 if-statements has 1024 paths to test. Adding just one more if-statement doubles the count to 2048. The second disadvantage is that many paths are impossible to exercise due to relationships of data.


# Other Metrics

## Function Coverage

This metric reports whether you invoked each function or procedure.

## Call Coverage

This metric reports whether you executed each function call. The hypothesis is that bugs commonly occur in interfaces between modules.

Also known as call pair coverage.

## Linear Code Sequence and Jump (LCSAJ) Coverage

The advantage of this metric is that it is more thorough than decision coverage yet avoids the exponential difficulty of path coverage. The disadvantage is that it does not avoid infeasible paths.

## Object Code Branch Coverage

This metric reports whether each machine language conditional branch instruction both took the branch and fell through.

## Loop Coverage

This metric reports whether you executed each loop body zero times, exactly once, and more than once (consecutively). For do-while loops, loop coverage reports whether you executed the body exactly once, and more than once.

## Race Coverage

This metric reports whether multiple threads execute the same code at the same time. It helps detect failure to synchronize access to resources. It is useful for testing multi-threaded programs such as in an operating system.

## Relational Operator Coverage

This metric reports whether boundary situations occur with relational operators (<, <=, >, >=). The hypothesis is that boundary test cases find off-by-one mistakes and uses of the wrong relational operators such as < instead of <=

## Comparing Metrics


## Summary

Coverage analysis is a structural testing technique that helps eliminate gaps in a test suite. It helps most in the absence of a detailed, up-to-date requirements specification. Condition/decision coverage is the best general-purpose metric for C, C++, and Java. Setting an intermediate goal of 100% coverage (of any type) can impede testing productivity. Before releasing, strive for 80%-90% or more coverage of statements, branches, or conditions.