# QS

Software is ready for release when:

* We can't find any bugs;
* We have finished testing (which are finite);
* When quality is high.
* When we can argue that it is dependable.

## Software quality

*Quality* encompasses both **what** the system does and **how** it does it; however, not all people agree on the same definition of quality.

### Quality attributes

*Quality attributes* are certain aspects of software quality that make a software have more or less quality. Some attributes are more of interest than others, based on the project.

Some attributes are:

* **Dependability**: the system offers the correct functionality under all possible (foreseen or not) outcomes;
* **Performance**: the system meets the timing requirements, i.e., the requirements of the system responding to some event;
* **Security**: the system protects the information and their users from unauthorized access;
* **Scalability**: the system can grow to process more concurrent requests;
* **Availability**: the system minimizes downtime and recovers **garciafully** from failures;
* **Modifiability**: it's possible to modify the system, adding new features, changing existing ones or removing existing ones easily;
* **Testability**: the system is easily testable by providing means of easily identifying faults in the system to reduce the probability of them happening again;
* **Interoperability**: the system is able to communicate with other systems easily, providing information and/or functionalities to 'em;
* **Usability**: the system is easy to use for each user and provides support to them, to increase satisfaction.

These attributes often **conflict**, i.e., improving one may harm the other.

It is important to decide what's the most important attributes for the system that we're building and which can be compromised a little bit.

A system may add a more secure cipher, which takes more time to compute. That increases **security**, but decreases **performance**.

#### Dependability

The goal of dependability is to establish that the system is:

* **Correct**: a program is correct if it is consistent with its specification;
* **Reliable**: statistical approximation of correctness, i.e., the likelihood of correct behavior for a certain period of time. It's improved when faults in frequently-used parts of the software are removed.
* **Safe**: ability to avoid hazards;
* **Robust**: ability to fail **garciafully**.


##### Measuring dependability

Because correctness is hard to prove and robustness/safety do not demonstrate functional correctness, *reability* is the only property used to measure dependability.

Reliability is the only property of dependability that is measurable.

The metrics used for measuring reliability are:

* **Availability**: calculated as $\frac{uptime}{totalTime}$ or $\frac{MTBF}{(MTBF+MTTR)}$;
* **Probability of failure on demand (POFOD)**: the likelihood that a request will fail. Calculated as $\frac{failures}{requests}$ (appropriate for when failures are serious);
* **Rate of occurrence of fault (ROCOF)**: frequency of occurrence of unexpected behavior. Calculated as $\frac{failures}{totalTimeObserved}$ (appropriate for when requests are made regularly);
* **Mean time between failures (MTBF)**: Average length between failures. Calculated as $\frac{\sum{timesBetweenFailures}}{numberOfFailures}$;
* **Mean time to recover (MTTR)**: Average length to repair failures. Calculated as $\frac{\sum{timesToRecoverBetweenFailures}}{numberOfFailures}$.

#### Performance

##### Measuring performance

There are many performance measuring methods, such as:

* **Latency**: time it takes to complete an interaction;
* **Responsiveness**: how quickly the system responds to routine tasks. Calculated as a probability (e.g., 95% of the time);
* **Turnaround time**: time to complete tasks. Calculated as a time to complete the task (e.g., 5s).
* **Jitter**: allowable interval of latency, because response time is non-deterministic;
* **Throughput**: number of transactions the system can process per unit of time. The greater the throughput the greater the response time (latency);
* **Deadlines in processing**: points where processing have reached a particular stage;
* **Number of events not processed**: events not process because the system is too busy.

#### Scalability

There are two types of scalability:

* **Horizontal**: adding software resources;
* **Vertical**: adding hardware resources.

#### Security


The users of the system (or the system itself) (**actors**) enforce security **mechanisms** (provided by the system) to allow or deny access to the resources to other actors. The system also provides **policies**, which define legitimate access to the resources and are applied by the mechanisms.

### Scenarios

A *scenario* is a description of an interaction between an external entity
and the system.

Scenarios are used in:

* **Designing the system**;
* **Stakeholder negotiation**;
* **Testing**.

A **good scenario** is credible, valuable, specific, precise and comprehensible.

#### Scenario format

SoS->System artifact: stimulus (action)
System artifact-->SoS: Response

The Source of stimulus (SoS) is an actor (system or user), which send a stimulus to a part of the system (a subsystem or artifact), which works under a specific **environment**; the system, in turn, responds with a response, which can be measured (through latency, throughput, etc.).

The system has a **state**, which affects how it is behaving at a certain point (such as information stored, current load, etc.).

#### Types of scenarios

There are several types of scenarios, such as:

* **Reliability scenarios**: used to minimize the number of observed failures;
* **Availability scenarios**: used to measure how the system responds to failure;
* **Performance scenarios**: used to measure system performance;
* **Scalability scenarios**: used to measure how adding and removing resources impact a system;
* **Security scenarios**: used to protect data from unauthorized access, while providing service to authorized users.





## V&V (Verification and validation)

The amount of *V&V* depends on **software purpose** (a software for a critical system needs much more v&v than a software for personal use, e.g.), **user expectations** and **marketing environment** (a software which a lot of competitors needs to be tested, as bugs and defects may cause the user to go to the competitor).

### Verification

*Verification* is a software engineering task, where we check if we are building the **product the right way**, i.e., if it conforms to its specification.

It is an experiment and there are many ways to verify it.

#### Types of verification

Verification can be:
* *Static*: analysis of pieces of the system, to discover problems. Can be made through:
* Proofs: posing hypotheses and making arguments;
* Inspections: manual checking the source code of pieces.
* *Dynamic*: exercising the system to argue that it meets the requirements. Can be made through:
* Testing: formulating sets of inputs to demonstrate that the ouput is in accord to the requirement;
* Fuzzing: generating random input to locate possible crashes or anomalies.

#### Testing

*Testing* is an experiment to check if the system conforms to the expected results given an input. There are many types of testing.

Testing cannot exhaustively try all inputs in finite time, so the tester must accept some degree of inaccuracy.

Testing is used to find:

* *Failure*: an execution which results in an incorrect output;
* *Faults*: the problem which caused the failure.

:::info
When we observe a failure, we try to find the fault.
:::

The main purpose of testing is to find faults.

:::success
“Program testing can be used to show the presence of bugs, but never their absence.”
- Dijkstra
:::

A *test suite* is a collection of *test cases*. A test suite is always executed together, but each test case is independent from one another.

A test case triggers a certain part of the software under test. A test case consists of:

* *Initialization*: any steps that must be taken before test execution;
* *Test steps*: the actual step that stimulates a certain part of the system and compares values against the oracle;
* *Tear down*: any steps that must be taken after test execution.

The input of a test case can be anything, from static to dynamic values. It can be:

* *Black-box*: the tester does not have access to the structure of the code;
* *White-box*: the tester has access to the structure of the code.

A test oracle determines if a test should **pass** or **fail**, i.e., it compares the output with the expected output.
Test creation and execution can be:

* *Manual*: Exploratory testing, alpha/beta testing, ...;
* *Automated*: Testing frameworks, interface frameworks, automated test case generation, etc.

##### Granularities of testing

We can test different granularities of software, such as (in decreasing order):

* *System*: the whole system, which can be interacted by using interfaces (such as APIs);
* *Subsystem*: parts of the system;
* *Units*: the lowest components of a system, which can be interacted by using method calls.

##### Unit testing

*Unit testing* is the process of looking for errors in an isolated part of the software.


Unit testing is usually done on functions, giving them an input and comparing their output with the expected output for the specific input on that function. To test it, many testing tools use **assert** commands, to help write tests.

*Assert commands* pass or fail the test if the output given by the SUT matches or not the given expected output for the given input.

###### In Java (JUnit)


###### In Python (unittest) (example)

Given the following function:

def isValidIpv4(address):
 """Check if an IPV4 address is valid.
 Parameters
 ----------
 address : str
 	The ipv4 address to check
 Returns
 ----------
 bool
 	True if the IPV4 address is valid and False otherwise
 """
 try:
   addressBytes = address.split('.')
   validAddressBytes = [b for b in addressBytes if int(b) >= 0 and int(b) <= 255]
   return len(addressBytes) == 4 and len(validAddressBytes) == 4
 except:
   return False

Possible tests for the function are:

def test_is_valid_ipv4(self):
  self.assertFalse(Helper.isValidIpv4(""))

def test_is_valid_ipv42(self):
  self.assertTrue(Helper.isValidIpv4("123.111.222.121"))

def test_is_valid_ipv43(self):
  self.assertTrue(Helper.isValidIpv4("123.12.222.121"))

def test_is_valid_ipv44(self):
  self.assertFalse(Helper.isValidIpv4("2.1.2.1.1"))

def test_is_valid_ipv45(self):
  self.assertFalse(Helper.isValidIpv4("2.1.2.1."))

def test_is_valid_ipv46(self):
  self.assertFalse(Helper.isValidIpv4("2.1.2"))

##### System testing

*System testing* is the process of testing the integration of units as a whole.


##### Testing percentages


###### Interface errors


##### Exploratory testing

Test cases are not created in advance; instead, testers embark on a journey through the app and create tests as they go; oh look they found a fockin "pikachu"!

### Validation

*Validation* is a software engineering task, where we check if we are building the **right product**.


## Product readiness

Because it is impossible to find all possible faults in a system in finite time, we have to assess when to stop V&V instead.

*Acceptance testing* has a controlled pool of actors in a controlled environment testing and providing feedback about the system. Ultimately, it is up to the users to assert that a system is functioning properly.


## Software testing myths