---
# System prepended metadata

title: 'UKAEA RSE Team Journal Club: 2023-02-02'

---

# UKAEA RSE Team Journal Club: 2023-02-02


:::info
- **Article:** <https://www.hillelwayne.com/post/metamorphic-testing/>
- **Location:** K1/0/85
- **Date:** 2023-02-02
- **Agenda**
  1. Brief intro `2 min`
  2. Discussion `43 min`
- **Participants:**
  - Matthew Bluteau (MB)
  - John Nonweiler
  - Kingsley Collie
  - Sanket Gadgil
  - Seimon Powell
  - Kristian Zarebski
  - George Gibbons
  - Callum Harrower
  - Dan Short
  - Harry Saunders
  - Aby Abraham
- **Host:** Matthew Bluteau (MB)

:::

## Brief Intro

- "As systems get more complex, bespoke manual tests become less and less useful"
  - Only hitting a single point in state space
- Generative testing and its more familiary subset Property-based Testing (PBT) are responses to this
    - In PBT, random inputs are generated and the outputs tested against a property of the function
    - The trouble is finding "good" properties for functions that actually provide some information about whether the function is behaving as expected
- Metamorphic Testing (MT) is an alternate (complementary?) approach to solving the problems of oracle and manual tests
    - Instead of considering function outputs in isolation, it looks at a whole set of outputs and enforces a _Metamorphic Relation_ between them
    - E.g. $$ \mathrm{for}\ x_1, x_2 : f(x_1) \sim f(x_2) $$
    - whereas in PBT: $$ \mathrm{for}\ x_1, x_2 :\\ f(x_1) \sim g(x_1)\\ f(x_2) \sim g(x_2) $$

## Discussion

- How do you find relevant MRs?

  - Could be applicable for certain tests - RestAPIs / deep learning tools
  - Could be less applicable for certain areas - bioinformatics

- Is it the same as property based testing? Is the MR just another property?

- Probably people are doing it anyway, just not using the specific name

- Literature on PBT focuses on inputs (how they can be reduced etc.), while MT focussing more on the outputs

- How do you prevent false positives? Could easily have a function that's wrong that satisfies the tests. Probably need some oracle tests still.

- Complementary, probably want an oracle test and then expand them out using MT


_The author identifies a problem with PBT as finding "good" properties for tested functions. Could the same also be said about finding "good" metamorphic relationships in the context of MT?_

Contention that MT is a subset of PBT (**controversial**): the property is getting identical output each time. The point was made that poor properties can yield non-deterministic output/random failures - although this could be considered the exact point of the testing. Either property (requirement?) is incorrect or test is incorrect.

_There still seems to be the issue that a function could be wrong but still satisfy the MR. Therefore, there still need to be oracle tests to "anchor" the function in reality. Thoughts?_

Consensus that testing strategies are complementary - PBT/MR is similar to getting the 'general' solution, oracle tests get the 'specific' solution. PBT/MR is probably good for catching some edge cases and exploring a large state space in complicated problems.

_An obstacle to implementing MT is whether there are any good tools available. Having only gotten the author's account of the case studies, it seems like metamorphic tests were implemented manually usually using just two inputs. There is a suggestion that existing PBT tools could be adapted. Any feeling for how difficult that might be? How realistic it is to use MT in the near future?_

Concluded that we can probably get quite a long way by just adding in sensible oracle tests when someone identifies an issue, prior to correcting it. If unit tests are well defined and inputs reasonably controlled, don't need to explore large solution space. However the point was also made that we could do quite a small amount of PBT/metamorphic testing and get a lot of the benefit (e.g. test using two inputs only) - kind of a Pareto on testing effort. Might be a useful way of thinking when doing lighter testing for other developers vs users.

Interesting point was made about how to determine the efficiency of PBT/MT - what is the metric which says how much of the code is tested? Coverage probably does not suffice. Perhaps coverage of the full set of properties (requirements?) defined for the problem. 