<style>
code:not(.hljs) {
color: #c9c;
}
.reveal-viewport li {
font-size: 32px
}
</style>
# Property testing with proptest
Slides: <https://hackmd.io/@tzemanovic/B12jdEk8p#/>
By: Tomáš Zemanovič
----
## Contents
<iframe src="https://giphy.com/embed/b4173kiWszeQnudzmS" width="240" height="195" frameBorder="0" class="giphy-embed" allowFullScreen style="float: right"></iframe>
- [What is property testing?](#What-is-property-testing)
- [Why do I property test?](#Why-do-I-property-test)
- [How to use Proptest effectively](#How-to-use-Proptest-effectively)
- [What is Proptest-state-machine](#What-is-Proptest-state-machine)
---
## What is property testing?
<iframe src="https://giphy.com/embed/XjlNyeZp5lDri" width="240" height="184" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>
- Searching for counter examples (invalidates a specified property)
- Difference from fuzz testing?
- Testing with deliberately valid (or invalid) inputs (composable generators)
- Successfully deployed in the industry applications that require high stability
----
### How does it work?
- Tests with random inputs
- generated via seeded randomness
- seed can reproduce a case
- On a discovered failure attempt to “shrink” the inputs to a minimal, reproducible example
- Proptest persists the seeds of failures (in text file) to perform regression testing (always ran before randomized tests)
---
## Why do I property test?
<iframe src="https://giphy.com/embed/eobOw9UZS8IqAorGso" width="240" height="240" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>
- <3 property testing from other languages (mostly FP)
- Practical for finding issues you might miss with unit testing
- Can help in reproducing a reported issue when it's not clear how to reproduce it
- Every CI run potentially covers new cases
- Fast feedback loop (no REPL)
----
### Proptest crate
- https://crates.io/crates/proptest
- thanks to the original authors, maintainer team and other contributors!
- for existing code mainly maintenance mode (trying to minimize breaking changes)
- accepts new features
----
### An Example
```rust=
proptest! {
#[test]
fn hello_world(input in vec(any::<u64>(), 0..100)) {
let mut reversed = input.clone();
// should be reversed twice
reversed.reverse();
assert_eq!(input, reversed);
}
}
```
```shell
Test failed: assertion `left == right` failed
left: [1, 0]
right: [0, 1].
minimal failing input: input = [
0,
1,
]
```
---
## How to use Proptest effectively
A quick start guide
<br/>
<iframe src="https://giphy.com/embed/6HypNJJjcfnZ1bzWDs" width="240" height="240" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>
----
### Getting started
- Completement don't replace unit tests as they:
- can be faster to grok and debug
- ensure that specific cases are always covered
- Don't randomize inputs that cannot affect execution path
- Think about cases you want to cover when writing generators
----
### How to get started
#### 1/9 - imports & boilerplate
- Add as a dev-dependency
```shell
cargo add proptest --dev
```
- Boilerplate setup
```rust [1-2,6,7,9-11|3-5,8,12|-]
#![cfg(test)]
mod test {
use proptest::prelude::*;
proptest! {
#[test]
fn my_test(
x in any::<u64>()
) {
assert!(x >= u64::MIN)
}
}
}
```
----
#### 2/9 - simple strategies (generators)
```rust
// Always `9_000`
let _const = Just(9_000); // Strategy<Value = i32>
// Same as `0..=u64::MAX`
let _arb_u64 = any::<u64>(); // Strategy<Value = u64>
let range = 0..10_usize; // Strategy<Value = usize>
// combine tuples (up to 12)
(range, any::<u32>()) // Strategy<Value = (usize, u32)>
```
----
#### 3/9 - traforming and combining strategies
```rust
use proptest::collection::vec;
let even = (0..=100_usize).prop_map(|x| x * 2);
let even_len_chars = even
.prop_flat_map(|x| vec(any::<char>(), x));
```
----
#### 4/9 - building strategies with `prop_compose!`
```rust
use proptest::collection::vec;
prop_compose! {
fn even_len_chars() // regular args
(x in even()) // strategies only
(chars in vec(any::<char>(), x))
-> Vec<char> { chars }
}
fn even() -> impl Strategy<Value = usize> {
(0..100_usize).prop_map(|x| x * 2)
}
```
----
#### 5/9 - methods vs. macro
```rust [1,8,12|2-4,9-11|5-6,13-14|-]
fn no_macro() -> impl Strategy<Value = String> {
even().prop_flat_map(|num| {
(vec(any::<char>(), num), "[a-z]{3,10}").prop_map(
|(chars, snd)| {
let fst: String = chars.into_iter().collect();
format!("{fst},{snd}") })})}
prop_compose! { fn with_macro() // <- regular args
(num in even()) // <- ↓ Composed strategies
(chars in vec(any::<char>(), num),
snd in "[a-z]{3,10}")
-> String {
let fst: String = chars.into_iter().collect();
format!("{fst},{snd}") }}
```
----
#### 6/9 - combining strategies with `prop_oneof!`
```rust
#[derive(Debug)]
enum Choice {
EvenNum(usize),
EvenLenChars(Vec<char>),
}
// The order matters - shrinks toward the first item
let choice = prop_oneof![
even.prop_map(Choice::EvenNum),
even_len_chars.prop_map(Choice::EvenLenChars),
];
// alternatively with `.prop_union()` and `.or()`
```
----
#### 7/9 - ensuring corner cases with `prop_oneof!`
- the default randomess distribution is uniform
```rust
let choice = prop_oneof![
// Number on the left is a weight -
// how likely this case is generated
1 => Just(0_u64),
1 => Just(u64::MAX),
1 => Just(u64::MAX - 1),
// 13 times more likely than other cases
13 => 1..u64::MAX - 1,
];
```
----
#### 8/9 - custom config
- in code
```rust
proptest! {
#![proptest_config(ProptestConfig {
cases: 10,
.. ProptestConfig::default()
})]
#[test]
fn slow_test(...) { ... }
}
```
- env vars (overrides)
```shell
PROPTEST_CASES=10 cargo test slow_test
```
----
#### 9/9 - avoid writing test inside decl. macro
```rust
proptest! {
#[test]
fn my_test(
x in 0..100_u64
) {
my_test_aux(x)
}
}
fn my_test_aux(x: u64) {
assert!(x < 100);
}
```
----
### Not covered
- [proptest-derive crate](https://crates.io/crates/proptest-derive) for deriving strategies
- [test-strategy crate](https://crates.io/crates/test-strategy) for alternative proc. macros
----
### Gotchas
- Deterministic reproduction
- `HashMap/Set` with default hasher may break deterministic reproduction
- Lots of output - read from the end
- Code changes may affect execution path of persisted regression seeds. If something is important ensure it's covered or make a unit test.
---
## What is Proptest-state-machine
<iframe src="https://giphy.com/embed/8lUDy1HTU9vRm" width="360" height="207" frameBorder="0" class="giphy-embed" allowFullScreen></iframe>
- abstract (reference) state machine testing
- useful for testing effectful code (e.g. mutable API, I/O, networking)
- based on eqc_statem from erlang
----
### How does Proptest-state-machine work?
- model the SUT in terms of an abstract state and transitions on this state
- compose strategy to generate transitions
- apply the transitions
- optionally constrain possible transitions with pre-conditions on the state
- test the state machine against the SUT
- assert transition's post-conditions after applying them
- global invariants (after every transition)
----
### How to use Proptest-state-machine? 1/7
- model the SUT in terms of an abstract state and transitions on this state
```rust
enum Transition {
...
}
impl ReferenceStateMachine for MyTest {
type State = Self;
type Transition = Transition;
fn init_state() -> BoxedStrategy<Value = Self>
...
```
----
### How to use Proptest-state-machine? 2/7
- compose strategy to generate transitions
```rust=
fn transitions(state: &Self::State)
-> BoxedStrategy<Self::Transition> {
// usually it's a choice
prop_oneof![
...
]
}
```
----
### How to use Proptest-state-machine? 3/7
- apply the transitions
```rust
fn apply(
mut state: Self::State,
transition: &Self::Transition
) -> Self::State
```
----
### How to use Proptest-state-machine? 4/7
- optionally constrain possible transitions with pre-conditions depending on the current state
```rust
fn preconditions(
state: &Self::State,
transition: &Self::Transition
) -> bool
```
----
### How to use Proptest-state-machine? 5/7
- test the state machine against the SUT
```rust
impl StateMachineTest for MyTest {
type SystemUnderTest = ...;
type Reference = Self;
fn init_test(ref_state: &Self::Reference::State)
-> Self::SystemUnderTest
...
```
----
### How to use Proptest-state-machine? 6/7
- test the state machine against the SUT
```rust
// assert transition's post-conditions after applying them
fn apply(
mut state: Self::SystemUnderTest,
ref_state: &Self::Reference::State,
transition: Transition
) -> Self::SystemUnderTest
// global invariants (after every transition)
fn check_invariants(
state: &Self::SystemUnderTest,
ref_state: &Self::Reference::State)
```
----
### How to use Proptest-state-machine? 7/7
- declare the test
- a strategy for initial state and up to 20 transitions
- sequential execution
```rust
prop_state_machine! {
#[test]
fn name_of_the_test(sequential 1..=20 => MyTest);
}
```
----
### Example output
- from `examples/state_machine_heap.rs`
```shell
Test failed: Popped value -1, which was less than 0 still
in the heap.
minimal failing input: (initial_state, transitions) = (
[],
[
Push(0),
Push(0),
Push(-1),
Pop,
Pop,
],
)
```
---
### Wrap up
- Start with small tests
- Try to generalize existing unit tests
- Much more details in [Proptest book](https://proptest-rs.github.io/proptest/)
----
### Thank you!
GitHub/Gitlab `@tzemanovic`
Mastodon `@tzemanovic@fosstodon.org`
https://tzemanovic.gitlab.io/about/
{"title":"Talk slides template","description":"View the slide with \"Slide Mode\".","contributors":"[{\"id\":\"94d404f4-5c56-4429-a21f-cd7b188c9e07\",\"add\":37340,\"del\":42462}]"}