# Testing - theory & good practices
This article gathers some theory and best practices I have gathered during my career. These practises are no dogma, but are here for a good reason.
If something's not clear, something's wrong or missing - please let me know. I would like this guide to be alive and up-to-date, not written in stone, so anytime we find better approaches/libraries/etc. - let me know, I'd be happy to update.
Enjoy!
## Why should I write tests?
_My code works, I've checked it manually_ - Paulo Coelho
1. **Reproducible results** - since it's written, there is no place for "manual" mistake (i.e. wrong order of actions, typo in IP address, etc.)
2. **Time savings** - once written, tests can be ran over and over again with little to no time effort (especially compared to manual testing)
3. **Up-to-date software documentation** - documentation that will not get out of sync with the implementation (otherwise the tests will fail)
## Test levels
1. **Unit tests** - test small "unit" or "component" of code in isolation; the "unit" might be single class, method, function etc.
2. **Integration tests** - test how few components are working together
3. **End-to-End tests** (_aka_ **E2E tests**) - full system tests, often used in aceptance testing; E2E tests should use system in a way the end user would (providing input through same channels, examing results through same outputs)
## Unit tests properties
* **Fast** - hundreds per second
* **Isolates** - if test fails the reason is known
* **Repeatable** - can be ran in any order at any time and still produce the same results
* **Self-validating** - gives simple anwser: fails or passed
* **Timely** - available quickly (ideally - before the production code)
## Good practicies
### Let them be runnable
Unit test should always be self-contained and independent from the outer world. It means other developers should be able to clone your repo and run unit tests without any manual adjustments. If someone need to set some specific environmental variables, run some service, write a specific config file - these are not unit tests, but rather some higher level tests, i.e. integration.
Ofcourse, there is a place for integration tests, which may require some manual setup, but in such case, all the steps and/or values should be **clearly** defined in `README.md`.
There is no good in tests that cannot be easily run. Because they don't get run. And they get out-of-sync with the code, rot and slowly die.
### Good names matter!
Test should have very explicit names, so they can be a living documentation and help reader understand the production code. Consider this simple function:
```python
def is_prime(number):
"""Return True if *number* is prime."""
if number < 2:
return False
for element in range(2, number):
if number % element == 0:
return False
return True
```
* Bad
```python
def test_is_prime():
assert is_prime(4) == False
```
* Good
```python
def test_that_4_is_not_prime():
assert not is_prime(4)
```
### Design of a test
In most cases, a unit test (or any other test) constist of three parts:
* _given_ - where we state preconditions
* _when_ - where the actual action happens
* _then_ - where we assert our expectation on results of the action
For instance:
```python
def should_sell_alcohol(age: int) -> bool:
return age >= 18
def test_should_sell_when_user_is_18():
# given
user_age = 18
# when
result = should_sell_alcohol(user_age)
# then
assert result is True
```
Often, expecially in simple cases, we can "merge" one or more sections, i.e
```python
def test_should_sell_when_user_is_18():
# given
user_age = 18
# when/then
assert should_sell_alcohol(user_age) is True
```
or even:
```python
def test_should_sell_when_user_is_18():
# given/when/then
assert should_sell_alcohol(18) is True
```
**NOTE**: the _given/when/then_ comments used in examples above are just to emphasise the existence of distinct sections; it's not needed to put them in real tests - separation by the new line should suffice.
### One test - one scenario
As tests can serve as a documentation of the implementiation, we should make sure that just one scanrio is tested in a singular test.
* Bad
```python
def test_split():
s = 'hello world'
assert s.split() == ["hello", "world"]
with pytest.raises(TypeError):
s.split(2)
```
* Good
```python
def test_that_split_separtes_string_on_white_chars():
s = 'hello world'
assert s.split() == ["hello", "world"]
def test_that_split_raises_type_error_when_called_with_wrong_arguments():
with pytest.raises(TypeError):
s.split(2)
```
#### parametric tests
To reduce code duplication, use parametric tests, i.e.:
```python
@pytest.mark.parametrize(
"test_input, expected",
[
("\xac\x1f\x00\x02", "172.31.0.2"),
("\x3e\xb3\x8c\xc5", "62.179.140.197"),
("\x3e\xb3\x8c\x85", "62.179.140.133"),
],
)
def test_ip_hex_to_str(test_input, expected):
assert helpers.ip_hex_to_str(test_input) == expected
```
### Using mocks
Mocks in Python are Test Doubles used where, for some reason, we cannot use the real object. It might be due to costly intialization, necesity to connect to a DB or remote host, or simly because the object is not implemented yet.
In the examples below it is assumed, that `m` and/or `mm` is defined as follows:
```python
from unittest.mock import Mock, MagicMock
m = Mock()
mm = MagicMock()
```
#### `return value`
Mocks can be progammed to return value upon call.
```python
m.return_value = 44
assert m() == 44
```
#### `side_effect`
Mocks can also be programmed to have a side effect, upon call. It comes in three flaovurs:
1. Iterable
```python
m.side_effect = [1, 2, 3]
assert m() == 1
assert m() == 2
assert m() == 3
```
When iterator is exhausted, `StopIteration` is raised.
```python
m() # raises StopIteration
```
2. Exception
```python
m.side_effect = ValueError #or ValueError()
m() # raises ValueError
```
It comes handy when we want to test error handling.
3. Calling arbitrary function
```python
def foo(x):
return x**2
m.side_effect = foo
assert m(7) == 49
assert m(13) == 169
```
or
```python
m.side_effect = lambda name: f"Mr. {name.capitalize()}"
assert m("janusz") == "Mr. Janusz"
assert m("ANDRZEJ") == "Mr Andrzej"
#### assertions on mock calls
Mocks provide set of methods for asserting that certain calls has/hasn't happened.
```python
def fetch_items(session, *ids):
url = "my.server/items"
if ids:
url += "?" + ",".join(f"q={id_}" for id_ in ids)
return session.get(url)
def test_that_fetch_items_gets_data_from_correct_url():
m = Mock()
m.get.returned_value = returned_items
fetch_items(m, 1, 2)
m.get.assert_called_once_with("my.server/items?q=1,q=2")
```
#### using `create_autospec()`
Mocks normally create attributes on demand (returning another `Mock`). But often we want to make sure that no new attribute is created. Consider:
1. We have a function that recognies user type
```python
class User(BaseModel):
name: str
type: str
class UserManager:
def grant_privileges(username: str)
...
...
def log_in(user: User, user_manager: UserManager):
... # some code
if user.type == 'admin':
user_manager.grant_privileges(user.name)
```
2. We test it
```python
def test_that_normal_user_doesnt_get_privileges():
normal_user = User(name="John", type="normal")
mocked_manager = Mock()
log_in(normal_user, mocked_manager)
mocked_manager.grant_privileges.assert_not_called()
```
3. We change the API of colaborator
```python
...
class UserManager:
def grant_root_privileges(username: str)
...
...
```
4. Simple mock gives us false positive - the test will still pass
5. With `create_autospec` we make sure that calling non existing methods will break the test:
```python
def test_that_normal_user_doesnt_get_privileges():
normal_user = User(name="John", type="normal")
mocked_manager = create_autospec(UserManager)
log_in(normal_user, mocked_manager)
mocked_manager.grant_privileges.assert_not_called() # here is still the old name
```
6. Now we are notfied by failing test:
```python
AttributeError: Mock object has no attribute 'grant_privileges'
```
7. We can adjust
```python
def test_that_normal_user_doesnt_get_privileges():
normal_user = User(name="John", type="normal")
mocked_manager = create_autospec(UserManager)
log_in(normal_user, mocked_manager)
mocked_manager.grant_root_privileges.assert_not_called() # with updated name, test is passing again
```
### Using `assertpy`
With `assertpy` we can make readable, explicit assertions (especially handy with `dict`s, `list`s, etc.), i.e
```python
@pytest.mark.asyncio
@freeze_time("2020.03.02 13:14:15")
async def test_produce_full_metrics_for_single_ip(snmp_client_mock, return_values_for_snmp_query):
ip = "84.10.1.230"
community = "top-secret"
program_mock(snmp_client_mock, return_values=return_values_for_snmp_query)
timestamp = int(time.time())
results = []
async for line in produce_full_metrics(snmp_client_mock, ip, community):
results.append(line)
assert_that(snmp_client_mock._mocked_query.call_count).is_equal_to(4)
assert_that(results).contains_only(
f"{timestamp};ip=84.10.1.230,hostname=pl-waw02a-tm1,interface=eth0,number=1,measurement=tmnxPortNetStats;droppedPacketsIngress=10,delayEgress=50",
f"{timestamp};ip=84.10.1.230,hostname=pl-waw02a-tm1,interface=eth0,number=2,measurement=tmnxPortNetStats;droppedPacketsIngress=0,delayEgress=5",
f"{timestamp};ip=84.10.1.230,hostname=pl-waw02a-tm1,interface=wlan0,number=1,measurement=tmnxPortNetStats;droppedPacketsIngress=5,delayEgress=15",
)
```
### Testing `datetime` or `time` related code
To ensure repeatable results, please use `freezegun.freeze_time()` decorator / context manager (example as above).
## References
* [Test levels by Google's engineers](https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html)
* [FIRST - Unit Tests properties](http://agileinaflash.blogspot.com/2009/02/first.html)
* [Test Double](https://martinfowler.com/bliki/TestDouble.html)
* [Cheatsheet for mocking](https://medium.com/@yeraydiazdiaz/what-the-mock-cheatsheet-mocking-in-python-6a71db997832)
* [Mock call assertions](https://docs.python.org/3.7/library/unittest.mock.html#unittest.mock.Mock.assert_called)
* [assertpy package](https://github.com/ActivisionGameScience/assertpy#the-api)
Sign in with Wallet
Connect another wallet