## Testing Pulp Using the New Pytest Fixtures
slides: https://hackmd.io/@pulp/HkMc_8XpF
presenter: Brian Bouterse
---
## Proposal
Have all new tests writen using pytest and its features
---
## Basic Stuff
* Have your tests discoverable by pytest
* Make a home for your fixtures
* Use fixtures to generate your bindings objects
* Use fixtures instead of setup/teardown
* Use assert instead of unittest.assertEqual
---
## Fancy Stuff
* Use local fixture server instead of fixtures.pulpproject.org
* Make assertions on server calls if you want
* Mark tests as `parallel`
---
## Why are we doing this?
[Background video from a pulpcon talk](https://youtu.be/4xi9JzUo7XI)
1. Keep fixtures with the commits that use them (one repo)
* version fixtures with code
* fewer commits
* cherry-pickable
2. Run locally, i.e. not use fixture.pulpproject.org
3. Make assertions from the webserver pulp is downloading from
4. Have less and better test code
---
## Thanks
* mdellweg for lots of collaboration
* pytest, aiohttp, and trustme communities
---
## Language issues
Fixtures here have two meanings
1. A pytest fixture - helps create conditions for test assertion
2. A data fixture - data served to Pulp downloaders via a webserver
Pleae ask if not clear which context.
---
## naming your tests for pytests
* prefix functions or methods with `test_`. [example](https://github.com/pulp/pulpcore/blob/main/pulpcore/tests/functional/api/using_plugin/test_sync.py)
* prefix classes as with `Test`.
* [Official pytest test collection docs](https://docs.pytest.org/en/6.2.x/goodpractices.html#test-discovery)
---
# Adding fixtures
1. Add a tests/conftest.py [see example](https://github.com/pulp/pulpcore/tree/main/pulpcore/tests)
2. You can organize via subimports
---
## Bindings as fixtures
1. Use pulpcore fixtures [from pulp-smash](https://github.com/pulp/pulp-smash/blob/47aaf2e28b575447eefc5c3e7c388dc971ff5f8c/pulp_smash/pulp3/pytest_plugin/__init__.py#L139-L147).
2. Use session scope (probably)
3. Make fixtures for your plugin's bindings [e.g. pulp_file's](https://github.com/pulp/pulpcore/blob/main/pulpcore/tests/conftest_pulp_file.py#L23-L45).
---
## Fixtures with cleanup
1. Use [`gen_object_with_cleanup`](https://github.com/pulp/pulp-smash/blob/47aaf2e28b575447eefc5c3e7c388dc971ff5f8c/pulp_smash/pulp3/pytest_plugin/__init__.py#L235-L252).
2. For example [pulp_file making a remote](https://github.com/pulp/pulpcore/blob/main/pulpcore/tests/conftest_pulp_file.py#L71-L77).
3. Used in pulp_file tests [like this](https://github.com/pulp/pulpcore/blob/70b7f31a7806f3701c3f746a63508a237c26c9a2/pulpcore/tests/functional/api/using_plugin/test_sync.py#L29-L43).
4. This is the [well-established pytest factory pattern](https://docs.pytest.org/en/latest/how-to/fixtures.html#factories-as-fixtures).
---
## Orphan cleanup fixtures
1. Use [`delete_orphans_pre`](https://github.com/pulp/pulp-smash/blob/47aaf2e28b575447eefc5c3e7c388dc971ff5f8c/pulp_smash/pulp3/pytest_plugin/__init__.py#L153-L156).
2. See [this example](https://github.com/pulp/pulpcore/blob/70b7f31a7806f3701c3f746a63508a237c26c9a2/pulpcore/tests/functional/api/using_plugin/test_sync.py#L29-L43).
---
## Use `assert` FTW
1. Stop using unittest.assertEqual, etc
2. Instead use `assert` [example](https://github.com/pulp/pulpcore/blob/70b7f31a7806f3701c3f746a63508a237c26c9a2/pulpcore/tests/functional/api/using_plugin/test_sync.py#L184-L187)
---
## Use local fixture server
1. Add fixtures to your repo [see example here](https://github.com/pulp/pulpcore/tree/85d366162602f28bab5562d75cdeb87f5d04cb20/pulpcore/tests/fixtures).
2. Create a `*_server_root`. [pulp_file example](https://github.com/pulp/pulpcore/blob/85d366162602f28bab5562d75cdeb87f5d04cb20/pulpcore/tests/conftest_pulp_file.py#L48-L50)
3. Make the fixture server. [pulp_file example](https://github.com/pulp/pulpcore/blob/85d366162602f28bab5562d75cdeb87f5d04cb20/pulpcore/tests/conftest_pulp_file.py#L65-L67)
4. Make a fixture to generate remotes. [pulp_file example](https://github.com/pulp/pulpcore/blob/85d366162602f28bab5562d75cdeb87f5d04cb20/pulpcore/tests/conftest_pulp_file.py#L70-L77).
5. Use the fixture server. [pulp_file example](https://github.com/pulp/pulpcore/blob/70b7f31a7806f3701c3f746a63508a237c26c9a2/pulpcore/tests/functional/api/using_plugin/test_sync.py#L29-L43).
---
## `gen_fixture_server`
1. Starts a threaded server
2. Function-scoped
3. Shuts down upon test-shutdown
4. fixture is actually [this data object](https://github.com/pulp/pulp-smash/blob/47aaf2e28b575447eefc5c3e7c388dc971ff5f8c/pulp_smash/pulp3/pytest_plugin/__init__.py#L56-L82)
5. Allows you to make assertions, like [this example](https://github.com/pulp/pulpcore/blob/70b7f31a7806f3701c3f746a63508a237c26c9a2/pulpcore/tests/functional/api/using_plugin/test_sync.py#L163).
---
## `aiohttp_fixtures_origin` setting
1. A pulp-smash setting, defaults to 127.0.0.1 [set here](https://github.com/pulp/pulp-smash/blob/677f50e67dfbf836034ea970c70e4bf6b4954fb9/pulp_smash/config.py#L562).
2. The CI overrides it for GHA [here](https://github.com/pulp/pulpcore/blob/85d366162602f28bab5562d75cdeb87f5d04cb20/template_config.yml#L11).
3. Used by `gen_fixture_server` [here](https://github.com/pulp/pulp-smash/blob/47aaf2e28b575447eefc5c3e7c388dc971ff5f8c/pulp_smash/pulp3/pytest_plugin/__init__.py#L103).
---
## http, TLS, mutual auth oh my!
* http only server [example](https://github.com/pulp/pulpcore/blob/main/pulpcore/tests/conftest_pulp_file.py#L65-L67).
* https only server [example](https://github.com/pulp/pulpcore/blob/main/pulpcore/tests/conftest_pulp_file.py#L60-L62). <-- use this one
* https only requiring client cert auth [example](https://github.com/pulp/pulpcore/blob/main/pulpcore/tests/conftest_pulp_file.py#L60-L62).
* Allows pulpcore to assert [like never before](https://github.com/pulp/pulpcore/blob/70b7f31a7806f3701c3f746a63508a237c26c9a2/pulpcore/tests/functional/api/using_plugin/test_sync.py#L29-L118).
---
## Running tests on CI in parallel
See [this discourse](https://discourse.pulpproject.org/t/parallel-test-runs-added-to-ci/320).
---
## Known gotchas
Checking in large binaries of fixture binary data is not good.
Two solutions:
* Make data fixtures generated at runtime
* Use git-LFS for side-lookup ref storage
---
## My Next steps
1. Allow fixture data to be generated dynamically for pulp_file
* Avoids checking in large binaries in source tree
* Allows each fixture to be unique to enable fullly parallel test running
2. Port all pulpcore tests to pytest + aiohttp data fixtures
3. Add proxy tests
4. Add additional tests for downloaders, e.g. basic auth
{"metaMigratedAt":"2023-06-16T18:07:48.411Z","metaMigratedFrom":"YAML","title":"Testing Pulp Using the New Pytest Fixtures","breaks":true,"description":"Learn how to use the new pytest fixtures","contributors":"[{\"id\":\"dc40d541-bddd-4823-82c8-6e5276fe233a\",\"add\":9410,\"del\":2564}]"}