# Planning Meeting: Pytest plugins, Sphinx extensions
**Date:** Friday 1:00 - 2:00 PM
**Topics:** Pytest plugins, Sphinx extensions
**Issues:** [#10](https://github.com/scientific-python/summit-2023/issues/10)

### Attendees
- Juanita Gomez
- (async) Dan McCloy
- Jarrod Millman
- Ross Barnowski (rossbar)
- Brigitta Sipőcz
- Paul Ivanov
- Lars Grüter (@lagru)
- Matt Haberland (@mdhaber)
- Pey Lian Lim (@pllim)
- Tyler Reddy
- Stéfan van der Walt (@stefanv)

## Relevant links

- https://github.com/astropy/astropy-sphinx-theme/issues/29
- https://github.com/numpy/numpydoc/issues/419#issuecomment-1216506339
- https://github.com/tylerjereddy/pytest-regex
- https://github.com/astropy/pytest-doctestplus/issues/107
- https://github.com/scikit-image/scikit-image/pull/6896#issuecomment-1514968114

## Ideas from issues

**Tyler:** I haven't found a plugin that allows regex usage in i.e., pytest -k "test_name". That could be pretty useful, though not really exclusively useful to scientific projects.

This helps you avoid running:

pytest -k "test_name_1"
pytest -k "test_name_2"

etc.
if the tests are spread over multiple modules but are affected by changes under iteration.

**Pey Lian:** I am particularly interested to see if any of these can be moved upstream from Astropy. Or if Astropy could retire some of them and use something else that is better maintained and more mainstream.

pytest

https://github.com/astropy/pytest-arraydiff
https://github.com/astropy/pytest-doctestplus
https://github.com/astropy/pytest-filter-subpackage
https://github.com/astropy/pytest-remotedata
Sphinx

https://github.com/astropy/astropy-sphinx-theme
https://github.com/astropy/sphinx-automodapi

**Daniel McCloy:** MNE-Python has a couple small sphinx extensions that might be of general interest:

- this one turns :gh:\`1234\` into a link to the issue/PR
- this one turns :newcontrib:\`Name Surname\` into a bold-formatted link and prepends "new contributor" to it
- this one joins a number and its unit with a non-breaking space so they aren't separated across linebreaks.

**Tyler:** FYI, I drafted a pytest-regex plugin in 2-3 hours or so, which already seems considerably more capable than the conventional -k. Of course there are no tests for the plugin itself yet, but I added examples on how to use it in the README over there, and it seems to do the right thing so far...

Anyway, just getting rid of the initial inertia on that ahead of the summit. FWIW, it sounds like the main pytest project may not be opposed to directly supporting it from one or two issues I scanned through, but prototying quickly externally at the summit that way may not hurt anyway, and perhaps they'd handle the absorption if they really wanted.

The other thing is that folks may want features that the main project wouldn't, so defining that first probably makes sense with a test in a very small external plugin codebase to start.

**Sebastian:** One thing related that I would be interested in also is infrastructure similar to/tagged on top of pytest-doctestplus to modify docs based on actual output. (That is to enable mass changes, I suppose whatever that should be should have something like a new_str = func(old_str, evaluated_str) to hook that in.)

(That is related to the fact we need to change how NumPy scalars print.)

**Daniel McCloy:** Another "neither pytest nor sphinx" thing that could be standardized/shared across projects is the website banner saying "warning this is an old version / the development version of this package, please switch to stable docs". The pydata-sphinx-theme has added an announcement banner that is configured in conf.py but it's not easy to get it to contextually appear/change messages depending on docs version (esp. when older versions of your docs were built with other themes / before the announcement banner was part of the theme).

@jarrodmillman describes the networkx approach here
MNE-Python uses this javascript that was originally adapted from sklearn but has diverged somwhat over the years
What are other folks doing? Is it worth trying to standardize/centralize this for the ecosystem?

*(Dan async): I'm traveling today and won't make the planning meeting. I'll expect to work on this JS version warning banner during the summit but feel free to suggest/assign me something else if you think it's higher priority. also happy to have a buddy to pair program the JS banner if someone is interested.*

-------------------------------------------------------

## Meeting notes

Brigitta:

- There is a need to move things like pytest-doctestplus upstream from astropy org to make it available to wider scientific Python community.
- Then maybe can replace some doctest stuff in numpy with pytest-doctestplus (https://github.com/astropy/pytest-doctestplus).
- Effort to move pytest-doctestplus upstream to pytest proper didn't work out (https://github.com/astropy/pytest-doctestplus/issues/100).

Jarrod: Moving things upstream does not mean ownership change; it is more about branding.

Stefan: Also a visibility issue, a cue that a tool is centrally used.

Jarrod:

- index/ registry of which of the scientific Python projects use what plugins.
- like Debian's "popularity contest"

pllim: Astropy infrastructure flowchart available at https://zenodo.org/record/7705958#.ZGfYu3bMKUk . If Scientific Python interested in some of them, Astropy will be happy to discuss upstreaming.

- Henry: [pytest-remotedata](https://github.com/astropy/pytest-remotedata) seems useful
- Brigitta: I'm happy to help with exploration of its adoption elsewhere, I use it all the time for astroquery and for pyvo. I'm already looking into its usage for scipy (pending I'm able to figure out to build scipy locally).
- Matt H: SciPy contributor quickstart: [Mac/Linux](https://scipy.github.io/devdocs/dev/dev_quickstart.html), [Windows](https://scipy.github.io/devdocs/building/index.html#building-from-source)

Henry: Has to maintain quality and archive the ones no longer useful/maintained.

- Henry mentioned something he uses, but has died. He would need to look it up. It was an opt-out plugin for pytest maybe? -> it was https://github.com/reece/pytest-optional-tests
- https://pypi.org/project/pytest-manual-marker/ (not the actual thing that died, but a newer version that does the same thing)
- Brigitta: markers work like this out of the box, e.g. see `bigdata` in https://github.com/astropy/astroquery/blob/main/setup.cfg
- Henry: No, they don't, I'm refering to opt-in, not opt-out. (Like the "slow" flag in pytest-astropy)

Ross: Page to advertise recommended tooling for such-and-such, to remove the problem of "tribal knowledge". But is it worth the payoff?

Jarrod, Brigitta: Maybe some tools are no longer necessary because the feature is now supported in pytest proper; so it is a matter of updating the tests than to use custom tools.

Henry recommends [Brian Okken's Python Testing with pytest](https://pragprog.com/titles/bopytest2/python-testing-with-pytest-second-edition/)
- [scikit-HEP testing guidelines](https://scikit-hep.org/developer/pytest)

Ross: data storage options, especially for tutorials
- Other orgs we can talk to to: ...

Tyler: ability to swap array-API backends -- making sure we can run tests on multiple backend architecture

pllim: On the sphinx side, I am also interested to revisit https://github.com/astropy/astropy/pull/14477 to adopt pydata-sphinx-theme in Astropy. Maybe it is nice to have theme experts next week near me.
- Stefan: Lars just helped switch skimage.