# [nteract] testbook - unit test framework for Jupyter notebooks

## Abstract

[`testbook`](https://github.com/nteract/testbook) is a pytest/unittest framework extension to unit-test Jupyter notebooks. Testing Jupyter notebooks allows data scientists and developers to maintain, reproduce and refactor code with ease.

Previous attempts at unit-testing notebooks involved writing the tests in the notebook itself. However this library, `testbook` will allow for unit-tests to be run against notebooks in separate test files, hence treating `.ipynb` files as `.py` files.

`testbook` will use the modern decorator and pytest fixture pattern which will fit more naturally into common unit testing workflows.

---

## Related Work

1. [Treon](https://github.com/ReviewNB/treon) is a test framework for Jupyter notebooks. It gathers and executes unit tests
written in Jupyter Notebook cells.

2. [nbval](https://github.com/computationalmodelling/nbval) is a py.test plugin which checks whether execution of the stored inputs match
the stored outputs of the .ipynb file.

---

## Why this project?

I firmly believe in the goal of this project and I can envision `testbook` becoming an essential tool in the research and development cycle for data scientists.

I have previously been involved in a project that led me to set up an entire testing suite for a library called [jovian-py][jovian]. Due to this, the team at Jovian are able to confidently develop, test and release their library. This experience is relevant and will help me to design the `testbook` testing API better for future users.

This project will be an exciting and challenging experience for me as I will get to work on designing and developing the library from scratch.

---

## Technical Details

<!--
[nbclient](http://nbclient.readthedocs.io/) will be used to interact with the Jupyter notebook and kernel.
-->

### Technical features of the library:

- Integrates with pytest
- Enable pre-running of specific cells before test
- Inject code snippets before and after test executes
- Inject Python mocks
- Display code-coverage for notebooks

### Implementation details

- `testbook` will support the pytest fixture pattern and the decorator pattern

#### pytest fixture pattern

from testbook import notebook_loader

def test_foo(notebook_loader):
  with notebook_loader('path/to/notebook') as nb:
      # Execute cells
      nb.execute_cell([2,3])

      # Assert that output is as expected
      assert nb.cell_output_text(3) == 'foo\n'

#### decorator pattern

from testbook import notebook

@notebook('path/to/notebook')
def test_foo(nb):
    # Execute cells
    nb.execute_cell([2,3])

    # Assert that output is as expected
    assert nb.cell_output_text(3) == 'foo\n'

- While the pytest fixture pattern is fairly straightforward to implement, the decorator
pattern will be slightly challenging when testing with pytest. The parameter passed in
with the decorator `testbook.notebook` is misinterpreted as a pytest fixture, hence we
will need to use built-in libraries like `functools` and `inspect` to [fix][2] this issue.

- We will need to wrap exceptions that arise from `nbclient`, and promote them to built-in
Python exceptions. This will make it possible to assert that a cell throws an `Exception`.
For example,

import pytest
from testbook import notebook_loader

def test_foo_throws_exception(notebook_loader):
  with notebook_loader('path/to/notebook') as nb:
      # Execute cells
      nb.execute_cell([2,3])

      with pytest.raises(ZeroDivisionError):
          nb.execute_cell(4)

<!-- - The latest development version of `nbclient` uses `asyncio` to make the Jupyter Kernel [asynchronously execute cells][async-cells]. This implementation is different from the current 0.1.0 release of `nbclient`. This will break the `testbook` APIs when the next release of `nbclient` occurs. -->

- Through the development of `testbook` with `nbclient` as the underlying library, many interface improvement pull requests will be submitted to `nbclient` to better facilitate patterns encountered in `testbook`.

## Schedule of Deliverables

#### Deliverables

1. Integrates with pytest (required)
2. Enable pre-running of specific cells before test (required)
3. Handle exceptions and promote to built-in Python exceptions (required)
4. Inject code snippets before and after test executes (required)
5. Inject Python mocks (required)
6. Documentation along with suitable examples (required)
7. Test suite (required)
8. Display code-coverage for notebooks (optional)

### Community Bonding Period

- #### May 4, 2020 - June 1, 2020
- Setup basic project workflows such as automated tests and coverage reports.
- Familiarize myself with the development version and release version (0.1.0) of the `nbclient` library.
- Go through `asyncio` documentation.
- Interact with the community and conduct a survey of how users would want a test framework for Jupyter notebooks to look like.

### Phase 1

- #### June 1 - June 15

- Enable pre-running of specific cells before test
- Handle exceptions and promote to built-in Python exceptions

- #### June 15 - July 3
- Support decorator pattern
- Inject code snippets before and after test executes
- Phase 1 evaluation submission

<div style="page-break-after: always;"></div>

### Phase 2

- #### July 4 - July 18

- Support decorator pattern continued
- Inject Python mocks

- #### July 19 - July 31

- Inject Python mocks continued
- Setup test suite - though tests will be added with every pull request, during this
time, I will make sure that all functionality has been tested.
- Setup code coverage for notebooks (optional deliverable)
- Phase 2 evaluation submission

- #### August 1 - August 14

- Setup documentation
- Continue to setup test suite
- Work on bringing code coverage up to 100%

- #### August 15 - August 31
- Continue to setup documentation
- Continue to work on bringing code coverage up to 100%
- Find early adopters to use the library and provide critical feedback

### Final Week

<!--
At this stage you should finish up your project. At this stage you should make
sure that you have code submitted to your organization. Our criteria to mark
your project as a success is to submit code before the end of GSoC. -->

- #### August 24 - August 31
- Try and release the library to PyPI
- Publish final blog
- Submit project summary and final evaluation

---

## Development Experience

- #### [jovian-py][jovian] - Share Jupyter notebooks online instantly with a single command

Wrote the entire unit-testing suite and brought coverage up to 100%.

[Letter of recommendation][google-drive-link] from co-founder and CEO, Jovian.ml.

| Pull Request | Link |
| ----------------------------------------------- | ---------------------------------------------- |
| Unit Testing | https://github.com/JovianML/jovian-py/pull/92 |
| Unit Testing 2 | https://github.com/JovianML/jovian-py/pull/96 |
| Unit Testing 3 | https://github.com/JovianML/jovian-py/pull/108 |
| Unit Testing Final and setup Codecov GH actions | https://github.com/JovianML/jovian-py/pull/125 |

- #### [numba] - NumPy aware dynamic Python compiler using LLVM

One bug fix and several improvements to `numba.typed.TypedList`

| Pull Request | Link |
| ----------------------------------------------------------------- | ---------------------------------------- |
| refine typed-list on unicode input to extend | https://github.com/numba/numba/pull/5295 |
| Added property `dtype` to `numba.typed.List` | https://github.com/numba/numba/pull/5235 |
| Added support for `np.asarray` to be used with `numba.typed.List` | https://github.com/numba/numba/pull/5231 |
| Fix `.strip()` to strip all whitespace characters | https://github.com/numba/numba/pull/5213 |

- #### [pandas] - Python Data Analysis Library

Wrote some unit tests for Pandas

| Pull Request | Link |
| -------------------------------------------------- | ----------------------------------------------- |
| TST: added test for df.loc modify datetime columns | https://github.com/pandas-dev/pandas/pull/28964 |
| TST: add test_series_any_timedelta for GH17667 | https://github.com/pandas-dev/pandas/pull/28942 |

---

## Personal Information

| | |
| --------- | --------------------------------------------------------- |
| Name | Rohit |
| Email | [e-mail] |
| Website | https://rohitsanjay.codes |
| LinkedIn | [Rohit Sanjay](https://www.linkedin.com/in/rohit-sanjay/) |
| Twitter | [Rohit Sanjay](https://twitter.com/imrohitsanj) |
| Phone | [phone] |
| Location | India 🇮🇳 |
| Time Zone | India (UTC +5:30) |

I'm a third year (junior) student of Electronics and Communication Engineering (minor in
Computational Mathematics) at Manipal Institute of Technology, Manipal, India.

I will be writing regular blog posts at [dev.to/rohitsanj](https://dev.to/rohitsanj)

<!-- ## Appendix -->

[google-drive-link]: https://drive.google.com/file/d/1u7XbLOGk_1sieogjRfqBeIJ5Xm8gAQIw/view?usp=sharing
[2]: https://github.com/nteract/testbook/issues/4#issuecomment-604096019
[pr]: https://github.com/jupyter/nbclient/pull/41
[async-cells]: https://github.com/jupyter/nbclient/commit/d4e09db6889e19883f2dd838636c7f6f4b8fe40f
[jovian]: https://github.com/JovianML/jovian-py
[numba]: https://github.com/numba/numba
[pandas]: https://github.com/pandas-dev/pandas