# CodeRefinery/ENCCS software testing workshop
## Links
- **Welcome!** Please open this HackMD and the course material!: https://hackmd.io/@coderefinery/testing-hackathon-2021/
- Course page:
https://coderefinery.github.io/2021-03-17-testing-hackathon/
- Workshop material: https://coderefinery.github.io/testing/
:::danger
Please:
- open HackMD (https://hackmd.io/@coderefinery/testing-hackathon-2021/)
- **rename yourself to "(n) Name"** in Zoom according to your chosen breakout room (Participant list→hover over your name)
- answer the icebreaker icebreaker icebreaker icebreaker icebreaker below.
:::
## Practical information to hackathon participants (March 24)
- We recommend that you sign up for the [CodeRefinery Zulipchat](https://coderefinery.zulipchat.com/) and join the "workshop-chat" stream, under which there's a "testing-hackathon" topic where we can chat after the workshop.
## Breakout room planning
Please choose a breakout room according to:
- which programming language you want to use for exercises
- whether you want to use GitHub or GitLab in the "automated testing" exercise
We will use the same breakout room for all the exercises in the workshop (until 12:00 CET).
If you're participating in a team and want to work with your team members during the exercises, make sure to write yourself into the same group. Please write your team name after your name (or registration number) as well.
We recommend max 6 people per group. If your team is bigger than 6 and want to be in the same room, please add lines in the room.
If you find rooms of your choice of language/tool are full, please add a new room room at the bottom with a new room number (21 or over) and combination of the language and tool in the same format as pre-defined rooms (e.g. "#### 21. Python-GitHub-5"). Please do not use other languages or tools than those used in pre-defined rooms.
If you're hesitant to write your name into an online document, please write your registration number instead (you can find it in your automatic registration reply email from support@coderefinery.org, e.g. "ref: #97" )
**1. Python-GitHub-1**
1. ref: #20
2. ref: #2
3. ref: #41, team AuBi
4. Tristan (not joined)
5. Inge (not joined)
6. Duke
7. Nadia (ref #40), team AuBi
**2. Python-GitHub-2**
1. ref: #22
2. Alessandro Marin
3. ref: #52
4. Mathias Bockwoldt
5. ref: #72
6. Arvid Bring
7. Ravindra Shinde
**3. Python-GitHub-3**
1. Shashank S. Harivyasi
2. Ilaria Carannante
3. Fabrizio Palumbo
4. Nkanyiso Mbatha
5. Johanna Frost Nylen
6. Martin Schappert
7. Diana (mentor)
**4. Python-GitHub-4**
1. Luiz Eduardo Christovam
2. Evgeniy Redekop
3. Vladyslav Shostak
4.
5. Christa Ringers
6. Annika Utz
7. Diana (mentor)
**5. Python-GitLab-1**
1. Nicolai A. B. Riis (CUQI)
2. Jakob S. Jørgensen (CUQI)
3. Amal Alghamdi (CUQI)
4. Roman (TU Delft)
5.
6.
**6. Python-GitLab-2**
1. Hassan Ahmed (ConcreteVision)
2. Saad Azhar (ConcreteVision)
3. Teemu Ojala (ConcreteVision)
4. Mustafa Khalid (ConcreteVision)
5.
6.
**7. Python-GitLab-3**
1.
2. ref: #51 (UiB)
3.
4.
5.
6.
**8. C++ -GitHub-1**
1. Farhan Rasheed (scivis)
2. Robin Skånberg (scivis)
3. Martin Falk (scivis)
4. Mark (mentor)
5.
6.
**9. C++ -GitHub-2**
3. ref: #38 (scivis)
4. Emma Broman (scivis)
5. Signe Sidwall Thygesen (scivis)
6. Emma Nilsson (scivis)
8. Chia-Jung Hsu
9. Roberto (mentor)
**10. C++ -GitLab-1**
1.
2.
3.
4.
5.
6.
**11. C++ -GitLab-2**
1. Pascal Stadler (SuperConga)
2. Niclas Wall-Wennerdal (SuperConga)
3. Patric Holmvall (SuperConga)
4. ref: #11
5. Rostyslav Skrypnyk
6. Johan (mentor)
**12. R-GitHub-1**
1. ref: #3
2. ref: #21
3. Haifa (ref: #59)
4. ref: #54
5. Thor (mentor)
6.
**13. R-GitHub-2**
1. Yusuf
2.
3. Even Myklebust (ref #55)
4. Haris
5. Thor (mentor)
6.
**14. R-GitLab-1**
1.
2.
3.
4.
5.
6.
**15. Julia-GitHub-1**
1.
2.
3.
4.
5.
6.
**16. Julia-GitLab-1**
1.
2.
3.
4.
5.
6.
**17. Fortran-GitHub-1**
1. Mary-Jane Bopape (SA)
2. Nkanyiso Mbatha (SA)
3. Nathi Xulu (SA)
4. Nohlahla Dlamini (SA)
5. Lesetja Lekoloane (SA)
6. Patience Mulovhedzi (SA)
**18. Fortran-GitHub-2**
1.
2.
3.
4.
5.
6.
**19. Fortran-GitLab-1**
1. Matthias Brakebusch (MIMICA)
2. Matthias Schwarz (MIMICA)
3. Hannah Imhof (MIMICA)
4. Luisa Ickes (MIMICA)
5. Johan (mentor)
6.
**20. Fortran-GitLab-2**
1.
2.
3.
4.
5.
6.
**21. Python-GitHub-5**
1. Jan Gerken (GCNN)
2. Yash Niranjan Poojary (chalmers)
3. Akhil Srinivas (chalmers)
4. Ekaterina Baibuz
5. Oscar Carlsson (GCNN)
6. Shivaprasad Gurram (chalmers)
**22. Python-GitHub-6**
1. Adams (ref:#5)
2. Xiaokang
3.
4.
5.
---
## Icebreaker question
In case you have already written some tests, what motivated you to do so?
In case you haven't yet, what motivated you to attend this workshop?
- to automatically find unintended breaks of functionality +1
- for test-driven development +1
- to "calibrate" and benchmark our numeric simulations, to reduce the risk of introducing new errors when modifying the code
- use unit tests to check simple code behaviour
- to re-activate tests in an R library I have been involved in
- I have, and I did it on a class-by-class level to get reassurance when refactoring code
- I have, and I did it on system level to avoid breaking existing user scenarios when adding new features/fixing bugs
- I have, BUT I haven't figured out how to do testing for serialised outputs such as file formats/images and proper usage of external API:s
- I tried to test some fortran code using a python wrapper. This works fine for simple functions but not for subroutines which depend on other function/subroutines. I hope to learn how to do this here.
- Never have. Not really a programming background; I wanna try to learn how to make my code publication ready, though.
- Trying to make sure our software won't break in the future :D (or at least finding out when it does)
- Starting to work on collaborative software development and need to feel that I can take responsibility for other people's use of code as well
- testing for boundary casestesting for boundary casestesting for boundary casestesting for boundary casestesting for boundary cases-
- I had to write some hairy code on a new kind of hardware and I needed it to work the same as it did on the existing supported hardware
- I haven't, I wanted to attend to improve my code for use in our Lab.
- I'm writing tests to verify and make sure the code do as I want it to do! And it makes the code a lot better.
- Learn accepted coding practices so its usable and can be read and understood by collaborators.
- I start learning CI through Galaxy bio-analysis platform
---
## Welcome and introduction
#### Code of conduct
https://coderefinery.org/about/code-of-conduct/
In order to foster a positive and professional learning environment we encourage the following kinds of behaviours:
- Use welcoming and inclusive language
- Be respectful of different viewpoints and experiences
- Gracefully accept constructive criticism
- Focus on what is best for the community
- Show courtesy and respect towards other community members
#### Presentation of course staff
- Anne Fouilloux
- Diana Iusan
- Johan Hellsvik
- Mark Abraham
- Naoe Tatara
- Qiang Li
- Radovan Bast
- Richard Darst
- Roberto Di Remigio
- Thor Wikfeldt
#### About CodeRefinery
- https://coderefinery.org/
- Funded by https://neic.no
- If you like what we do, sign up for the [CodeRefinery newsletter](https://coderefinery.org/), reach out via https://twitter.com/coderefine and/or join us on [CodeRefinery chat](https://coderefinery.github.io/manuals/chat/): you can listen in, follow certain threads, participate, and influence.
- Get notified, or have your friends and colleagues, [be notified of other workshops](https://coderefinery.org/workshops/upcoming/#notify-me).
#### About EuroCC National Competence Center Sweden - ENCCS
- https://enccs.se/
- Funded by the EuroHPC-JU, Swedish Research Council (Vetenskapsrådet) and the Swedish Innovation Agency (Vinnova)
- Provide training on HPC, AI and HPDA topics to academia, industry and public sector
- EuroHPC JU Systems Access Proposal Support
- Support the development of key scientific HPC software in Sweden to adapt the software to forthcoming (pre) exascale EuroHPC systems
- To follow ENCCS, sign up for the [newsletter](https://enccs.se/newsletter) or follow the project on [Twitter](https://twitter.com/EuroCC_Sweden)
#### Breakout rooms
- Please write your name (or registration ID) to a breakout room according to programming language and GitHub/GitLab preference! See above
- We will have 3 breakout room sessions for exercises
- Workshop helpers will move between breakout rooms to answer questions and solve problems.
- But you can always ask questions in this hackMD!
- In the first breakout room session, spend a couple of minutes introducing yourself to your "room mates" (if you don't already know each other)
#### About the material
- All material is online and CC-BY: https://coderefinery.github.io/testing/
- feel free to reuse or contribute!
- Material contains more than we will discuss.
- Exercises are available in multiple languages, but not all exercises are available in all languages.
- In the section *"Automatic testing with GitHub Actions or GitLab CI"*, you will use either GitHub or GitLab. Everyone in a breakout room should use the same platform
#### How to ask questions and follow along: HackMD
- HackMD is our base communication system:
- You can ask questions at the bottom
- You can check towards the bottom for a link to our current section
- [HackMD mechanics](https://coderefinery.github.io/manuals/hackmd-mechanics/)
- Anytime, create a new bullet point at the bottom to ask a question. Someone will be watching and respond
- HackMD should contain detailed infromation on exercises and breaks (always at bottom).
- You can also ask questions by voice anytime
#### Certificates
- Certificates available upon request to support@coderefinery.org.
---
## Questions and answers
(new questions to the bottom of this document, please)
- Is this how I should ask a question?
- yes, and an answer will be written like this
- Is it possible to change the room now? I want to be in Github C++.
- if there is space in your preferred room, go ahead :)
- I don't understand what is the difference between gitlab and github? I have also a gitlab account and could go into a Python gitlab room.
- They are two online platforms for collaborating on code. They work quite similarly, but are owned by different companies.
- Yes, I know :-) But I use the same git commands to push.
+ Both use git for version managment (hence the same commands), but the features are different between the two platforms (testing, actions, etc)
- The exercise will show how to run tests automatically on the two platforms when you push changes. (Behind the scenes, git knows which repository you originally cloned, whether gitlab, github or something else. So after you clone, very little differs.)
- for the code refinery workshop it was possible to get graduate school credit (at Tu Delft). Is it possible to get a certificate for this workshop as well to get GS credit?
- certificates can be requested via email to support@coderefinery.org
- if credits can be given from university depends on your university
- great thanks!
- What is the status of room Python-Github-6, please? There is only one person.
- (host) I see that 4 people who wrote their name/reg. # in rooom 6 have come to this Zoom room at least.
- How does the assert function work?
+ `assert(cond, message)` checks whether the given condition evaluates to true. If it is false, the message is printed (sometimes along with the condition) and the program terminates. In case of true, the program continues.
- Is it best to include tests in the same script file or to organize specific test files ?
- It will depend on the facilities your language offers for testing. In Python, for example, you can do both and `pytest` will be able to find and run them all. In C++, they need to be in other files. Personally, I prefer tests to be in a separate folder, regardless of the language I am using. If you're starting to work on a new project, it's very instructive to look at the tests to understand what the code does and with which functions in the codebase. If tests are in separate files in separate folders, then this exploration is much easier.
- Will we be trying mock in Python?
- There are no explicit exercises on mocking, but we can discuss examples later on, I think.
- What does CD stand for, please? I saw CI/CD, I think.
- Continuous integration (CI) and continuous delivery (CD). The former is running tests automatically on a regular schedule: usually, but not necessarily on every commit pushed to the remote repository. The latter is the automated deployment of the "artifacts" of your code on a regular schedule (could be on every push). The deployment might be uploading a Python package to the Python package index (PyPI) or running a new version of a web application on a server that your users interact with.
- (moved from above) If i have functions that do not perform very specific calculation, but they do operate and modify a class of inputs (i.e. high pass filter on an image), how do i test them?
- In the test code, I would save a reference input and reference output files, run the code on the former and then check that the new output matches the reference one.
- Consider also refactoring the original code, so that you do not modify the input but rather make a copy and return it as output. This will incidentally help the modularity and reusability of the code. However, if the resources (memory, CPU) to copy are very expensive, you might not follow this strategy.
- Another option is to create a very small input set where you already know the result (or can compute it yourself). For example sampling or filtering a small 2D grid. Or an image where the high pass filter returns a trivial result, like all black.
- Can I customize the name of the test in pytest?
- yestest?
- yes
- What does the pytest return exactly?
- Each function named ``test_`` should return either ``True`` or ``False``. Running pytest aggregates the results and prints them to screen nicely.
- I had the test to fail. Function returns a-b. The test should assert add(2,3)==5. I get
```
assert add(2, 3) == 5
E assert -1 == 5
E +-1
E -5
```
There is a + in front of 1, not --1 as shown in the example. Same for 5. We wondered if there is some version control in the pytest.
- The ``+`` and ``-`` work as in a patch you obtain from version control: "I expected 5 (``-``) but found -1 (``+``)" I am not sure why it's different in the published material.
- I still don't understand. The `+` indicates what, please, for example?
+ The test was expecting the value ``5`` to be present, but found the value ``-1``. This means that ``-1`` was *added* to the output, hence the (``+``) by its side, while ``5`` was *removed*, hence the (``-``) in front.
- What is a good way to set the acceptable tolarance for floating point comparisons across an entire test suite? Should/could this be an external flag or part of the tests? And should one add checks to verify the precision of the interpreter running the tests?
- That really depends on what you are testing for. By default, floating point numbers in Python are double precision, so the predefined thresholds in pytest and numpy are geared towards that.
- I suggest against this, unless the kind of logic that computes float-point numbers is always the same. You don't want to force yourself to have very sloppy testing of numbers that are very reproducible, just because some of the other numbers are computed in a way that is less reproducible.
- Ideally, you understand the algorithm that computes the floating-point number and can reason about what range makes sense. For example, accumulating a sum over a vector has an absolute bound on the range of results, depending on the order of accumulation, but that bound depends on the values in the vector... (But this is a big topic in its own right!)
- I can't get the pytest working
- please write a PM to someone with (CR) in name in zoom
- or try to explain here what is not working
- Why do other calculations like 0.1 + 0.4 work correctly?
- In that case, the result (0.5) can be represented exactly as a floating-point number, and it’s possible for rounding errors in the input numbers to cancel each other out - But that can’t necessarily be relied upon (e.g. when those two numbers were stored in differently sized floating point representations first, the rounding errors might not offset each other).
- In other cases like 0.1 + 0.3, the result actually isn’t really 0.4, but close enough that 0.4 is the shortest number that is closer to the result than to any other floating-point number. Many languages then display that number instead of converting the actual result back to the closest decimal fraction.
- https://floating-point-gui.de/basic/
- How to set >>> getcontext().prec can be used to solve
- (from zoom chat)how do you test when you using Spyder?
- ``!pytest -v example.py``
- Is it the same like `!ls` in a notebook, i.e. executing a terminal command?
- yes, same idea, to run non-python commands
- I am not able to rename myself with the group name in Zoom.
- can you not hover over your name in the Participants list, click "More", and then "Rename"?
- Answer: yes, thank you.
- Great!
- How to run tests in Jupyter notebook? Do we need to run through command prompt ?
- you could use ``!pytest test.py`` if you have your tests in a separate script (test.py) from within a codecell in the jupyter notebook
- there seem to also be plugins for jupyter notebooks, such as https://pypi.org/project/pytest-notebook/ (no experience with those)
- We saw an exercise using the `tempfile` library to create a temporary file. How to make sure that the temporary file gets deleted even if a test fails? I am working on something like this:
```
def create_file_with_text(text='', fname=None):
if not fname:
_, fname = tempfile.mkstemp()
with open(fname, 'w') as f:
f.write(text)
return fname
@pytest.fixture(scope='module')
def temp_file(self, text='', fname=None):
fname = create_file_with_text(text='', fname=None)
yield fname
os.remove(fname)
def test_something(temp_file):
pass
```
- We have been running the tests using pytest. How to set up the test file to be run from an IDE (e.g. pycharm) for the purpose of debugging the tests? Something like this seems to work
```
if __name__ == "__main__":
pytest.main(['-v',
'ignore:test_asserts.py::test_find_file_with_extensions',
'-Wignore',
__file__])
```
- Any tip on which warnings we can suppress and how?
# Exercise until 10:45
- How big allocation does a repo get in terms of testing? Could too long test times be limited by Github?
- often it's ~ 1 hour per task/workflow and often you can run several of them at the same time. if the test set takes hours, then it won't fit there but then you can connect your own test runner (can be an idle computer or some cloud computing resource) to e.g. GitLab CI and then they can be in principle arbitrary long.
- Could you comment on what Mark means with targeting different use cases by github and gitlab, please? Thank you.
- They're useful for the same use cases. They just are built differently by different people and the feature sets suit the people who paid for them to be built. For our purposes, they're the same.
- Not sure about pt 4: "Also browse the “Actions” tab and look at the steps there and their output."
- There you can see what actions have run, e.g. https://github.com/mabraham/testing-workshop-mark/actions
- Ok, I think my tests are not running, that's why I cannot see anything (see the point below)
- Make sure you followed the steps to enable automated testing.
- Is it possible to run the GitHub actions inside a singularity container
- You should check the GitHub marketplace to see whether there's a pre-packaged action. It is possible to run inside a Docker container, so it should be possible to run within Singularity.
- this exercise is not working for me, it seems like I'm missing a "runner". I'm using my TU Delft gitlab account, perhaps there are no "runners" configured.
- right. we were somehow assuming that gitlab would be gitlab.com. we need to clarify this for later workshops that other gitlab servers may not be connected to runners yet.
- to investigate, in the left-hand menu, scrolling down to settings, then find "CI/CD" submenu under it. Then you will see if the shared runners are available.
- To allow actions in your fork of a repository on GitHub you need to activate them by clicking Actions and enabling them. What are the risks of enabling actions a forked repo under your GitHub account?
- I don't see any risks. This can be useful if you want to test your changes automatically before sending a pull/merge request to "upstream". Note that you don't need to activate actions on the fork to get pull requests upstream automatically tested.
- Can we have more time?
- some of us can stay a bit after the workshop, if you would like to continue this exercise (breakoutrooms will stay open)
- (moved from zoom chat) What should one consider regarding resource limitation on GitHub actions?
- https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits
You have a hard 10GB limit on disk space.
- Why does a test can take hours, please?
- a test could be an end-to-end test and test the whole code (we will see that later) and depending on the program it can take arbitrarily long. but it is good if tests only take seconds or minutes since it can make development easier
- GitLab only creates a merge request across different branches. Gives an error when the merge is from master to master. Is that right?
- Yes, merges and pull requests only make sense between different branches (ie including a branch of the same name in a different repo)
- Pt 5 does not break the tests in [my repo](https://github.com/aless80/test-aless80). My hunch is that he tests are not running because of some silly syntax mistake.
- You should have used the "Python package" action. The one you are using is for publishing the pacakge to PyPI and will only run when you create a new release.
- thank you, what now? can we talk?
- solved
- Thank you. I had a yml triggering every release, which is not good. Therefore I created a new workflow with of type "Python application", then the .yml file was created in the wrong place, so I had to move it to .github/workflows
- in github/workflows/python-app.yml i got n error "every step must define a `uses` or `run` key" what that means?
- probably you made a typo, like wrong numbers of spaces in the indenting. Copy the example carefully and it will work.
- I also had it, it was due to a dash before the `run` command
- in the background, does the testing takes place in a container? (I noticed `ubuntu-latest` in the yml file.) Would this entail all the (environmental) cost of spinning up a container?
- yes indeed. it creates a container, runs the tests, and then destroys the container. and you can also use own containers instead. That adds a tiny amount of overhead compared with no container, but you don't have an option about that. Running automated tests does indeed have an environmental cost, as a computer somewhere is doing more work than it would otherwise have done.
- thanks!
- If we have multiple code files in repository then we need to add line for every file including test in Github Action workflow?
- this depends on the project language and test runner. if this is about pytest, then it can also gather all tests in a python package so it is not necessary to add one line for each file. (if i understood the question correctly)
- For example, a C++ project using CMake with CTest would register each test binary (or python script) with CTest, which would be run by "make test." Now only that line needs to be part of the workflow .yml file, but that's your choice.
- Why does github asks me again all the time for login/password? Last week I worked on a different repo on github, now I am working on the repo for the coderefiner workshop and it has started to ask me before every push. How does git know to push to which repo when I work on several, please?
- If you cloned using SSH (`git clone git@....`), it will use public-key authentication and *not* ask for a password. If you cloned using HTTPS (`git clone https://`), then it will ask for a password. When working on a repo that is not yours, the default is anonymous HTTPS clone.
- Thank you. I was lazy and did `git clone https://` I did not know about the git@ prefix. Can I fix this still, please?
- You can get your hands dirty with `git remote rm origin` and then `git remote add origin git@whatever-it-is` and then teach each local branch which branch of the remote it should track. Much easier to just clone with `git@whatever-it-is` separately (or follow the advice below!)
- or: `git remote set-url origin git@someaddress` which will change to `git@someaddress` from whatever it was before
## Exercise until 11:47
Please also write here questions or surprising/interesting findings which can be good for other groups.
- Any test case to test the factorial can only test a finite sample. Testing the first few instances is prone to missing deviations at large n. I think it might make sense to test whether factorial(n+1)=(n+1)*factorial(n) where n can be a random integer greater than 1 (maybe not too large ...).
- good approach! sometimes you have two functions that implement the same thing. a simple, obviously correct, slow function, and a more complex fast function. and then it can be useful to compare both for a random set of input values and design a test this way.
- yes, that would be a better approach: it's called *property-based testing* and in Python you can use the Hypothesis library to help you with generating test sets.
- sometimes you can even test exhaustively every possible input value and compare it to some source of truth (e.g. that a fast, complex, hard-to-read implementation does the same as a slow, simple, comprehensible one)
- Can I use print statements with the pytest test functions?
- Yes. More info here: https://docs.pytest.org/en/stable/capture.html?highlight=print%20statement
- `pytest -s myfile.py`
- When testing randomness in Python. Is there a good way to fix the seed within the function, but not change the global seed that is used for the other tests? (One idea I just had to was to extract the current seed, set a seed (1), run function, then restore the original seed?)
- I believe you need to use an external library implementing pseudorandom generators or even use a completely different generator.
- Refactor the function so that it does not have the external dependency on a global rng, but rather a specific one. Now you can control that in the test, while letting the normal code use the global one. (y)
- Why does pytest not find my file? Terminal output and file content below.
```
username@machine:~/pytest-example$ pytest test_yahtzee.py
==================================================================================== test session starts =====================================================================================
platform linux2 -- Python 2.7.18, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
rootdir: /home/username/pytest-example
collected 0 items =============================================================================== no tests ran in 14.43 seconds ================================================================================
```
```python
# The following is test_yahtzee.py
###########################################################
from yahtzee_library import *
num_games = 1000000
winning_games = list(
filter(
lambda x: x == 5,
[yahtzee() for _ in range(num_games)],
)
)
#print(100 * len(winning_games)/num_games)
assert abs(len(winning_games)/num_games - 0.046) < 1
###########################################################
```
- it looks for functions that start with `test_` and runs them
- But the file is called test_yahtzee.py
- inside there you need to create a function `test_something()` and move the test under that function
- Thanks! :)
- What if the code imports external packages? Referring to "4. Function with an external dependency". What if it's not a simple value, but a complicated function
- It's normally a good idea to wrap external packages with a layer that you control that gives you the ability to e.g. swap it for another package (or version of the same package), either for testing or actually switching packages. So if you do that, then the dependency problem is easier. Also, you can refactor the code from ``code_to_test()`` to look like ``value = get_from(dependency); code_to_test(value);``
- I read in the past that an impure function is also when, for example, the text input `mytext` is modified and the function returns `mytext`. Do you see problems with this types of functions?
- any function which accesses the disc or keyboard or internet or database is inherently impure since the disc and keyboard and internet are "time-dependent" and will change from call to call. but functions which access files can still be tested (see one example in test design) by creating a temporary file or temporary database, run the function, verify the result, then destroy the temporary file/database. in practice i would recommend to move the I/O (file/internet/database-access) to the "outside" of your code and keep the "inside" of your code pure and stateless and easy to test.
- but this function, which does not access the disc or anything, is it impure and potentially problematic? Notice that it modifies the input variable only inside the scope of the function and then returns it. I am asking because it is relevant to the code I am working on.
```
def add_x_to_input(mytext):
mytext = mytext + 'x'
return mytext
mytext='some text'
mytext = add_x_to_input(mytext)
```
- this function can be considered pure: given an input it will always produce the same output. and it does not modify any variables outside the function. although it looks like it would modify `mytext`, it modifies its copy and returns a new value which you then choose to put into the same "container" but the function itself does not modify the input `mytext` hence I would call this one pure.
- Right, cause strings are immutable. Cool, thank you.
- How do you test functions that render file formats, such as images?
- understand what you expect the behaviors to be
- have a set of test images and record the results so you can compare them pixel-by-pixel in future, but you'll need a human to decide up front that those results actually are good
- i find this approach useful: what do you look at to decide whether the generated image is correct? now write these criteria down. can you put these criteria into code? (this is the difficult part). once you have the criteria written in any language, you have a test. i know it's not easy but defining and writing down these validation criteria can be a useful exercise.
- more practically: you could store the checksums of generated images for a series of input parameters and check whether you regenerate the same result by comparing the checksum.
- More on the previous question: say that my program converts some text file to a text file e.g. html. how to write tests there? For example, testing that a link in my output is written in the correct syntax and points to some paragraph in the output itself, but avoid whitespaces or whatever changes in the html syntax that are not relevant.. One could probably do this, but if you know of any syntax-aware library (for html or markdown or python) or tips let me know
- just to make sure I understand: you are looking for a library that can validate html and markdown and check internal links?
- Well, maybe. I am wonder how you would write tests for software that converts from one text format to another, e.g. to html (but not only). I cannot diff some reference html file vs the html output as is because, for example, some .js file in the `<head>` will change in time, I don't want line breaks or comments to make tests fail, etc. I should probably massage the html output/use regex before diff-ing. It sounds like a hassle.
- I discussed it with Thor and I should probably start testing on my low-level functions in my software and go from there.
- Interesting: I asked what was the difference between `master` and `main` in git.
- It is the same, but the `main` syntax is more inclusive
- Do you have any possible answer to the section Designing an end-to-end test [here](https://coderefinery.github.io/testing/test-design/)?
- sorry we need to create a solution and I only created the problem yesterday but in this particular case I would keep a folder with inputs and outputs and the end to end test would loop through all inputs, run the code, and compare the produced output with each reference output. in this case one could compare strings precisely, whether they are identical. if the code produced floats, one would have to compare with numerical tolerance. sometimes code produces results but also prints times and dates and timings and then one needs to do some filtering.
- How do you test functions that rely on external API calls? Maybe with a focus on API:s that may be subject to change, is that defensive programming rather than testing?
- here my focus would be to test this often/frequently so that once things change, i notice this. one can run scheduled tests on github which are run every day/week/month. there are also tools that focus on testing API changes/degradation (i am not using them myself but only saw that they exist)
- You'd need both. The tests help you understand what has changed. Then your code can be changed to react accordingly, e.g. support multiple versions. Key here is that you wrap the external API in a layer that you control. That lets you make observations (ie run tests) before it gets into the core components of your code. Maybe you can introduce a change in the layer that means your core components are insulated from the change to the external API.
Discussion from group 1: What are pure/impure functions in practice? Difficult to test functions that render files or rely on external API:s. Randomness is complex and inticate. We chose test-driven development.
Test-driven development is like writing specification as testing code before writing the actual implementation code. Check the written conditions/criteria and translate them into tests according to the testing framework. E.g. "takes an integer argument" -> test_int_fizzbuss(). Writing the test forces you to learn more about the problem/what the implementation needs, e.g. validation of the type of argument and raising an error, e.g. TypeError for bad arguments.
## Quick feedback
Please write down one thing you liked and one thing you think should be improved!
Good:
- Nice overview on existing testing strategies
- Great concepts and test design sections
- Very good onboarding process. I was scared of implementing testing, now I look forward to it.
- Great intro with good pace, will join similar things in the future
- Great to have an instance to work this excercises out with others.
- Great organization, use of HackMD and instructing
- Exercises are good but sometimes get complex. A universal online environment for everyone should be identified.
- Radovan's ending comments on how to get started on existing codebase were really were good. (Maybe add them to the docs?)
- Good overview about testing, gives me some inspiration for testing!
- Very nice documentation but needed to be updated for version as we were wondering if differencies were due to different versions, however very comprehensive
- Nice to have a session dedicated to testing and very interesting content !
- Very nice that you catch questions from the chat and bring them to the speaker's attention. Thanks!
- Good and quick answers to the questions asked in the HackMD.
To improve:
- CI/CD did not work for institutional gitlab accounts with no configured runners
- not Familiar with Github so struggled there
- maybe add example of test cases for one particular function (e.g. range checks for list input, invalid indices)
- did not have a helper in the room when we needed one in the difficult second part
- It was great! But (this is more of a question), will there be more advanced labs in the future (asking for a friend, who is familiar with the content of todays lesson). Also mocking, could be added, which I do not understand at all. Otherwise great, this is really needed in research!!!
- It would be helpful to have a session on integration tests and end-to-end tests, in particular, since you recommend this as the first step
- Maybe reduce diversity of platforms/languages to focus more on testing as a concept. An instructor-led discusscion of the last segment (test-design) would have been more helpful.
- The last part on "test design" became way too rushed - try to reduce the number of exercises here, or split them over more breakout exercise sessions. A lot of attendants didn't even have time to get the examples to compile/their testing environment to work. When we finally started screen sharing an environment that worked, the time was out, and we had barely covered anything.
- too short perhaps +1
- More time for exercises+1
- I think as well, it was way to short. We also did not have a mentor, but we worked well as a team, but this slowed us down.
- There should be guided exercises in Breakout rooms and less exercises so concepts can be learned qualitatively not quantitatively.
- Perhaps have some examples of really good tests
- The exercise of Test design section needs more time. Probably double.
- No "real" mentor in our room, so felt a bit lonely to take decisions
- Would have liked more time on the last excercise since that was the most interesting one for me :)
- Is there a document or a section for feedback? I would like to write it before I forget :)
- *I will write it here. Please move it later to the appropriate place.* I had to create a Gitlab account and the instructions in the lesson on Automated testing and CI for pushing to the new repo did not match the steps needed. I suspect it was because people creating the instructions in the lesson alreay had an account. So maybe you would like to consider starting from scratch and creating a new account for those instructions.
- thanks for this feedback. indeed this was steep learning and we need to improve this.
- Would be good to have some more detailed instructions on the setup requirements for either github or gitlab (as often people have different defaults). More guidance with the exercises would help.
- missing code repo were all groups could put their creative work
- Would it be helpful to somehow differentiate breakout rooms according to participant skill level? I understand you could be more clear about prerequisites (e.g. everyone should have done a fork/clone/pull request on someone else's repo *before* the workshop if this is expected knowledge), but differentiating groups on skill could possibly facilitate the level of instructor time or how it is allocated. Some groups with more advanced participants may want to move forward on their own, with possibility for expert support, where other groups could aim to cover the basics and have support focused on this. Understand this is always a challenge though!
- In one group I was in, the focus was a lot on just "getting done" for the first exercise, under silent and independent work. Could you perhaps clarify that participants shouldn't be afraid of asking questions, and that if you are done early, it is welcome to offer to help others who need it (if this is indeed the idea)? An ideal experience from the learner is probably to be in a group with people of the same skill level (approx), to know each other a little bit (to not be afraid to ask), and to get good help.
---
*Always ask questions at the very bottom of this document, right above this. Switch to view mode if you are only watching.*
Sign in with Wallet
Connect another wallet