---
title: Hardware testing
description: In this lesson we introduce concepts, tools and techniques to focused on verification and testing of hardware. We emphasize the importance of hardware testing in different phases and project activities.

authors: Jerry de Vos, Jose Urra
---

# Hardware testing

[](https://creativecommons.org/licenses/by/4.0/) [](https://doi.org/10.5281/zenodo.7195686)

**Date of release:** 05-09-2022

:::warning
:warning: This lesson is an extension of our hardware design and prototyping lesson. In this material, we focus on testing for verification and functional testing.

:construction: Consider this lesson a beta version, by this we mean that it can be considerably enhanced and improved by persons with experience in hardware testing and verification.
:::
Successful, scalable, and robust hardware designs and products require thorough testing along a project lifecycle from early development stages up to production. Strict testing setups are needed for those artifacts with potential safety flaws such as high currents, and high mechanical forces that may harm end users or users in a production line just to name a few examples. **The more adopters and users your project has the more testing becomes a critical factor, the same applies to safety considerations.**
:::info
**By the end of this lesson you will be able to**
- Design a hardware test that is relevant to your project
- Test your hardware

**Learning objectives**
- Apply testing as part of all the phases of your project (from early concept development to refinement and production)
- Get familiarized with the concept of test-driven development

**Prerequisites**
- Be familiar with hardware design and prototyping

:bulb: (We recommend you to review our **[hardware design and prototyping lesson](https://hackmd.io/1kONu5gwSS6knAGnwcO5KQ?view)**) where we touch upon concepts that are related to prototyping and testing including validation and verification.
:::

## Introduction
Testing of products that may include software and hardware is an established practice in many industries. This is especially the case in industries where reliability, safety, robustness, and performance are critical. (Think of product certifications for washing machines, microwave ovens, autonomous vehicles, planes, etc.)

Today testing has become so essential to open source software development that it has become an indicator of project maturity and reliability. Furthermore some software developers even design tests before writing code, this approach is called **test-driven development.** Hardware testing is a bit more complex due to the specificities of testing environments required and different types of test domains (material, integrated circuits, mechanical, etc). But the concept of test-driven development has been increasingly adopted in hardware development as well.

:::warning
:point_up: Image retrieved from https://wikispeed.com/car
This image shows a car developed by wikispeed.
> Wikispeed innovates by applying scrum development techniques borrowed from the software world. They use open-source tools and lean management methods to improve their productivity. [Read more...](https://en.wikipedia.org/wiki/Wikispeed)

> From Test Driven Development we start with failing tests and then develop solutions. This allows us to quickly identify if current work is not targeted to passing a test or causing problems elsewhere in the system, which avoids waste.[Wikispeed development process](https://wiki.p2pfoundation.net/WikiSpeed#The_Development_Process)
:::



Image retrieved from [fiber laser/splitter unit and the acousto-optic (AO) modulator](https://www.researchgate.net/publication/258688392_Hardware-in-the-loop_projector_system_for_light_detection_and_ranging_sensor_testing/figures)
Example of a hardware testing setup

## Putting hardware testing in the context
As explained in our [hardware design and prototyping lesson](https://hackmd.io/1kONu5gwSS6knAGnwcO5KQ), testing takes place in the context of a project with a specific framing, set of resources, and capabilities, more or less defined time constraints, core hypothesis/concepts/ideas behind the design, target specifications and desired outcomes (alpha, beta, or production-ready design). In the context of hardware design verification tests are focused on checking the functionality of a full design or certain features of the design (for example a component or unit, vs the whole design).

Within this context, hardware verification testing is a subset of testing activities explained in the [design and prototyping lesson](https://hackmd.io/1kONu5gwSS6knAGnwcO5KQ). If you have gone through the exercises of the lesson mentioned you already have a prototyping plan.

### Different types of tests
This section is meant to give you an idea of the range of test classifications and definitions you can find on the internet just to give you an idea of how broad the subject can be, then we will try to keep it simple to get you started.

Image retrieved from [Best Practices for Functional Testing](https://www.globalapptestinghttps://www.globalapptesting.com/best-practices-functional-testing.com/best-practices-functional-testing)

Image retrieved from [How do you define functional testing with two popular and seemingly contradictory explanations?](https://stackoverflow.com/questions/48614190/how-do-you-define-functional-testing-with-two-popular-and-seemingly-contradictor)

Image retrieved from [How do you define functional testing with two popular and seemingly contradictory explanations?](https://stackoverflow.com/questions/48614190/how-do-you-define-functional-testing-with-two-popular-and-seemingly-contradictor)

### Some useful definitions
**Test bench**
> A **test bench** or **testing workbench** is an environment used to verify the correctness or soundness of a design or model. [Read more](https://en.wikipedia.org/wiki/Test_bench)

:::spoiler **Different types of testing environments**

> - **Simulation:** the tests are performed on a computer. It's fast, and affordable and it can be done in-house. For example, you can perform thermal or aerodynamics analysis inside different 3D modeling software.
> - **Laboratory conditions:** This testing environment is common for chemical and technical testing.
> - **Real-life conditions:** These tests are usually performed after you test in laboratory conditions. They are more relevant during late development phases as they require advanced prototypes.
> An additional meaning for "test bench" is an isolated, controlled environment, very similar to the production environment but neither hidden nor visible to the general public, customers, etc. Therefore making changes is safe, because final users are not involved.
[Retrieved from](https://wikifactory.com/+wikifactory/stories/hardware-testing-during-early-development-phases)
:::
---

**Behavior**
What a system will do in response to its external environment without referring to details on implementation (e.g., use of technologies). Given a certain input of data, power, or force it should behave in a certain way. For example produce a signal like raising an alarm, or powering of a board if a peak of power is reached, etc.

**Sanity test**
A sanity check or sanity test is a basic test to quickly evaluate whether a claim or the result of a calculation can possibly be true. It is a simple check to see if the produced material is rational...The point of a sanity test is to rule out certain classes of obviously false results, not to catch every possible error. [Read more](https://en.wikipedia.org/wiki/Sanity_check)

**Unit test**
> Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation. [Read more...](https://www.techtarget.com/searchsoftwarequality/definition/unit-testing#:~:text=Unit%20testing%20is%20a%20software,developers%20and%20sometimes%20QA%20staff.)

**Integration test**
> System integration testing involves the overall testing of a complete system of many subsystem components or elements. The system under test may be composed of hardware or software, hardware with embedded software, or hardware/software with human-in-the-loop testing. [Read more...](https://en.wikipedia.org/wiki/System_integration_testing)

**Test cases**
> A test case is a detailed procedure that fully tests a feature or an aspect of a feature. Whereas the test plan describes what to test, a test case describes how to perform a particular test. You need to develop a test case for each test listed in the test plan.

**Test Driven Design**
Test-driven development (TDD) is a system development process relying on system requirements or target specifications being converted to test cases before the software is fully developed, and tracking all software development by repeatedly testing the software against all test cases. This is as opposed to software being developed first and test cases being created later.

### Getting started with hardware testing
:::warning
This approach to testing has been extracted from Neil Johnson's slides and blogpost on TDD (Test Driven Development) And A New Paradigm For Hardware Verification:
- [Slides](https://www.agilealliance.org/wp-content/uploads/2016/01/TDD-and-a-new-paradigm-for-hardware-verification.pdf)
- [Blogpost](http://agilesoc.com/2011/11/14/tdd-and-a-new-paradigm-for-hardware-verification/)
:::

**Split the entire test effort into a series of steps**
- Step 1: Basic sanity of primary behavior (the overall concept of your project)
- Step 2: Sanity of first major feature set (The most important features and specs of your project)
- Step 3: Sanity of second major feature set
- Step 4: Sanity of third major feature set
- Step 5: Sanity of combined feature sets
- Step 6: Functional coverage of combined feature sets
- Step 7: Verify misc features
- Step 8: Clean-up/release
    - Incremental implementation of the testbench to support each step
    - testbench never includes more than the minimum required
    - choose techniques that suit each step
        - directed tests to start
        - random tests later
:::success
As you can see this is an extended version of developing a progress board for tests explained in the lesson: [Hardware design and prototyping](https://hackmd.io/1kONu5gwSS6knAGnwcO5KQ)
:::

### Designing test cases (verification experiments)
::: warning
:warning: This entire section has been retrieved from the following source: [Designing Test Cases, by One Stop testing](http://www.onestoptesting.com/test-cases/designing.asp)
:::

A test case is a detailed procedure that fully tests a feature or an aspect of a feature. Whereas the test plan describes what to test, a test case describes how to perform a particular test. You need to develop a test case for each test listed in the test plan.

**A test case includes:**

- The purpose of the test.
- Special hardware requirements, such as a modem.
- Special software requirements, such as a tool.
- Specific setup or configuration requirements.
- A description of how to perform the test.
- The expected results or success criteria for the test.

Test cases should be written by a team member who understands the function or technology being tested, and each test case should be submitted for peer review

## Hardware tests examples
Hardware testing can take up many different shapes and sizes, just to give you an idea of this, checkout how SpaceX has many millions of dollars going up into flames:

{%youtube bvim4rsNHkQ %}

This was what took the company to a point where now it is possible to land a rocket consistently.

---

When you are making early prototypes you end up adding changes and creating new versions. Not testing these can result in "integration hell". The following video goes through this at the minute 8:30.

{%youtube vsTTXYxydOE %}  

---
If you make hardware commercial hardware products, automatic testing can be critical. Here is a great video demonstrating why automatic testing is important for an open hardware pick and place machine:

{%youtube g2478l8GVkk %}

Fore more in-depth testing reads you can check the blog posts of the Flipper development:
https://blog.flipperzero.one/electronics-testing/

---
# Exercises
## Exercise 1: Include verification tests in your testing plan
:::warning
If you have gone through the previous lesson: [hardware design and prototyping lesson](https://hackmd.io/1kONu5gwSS6knAGnwcO5KQ) then you have alrady created a testing progress board containing your tests/experiments.
:::

:::spoiler Testing process and board summary

:point_up_2: Image retrieved from the book: "Value proposition design: How to create products and services that customers want"

**Progress board**
This tool simply aims to structure your testing process by starting from left to right with your core ideas broken down into prioritized hypotheses (concepts, ideas, sub-problems, sub-solutions). Then we have a backlog of tests that can also be prioritized using the prototyping criteria we have shared in previous sections and ends up with learnings and knowledge that are captured in better design specifications and choices.


:point_up_2:  Image retrieved from the book: "Value proposition design: How to create products and services that customers want"

:::

___

**Instructions**
- Revisit your testing board
- Add sanity checks or tests that aim to verify target features and/or specifications considering the steps above explained
- Do a testing board with your testing overview for verification tests (if you haven't)

## Exercise 2: Design a test to check the sanity of the primary behavior of your project
**Instructions**
- Use the template below to describe a test case
- (Consider adding this as a task on your project board)

Test Case Design

Test Case Name:

Test description:
The description of the test case you are going to test.

Revision history:
Each test case has to have its revision history in order to know when and by whom it is created or modified.

Function to be tested:
The name of the function to be tested.

Environment:
It tells in which environment you are testing.

Test Setup:
Anything you need to set up outside of your application for example printers, network, and so on.

Test Execution:
It is a detailed description of every step of execution.

Expected Results:
The description of what you expect the function to do.

Actual Results:
pass / failed

If pass - What actually happens when you run the test?

If failed - put in the description of what you've observed.

## Exercise 3 (optional): Design a test bench for those tests that you think you need to perform several times
**Instructions**
- Make a bill of materials for your testing environment
- Design your testing environment (make a sketch or a draft)
- (Consider adding this as a task on your project board)

# References
:::warning
:construction: References need to be shortened.
:::
[1] I. Pomeranz, N. R. Saxena, R. Reeve, P. Kulkami, and Y. A. Li, “Generation of test cases for hardware design verification of a super-scalar Fetch Processor,” in Proceedings International Test Conference 1996. Test and Design Validity, Oct. 1996, pp. 904–913. doi: 10.1109/TEST.1996.557152.

[2] “Hardware Verification, Testing and Maintenance —.” https://aceproject.org/main/english/et/ete05a.htm (accessed Sep. 04, 2022).

[3] “System integration testing,” Wikipedia. Jun. 01, 2022. Accessed: Sep. 04, 2022. [Online]. Available: https://en.wikipedia.org/w/index.php?title=System_integration_testing&oldid=1091007643

[4] “What is Unit Testing? Definition from WhatIs.com.” https://www.techtarget.com/searchsoftwarequality/definition/unit-testing (accessed Sep. 04, 2022).

[5] https://user.eng.umd.edu/~austin/enes489p/lecture-slides/2012-MA-Behavior-and-Structure.pdf (Accessed Sep. 04, 2022).

[6] “When you want to unit test... abstract the hardware,” ElectronVector - Test-First Embedded Software. http://www.electronvector.com/blog/when-you-want-to-unit-test-abstract-the-hardware (accessed Sep. 04, 2022).

[7] https://www.ee.ucl.ac.uk/~zceedjm/AcademicSite/blog/unit-testing.html (Accessed Sep. 04, 2022).

[8] “Behavior (glossary) - SEBoK.” https://www.sebokwiki.org/wiki/Behavior_(glossary) (accessed Sep. 04, 2022).

[9] “Designing Test Cases | The Purpose Of The Test | Special Hardware Requirements.” http://www.onestoptesting.com/test-cases/designing.asp (accessed Sep. 04, 2022).

[10] Shoo, “Answer to ‘How do you define functional testing with two popular and seemingly contradictory explanations?,’” Stack Overflow, Feb. 06, 2018. https://stackoverflow.com/a/48641759 (accessed Sep. 04, 2022).

[11] J. Hallyburton, “How do you define functional testing with two popular and seemingly contradictory explanations?,” Stack Overflow, Feb. 04, 2018. https://stackoverflow.com/q/48614190 (accessed Sep. 04, 2022).

[12] S. Bashkyrtsev, “Answer to ‘How do you define functional testing with two popular and seemingly contradictory explanations?,’” Stack Overflow, Feb. 06, 2018. https://stackoverflow.com/a/48642473 (accessed Sep. 04, 2022).

[13] gd-admin, “Function Testing,” https://www.fumaxtech.com/. https://www.fumaxtech.com/function-testing/ (accessed Sep. 04, 2022).

[14] https://www.abi-agile.com/en/wp-content/uploads/2022/04/scrummaster-theagiletrainingseminarforbusinessperformance-sample.pdf (Accessed Sep. 04, 2022).

[15] nosnhojn, “TDD And A New Paradigm For Hardware Verification,” AgileSoC, Nov. 14, 2011. http://agilesoc.com/2011/11/14/tdd-and-a-new-paradigm-for-hardware-verification/ (accessed Sep. 04, 2022).

[16] “WikiSpeed - P2P Foundation.” https://wiki.p2pfoundation.net/WikiSpeed#The_Development_Process (accessed Sep. 04, 2022).

[17] Abhishek Tiwari, “Test Bench Development,” 23:46:33 UTC. Accessed: Sep. 04, 2022. [Online]. Available: https://www.slideshare.net/Abhishekt11/test-bench-development

[18] “Democratizing the Production of Interactive Hardware | Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology.” https://dl.acm.org/doi/pdf/10.1145/3379337.3422877 (accessed Sep. 04, 2022).

[19] J. Millitzer et al., “Recent Developments in Hardware-in-the-Loop Testing,” in Model Validation and Uncertainty Quantification, Volume 3, Cham, 2019, pp. 65–73. doi: 10.1007/978-3-319-74793-4_10.

[20] “Development + Priorities Narrative,” Google Docs. https://docs.google.com/presentation/d/1YH4BHfyPbEUv1HSPqm5NVupav2ErisjSWUZxQ3JzW2g/embed?start=false&loop=false&delayms=3000&usp=embed_facebook (accessed Sep. 04, 2022).

[21] “Hardware Development and the Agile Manifesto | LinkedIn.” https://www.linkedin.com/pulse/20141201172054-5203931-hardware-development-and-the-agile-manifesto/ (accessed Sep. 04, 2022).

[22] “A Method Is Not A Methodology,” AgileSoC, Jul. 06, 2011. http://agilesoc.com/articles/a-method-is-not-a-methodology/ (accessed Sep. 04, 2022).

[23] “Hardware Testing during Early Development Phases by +wikifactory,” Wikifactory. https://wikifactory.com/+wikifactory/stories/hardware-testing-during-early-development-phases (accessed Sep. 01, 2022).

[24] “Test Environment for Software Testing.” https://www.tutorialspoint.com/test-environment-for-software-testing (accessed Sep. 01, 2022).

[25] “Wikispeed - Doing Awesome with Agile,” InfoQ. https://www.infoq.com/news/2012/07/wikispeed-wellington/ (accessed Sep. 01, 2022).

[26] “WikiSpeed - P2P Foundation.” https://wiki.p2pfoundation.net/WikiSpeed (accessed Sep. 01, 2022).

[27] S. Jiang, Y. Ou, P. Pan, K. Cheng, Y. Zhang, and C. Batten, “PyH2: Using PyMTL3 to Create Productive and Open-Source Hardware Testing Methodologies,” IEEE Design & Test, vol. 38, no. 2, pp. 53–61, Apr. 2021, doi: 10.1109/MDAT.2020.3024144.

[28] S. McFarlane, “Hardware Testing Process – How to test products during production,” Viewpoint Systems. https://www.viewpointusa.com/TM/ar/hardware-testing-process/ (accessed Sep. 01, 2022).

[29] “Hardware Verification, Testing and Maintenance —.” https://aceproject.org/main/english/et/ete05a.htm (accessed Sep. 01, 2022).

[30] “Hardware Testing 101 | Product Development,” Fictiv. https://www.fictiv.com/articles/hardware-testing-101 (accessed Sep. 01, 2022).

## Thanks to
This lesson is made possible by:
Jose Urra, Jerry de Vos, Andjela Tomic, Santosh Ilhamparuthi

## Contact

Link to [Academy website](https://www.openhardware.academy/)