NTU Software Testing Notes

# NTU Software Testing Notes ###### tags: `NTU_ST` `Software Testing` **All content in this presentation is refer to [Pro. Farn Wang Website](http://cc.ee.ntu.edu.tw/~farn/courses/ST/2021.Spring/)** :::spoiler Click to open TOC [TOC] ::: ## 6.1 Regression Testing * Definition > The process of re-testing software that has been modified > 重複執行既有的全部或部分的相同測試 - by [Esther](https://medium.com/@esther.tsai/%E5%9B%9E%E6%AD%B8%E6%B8%AC%E8%A9%A6-regression-testing-35d69b996481) * Note that: > Most of our testing effort is regression testing > Regression tests must be automated * Type of tools * Capture / Replay: Capture values entered into a GUI and replay those values on new versions(抓取輸入到GUI的value並replay到新的版本) * Version control: 追踪測試集合、預期結果、測試來源、使用的標準及其過去的有效性(Keeps track of collections of tests, expected results, where the tests came from, the criterion used, and their past effectiveness) * Scripting software: 管理以下流程，包含獲取測試輸入、執行軟體、獲取輸出、比較結果和生成測試報告等等(Manages the process of obtaining test inputs, executing the software, obtaining the outputs, comparing the results, and generating test reports > Tools are plentiful and cheap * Managing Tests in a Regression Suite * Test suite會不斷累加新的test * Test suites通常在晚上執行(固定、一段很短的時間) * Policies for Updating Test Suites * 要保留哪些測試? - 確保始終滿足覆蓋標準(**coverage criterions**) - Add a new test for every problem report * 要刪除那些測試? - 無法滿足覆蓋率的測試 - 從未發現故障的測試（**Risky!**） - 與其他測試發現相同錯誤的測試（**Risky!**） * Reordering strategies - If a suite of N tests satisfies a coverage criterion, the tests can often be reordered so that the first **N-x** tests satisfies the criterion – so the remaining tests can be removed * When a Regression Test Fails * Regression tests are evaluated based on whether the test result on the new program `P` is equivalent to the test result on the previous version `P-1`(若不同則代表fail) * Fails represent 3 possibilities * The software **has a fault** – Must fix the fix * The test values are **no longer valid** on the new version – Must delete or modify the test(測試端無效可能是待測程式也有可能是new version) * The expected output is no longer valid – Must update the test(輸出端無效可能是發現新的問題要測試) * Evolving Tests Over Time * Changes to **external interfaces** can sometimes cause all tests to fail * Modern capture / replay tools **will not be fooled by trivial changes** like color, format, and placement * Automated scripts can be changed automatically via global changes in an editor or by another script(一些global configuration之類的) * 但是小小的改動成本會慢慢累加 * Choosing Which Regression Tests to Run * Change Impact Analysis(CIA) - 一個小小的改動會對整體的測試造成多大的影響? * Strategy: | Strategy | Pros | Cons | | -------- | -------- | -------- | | Run all tests| More safety| High cost of time| | Based on priority/risk| 可以管控到重大風險的議題| Hard to implement and also need more skills on **evaluating risk**| | Based on the function(重要或常用)| The most effective way to improve system reliability on a budget| Hard to implement and also need more skills on evaluating **what is a important function**| | Run revise part| Trivial and small scale| Not comprehensive enough| | Selective repeat test (確認正確性和周邊是否受到影響)| Fast and effectively| More skills on analyze the effective part (需要較多技巧在分析上)|  ## 6.2 Integration and Testing ![unintegrate test](https://user-images.githubusercontent.com/88981/52933895-c0d47600-338f-11e9-9034-11e1ad0c42f1.gif) * Big Bang - Throw all the classes together, compile the whole program, and system test it - **Risky!** * Usually method: start small, with a few classes that have been tested thoroughly (從小地方開始) * Add a small number of new classes * Test the connections between the new classes and pre-integrated classes * Integration testing: testing interfaces between classes * Should have already been tested in isolation **(unit testing)** * Methods, Classes, Packages * Integration can be done at the method level, the class level, package level, or at higher levels of abstraction * We use the word **component** in a generic sense which is a piece of a program that can **be tested independently** * Integration testing is done in several ways * Evaluating two specific components * Testing integration aspects of the full system * Putting the system together “piece by piece” * Software Scaffolding(鷹架) 是為support integration和testing而創建的extra software components ![process of software scaffolding](https://imgur.com/5V4zxEw.png) * Stubs(票根/存根聯): 單純模擬尚未實踐或整合的method，被call的result * Drivers: 單純模擬make a call to 正在測試的component(CUT)的方法 * Stubs * The first responsibility of a stub is to allow the CUT(Component Under Test) to be compiled and linked without error * Approaches: 1.Return constant values from the stub 2.Return random values 3.Return values from a table lookup(回傳查表) 4.Return values entered by the tester during execution(回傳測試者輸入) 5.Processing formal specifications of the stubbed method * Note that 1 ↦ 5 is more costly / more effective * Drivers Many good programmers add drivers to every class as a matter of habit * Instantiate objects and carry out simple testing(實例化+簡單測試) * Criteria from previous chapters can be implemented in drivers * Test drivers can easily be created automatically * Values can be hard-coded or read from files * Class Integration and Test Order (CITO) * Which order to integrate was pretty easy: * Test the “leaves” of the call tree(測試leaves) * Integrate up to the root(整合到root) * **Goal is to minimize the number of stubs needed** * OO會使其更複雜 - dependencies (call, inheritance, use, aggregation) or circular dependencies : A inherits from B, B uses C, C aggregates A... * Which order should we integrate and test? * Must "break cycles” * Common goal: least stubbing ## Reference * [回歸測試 regression testing](https://medium.com/@esther.tsai/回歸測試-regression-testing-35d69b996481) * [一次搞懂單元測試、整合測試、端對端測試之間的差異](https://blog.miniasp.com/post/2019/02/18/Unit-testing-Integration-testing-e2e-testing)