1 / 34

Background

A Systematic Approach for Increasing the ROI of Software Test Automation Vahid Garousi, Associate Professor Senior Software Consultant. Background. Many test teams adapt software test automation But unfortunately, often times, the automation experience ends with negative outcomes

sheila
Download Presentation

Background

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Systematic Approach for Increasing the ROI of Software Test AutomationVahid Garousi, Associate ProfessorSenior Software Consultant

  2. Background • Many test teams adapt software test automation • But unfortunately, often times, the automation experience ends with negative outcomes • e.g., it becomes costly to maintain automated test suites, in parallel to changes in the production code • Excited then disappointed…! • Return on Investment (ROI) from test automation • Reason: following improper test automation strategies • Result: In such cases, project managers decide to ignore automated test suites in their entirety and decide to not use test automation again.

  3. Objective • Goal: to systematically answer the following questions: • When to automate (test cases)? • What to automate? (test cases, methods, class, etc.) • So that the ROI of test automation is increased

  4. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved • Q/A

  5. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved • Q/A

  6. Overview of automation across the software test process • Test automation does NOT mean 100% automation in all testing tasks! • Let’s review the different tasks in SW testing:

  7. Automation in Test-case Design • Test-case Design (Criteria-based): • Design test input values to satisfy coverage criteria (e.g., line coverage, or requirements coverage/traceability) • Usually done manually, but can be automated (combinatorial test tools) • Test-case Design (Human knowledge-based): • Design input test values based on domain knowledge of the program and human knowledge of testing • Also called exploratory testing • Almost always done manually. We can hardly develop a program to document domain knowledge (example: power engineers)

  8. Automation in Test Scripting • Test Scripting: • Writing the test cases either for manual or automated test execution • Can be done manually or automated • Do you remember (have!) thick documents of manual test scripts (hard or soft copy) • Can we have tools to auto generate automated test scripts?! (yes, there are some!)

  9. Automation in Test Execution • Test Execution: • The common interpretation of “automated testing”! • Run tests on the software and record the results • Can be done manually or automated

  10. Automation in Test Evaluation and Result Reporting • Test Evaluation (oracle): • Evaluate results of testing (pass/fail), a.k.a. test verdict, and report results to developers • Again, can be done manually or automated • Test Result Reporting: • Again, can be done manually or automated

  11. The big picture! • There are often inter-dependencies among the following decisions Leads to… Leads to…

  12. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved

  13. Successful Test Automation w.r.t. ROI • Classical chart on ROI: Upfront cost of automation (development of automated test suite, etc). Only if the decision to automate (and how much of it) has been made properly, then we will see this cost saving! Payoff point (“sweat spot”!)

  14. The “Ugly Side” of Test Automation • To rush into automation just for the sake of it… Upfront cost of automation (development of automated test suite, etc). • No payoff point ?? Costs to maintain the test suite is growing, since an IMPROPER test automation strategy was followed Project manager: Let’s throw out the automated test suites and never use test automation!

  15. ROI Experiences in Test Automation: Comparison • “The Good, the Bad and the Ugly”

  16. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved

  17. When is test automation has the highest ROI? (i.e., “worth it” in terms of its business value) • Carefully select which of the following activities should be automated (and how much, it is not Boolean!, but rather a fuzzy decision), for which parts of the system, and which test activities should stay manual… so that the sum of all testing costs are the least expensive possible in the entire SW life-cycle • i.e., choosing the “numbers” (knobs) on the following diagram • We are not advocating 100% pure end-to-end automation! • One good decision for a hypothetical team: 80% 20% 20% 60% 40% 80%

  18. Best Mix of Manual/Automated Testing • We need both, but how much of each for each type of testing (acceptance, performance, etc.), system-level use-case, class, component, sub-system, etc. (many levels of granularity) • A kind of optimization problem… • Example test activity: test scripting … (development) of manual / automated test suites • Let’s start with some heuristics: • Volatility factor: If we expect a lot of change for a part of system (class, use-case, etc.), and thus the need for re-testing, we should thinking of automated testing for it • Cost factor: If manual testing of a part of system is COSTLY, then automate it.

  19. Choosing when and what to automate • Reminder: Carefully select • whichof the following activities should be automated (and how much, it is not Boolean!), • for which part of the system, • and which activities should stay manual… • so that the testing activities are the least expensive possible in the “long-run” Cost of manual testing: Low Expected volatility: Low Cost of manual testing: High Expected volatility: High

  20. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved

  21. Our experience • Our experience in measuring ROI and successful test automation • A selected project: • Automated Unit and Functional Testing of a SCADA Software (2009-2012) • SCADA: Supervisory Control and Data Acquisition system

  22. Automated unit and functional testing of an in-house developed SCADA software • Software under test: A commercial large-scale Supervisory Control and Data Acquisition (SCADA) software system • Named Rocket, developed by based in Calgary, Canada • Has been developed using Microsoft Visual Studio C# • Has now been deployed in several locations across Canada and the US

  23. Black-box Unit Testing (BBUT): Challenges • If we apply the equivalence classing, we will get 19,683 test cases for only this function block. Bad news ;( • Challenge 1: Coding of test cases (in NUnit): Too much effort • Challenge 2: Coupling of test cases to test input data • Challenge 3: Manual generation of expected outputs (test oracle)

  24. Black-box Unit Testing (BBUT): Challenges • One possible solution → Automated generation of NUnit test code • There are some tools out there: • Microsoft Pex, JML-JUnit, JUB (JUnit test case Builder), TestGen4J, JCrasher, NModel • After evaluating them for our purpose, unfortunately none was suitable (details in our article) • Decision: to implement our own test tool!

  25. AutoBBUT - Example UsageReminder: Test code is automatically generated, saving many hours

  26. Effectiveness in Detecting Defects Most of the defects were obviously not caught by manual testing conducted by various developers and users during the development process. There were mostly on the boundary value conditions, overflow cases and invalid inputs, denoting the importance of robustness testing in such applications.

  27. Return On Investment (ROI) of Test Automation • Cost and benefit drivers in the project:

  28. In-depth Analysis of the ROI of Test Automation • Careful measurements were conducted • Summary: We could get a time saving of: ROI = -120 hours (AutoBBUT’s development time) -3 hours (test code inspection and completion) +87 hours (initial development of manual test suite, if it was to be done) +87*6 hours (test code maintenance, since the system evolved 6 times) ROI = 486 hours = 60 man-days • More details in: our recent article in the IEEE International Conference on Software Testing, Verification and Validation (ICST), Industry Track, April 2012

  29. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved

  30. End-to-end Activities of Manual/Automated Testing and the Costs Involved

  31. End-to-end Activities of Manual/Automated Testing and the Costs Involved

  32. End-to-end Activities of Manual/Automated Testing and the Costs Involved

  33. End-to-end Activities of Manual/Automated Testing and the Costs Involved To increase the ROI of test automation, we need to systematically decrease the costs in each activity, e.g.: Benefiting from “test patterns” in development of test code to increase its maintainability

  34. Outline of the Presentation • A brief overview of automation across the software test process • Successful vs. unsuccessful Test Automation • When is test automation has the highest ROI? • Choosing when and what to automate • Experience from one of our projects in measuring ROI and Successful Test Automation • End-to-end Activities of Manual/Automated Testing and the Costs Involved • Q/A

More Related