Model-based Statistical Testing

1. Risky Business Model-based Statistical Testing Mathias Winder

2. Content • An Usage Model based on UML-Statemachines • Risk-based Modeling • The Idea of Risk-based Testing • Risk- and Model-based Testing • Graph-oriented Testcase Generation • UML-Statemachines and Graphs • Markov Chain Usage Models • Generating Transition Probabilities • Risk-based Priorization

3. An Usage Model based on UML-Statemachines • UML-Statemachines are very powerful  Difficult to automate  Reduce number of different elements • Subset with enough elements to define (semantically correct) Usage Models: • States: • DoActivity define activities processed while state is active • Composite States and Submachine States contain (or refer) other Regions • Final States defines the end of the usage • Pseudostates: initial, junction, choice • Transitions: • Trigger: an event (e.g. Mouse Click) firing the transition • Guard: a condition, that the transition can be fired • Effect: a behavior caused by the transition

4. Example: Alarm System black box model for an alarm system doActivity refers an function which checks some behavior in the black box help functions: CheckStatus(status) CheckFlashAndSound(flash, sound) help variables: $Status („Armed“/“Unarmed“) $Flash (true/false) $Sound (true/false)

5. Content • An Usage Model based on UML-Statemachines • Risk-based Modeling • The Idea of Risk-based Testing • Risk- and Model-based Testing • Graph-oriented Testcase Generation • UML-Statemachines and Graphs • Markov Chain Usage Models • Generating Transition Probabilities • Risk-based Priorization

6. The Idea of Risk-based Testing • Large Scaled Systems are very complex and normally there aren`t enough resources to test every feature of the System in detail • Some features are more important than others: • more complex implementation  probability of failure • damage, if the feature fails: • for costumer (use of the software, costs, ...) • for company (maintainance, ...) • Idea: Use metrics to estimate the Risk of the features and focus on more risky areas while testing • e.g., risk = (probability of failure) * damage

7. Risk- and Model-based Testing • Annotated Risks on Usage Models can be used to achieve Testcases focused on risky areas • Goal: • Find the most interesting Paths/Testcases • A Risk raises the interest, but a Risk get`s less interesting if it`s repeated more often (How to estimate interest?) • Standard Workflow: • Create Usage Model and assess Risks for important features (for single transitions or whole regions) • Generate Testcases (e.g. using Random Walk) • Estimate and priorize Testcases based on Risks

8. Risk- and Model-based Testing (2) whole region single transitions

9. Content • An Usage Model based on UML-Statemachines • Risk-based Modeling • The Idea of Risk-based Testing • Risk- and Model-based Testing • Graph-oriented Testcase Generation • UML-Statemachines and Graphs • Markov Chain Usage Models • Generating Transition Probabilities • Risk-based Priorization

10. Graph-oriented Testcase Generation • Idea: Use well-known graph-algorithms to derive Testcases from Usage Models: • coverage-based algorithms (e.g. Chinese Postman) • search-based algorithms (e.g. A*) • random-based algorithms (e.g. RandomWalk) • But: • Usage Model have to be interpreted as Graph • Usage Model are normally not static (there are conditions like Guards/Precondition/...) •  Adaption of algorithms

11. UML-Statemachines and Graphs • UML-Statemachines aren`t graphs: • Subregions (Submachine and Composite States) • Guards, which can change during usage/generation • Transformation of Statemachines: • Adaption of Graph-based Algorithms

12. Markov Chain Usage Models • Annotate transitions (edges) with probabilities and use them as heuristic to optimize the Testcase-Generation: • Approach 1: use probabilities to define relative frequency of different paths • Approach 2: use probabilities to achieve a „fair“ (uniform) distribution (every path should have the same probability)

13. Markov Chain Usage Models (2) • Fair Transition Probabilities: • Without Probabilities: • e.g. 0,1,3,0,2,4,0,1,3,0,2,5 (transition (1,3) is preferred) • With Probabilities: • e.g. 0,2,4,0,1,3,0,2,5 (uniform distribution) • Probabilities can be generated automatically

14. Generating Transition Probabilities • Basic Approach: • Generate a path using the Chinese Postman Problem (with a fictitious transition between final and intial state) • Count the use of every Node n: N[n] = #n in path • Count the use of every Transition t: T[t] = #t in path • Probability of Transition t: P[t] = T[t] / N[t.Source] • E.g. (example from previous slide): • Path: initial,0,2,4,final,initial,0,1,3,final,initial,0,2,5 • Probability for transition t(0,2): • P[t(0,2)] = T[t(0,2)] / N[0] = 2/3 = 0.67 • Problem: Chinese Postman is NP-Complete •  We need good approximations

15. Generating Transition Probabilities (2) • Other Solutions: • Use Approximation for Chinese Postman (result depends on the quality of the approximation) • Chow`s Algorithm: • build a test tree doing a breath-first traversal (stop traversal if reaching final state or if transition is already traversed) • count N[n] and T[t] by traversing tree from root to leaves • e.g. [0,1,3], [0,2,4], [0,2,5] • P[t] = T[t] / N[n] • e.g. P[(0,2)] = 2 / 3 = 0.67 • ...

16. Generate Transition Probabilities (3) • Adopting CP Solution on Alarm System Example: 0.91 0.09 0.31 0.31 0.52 0.52 0.39 0.39 0.09 0.09 0.09 0.10 0.21 0.09 0.39 0.52 0.60 0.09 0.09 0.46 0.29 0.45 0.09 0.31 0.60

17. Content • An Usage Model based on UML-Statemachines • Risk-based Modeling • The Idea of Risk-based Testing • Risk- and Model-based Testing • Graph-oriented Testcase Generation • UML-Statemachines and Graphs • Markov Chain Usage Models • Generating Transition Probabilities • Risk-based Priorization

18. Risk-based Priorization • Assume we have a (big) list of generated testcases: • Which testcase should be executed first? (depending to ressources we maybe cannot execute all) • Priorize testcases in the list to decide which testcases are executed first • Static Risk-based Priorization: • For each Testcase tc: Calculate the Risk Exposure RE which is the sum of Risks achieved by the Testcase • (RE[tc] = Sum(R in tc)) • Order the Testcases by RE

19. Risk-based Priorization (2) • Static Risk-based Priorization on Alarm System Example: • Example Testcase/Path (red marked): • RE = R(„2s elapsed“) + Risk(Alarm) = 5 + 5 = 10

20. Risk-based Priorization (3) • Dynamic Risk-based Priorization: • Given: list of testcases L • Create Empty List P • Use a Heuristic, which assess the interest of testing a feature depending on risk and the count, how often the Risk was used in prior testcases: • e.g. Interest(R) = R * 0.5^PriorUsingsOfR • While list of testcases is not empty: • Find testcase TC in L with the hightest interest (Sum of Interest) • Remove TC from L and add TC to P • P is the Priorized List

21. Short Case Study • Alarm System with mutant in risky area: • mutant: sound does not turn off automatically • Chinese Postman for Transition Probabilities • Testcases generated by Random Walk (using Probabilities as Heuristic) and priorized with dynamic Priorization (Interest(R) = R * 0.5^PriorUsingsOfR) • Mutant found by priorized testcases 1,2 and 9 (Total: 15)

22. References • [1] A. Feliachi and H. Le Guen, „Generating transition probabilities for automatic model-based test generation“ (2010) • [2] H. Stallbaum, A. Metzger and K. Pohl, “An Automated Technique for Risk-based Test Case Generation and Prioritization” (2008) • [3] Y. Chen and R. Probert, „A Risk-based Regression Test Selection Strategy” (2003) • [4] S. Amland, „Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application” (2000) • [5] J. Whittaker and M. Thomason, „A Markov chain model for statistical software testing” (1994)