A Tool for Risk-Based Testing

1. A Tool for Risk-Based Testing Linda Sherrell, Sarah Bowen, and Hima Puppala

2. Importance of Risk Management • Increased interest in industry • Key process area of CMMI • Typical methods • Majority emphasize final product • Most of risk assessment occurs during requirements • Some analysts feel this is too narrow a view

3. Risk Management Process • Risk identification – uncovering of risks • Risk analysis – quantification of risks • Risk planning – contingency and mitigation plans • Risk monitoring – continuous assessment and adjustment of risk mitigation plans

4. Risk Identification Techniques • Taxonomies • Decision-driver analysis • Assumption analysis • Decomposition • Generic risk lists • Risk catalogs • Quality criteria categories

5. Risk Analysis • Two major techniques • Cause-effect analysis • Risk Exposure • Risk Exposure = Probability * Impact • Prioritization of risks based on risk exposure or individual values • Probability and impact values may be either quantitative or qualitative

6. Heuristic Risk Based Testing • Inside-out approach • Begin with situation details • Then perform risk identification • Outside-in approach • Use predefined risk list • React to risks in current circumstance

7. Project Goals To provide an easy-to-use tool to support heuristic risk based testing To allow the testing team to compare current risk assessment values to previous data Tool Characteristics Supports the first two steps of risk management Provides users with generic lists for both product risks and process risks QUART-ET (Quick Assessment of Risks Tool for Engineering Testing)

8. Generic Risk Lists • High-level risk categories • Detailed risk factors • Example: Complexity • Adding a new feature/module that is highly complex • Modifying a new feature/module that is highly complex • Modifying a feature/module with a large number of lines of code

9. Failure history Internal dependency External dependency Complexity Distributed components General Data Unfamiliar Resources Global Environment/Configura-tion/System setup Product Risk Categories

10. Process Risk Categories • Requirements discipline • Modifying requirements late in process • Lack of testing personnel on requirements team • Design discipline • Modifying design late in development • Testing discipline • Lack of full regression testing • Lack of security testing • All disciplines • Lack of backward traceability • Noncompliance with company standards

11. Conclusions • QUART-ET can be used to foster a culture that supports risk management • The process supported by QUART-ET complements prioritization of testing items • Tool can be extended to handle additional risk management process steps

12. References • Bach, J. (1999, November). Heuristic Risk-Based Testing. Software Testing and Quality Engineering Magazine, pp. 1-9. Retrieved from http://www.satisfice.com. • Boehm B.W. (1991). Software Risk Management: principles and practices, IEEE Software, vol. 8, pp. 32-41. • Boehm, B. and R. Turner (2004). Balancing Agility and Discipline: A Guide for the Perplexed. Boston: Addison-Wesley. • Carr, M. J (1997). Counterpoint: Risk Management May Not Be for Everyone. IEEE Software, vol. 14,  no. 3,  pp. 21,23. • DeMarco, T. and T. Lister (2003). Waltzing with Bears: Managing Risk on Software Projects. New York: Dorset House.

13. References (cont.) • Gotterbarn, D. (2001). “Reducing Software Failures: Addressing the Ethical Risks of the Software Development Cycle”. In T.W. Bynum et al., eds. Proceedings of Ethicomp 2001: The Fifth International Conference on the Social and Ethical Impacts of Information and Communication Technology. Vol 2. Gdansk, Poland: Wydawnictwo Mikom, pp. 9-21. • Hall, E. (1998). Managing Risk: Methods for Software Systems Development. Reading, MA: Addison-Wesley. • Lister, T. (1997). Point: Risk Management Is Project Management for Adults, IEEE Software, vol. 14,  no. 3,  pp. 20-22. • Ould, M. A. (1999). Managing Software Quality and Business Risk. Chichester: John Wiley & Sons.