Testing

1. Testing Course notes for CEN 4010

2. Outline • Introduction: • terminology and philosophy • Factors that influence testing • Testing techniques

3. Why Do We Test? • We test to find bugs • “Testing is the process of executing a program with the intent of finding errors” • A good test intends to find bugs • “A good test case is one that has a high probability of detecting an as-yet undiscovered error” • A test that finds no bug is a failure • “A successful test case is one that detects an as-yet undiscovered error”

4. Testing vs. Debugging • Testing != Debugging • While testing may reveal a symptom of an error, it may not uncover the exact cause of the error • Debugging is the process of locating the exact cause of an error, and removing that cause

5. Our Very Real Problem • We can never say for an arbitrary program: • This software has no errors • This software works only as intended • This software is safe Testing proves the presence, not the absence, of bugs Absence of evidence is not evidence of absence

6. Outputs (i.e. Results) Valid Invalid Good Goal BUG!!! Valid (includes exceptions) Inputs Should always Invalid produce exceptions! 4 dimensions of the Testing

7. Where Does Testing Fit?

8. When to test ?

9. What other Tests Are There?

10. Test activities: Analysis • Test the Models against the Problem specification • “Play computer” through the models • Analysis models must accurately describe the problem to be solved, and the boundaries of the problem domain

11. Test activities: Design • Test the Models against the Solution domain. • “Play computer” through the models • Incorporating all required “design” classes • GUI frameworks • Third-party software and class libraries • Collection/Container classes • Operating environment classes (wrappers) • Interface classes to external resources. E.g., • RDBMS, • network, • communications.

12. Test Metrics • Traditional Metrics: • No. of bugs per 10,000 Source Lines of Code • IEEE Standard 982.1 • McCabe Complexity Measure • Halstead Software Science Measures • …

13. Test Techniques: • The “Best of all Possible Worlds” technique: • Run it...and make sure it doesn’t crash, then ship it! • For the rest of us: • Desk Checking • Inspections &Walkthroughs • Black-box/White-box testing • Boundary testing • Path testing • Code coverage • Error message testing • Top-down & Bottom-up • …

14. Desk Checking • Simple, Labor-intensive • Procedure • Developer has his/her design or code manually reviewed by another developer • “a second pair of eyes” • Developer must justify decisions to reviewer. • Developer reviews/accepts/rejects reviewer’s recommendations.

15. Inspections & Walkthroughs • Formal or informal as needed. • Well-known technique. • Procedure: • Group gather to review artifacts (docs to code). • Group composed of “role” representatives (e.g. “user”, “QA”, “Standards”, etc.). • Rules of conduct apply to participants • Responsibility for verification and validation of artifact is removed from the developer.

16. Inspections • Author narrates, statement by statement, the logic of his program. • Reviewers listen to the author, raise questions and ask for clarification in their attempt to uncover errors. • Ironically, most of the errors are discovered by the author as he/she “teaches” the program to the group.

17. Walkthroughs • Programmer does not narrate to the group. • One of the group plays the role of “tester”. • Tester comes to review armed with a small set of simple test cases to apply to the program. • Group “play computer” by performing manual simulation of the system using the test data. • Data are important as a vehicle to stimulate discussion. • Most errors are uncovered by questioning of the author rather than “execution” of data.

18. Inspections & Walkthroughs • When should I perform one? • Anytime, so long as the product under review is complete (i.e. tangible, understandable, and objective). • After each milestone with delivered artifacts (e.g. Requirements Spec., Analysis Models, Design Spec., pseudo-code or flowcharts prepared, program code written, etc.). • Yourdon’s milestones [YOUR79]: • after the design artifacts are completed • after the code is prepared but not compiled • after the first compilation • after the first “clean” compilation • after the first test data set have been executed successfully • after the programmer thinks all test cases have been executed successfully

19. Inspections & Walkthroughs • Who should participate? • Author/developer/presenter/producer • Moderator/coordinator • Scribe/secretary • Standards Bearer • Maintenance Expert • User Representative • GUI Expert

20. White-Box Testing • White-box testing is the testing of the underlying implementation of a piece of software (e.g., source code) without regard to the specification (external description) for that piece of software. • The goal of white-box testing of source code is to identify such items as • (unintentional) infinite loops, • paths through the code which should be allowed, but which cannot be executed, and • dead (unreachable) code.

21. Black-Box Testing • Black-box testing is the testing of a piece of software without regard to its underlying implementation. • Specifically, it dictates that test cases for a piece of software are to be generated based solely on an examination of the specification (external description) for that piece of software. • The goal of black-box testing is to demonstrate that the software being tested does not adhere to its external specification. (Note that if there is no "external specification" it will be difficult to conduct black-box testing.)

22. Black Box Tests Boundary-value Analysis Equivalence Partitioning Cause-Effect Graphing Error Guessing Error Message Generationfrom [MYER79] White Box Tests Statement Coverage Decision/Condition Coverage Multiple-Condition Coverage Test Categories

23. Black Box Tests • Boundary-value Analysis • Uses test cases “generated on, and immediately around, the boundaries of the input an output for a given piece of software.” • Equivalence Partitioning • Divide test sets into equivalence partitions: “collections of items which can all be regarded as identical at a given level of abstraction. I.e., a set of data items which will all evoke the same general behavior from a given software module.” • Cause-Effect Graphing • Generates test sets based on input combinations, specified by a combinatorial logic network or Boolean relations, and the expected outputs from those input combinations.

24. Black Box Tests • Error Guessing • An intuitive and empirical technique of selecting test cases which elicit failures. • A.k.a. “a gift”. • Error Message Generation • Selecting test cases which will elicit the error messages defined in the software under test.

25. White Box Tests • Statement Coverage • a.k.a. “basis path testing” • Use test cases which cause each executable statement to be executed at least once. • Decision/Condition Coverage • All statements are executed at least once, and all binary decisions have a TRUE and FALSE outcome at least once, all exceptions are raised at least once, and all possible interrupts are forced to occur at least once.

26. White Box Tests • Multiple-Condition Coverage • Use test cases such that all possible combinations of condition outcomes in each decision, and all points of entry, are invoked at least once.

27. Path Testing • Why can’t we just test all possible paths? • The flowchart on the right has approximately 100 trillion possible paths that may be executed!!! • At 1ms/test, working 24 hrs/day, 365 days/year: it will take 3170 years to test this simple structure.