1 / 35

Model Based Testing

Model Based Testing. Texas A&M University Department of Computer Science Sameh S. Sharkawi. Outline. Introduction What is MBT Motivation behind MBT How MBT Works Benefits of MBT Problems of MBT Current State of MBT in Industry Software Tools for MBT Conclusion References.

teenie
Download Presentation

Model Based Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Model Based Testing Texas A&M University Department of Computer Science SamehS. Sharkawi

  2. Outline • Introduction • What is MBT • Motivation behind MBT • How MBT Works • Benefits of MBT • Problems of MBT • Current State of MBT in Industry • Software Tools for MBT • Conclusion • References

  3. Introduction (What is MBT) • Definition “Model-based testing refers to software testing where test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT) .” * * Wikipedia Definition of Model Based Testing.

  4. Introduction (What is MBT Contd.) • MBT is an approach in which you define the behaviorof a system in terms of actions that change the state of the system. • Such a model of the system results in a well-defined Finite State Machine (FSM) which helps us to understand and predict the system’s behavior

  5. Introduction (Motivation behind MBT) • Testing is expensive • 30-50% of development costs • Poor testing is VERY expensive • Down time • Maintenance costs • Rework • Law suits • Model Based Testing promises • Increased effectiveness of testing • Similar or decreased costs • Reuse of design artifacts

  6. Introduction (Motivation behind MBT Downtime Costs) • Brokerage operations $6,450,000 • Credit card authorization $2,600,000 • Ebay (1 outage 22 hours) $225,000 • Amazon.com $180,000 • Package shipping services $150,000 • Home shopping channel $113,000 • Catalog sales center $90,000 • Airline reservation center $89,000 • Cellular service activation $41,000 • On-line network fees $25,000 • ATM service fees $14,000

  7. How MBT Works

  8. How MBT Works? • The model is usually an abstract, partial presentation of the system under test's desired behavior. • The test cases derived from this model are functional tests on the same level of abstraction as the model. • These test cases are collectively known as the abstract test suite

  9. How MBT Works? Contd. • The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. • Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. • This is done by mapping the abstract test cases to concrete test cases suitable for execution.

  10. How MBT Works? Contd. • Deriving Test Cases Algorithmically • Test Case Generation by Theorem Proving • Test Case Generation by Constraint Logic Programming • Test Case Generation by Model Checking • Test Case Generation by Symbolic Execution

  11. How MBT Works? Contd. Spec Explorer (UW-MSR Summer Institute '04) Model Provides expected results for Generates PassFail Test Cases Test Oracle Provides actual results for Run Implementation

  12. Benefits of MBT

  13. Benefits of MBT • Starting from specification • Involves testers early in the development process • Teams testers with developers • Forces testability into product design • Building the test interface • Finds design and specification bugs - before code exists • The model is the test plan - and is easily maintained • Automated test suite generation • Coverage is guaranteed - increases testing thoroughness • Zero test suite maintenance costs • Automated test suite execution • Finds code and interface bugs • Includes a framework for the testing of distributed applications • Reduces test execution costs

  14. Benefits of MBT Contd. • These benefits can be summarized as: • Machine • Time

  15. Benefits of MBT Contd. • Machines • A typical test engineer • Earns €41 500 ($50 000 USD) per year • Works 40 to 50 hours per week • A typical test machine • Costs €250 ($300 USD) per year to buy and operate • Can work 100 hours per week

  16. Benefits of MBT Contd. react.cs.uni-sb.de/mbt2006/talks/mbt4masses.pdf

  17. Problems of MBT

  18. Problems of MBT

  19. Problems of MBT Contd.

  20. Problems of MBT Contd.

  21. Problems of MBT Contd.

  22. Problems of MBT Contd. • Process shift • Up front investment in test • Personnel shift • Higher education and sophistication Spec Explorer (UW-MSR Summer Institute '04)

  23. Current State of MBT in Industry

  24. Current State of MBT in Industry • IBM Research • Most projects were done by PhD Holders • Very Little was successful (Just in the Lab, never actually went into market) • Too Complicated to use and analyze • Microsoft Research • The Spec# programming system is a new attempt at a more cost effective way to develop and maintain high-quality software

  25. Current State of MBT in Industry • Spec# is a computer language which includes and extends C# • Adds pre/post conditions, contracts • Adds high-level data types with convenient notations • Adds logical quantifiers like FORALL and EXISTS • A Spec# model is just a program! • You can run it like a C# program • It can call framework code • NEWYou can explore it (run all of its possible behavior) which is how we do MBT with Spec# Spec Explorer (UW-MSR Summer Institute '04)

  26. Current State of MBT in Industry (Spec Explorer) • A model exploration and testing tool for .NET: • Authoring of models in Word • Model exploration (generating FSM) • FSM visualization • FSM traversal and test suite generation • Automatic implementation binding • Online test-suite execution and offline test-suite code generation Spec Explorer (UW-MSR Summer Institute '04)

  27. Current State of MBT in Industry (Spec #) • Model: Calculator (viewpoint: only starting and stopping, and switching scientific mode) bool Running = false; bool Scientific = false; [Action] void SetRunning(bool newRunning) requires Running != newRunning; { Running = newRunning; } [Action] void SetScientific(bool newScientific) requires Running; requires Scientific != newScientific; { Scientific = newScientific; } System State Spec#: Pre-Condition describes when action is enabled State Update Tells Spec Explorerthis is an action. Action invocations do appear in tests. Spec Explorer (UW-MSR Summer Institute '04)

  28. Software Tools for MBT

  29. Software Tools for MBT • Conformiq Test Generator is a model-based testing tool using UML state charts that represent testing strategies • Leirios Test Generator is a model-based testing tool that generates tests automatically from deterministic system specifications • Reactis Tester is another model-based testing tool that focuses on control systems • TGV is a tool for the generation of conformance test suites for protocols • TorX is also a prototype testing tool for conformance testing of reactive software • Lurette is an automated testing tool of reactive programs written in Lustre • AsmL Test Tool can generate tests directly from an AsmL model • AutoFocus (in german) is a graphical tool for developing and modeling distributed systems with integrated testing facilities

  30. Software Tools for MBT (AsmL) • AsmL is the Abstract State Machine Language. It is an executable specification language based on the theory of Abstract State Machines. The current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. It is fully interoperable with other .NET languages. AsmL generates .NET assemblies which can either be executed from the command line, linked with other .NET assemblies, or packaged as COM components.

  31. Conclusion

  32. Conclusion • Modeling is not easy • What details do you leave out? • How do you check the model? • There are no recipes for success • What needs to happen • Easier notation • Incremental approach • Pilots and examples

  33. Questions • How do we guarantee that the generated expected outputs and inputs are enough and cover all the cases? • How to make for the errors and exceptions that may occur due to running the program on different environments and different OSs? • How is automated generation of inputs and outputs work when an actual human interaction is required as part of the system testing? • Does MBT cover stress testing? • Does MBT cover how user friendly the system is or it only covers functionality?

  34. References

  35. References • http://en.wikipedia.org/wiki/Model-based_testing • http://blogs.msdn.com/nihitk/articles/144664.aspx • http://research.microsoft.com/specsharp/ • http://react.cs.uni-sb.de/mbt2006/talks/mbt4masses.pdf • http://react.cs.uni-sb.de/mbt2006/talks/ModelBasedTestingSoberEvaluation.pdf • http://aetgweb.argreenhouse.com/papers/1999-icse.pdf • Colin Campbell, Wolfgang Grieskamp, Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes “Spec Explorer:An Integrated Environment for Model-Based Testing.” Foundations of Software Engineering Microsoft Research, Redmond

More Related