1 / 21

PVR Murthy Andreas Ulrich Siemens AG, Corporate Technology

Essential Criteria on MBT to Ensure Quality of Software in Industry . PVR Murthy Andreas Ulrich Siemens AG, Corporate Technology. PVR Murthy Andreas Ulrich Siemens. Agenda. Model-based testing (MBT) Case study

olwen
Download Presentation

PVR Murthy Andreas Ulrich Siemens AG, Corporate Technology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Essential Criteria on MBT to Ensure Quality of Software in Industry PVR Murthy Andreas Ulrich Siemens AG, Corporate Technology PVR Murthy Andreas Ulrich Siemens

  2. Agenda • Model-based testing (MBT) • Case study • Effective criteria for evaluation of MBT tools • Analysis and discussions • Summary

  3. Test Suite Test Suite Executable Scripts Executable Scripts Introduction to Model Based Testing Modeling Generating Executing Analyzing Results Coverage Algorithm Generate Test Cases Mental Model Create Executable Requirement or Design Documentation and/or Generate Executable Oracles • Decide whether to • Generate more tests • Modify the model • Stop testing • Estimate • Reliability & other • quality measures Test pass or Fail Verify Results Run Scripts Execute Manually Application Under Testing

  4. Model-Based Testing • Model-based testing (MBT) is an evolving test generation technique. • Uses design artifacts (models) of a system under test. • Generate test cases automatically. • Suitable for adequateand thorough testing of critical and complex software.

  5. MBT tools under study • A number of MBT tools are available • Commercial tools, e.g. T-VEC, Qtronic, Spec Explorer etc. • Academic tools, e.g. NModel, TGV/CADP etc. • We selected two tools for our study • Conformiq’sQtronic V2.0 • Microsoft’s Spec Explorer2010 V3.0 • They represent two different modeling paradigms • Qtronic uses both textual and graphical notations while describing the system model (based on hierarchical state charts). • Spec Explorer uses guarded state update rules in C# (based on abstract state machines, called model programs) and a coordination specification.

  6. Conformiq’s Qtronic • Models are expressed as a collection of textual files and/or graphical models. • The textual notation is defined using QML • Graphical models are created using either Conformiq Modeler or a third party UML editor. • Generation of test cases based on state space analysis • Can translate test cases into executable test scripts

  7. Microsoft’s Spec Explorer • Uses Visual Studio programming environment to model system behavior. • Input is a set of .NET model assemblies and a coordination script. • Generates test cases based on exploration of the model. • Exploration systematically discovers all possible states and transitions. • Exploration can be visualized through a state space graph. • Test cases are a collection of C# files executable in MSTest

  8. We have considered an application from Siemens medical domain. Considered SUT:An automated exposure control software. Make large images with an automatic sequence of many small images Modeled as a UML state chart with 11 system states, 24 transitions and 72 transition paths. Digital radiographic system Case Study

  9. Evaluation Criteria • For a Siemens industrial project in the healthcare domain • High quality, reliable and robust software is indispensable • It therefore requires rigorous and effective testing • Effective and rigorous testing of critical and highly reliable software requires a good MBT tool / methodology. • We have identified eight important parameters for evaluating the considered MBT tools. • Based on the Siemens requirement in the domain of highly critical and reliable software. • Identified parameters are subjective.

  10. Criterion 1: Model Representation • Aspects considered • Type of model (tester or design model). • Kind of notation (standard or proprietary). • Ease of creation and editing (expressiveness, reusability, etc) • Modeling advanced features (hierarchy, concurrency). • Findings

  11. Criterion 2: Model Validation • Aspects Considered • Detecting requirement defects. • Consistency checking. • Checking design errors in the models. • Findings

  12. Criterion 3: Test Generation Strategy • Aspects considered • Tool support for choosing test coverage criteria. • Efficiency of test case generation algorithm. • Test generation control to avoid test case explosion • Findings

  13. Criterion 4: Test Data Generation • Aspects Considered • Automation in test data generation. • Accuracy, thoroughness, cost of testing. • Findings

  14. Criterion 5: Concurrency support • Aspects Considered • Ability to test concurrent system behavior. • Findings

  15. Criterion 6: Testing Levels • Aspects Considered • Unit, integration, or system testing. • Findings

  16. Criterion 7: Regression testing • Aspects Considered • Capability to generate test cases for regression testing. • Test case prioritization, Test case optimization • Findings

  17. Criterion 8: Usability • Aspects Considered • Initial learning curve, • Availability of documentation and on-line help. • User friendliness. • Findings

  18. Relevance of criteria for considered case study • Selection of relevant criteria for applying MBT in large industrial projects • Leads to an informed selection of the right MBT approach and tool • Highlights weak points of the selected tooling

  19. Outlook – Suggested improvements for future generation tools • Integration of different testing levels in a single framework. -- Unit, Integration testing and System testing • Need more rigorous automatic test case selection techniques. -- Sophisticated constraint specification over models • More sophisticated automatic test data generation. --Applying right combination of standard techniques for effective fault revealing data -- Handling primitive as well as structured data types. • Improved tool interoperability for different test artifacts. -- Migration from one environment to another

  20. Summary • In addition to the criteria mentioned already as a basis • for understanding MBT approaches for application in • industry, criteria for effective MBT may also include • fault modeling support for applications in different domains • model debugging support • guiding modelers towards incorporating safety and liveness properties of applications in models • facilitating expression of test design , test verdict and data concepts from UML Testing profile. • support for modeling non-functional requirements

  21. Thank You – Any Questions?

More Related