Verification and Validation in the Context of Domain-Specific Modelling

1. Verification and Validation in the Context of Domain-Specific Modelling Janne Merilinna

2. Conclusion(OOPSLA DSM’09 group work) • Sun was shining • The beer was good • Loud discussions • Which was good • However • To be continued in the next SPLASH (OOPSLA) DSM’10

3. Overview • Definition of Verification and Validation (V&V) • What to V&V in the context of DSM • V&V adapted into the context of DSM • (some) Means to facilitate V&V • Conclusions

4. Definitions 1/2 IEEE Standard Dictionary: Validation ‘the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.’ Model Validation ‘the process of determining the degree to which the requirements, design or implementation of a model are a realization of selected aspects of the system being modelled.’

5. Definitions 2/2 IEEE Standard Dictionary: Verification ‘(1) the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. (2) Formal proof of program correctness.’ Model Verification ‘the process of determining the degree of similarity between the realization steps of a model; for example, between the requirements and the design or between the design and its implementation.’

6. The Process to V&V • A process of transforming end-user requirements to a system fulfilling such requirements with a DSM approach • The development process includes phases for developing • Modelling infrastructure • Metamodels, • Generators, and • Application models • which are transformed into the target format running on a software platform.

7. Verification Adapted to the DSM Context-Metamodel 1/2 • The verification of metamodels (MM) means • Determining if a metamodel encapsulates • the problem domain, and • the rules of the target platform that are necessary to take into account in the metamodel

8. Verification Adapted to the DSM Context-Metamodel 2/2 • Verifying the domain coverage requires neural processing but guidelines for language development help • Verifying that only correctly designed applications can be modelled • ‘Correctly designed’ denotes that the modelled applications are syntactically correct and semantically sound in the sense that they appear to be correct from the interpreter perspective, i.e. the application models can be generated into working executables running on a set platform • Developing an extensive test suite of application models might help

9. Verification Adapted to the DSM Context-Model 1/2 • The verification of models (M) means • Determining whether a model encapsulates a product under development based on the requirements set for it

10. Verification Adapted to the DSM Context-Model 2/2 • The verification of models means scrutinizing whether the requirements set for an application should be possible to be covered by a model simply by considering the model itself • Maintaining a traceability-link between the requirements and models help • Analysing the models with advanced analysis tool might also help • Transforming a model to a format understood by analysis tools is a risk!

11. Verification Adapted to the DSM Context-Generator 1/3 • The verification of generators (G) means • Determining whether the generators produce correct output (GTR) from the input models

12. Verification Adapted to the DSM Context-Generator 2/3 • The verification of code generators [Kuster et al.] • The syntactic correctness of a model transformation: in order to ensure that the outcome of the transformation is syntactically correct • The termination and confluence of a model transformation: in order to ensure that a model always produces the same outcome • The safety and liveness properties: in order to verify that e.g. specific structural properties are preserved during the transformation. • The semantic equivalence of a model transformation: in order to ensure that the semantics are preserved during the model transformation

13. Verification Adapted to the DSM Context-Generator 3/3 • Syntax and format in general • Metamodel coverage of the generator should be checked • Semantics • The semantics between the source model and target model has to be preserved during the transformation • A third metamodel to connect the both worlds? • -> Verification done when verification of GTR is considered?

14. Verification Adapted to the DSM Context-GTR 1/2 • The verification of generated textual representation (GTR) means • Evaluating whether or not the intention of a model is realised in the outcome, i.e. in the GTR produced by a generator

15. Verification Adapted to the DSM Context-GTR 2/2 • Notice: The intention of the model should be considered to be a fact, thus the model must be considered to be correct from the modeller’s perspective in order to make any sense in the GTR verification • Maintaining traceability-link between GTR and models should help to verify that the intention is realised • Testing should not be overlooked although testing cannot formally state that the GTR always functions as planned

16. Validation Adapted to the DSM Context 1/2 • The validation is the process of evaluating the GTR, including the target platform, to determine whether it satisfies the specified requirements • It must be noted that one application model can only be seen as one validation case for the whole modelling infrastructure • Multiple application models are required!

17. Validation Adapted to the DSM Context 2/2 • We see validation as the challenge of maintaining a bidirectional traceability link between the end-user requirements and the generated deliverable • Maintaining a bidirectional traceability link between requirements and GTR should help

18. Conclusions • There are methods, techniques and some tools for verification and validation • No comprehensive methodology backed up with tools still exists especially intended for DSM • And then what? • Methods and tools for the language developer • Metamodel • Method to check what kinds of erroneous models can be modelled? • Code generator • Method to check what parts of the metamodel the code generator can reach? • Method and tools for the language user • Bi-directional traceability link • From requirements databases (DOORS, OSRMT) to domain-specific models to source code and back • Comprehensive test-suite generation from application models