1 / 18

A conformance test suite for TTCN-3 tools

A conformance test suite for TTCN-3 tools. Benjamin Zeiss, T-Systems Andras Kovacs, Broadbit Bogdan Stanca-Kaposta, Testing Technologies. Agenda. Motivation STF 409 Methodology Discussion Challenges Conclusion. Motivation (1/2). Many TTCN-3 compilers (~7 commercial, 2 internal)

kevork
Download Presentation

A conformance test suite for TTCN-3 tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A conformance test suite for TTCN-3 tools Benjamin Zeiss, T-Systems Andras Kovacs, Broadbit Bogdan Stanca-Kaposta, Testing Technologies

  2. Agenda • Motivation • STF 409 • Methodology • Discussion • Challenges • Conclusion A conformance test suite for TTCN-3 tools

  3. Motivation (1/2) • Many TTCN-3 compilers (~7 commercial, 2 internal) • High interest in the language • Complex standard: hundreds of pages • Estimation of ~5000 requirements • No way to check the compliance of TTCN-3 tools • TTCN-3 is based on ETSI standards • A conformance test suite for TTCN-3 tools is needed! • Quality of a test tool is critical • Errors in the test tool may influece test results. A conformance test suite for TTCN-3 tools

  4. Motivation (2/2) • Relevance: • For users: • Is my tool standards compliant? • Can I switch tools easily / vendor lock-in? • For tool vendors: • Can I improve the standard compliance of my tool? • Possibly later: compliance certification • For standard developers: • Where is need for clarification? A conformance test suite for TTCN-3 tools

  5. STF 409 (1/2) • Specialist Task Force initiated by ETSI MTS • Manpower: 3 persons w. 20 days each (60 days total) • Benjamin Zeiss (T-Systems/University of Göttingen) • Andras Kovacs (Broadbit) • Bogdan Stanca-Kaposta (Testing Technologies) • Dates: October 2010 – Early 2011 • Work is based on a framework given by TTCN-3 STF. • Task: • Devel. of a conformance test suite for TTCN-3 tools. • Creation of a thorough documentation for the ATS. A conformance test suite for TTCN-3 tools

  6. STF 409 (2/2) • Three deliverables: • ETSI TS 102 950-1 V1.1.1 (2011-04) – ICS • ETSI TS 102 950-1 V1.1.1 (2011-04) – TSS & TP • ETSI TS 102 950-2 V1.1.1 (2011-04) – ATS & IXIT • Approved by MTS in April 2011. • Downloadable from the ETSI download area right now! • Time spent: • 10 days each (30 work days) • Development of conformance tests. • 10 days each + extra time (30+ work days): • Validation, fixing of problems, documentation, tool development. A conformance test suite for TTCN-3 tools

  7. Methodology (1/7) • Five main steps: • Create clause priority list. Discuss with MTS. • Develop conformance tests. • Create documentation framework. • Generate documentation content. • Validate and fix. • Step 1 (creation of clause priority list): • Not all clauses can be covered. Selection has to be made. • How to prioritize? • Vendor feedback + most basic functionality. • There is no perfect selection! A conformance test suite for TTCN-3 tools

  8. Methodology (2/7) • Step 2 (development of conformance tests): • The ATS is not the test automation, but the test input! • In most cases: 1 module = 1 conformance test. • Two steps: • Specification of the test purpose: @purpose + @verdict • Implementation of the test in TTCN-3. • Conformance verdict: • Expected tool output is specified in the document tags (@verdict). • Purpose: • Also specified in the document tags (@purpose). • Test automation is not provided by the ATS! • Can be scripted easily by extracting the document tags. • Example Java source code can be provided! • Test suite structure: organized according to clauses. A conformance test suite for TTCN-3 tools

  9. Methodology (3/7) • Step 2 (development of conformance tests) cont.: • Expected output for verdict determination: • @verdict pass accept/reject [expectedoutput] • Examples: • @verdict pass reject • @verdict pass accept, ttcn3verdict:pass • @verdict pass accept, noexecution • Purpose specification: • @purpose documentreference, description • Documentreference: part:clause • Examples: • @purpose 1:5, Ensure that when the IUT loads a module containing some definitions before the module declaration then the module is rejected. A conformance test suite for TTCN-3 tools

  10. Methodology (4/7) • Step 2 (development of conformance tests) cont.: • Complete TTCN-3 test input example: /*************************************************** ** @purpose 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. ** @verdict pass accept, ttcn3verdict:pass ***************************************************/ module Sem_050202_Uniqueness_001 { importfrom Sem_050202_Uniqueness_001_import { constall; } typecomponent GeneralComp {} function f_funcScope() { varboolean repeatedIdentifier := true; if(repeatedIdentifier==true) { setverdict(pass); } } testcase TC_Sem_050202_Uniqueness_001() runson GeneralComp { f_funcScope(); } control { execute(TC_Sem_050202_Uniqueness_001()); } } module Sem_050202_Uniqueness_001_import { // repeated indentifier // from imported module // is allowed constinteger repeatedIdentifier := 0; } A conformance test suite for TTCN-3 tools

  11. Methodology (5/7) • Step 3 (creation of documentation framework): • Already provided by STF393 (Proforma). • Split single document into three documents. • Step 4 (generate documentation content): • Only 60 days manpower, little time for documentation. • Working on a single source saves time. • Solution: • Write test purposes and expected tool outputs directly into the test inputs. • Generate tables for the documentation from the ATS: • Test purpose tables. • Test suite structure. • Much easier to maintain! • Documentation generation tools developed as part of STF409. A conformance test suite for TTCN-3 tools

  12. Methodology (6/7) • Step 5 (validation and fixing): • Validation and fixing took more than half of the development time! • STF internal validation with 4 tools: • TTworkbench: compile-time and execution. • TestCast: compile-time and execution. • IBM: some initial compile-time validation. • MTP: as reports are sent in. • Lots of feedback from Elvior and MTP! A conformance test suite for TTCN-3 tools

  13. Methodology (7/7) • Executing the ATS: TTCN-3 File TTCN-3 File TTCN-3 File Sem_060101_TopLevel_001.ttcn ATS ** @verdict pass accept, ttcn3verdict:pass Test input Sem_060101_TopLevel_001.ttcn NegSem_060303_component_types_001.ttcn … Expected output TTCN-3 Tool (test subject) Tool output Rejection as invalid Rejection to execute Execution with results (e.g., pass/inconc/fail) Tool output evaluation pass fail A conformance test suite for TTCN-3 tools

  14. Discussion pass TTCN-3 Conformance ATS TTCN-3 Tool Meaning? • Compliance does not imply safety from a vendor lock-in! • Testing is never complete! • With v1.1.1 of the ATS, about 32% of the core standard clauses are covered with at least one test. • Ideally, we should have many tests for each clause. • Compliance means less likelihood for vendor lock-in. • With better coverage and higher number of test cases, we improve the quality of this statement. • Compliance reduces the likelihood for major tool errors • Tests range from simple arithmetic operations to communication operations and verdict handling. • ATS checks syntax and semantics. • ATS does not check the correctness of every single tool message, but of the overall output. A conformance test suite for TTCN-3 tools

  15. Challenges (1/2) • Various standard clarifications were necessary: • 9 CRs for TTCN-3 core languages have been written. • Unclarified test cases have been excluded from the delivery ("on-hold"). • Tools sometimes handle the standard differently. • Tests need to be expressed as tool independent as possible. • Time management: • Proper validation and fixing is extremely time consuming. • More than 10 voluntary days have been actually spent. • Responsibilities: • Everyone mixes up STF393 and STF409 responsibilities (proforma). A conformance test suite for TTCN-3 tools

  16. Challenges (2/2) • Version numbers: • To which core language standard version does the conformance test suite relate to? • How to maintain compatibility across different language versions? • Against which version should the conformance test suite be developed in the future? • Target quality: • The target quality of the test suite should be fixed right in the beginning (Terms of Reference). A conformance test suite for TTCN-3 tools

  17. Conclusion • The conformance test suite is available now! • ATS Metrics: • Total number of conformance tests: 733 • Number of positive syntactic conformance tests: 110 • Number of negative syntactic conformance tests: 34 • Number of positive semantic conformance tests: 335 • Number of negative semantic conformance tests: 254 • Total number of clauses in the TTCN-3 standard: 342 • Total number of clauses (at least partially) covered: 110 • About 140 pages of test purposes. • Free to use for tool developers. Use it to improve your tool! • Bug reports: http://t-ort.etsi.org A conformance test suite for TTCN-3 tools

  18. Special thanks: Tool validation and comments Elvior, MTP, Conformiq, Ericsson, ISPRAS Jens Grabowski / University of Göttingen Questions? Feedback? A conformance test suite for TTCN-3 tools

More Related