Verification & Testing: SAP HANA Case Study

1. Verification & Testing: SAP HANA Case Study Anthony J H Simons (University of Sheffield)Wolfgang K R Schwach (SAP Karlsruhe)

2. Overview • Verification and Testing Toolset • Service Specification • Validation and Verification • Test Generation and Grounding • Test Execution on SAP HANA Cloud App • V&T tools: Anthony Simons, Uni. Sheffield • App & grounding: Wolfgang Schwach, SAP Anthony J H Simons, University of Sheffield

3. Verification & Testing Tools Anthony J H Simons, University of Sheffield

4. Specification • Why? • Gentle pressure towards common standards • Provide basis for formal service certification • What? • Describe service interfaces and behaviour • State-based model of service API • IOPE model of detailed behaviour • XML language  Java metamodel Anthony J H Simons, University of Sheffield

5. Holiday Booking Service select/fail save/failsave/duplicate save/overflow View Days back/okdelete/ok bookDays/ok Book Days back/oksave/ok select/high select/low Delete Days choose/onechoose/four choose/seven choose/onechoose/four choose/seven Anthony J H Simons, University of Sheffield

6. XML Specification – FSM Anthony J H Simons, University of Sheffield

7. XML Specification – IOPE Anthony J H Simons, University of Sheffield

8. Validation & Verification • Why? • Designers write incomplete specifications • Designers write incorrect specifications • How? • Validation tool explores state-transition diagram • Reports unhandled events, missing transitions • Verification tool simulates all input/memory space • Reports blocking/nondeterministic operations • Blocking: input exists which triggers no operation • Nondeterminism: some input triggers many operations Anthony J H Simons, University of Sheffield

9. Verification Fault Analysis off-by-one indexing error Anthony J H Simons, University of Sheffield

10. Test Generation • Why? • More thorough coverage than manual testing • Assures full compliance to the specification • How? • Generation tool explores the whole specification • Creates all interleaved sequences up to depth k • Reduces test-suite size by optimising • Removes infeasible sequences (blocked in the spec) • Removes redundant sequences (equivalence-classes) • Compresses executable suite (multi-objective tests) Anthony J H Simons, University of Sheffield

11. Generated Test Suite multi-objective test Anthony J H Simons, University of Sheffield

12. Test Grounding • Why? • High-level test suites are technology-neutral • Platform-specific translations required • How? • Platform providers offer bespoke groundings • Broker@Cloud framework provides hooks • XML test suite parsed as a Java metamodel instance • Visitor Design Pattern is used to generate concrete tests • Bespoke grounding to e.g. Java, SOAP, REST, Selenium Anthony J H Simons, University of Sheffield

13. Example Java Grounding auto-gen JUnit test assertions Anthony J H Simons, University of Sheffield

14. SAP HANA Case Study • User may book up to 5 days vacation • blocks of days can be booked, selected, deleted • prevents overlapping blocks, more than 5 days • App designed using the SAP UI5 toolkit • visually pleasing point-and-click interface • operations nominally match the earlier specification • but a rich-client web app API is hard to test • Deployed on the SAP HANA cloud platform • temporarily available at URI: https://brokercloud-d050960trial.dispatcher.hanatrial.ondemand.com/ Anthony J H Simons, University of Sheffield

15. App Running on HANA Anthony J H Simons, University of Sheffield

16. Selenium Test Driver • Selenium: a tool for web-testing • normally used to capture-and-record a user's point-and-click gestures interactively • can replay recorded test scripts, stored as HTML tables, to run web app regression tests • Leverage to test a rich-client cloud app • given the Selenium DOM for the web app • translate high-level test suite into Selenium test scripts • use the Selenium engine to execute the test scripts Anthony J H Simons, University of Sheffield

17. Selenium Grounding Rules Anthony J H Simons, University of Sheffield

18. Executable Test Scripts Anthony J H Simons, University of Sheffield

19. Executing the Tests Anthony J H Simons, University of Sheffield

20. Subtle Faults Detected • Functionality • service failed to notify duplicate holiday booking as a separate error case • service failed to distinguish overflow and duplicate holiday booking responses • Security • service executed operations in extra states, when the operations should be ignored • service revealed response information, when the request should be ignored Anthony J H Simons, University of Sheffield

21. Beyond State-of-the-Art • Extends leading SXM theory for MBT • Adds reasoning about the full SXM model • Adds validation and full verification capability • Adds test optimisation capability • Exceeds all industrial testing methods • Full coverage of the specification achieved • All positive and negative test cases created • Identifies top 10% uniquely discriminating tests Anthony J H Simons, University of Sheffield

22. Questions • Acknowledgement FP7-ICT-2011-8 grant agreement no: 318392 • Project website www.broker-cloud.eu Anthony J H Simons, University of Sheffield

23. References • Broker@Cloud D20.1 State of the art and research baseline, http:www.broker-cloud.eu, 2013. • Broker@Cloud D20.3 Requirements for continuous quality assurance and optimisation in cloud brokerage, http:www.broker-cloud.eu, 2013. • AJH Simons and RE Lefticaru, Specification, verification and testing of brokered software services in the cloud, IEEE ICST 2015, Graz (submitted, under review) • AJH Simons and RE Lefticaru, Stream X-Machine based verification and testing of cloud services, 2nd Int. Workshop on Cloud Service Brokerage, ESOCC, Manchester, 2 Sept., CCIS, 2014. • M Kiran, A Friesen, AJH Simons and WKR Schwach, Model-based testing in cloud brokerage scenarios, 1st Int. Workshop on Cloud Service Brokerage, ICSOC, Berlin, 2 Dec., LNCS, 2013. Anthony J H Simons, University of Sheffield