Software Quality Assurance Automated Testing Techniques

1. Software Quality AssuranceAutomated Testing Techniques Michael Biwer

2. Apology • I did not communicate to Dr. Yang my Abstract and References by Monday 8:00AM. • This is because I completely forgot and dropped the ball on the simplest part of this course. • This is why there had been a late communication saying that I was to present today. • Thankfully, Dr. Yang and me worked out a deal.

3. Agenda • Software Quality Assurance:Automated Testing • Overview • Testing Methodology • Overview of types of tests • Automated Testing Software • Versus Manual Testing, Pros, Cons, And Basic Functionality • Types of Automated Testing Framework • Modular, Data-Driven and Keyword-Driven Frameworks.

4. Definition • “The management and performance of test activities, to include the development and execution of test scripts so as to verify test requirements, using an automated test tool”[2].

5. History • Began to evolve with the rest of software development as pressure was put on developers to: • Reduce time, cut costs and increase quality. • This began to create more incremental builds: • Shown with the increase of Rapid Application Development methods such as Agile. • Posed the difficulties of faster changes and additions to project requirements. • Higher customer appreciation of quality control. (ISO 9000) • Therefore, Software Quality Assurance: Automation Testing began to evolve and third party companies began to create automation suites.

6. Developer TestingVS.Quality Assurance Testing • Developer Testing: • Ensures that the all of the projects requirements have been met through testing the processes. • Usually, done through testing the code from an internal standpoint. • Overall, looks to ensure that the process work correctly. • Quality Assurance Testing: • Ensures that all of the project requirements have been met through testing the product. • Usually, done through testing the product from an external standpoint. Most instances mimic’s users view of the product. • Overall, looks to improve the product as a whole.

7. Manual Testing Vs. Automated Testing • Manual: • Labor intensive, testing take many man-hours to complete. • Does not require software knowledge or skills from the testers. • Repetitive tests still require many hours to complete and may be narrowed to fit a time schedule. • Automated: • Capital intensive. • Large upfront cost for the software. Scalability might require hardware upgrades. • Will not receive any benefits until the 2 – 3 project, because the creation of the test framework. • Repetitive testing shows rates of returns around the 10th time it is run. • Requires more specialist knowledge bases for the automation engineer.

9. Why? • Proper defect tracking. • Aim to enhance the overall quality of the product, not to just ensure it works correctly. • Different eyes on the testing ensures that project requirements were communicated efficiently between customer and development team. • Because not all developers are good testers, it is good to have a specialist test the product to ensure consistent quality. • Ensures that the customer has input, in order to “make sure that they will be happy with the end result.”

10. Pros • 24/7 Testing. • Repetitive testing is easily accomplished and cost effective in long running projects or in maintenance. • In depth reporting tools. • Consistent testing. • Fewer, more skilled workers.

11. Cons • Cost • Heavy upfront cost with the purchase of the software and the time devoted to the creation of a automation framework. • In addition, you will not see a return on investment right away. • Must pay your employees more. • You will require an automation engineer which has software programming knowledge. • You will have to train your automation testing team in the automation software tool you choose.

12. Overview of Testing Types • Two overall types of testing: • White-Box Testing: • Testing that is done from an internal viewpoint of the application. • Black-Box Testing: • Testing that is done from an external viewpoint of the application with only the working knowledge of how the application “should” work. Input Process Output Input Output

13. Overview of Testing Methods (cont..) • Unit Testing • Tests that target specific code to ensure that it can operate separately from the system given the correct inputs. • Integration Testing • Testing that aims to test the applications code as a whole to ensure that the components interact correctly. • System Testing • Testing that makes sure that the application meets the requirements set down for the software and that it doesn’t interfere with any other platforms/applications. • Acceptance Testing • Testing where either the user or developer/QA team is given a build of the program in order to verify the requirements have been met.

14. Overview of Testing Methods(cont…) • Smoke Testing: • “Build Verification Testing”, Ensures that no serious problem has occurred between build of the application. • Regression Testing • Testing normally performed after a major code overhaul has been made to make sure that there are no persisting defects and that no functionality has been lost. • Functional Testing • Testing that only cover a particular flow or process within the program. • Non-Functional Testing • Testing that confirms the overall quality of the product in reference to the user. • Performance Testing • Testing that checks the scalability, reliability and resource usage of certain aspect of the program. • Usability Testing • Testing of the Graphical User Interface (GUI) works correctly in the given layout.

15. When to Automate a Test • It makes the most sense to automate tests for products that are going to have continuous development or maintenance. • This ensures that the test cases can be reused, increasing the return on investment of the automation software. • Such as for the purposes of regression or smoke tests. • Side Note: • Developers often use automated testing methods for unit or integration tests. (Junit Tests.)

16. Basic Automation Software Features • Record/Playback • Process: • Click the play button in the automation suite. • The Automation Software tracks your mouse movements, clicks and keyboard actions. • Click the stop button when the test completes. • The Software then compiles the movements into a test cases for you to insert into a test plan. • You are then able to perform those exact actions multiple times. • Sometimes features of the automation suite include auto-generation of record/playback tests into test scripts.

17. Basic Automation Software Features • Record/Playback (cont…) • Benefits: • Simple, does not require a software background to utilize. • Quick, there is not much set up to create this type of test. • Can be used for repetitive testing on a basic level. • Drawbacks • The types of tests that you can performs are limited to external options. • If you have to change the test data, you must redo the entire test. • Therefore, there can be a lot of maintenance.

18. Example of Record/Playback • http://youtu.be/facNRr8SHpk?t=45s

19. Basic Automation Software Features • Object Recognition • Allows you to have an external look at the objects that are currently being displayed in the application being tested. • Aka. GUI. • You are able to store these objects in an “Object Repository” in order to easily reference them or you are able to “Call” them directly within your test scripts. • They are formed in a hierarchy of “container” objects. • Automation software with this ability gives you libraries which allow you to perform functions and retrieve/change properties of the objects.

20. Basic Automation Software Features • Object Recognition(cont…) • Benefits: • Allows you to set up more in-depth specific tests within test scripts. • Drawbacks: • Requires a test script framework to create test on these objects. • Creates large initial overhead. • Problems occur when custom objects/properties are created by the developers which the automation suite cannot recognize. • This would require you to create your own “Object Script Extender” to recognize the object/property.

21. Example of HP QuickTest Pro Object Recognition

22. Basic Automation Software Features • Test Script Execution Framework • Allows you to create libraries in order to test any portion of your application. • Code • GUI • API’s • Only limited to the amount of coding you want to do. • These scripts are accessed through drivers and can be used across many different test cases. • Commonly these test scripts are created using VBScript, Java, Perl, or CGI. • The automation tools have pre-loaded libraries to give you extra capabilities with testing objects.

23. Simple test script example. • Uses a direct calling of objects to interact with them.

24. Basic Automation Software Features • Test Cases Reports • Able to view the passes and the fails of your automation tests in a report format. • Since this is based on your test scripts, you only get as much detailed information as you allow the test case record to collect from the tests.

25. HP QTP: Example Report

26. Components • Test Plan • A set of test cases which verify similar requirements of the application. • Test Case • A test to verify requirements of the application. • Driver Scripts: • The overall scripts that call your test scripts that make up your test cases. • In essence, will reflect your test plan. • Test Scripts: • Code which allows for the tests actions to be carried out. • Test Data: • The data that you will be testing in your application.

27. Meta-Language Development • Meta-Language is a abstract language used in running scripts. It has been simplified for the average user with no knowledge of programing to use. • This allows testers that only have the business knowledge of the system to utilize testing tools. • Allows for manual testers to migrate onto the new system easily. • This is essential for the testers to take ownership of the test cases. • It is used at the highest visible level for the users to see, the Driver Script. • In essence however, the Meta-Language, will be a list of parameters that will be parsed/translated and used in the test scripts.

28. Where the automation tester inputs the test parameters. • HP QTP Example: Input parameters that the automation engineer has set for this Driver Script

29. Types:Modular Framework • Creating specific test scripts in a library which are used within the Driver Scripts to perform the tests. • IE: In the windows calculator, having a script that would verify that 5 + 5 = 10. • Benefits: • Quick and easy to put together. • Drawbacks: • Difficult to maintain. • Less reusable code. • Data is within the test scripts. • Can easily have multiple copies of the same code. • Because the test scripts have all of the control, this means that the ownership of the test cases lie in the automation engineer.

30. Types:Modular Framework (cou…) Example of framework setup: Example of Psuedocode for a Modular Framework Driver Script: Main Read Parameters Window_1 Functions Function_1 Call Function_2 Call Function_3 Call Window_2 Functions Function_4 Call Function_5 Call Function_6 Call End Main

31. Example of Driver Script Parameters:

32. Types:Data-Driven Framework • This framework looks to fix the problem of data being within the test script. The input is commonly moved to the driver script or to an external database or spreadsheet which is fed into the test scripts. • IE. In the windows calculator, you have a spreadsheet with numbers which are run through a script that adds them together. • Benefits: • Gives ownership of the test cases to the testers. • Maintenance is more manageable. • Drawbacks: • Test case information can be split into different areas confusing large operations. • Scalability is a real problem.

33. Types: Data-Driven Framework (cou…) Example of framework structure: Pseudocode for Data-Driven Driver Script: Main Read in Parameters Perform required Script with given parameters Script_1 Call Script_2 Call Script_3 Call End Main

34. Example of the Data-Driven Driver Script:

35. Types:Keyword-Driven Frameworks • This framework looks to provide maximum ownership to the testers and maximum flexibility. • Does this by providing a object by object look at the test cases and providing actions that can be taken out on those objects. • Benefits: • Provides all ownership to the testers. • Maximum flexibility and highly scalable. • Maintenance is on the testers and is standardized. • Drawbacks: • Large overhead time for creating the framework. • A driver file must be created for each “window” of the application. • An iterative approach which can increase test case execution times.

36. Types: Keyword-Driven Framework (cou…) Pseudocode for Keyword-Driven Driver Script: Main Read in Parameters Parse parameters into correct Object+Function+Data groups For Each Object in window Selected Action Object_1 Action Calls Object_2 Action Calls Object_3 Action Calls End Main

38. Types: Hybrid Framework • Combines the previous frameworks together in various levels to provide maximum testing capabilities. • Normally involves connecting to a test database for various inputs. Example of Hybrid Framework Psuedocode: Main Read Parameters Group Parameters Link to database Select a specific window For each object in a specific window Perform the needed scripts End Main

39. What we discussed • Software Quality Assurance: Automated Testing • Overview • Testing Methodology • Overview of types of tests • Automated Testing Software • Versus Manual Testing, Pros, Cons, And Functionality • Types of Automated Testing Framework

40. References • Chaos manifesto 2012: The year of the executive sponsor. (2012). Retrieved from http://versionone.com/assets/img/files/CHAOSManifesto2012.pdf • Elfriede, D., Rashka, J., & Paul, J. (2008). Automated software testing: Introduction, management and performance. (13 ed.). Castleton, New York: Hamilton. Retrieved from http://books.google.com/books?id=kl2H0G6EFf0C&lpg=PR5&ots=wN_PrWrBx1&dq=automation software test qa&lr&pg=PA43 • Mosley, D., & Posey, B. (2002). Just enough software test automation. Upper Saddle Hill, new Jersey: Prentice Hall PTR. Retrieved from http://books.google.com/books?id=PEBvfWESIt4C&lpg=PR15&ots=xGxZPGvp2P&dq=software test automation &lr&pg=PA3 • Laukkanen, P. (2006). Data-driven and keyword-driven test automation frameworks. (Master's thesis)Retrieved from http://eliga.fi/Thesis-Pekka-Laukkanen.pdf • Software testing. (n.d.). Retrieved from http://en.wikipedia.org/wiki/Software_testing • Allott, S. (1999, 10). Automate your tests - you won’t regress it!. Seventeenth annual pacific northwest software quality conference, Oregon Convention Center Portland, Oregon. Retrieved from http://www.snapshots.pnsqc.org/2011-theme/pnsqc-04-06-2011/proceedings/pnsqc99.pdf • Kumar, S. (Designer). (2007, April 9). File:SampleCode.JPG [Print Photo]. Retrieved from http://en.wikipedia.org/wiki/File:SampleCode.JPG • Kiranpaul , K. (n.d.). Quick test professional tutorials - qtp fundamentals, qtp features, vb scripting examples in qtp, qtp framework and vb script.. Retrieved from http://qtp4free.blogspot.com/2010/05/object-spy-in-qtp-92.html • Jain, A. (Photographer). (2013, July 5). Good Looking and Enhanced Results Viewer [Web Graphic]. Retrieved from http://2.bp.blogspot.com/_iTz5d0BSy6g/TPY5-7SOIGI/AAAAAAAAAPM/wxlEl9BHMqM/s1600/QTP_11_RV_SS.png • Walter , T. (Photographer). (2012, July 6). Add Parameters to specify the Ranorex Automation [Web Graphic]. Retrieved from http://www.ranorex.com/blog/wp-content/uploads/2012/07/Add-Parameters-To-Test.png • Tutor. (2012, Aug 6). Software testing life cycle (stlc) – complete tutorial. Retrieved from http://www.sdlc.ws/software-testing-life-cycle-stlc-complete-tutorial/