1 / 34

Structured Exploratory Testing – Method in Madness

Structured Exploratory Testing – Method in Madness. Vivek Devarajan. Senior Consultant, Wipro Technologies. 1. Scripted versus Non-Scripted Testing. Some Exploratory Testing Ideas. Session Based Test Management. Benefits of Exploratory Testing. Case Studies. 2. 3. 4. 5.

roxy
Download Presentation

Structured Exploratory Testing – Method in Madness

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Structured Exploratory Testing – Method in Madness Vivek Devarajan Senior Consultant, Wipro Technologies

  2. 1 Scripted versus Non-Scripted Testing Some Exploratory Testing Ideas Session Based Test Management Benefits of Exploratory Testing Case Studies 2 3 4 5

  3. Scripted versus Non-Scripted Testing

  4. Scripted versus Non-Scripted Testing Leveraging the benefits of both Testing Styles Madness Methodical • Elaborate Documentation • Static Test Suite • Higher Testing Costs • Limited Defect Detection • Concise Documentation • Dynamic Test Optimization • Lower Testing Costs • Better Defect Detection • No Accountability • No Predictability • High Possibility of Defect Slippage Scripted Testing Freestyle Testing Structured Exploratory Testing / Session Based Test Management

  5. Some Exploratory Testing Ideas

  6. Exploratory Testing using Question Patterns • While testing a system, how often have you felt “I have tested something similar before?” • The Domain Factor • On the other hand, how often have you been bogged down by an application, just because it was from a different domain that you are not so familiar with? • The Reuse Factor • Is there a way to develop testing ideas once and reuse them across domains / applications? • The Knowledge Factor • Is there a way an experienced tester or domain expert can impart his/her experience to others? • The Exploratory Testing Context • Is there a way to explore the application systematically and find bugs?

  7. Question Patterns and Exploratory Testing • Question Patterns: A Set of interrelated Questions grouped together, focused on an aspect of the system under test that • Enable someone to understand the application better • Can be used as a technique for requirements capture • Patterns help the tester explore the application systematically and find bugs • Example: Parent and Child Entity Pattern • Scenario A).Consider an online shopping application where the user can add / modify / delete multiple items in his shopping cart order. In this case, the shopping cart is a business entity which has multiple child entities (items) within itself (a one-to-many relationship between the cart and the items) where the child entity (item) can be added/updated or deleted. • Scenario B). Consider an application where the user can apply for an insurance policy online, to which he can add / modify / delete multiple beneficiaries. In this case, the policy is a business entity which has multiple child entities (beneficiaries) within itself (a one-to-many relationship between the policy and the beneficiaries) where the child entity (beneficiary) can be added/updated or deleted.

  8. Question Patterns and Exploratory Testing • Questions that can be asked: Parent and Child Entity Pattern • What are the business entities involved? • What is the relationship between them? (One to Many, One-to-One, etc) • What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search) • What are the field level and business validations for each entity? • Answers to such questions enable testers to find gaps in requirements (or even capture requirements). • Answers to such questions also serve as a guideline and a checklist during exploratory testing.

  9. Question Patterns and Exploratory Testing • Questions that can be asked: Parent and Child Entity Pattern • What are the business entities involved? • What is the relationship between them? (One to Many, One-to-One, etc) • What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search) • What are the field level and business validations for each entity? • Answers to such questions enable testers to find gaps in requirements (or even capture requirements). • Answers to such questions also serve as a guideline and a checklist during exploratory testing.

  10. Exploratory Testing by Looking for Specific Problems • Purpose of Testing – Looking for Bugs/Problems that could impact the stakeholders • Defect Seeking v/s Defect Finding • Make a list of possible issues • Design tests to find ways to stimulate the possible issues identified

  11. Exploratory Testing by Analyzing Data Flow • Most Applications process / manage data • Identify the input and output data • Verify the output data for various combinations of input data (including green path values, invalid values, boundary conditions, large values, etc) • Try modifying the data at various stages in the flow and see the impact • Identify the right combinations / representatives of data

  12. Exploratory Testing by Analyzing Flows • Identify end-to-end flows and branching • Depict the branching graphically, if needed • Try parallel flows • Try varying the sequencing / timing of the flows • Analyze the impact of input data values on the flows • Check the main flows as well as alternate/exceptional flows • Check for state transition flows

  13. Session Based Test Management – Accountable and Manageable Exploratory Testing

  14. Session Based Test Management - Approach • Test Execution is divided into time-boxed Test Sessions, where the tester does (uninterrupted) exploratory testing with a specific charter • Tester takes brief notes about the testing done and the issues found • Tester uses a checklist of testing ideas as a guide for testing • Further tests are dynamically optimized depending on the results of the current test • The Test Lead conducts a ‘Debrief’ session with the testers on the test findings and plans further executions based on the findings

  15. Session Based Test Management – Session Contents • The contents of a typical test session are as follows: • Charter: Contains the mission of the given session. Typically it mentions the application areas to be covered / explored, at a high level. • Tester name, Session Time Start and End • Task Breakdown: Time taken for Test execution v/s Bug Reporting, Charter (Original Session Charter) v/s Opportunity (Diversions taken) • Test Notes: Notes of what the tester executed, in brief. • Bugs/Issues found

  16. Session Based Test Management – Accountability • Tester’s Activities: Tracked via Test Session reports • Coverage and Testing Progress Tracking: • High Level Coverage: Number of functional areas and quality criteria covered • Granular Coverage: Number of checklists/items/test ideas covered for each functional area • Traceability: Each test idea and item in the checklist can be mapped to requirements

  17. Benefits of Exploratory Testing

  18. Exploratory Testing - Benefits • Lesser Preparation and lesser effort • Quicker Bug Detection • Leverages on Tester’s Abilities, experience and creativity • Works even if requirements are not fully frozen

  19. Skilled Testing v/s Deskilling – The Classic Debate • Deskilling via Detailed Test Cases leads to higher test design and maintenance costs and limited bug detection capabilities • Testers with lesser experience can be trained in skilled exploratory testing • Skilled Testing is more cost effective and results in better product quality

  20. Case Studies

  21. Case Study 1 – The Problem • A Police IT Application for a Government Customer • The Context: • Previous customer concerns over quality in beta releases • Large volume of functional tests to be executed in a short time span • - Conventional Testing Estimations: 3000 test cases to be designed and executed in 2 months (including KAP phase), requiring 14 testers • Inexperienced Testers with no domain knowledge • Customer more focused on bugs/issues than in testing processes

  22. Case Study 1 – Testing Approach and Results • A Police IT Application for a Government Customer • Question Patterns used to arrive at checklist of testing ideas • Entity Relationship pattern • State Transition Pattern • Master Data – Transaction Data Pattern • Checklists consisting of Business Conditions and Testing Ideas • Initial Test Sessions focused on exploration of the business operations of entities • Deep Dive Test Sessions focused on finding specific problems • Test Sessions for End to End Testing focused on Application Flow and Data flow • Results: 65% reduction in testing efforts and 60% better defect detection

  23. Case Study 2 – The Problem • An Order Management Application for a Healthcare customer • The Context: • The application is intended to enable healthcare providers and pharmacies to place orders quickly • Bugs not being found early enough • High cost of testing • Frequent patch releases with incomplete functionality • Mix of experienced and inexperienced testers, with fair domain knowledge • Parallel run of conventional testing and Session Based Test Management

  24. Case Study 2 – Testing Approach and Results • An Order Management Application for a Healthcare customer • Question Patterns used to arrive at checklist of testing ideas for regression testing • Checklists consisting of Business Conditions and functional changes to be checked for • Initial Test Sessions focused on the exploration of new functionalities • Subsequent Test Sessions focused on user experience • Results: 50% reduction in test design efforts and quicker defect detection

  25. Thank you Vivek Devarajan Senior Consultant, Wipro Technologies vivek.devarajan@wipro.com

  26. Appendix

  27. Artifacts Created • Checklists Created for Session Based Test Management: • Sample Test Session:

  28. Session Based Test Management SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles Exploratory Testing Freestyle Testing Scripted Testing Session Based Test Management

  29. Session Based Test Management SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles

  30. Comparison of Structured v/s Freestyle Testing Scripted Testing Freestyle Testing Benefits: • Very Low Costs for Test Creation/Maintenance • Highly effective in quick Defect Seeking, especially effective for tricky bugs • Does not need well documented requirements Benefits: • High Predictability • Deskilling (Test Execution) • High Accountability Drawbacks: • Very High Costs (Resources + Tools) for Test Creation as well as Maintenance • Limited Defect Detection Capabilities (Lack of Proactive/Creative Approaches) • Needs well documented requirements Drawbacks: • Low Accountability • Relies on the creativity of the Tester

  31. Session Based Test Management SBTM - Leveraging the benefits of both Testing Styles Scripted Testing Freestyle Testing • Elaborate Documentation • Static Test Suite • Higher Testing Costs • Limited Defect Detection • Concise Documentation • Dynamic Test Optimization • Lower Testing Costs • Better Defect Detection • No Accountability • No Predictability • High Possibility of Defect Slippage Automated Testing Session Based Test Management Exploratory Testing

  32. Evolution of Session Based Test Management Session Based Test Management Exploratory Testing Evolution Adhoc / Freestyle Testing

  33. Salient Features of Session Based Test Management • Application Learning, Test Design and Test Execution happen in Parallel • A Structured, measurable and accountable method of Exploratory Testing • Crisp Checklists to guide the tester on what needs to be tested • Execution happens in Test Sessions (Uninterrupted Test Execution Time) with specific charters, after which tester takes notes and makes a report • Tester takes a different execution route, if required, depending on issues found in a Test Session • Future test sessions are planned based on results from current session

  34. Metrics for Session Based Test Management Test Session Metrics Execution Metrics Defect Trend Cumulative Defect Trend

More Related