1 / 20

Automation @ Squirrel Systems

Automation @ Squirrel Systems. Our automated solution at a glance By: Justin Hollander. About the Presenter. Employed at Squirrel Systems since August 2007 as a QA Developer/Analyst

xenon
Download Presentation

Automation @ Squirrel Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Automation @ SquirrelSystems Our automated solution at a glance By: Justin Hollander

  2. About the Presenter • Employed at Squirrel Systems since August 2007 as a QA Developer/Analyst • Volunteered one year in Guatemala to help build an online system for the Central American district. Trained staff and volunteers to use the systems. Prepared manuals and other documentation • BCIT Alumni and currently attending BCIT

  3. About the Presentation • Benefits of Automation • Description of automation needs at Squirrel Systems • Design goals vs. cost considerations • Previous attempts at solving the automation / autoretest problem • The current design of our autoretest solution • Problems we overcame • Future Considerations • Questions and Answers

  4. Our Automation Needs • Intended to reduce overhead of manual certification checklists • To allow other testers to do more functional / exploratory testing on other parts of the system that cannot be tested in an automated fashion • Consistency in testing. Day in and day out.

  5. Previous Attempts • Attempt 1: Batch processing, relied on playback utilities, with an Access Database(DB) to manually track successes and failures of the running scripts • Attempt 2: Improved batch processing(more readable scripts), relied on playback utilities, without the Access DB to track the successes and failures of the running scripts

  6. Problems with Attempt 1 • Script information and state were manually tracked • Batch scripting is not as flexible or powerful as a programmatic solution to get the options you need for a comprehensive automated solution • Results were not archived and therefore could not be linked • Output was in a raw text only format and very cumbersome to process manually • Too many false positives as comparison was blind, and not intelligent, nearly impossible to trust results

  7. Problems with Attempt 2 • Same as attempt 1, except now, without the Access DB, there was no way to track the running scripts • Solution was scaled back due to resource constraints and the project had a hard time getting traction and support • Ultimately, the project was suspended

  8. What we Learned • Automation/autoretest is more complex than we thought • There is no “easy” solution • Externalizing our test input data while using our playback utilities to retest the system is impossible • At a minimum; output data must be analyzed better to eliminate false positives • Batch scripting isn’t working • A more dedicated approach is required

  9. Autoretest, TAKE 3! • Wedecided to try this again • Decision to scrap previous work and start again from design phase • Requirements were well understood, however, the right skill-set was missing until now

  10. High level Design vs. Cost Design goals Cost Considerations • To use existing playback tools within a framework to retest recorded ‘tracks’ • Use intelligent comparison algorithms to rule out false positives • Output results in a human friendly format in order to ensure a quicker analysis of the potential problems • Externalize the test input data from the system • Re-usable parts to extend framework to all parts of POS system • Limited to 40 man-hours a week(Basically me ) • Development team on a tight schedule and have limited resources for this project • No money for off-the-shelf solution • Management wanted a solution rather quickly(6 months) • Money already invested; 2 previous attempts in solving this in last 5 years have failed

  11. The Current Design • Client/Server model • Written in Java with easy to use administrative GUI • Externalized data as much as possible • Used intelligent output comparison algorithms • One DB configuration supports one or more test scripts • Supports XML processing • Comparison output formatted in HTML • Automatically detect if software that is being tested is the current build. Will update if needed

  12. The Current Design cont.. • Test outputs and comparison results archived to achieve the possibility of linking of historical data to present results • Script information stored in a relational SQL DB • Script information linked to GUI using custom written SQL driver • Script states linked to scripts via GUI and they are not static • Client has a head-less mode supported by multi-threaded timer mechanism • Extensible due to its polymorphic nature and can easily support additional forms/levels of testing

  13. Screenshot of GUI

  14. Screenshot of formatted Output

  15. Screenshot of XML Utilization

  16. Code Sample – DB Compare private static void processRows(Vector<String> vB, Vector<String> vR) { if (vB.toString().equals(vR.toString())) //table contents equal?? { vB.removeAllElements(); vR.removeAllElements(); } else //everything not equal probable bug or code change in squirrel { if (vB.size() == vR.size()) //just mismatches ?? { for (int i = 0; i < vB.size(); ++i) { if (vB.get(i).equals(vR.get(i))) { vB.set(i, "VALID"); vR.set(i, "VALID"); //the only ones not tagged are the mismatches and the can get filtered out while still preserving the row number } } } else if(vB.size() > vR.size()) //missingData ?? validateDifferentSizeTable(vR,vB); else if(vB.size() < vR.size()) //extraData, this should be rare validateDifferentSizeTable(vB,vR); }// that should leave us with any differences for the HTML Manager to process further. }

  17. Problems Solved with this implementation • All DB false positives have vanished • We are able to use our scripts longer, as it is easier to trust output with limited false positives • Output is easy on the eyes with HTML colour coded output • Playback utilities have been made more robust • System is much more configurable than before

  18. Considerations for the Future • Remote Web Administration using JSP and Java servlets • Make better use of RMI technology and turn this into a distributed system from a client/server model • Completely externalized data • Build robots to do the maintenance on the system 

  19. Questions and Answers

  20. THANK YOU ALL

More Related