Software Development QA Best Practices

1. Software DevelopmentQA Best Practices Suzette Hackl, CSM Senior Project Manager Skyline Technologies, Inc. May 20, 2010

2. Agenda • Definition of “Best Practices” • Software Quality Assurance as it relates to: • Process • People • Technology • Application to SCRUM • Questions?

3. “Best Practices” • A collection of Software Quality “What Works Now” practices. • Suzette’s collection of practices come from many sources and span across industries, sectors, and experiences. • Recognizable, Obvious, Debatable….. • The list of practices in the QA space is large. A large laundry list is hard to conceptualize and translate into implementation. I have grouped my culled down list by People, Process, and Technology and categorized them by: • Basic: The basic are exactly that. They are the training wheels you need to get started and when you take them off, it is evident that you know how to ride. When you eventually take off your training wheels and ride that does not mean that you always remember how to ride. Each one of these basic practices is “needed” for successful quality assurance. Removing or discontinuing one of these basic practices will degrade quality of your systems and potentially put business at risk. • Foundational: The foundational practices are the rock and mortar that will hold your “Quality” program together. They have significant value adds to an organization. Unlike the basic, they are probably not as well known and therefore need implementation help. • Incremental: The incremental practices are to be judiciously applied in special conditions and bring many returns in terms of value. Individually they do not provide broad gains across the board of testing because of their specialized nature.

4. Agenda Definition of “Best Practices” Software Quality Assurance as it relates to: Process People Technology Application to SCRUM Questions?

5. QA Processes • Standard & Consistent Requirements for Test Planning • One of the roles of software testing is to ensure that the product meets the requirements of the client. Capturing the requirements therefore becomes an essential part not only to help develop but to create test plans that can be used to gauge if the developed application is likely to meet the customers’ needs. Therefore, requirement management and its translation to produce test plans is an important step. The premise of this practice is that requirements are consistently and clearly documented so as to be interpreted correctly by the development and QA team members. • Standard & Consistent Functional Specifications or Solution Design Documentation • Functional Specifications are a key part of many development processes. While it is a development process aspect, it is critically necessary for software functional testing. The advantage of having a functional specification is the test generation activities could happen in parallel with the development of the code. This is ideal from several dimensions. Firstly, it gains parallelism in execution, removing a serious serialization bottleneck in the development process. By the time the software code is ready, the test cases are also ready to be ran against the code. Secondly, if forces a degree of clarity from the perspective of a designer and an architect so essential for the overall efficiencies of development. Thirdly, the functional specifications become documentation that can be shared with the business to gain an additional perspective on what is being developed. • Common Glossary of QA Terminology • This may seem to be the most basic of all concepts but it is one of the most important. Having clearly defined terms communicated across the organization will go a long way in the acceptance and engagement of Quality Assurance

6. QA Processes - continued Reviews & Inspection This best practice includes code reviews, Internal and external QA Audits, and Project Audits. Test Monitoring & Governance The premise of test monitoring is to ensure quality key performance indicators are being met. This is done thru verification across project phases and deliverables. By test monitoring, QA assures software completeness and readiness for delivery.

7. QA Processes - continued Formal Entry and Exit Criteria The idea is that every process step has a precise entry and precise exit criteria. These are defined by the development process and are watched by management to gate the movement from one stage to another. The preciseness of these criteria is determined by each individual project leadership. However, this practice allows much more careful management of the software development process. Functional Test - VariationsMost functional tests are written as black box tests working off a functional specification. The numbers of test cases that are generated usually are variations on the input space coupled with visiting the output conditions. Functional testing involves writing down different variations to cover as much of the state space as one deems necessary for a program. The best practice involves understanding how to write variations and gain coverage which is adequate enough to thoroughly test the function. Given that there is no measure of coverage for functional tests, the practice of writing variations does involve an element of art User Validation and AcceptanceThe idea is to release an application to a limited number of users and get feedback to fix problems before implemented to the entire user community. This best practice has everything to do with getting functionality in the hands of users as early in the process so as to reduce cost / business interruptions and post implementation issues

8. QA Processes - continued Automated Test ExecutionThe use of automated testing to support rapid testing, capturing issues earlier in the development lifecycle, and providing detailed logs of frequent test results (sometimes through nightly builds). The automated test environment must be carefully setup to ensure accurate and consistent testing outcomes. Automatic testing should be run at code check-in to minimize changes of disrupting a working build. The goal of automated test execution is to minimize the amount of manual work involved in test execution and gain higher coverage with a larger number of test cases. The automated test execution has a significant impact on both the tool sets for test execution and also the way tests are designed. Integral to automated testing is the verification of current operations and logging of test failures with diagnosis information “Nightly” Builds & Continuous TestingThe concept captures frequent builds from changes that are being promoted into the change control system. The advantage is firstly, that if a major regression occurs because of errors recently generated, they are captured quickly. Secondly, regression tests can be run in the background and cycled when it makes sense. Thirdly, the newer releases of software are available to developers and testers sooner. Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It reduces the time and energy required to keep code well-tested, and prevents regression errors from persisting uncaught for long periods of time.

9. QA Processes - continued Integration Testing with User Scenarios As we integrate multiple software products and create end user applications that invoke one or multiple systems, the task of testing the end user features gets complicated. One of the viable methods of testing is to develop user scenarios that exercise the functionality of the applications. These are called user scenarios. The advantage of the user scenario is that it tests systems in the ways that most likely reflect a user’s everyday usage Software Development Lifecycle & Incorporating QAQuality Assurance encompasses the entire software development process, which includes processes such as software design, coding, source code control, code reviews, change Management, configuration management, and release management. There are specific phase activities that should be conducted during the software life cycle. At the conclusion of each phase there is a key gate check and management decision that the existing phase has been completed and approval to proceed to the next phase. Below is an example of how this applies to SCRUM.

10. QA Processes - continued Quality Assurance Benchmarks and Performance Indicators Establish and collect useful metrics that will facilitate decision making. Benchmarks and Performance Indicators (PI´s) cannot be meaningfully formulated unless there is a direct connection between them and the related policies and goals. Usability Testing Usability testing means a way to measure how people (intended/end user) find it (easy, moderate or hard) to interact with and use the system keeping its purpose in mind. It is a standard statement that "Usability testing measures the usability of the system". Some would argue that application usability perceived by the user community is the utmost factor that determines success. Usability testing needs to not only assess how usable a system is but also provide feedback on methods to improve the user experience and thereby gain a positive quality image. Test Case Quality and Re-usability This practice is two-fold. Firstly, the valuation of a test case’s quality. Secondly, the value of re-using test cases where applicable. Secondly, the re-usability of test cases. Pairing There are various methods of pairing skilled individuals of a project team to produce higher quality of work more efficiently; pairing developers, teaming testers with developers, etc. It’s been recognized for a long time that the close coupling of testers with developers improves both the test cases and the code that is developed

11. QA Processes - continued Risk Management and ConfidenceThe notion of this practice is to focus testing on risky areas and to find the important bugs; you have to practice some kind of Risk Management. There is a factor of confidence used here. The practice of accessing a risk factor to tests can also help with resource modeling. For example; testing of high risk, low confidence functionalities make more sense to be staffed by those QA Team members with high domain knowledge while low risk, high confidence functionalities could be staff by lower priced resources.

12. QA Processes - continued Bug-Tracking and Resolution with Root Cause Analysis Estimating Traceability The linkages between test cases, their parent design elements, and their parent requirements are maintained through a technique termed Requirements Traceability. In essence, it is necessary to be able to trace the linkage from a test case all the way through to a grandparent goal.

13. Agenda Definition of “Best Practices” Software Quality Assurance as it relates to: Process People Technology Application to SCRUM Questions?

14. People Software Quality Ownership The premise of this practice is that business owns the task of determining the quality expectations and KPIs of an application. Quality Control owns the task of assuring that software meets those quality guidelines. Clearly Defined QA Roles & Expectations QA Professional/Senior Lead (QA Analyst III) QA Lead (QA Analyst II ) QA Analyst (QA Analyst I) SCRUM Team Member QA Analyst Domain Knowledge Testers require domain knowledge and have to be at least as skilled as an expert user/power user. Good Writing Skills are Mandatory Writing understandable and repeatable test scripts, which also contains information about the idea and the intention of the test, the expected results and technical background, isn’t easy. There's also a fine line between too detailed test scripts, which are difficult to maintain, and too superficial ones. This is truly an “art” and not a science and takes experience. A good test case is one which brings you closer to your goal. There’s no simple formula or prescription for generating “good” test cases.

15. People - continued Software Delivery & Capacity PlanningA unit of work represents both the functionality being delivered and the quality of the functionality. We see that each work unit has both functionality and a quality component. We don’t want to intentionally plan to deliver functionality without quality QA Members have broad and comprehensive knowledge in IT The premise of this best practice is that QA folks mustn’t be experts or gurus, but they should be able to warm up fast to many technologies. For example different operating systems, networks, Internet, databases, program languages, etc. Quality Assurance is not only closely related to development but also support, technical writing and field service. Therefore QA engineers should have some knowledge and experience in at least one of these areas. There shouldn't also be any borders or constraints that prevent or discourage QA people to switch to development, technical writing, field service or support - and vice versa.

16. People - continued QA people need to be good communicators and team players They must be diplomatic (but sometimes tough), convincing and able to present. QA Job Satisfaction - Paradigm shift Awareness of the value of Quality Assurance to an organization greatly impacts job satisfaction for a QA professional. This includes the understanding and acceptance of testing limitations.

17. Agenda Definition of “Best Practices” Software Quality Assurance as it relates to: Process People Technology Application to SCRUM Questions?

18. Technology System Intellectual Intelligence This practice is related to the system knowledge held internally by an organization and its owning group. This knowledge ranges from architectural design to code practices and standards used in the application. This is a control measure for protection. Augmenting development teams with consultants is a common practice as a means of meeting surges and recessions in business demands. Nevertheless, retaining an underlying knowledge level is important for continuity and risk mitigation. The level of intellectual intelligence retained is unique to the specific software application, what business functions, and the value of these functions. Application Ownership We have already discussed the concept of quality ownership. The practice is the answer to the question “Who is the person that has the ultimate ownership of the application?” It is the one single ring able neck and product owner as described in SCRUM. IT has the responsibility for ensuring the system is maintained, sustained, and operational. Automation Testing ToolsUnit test automation, System or Functional test automation, Performance test automation, Regression test automation Technical Infrastructure and Support Test Environments & Supporting applications and job scheduling tools

19. Technology Utilization of single platform to support QA Processes The concept of a single platform to support multiple QA processes is not new. There are studies that show how quality attributes can be lost between systems resulting in delivery of a product which is not what the client expected. The use of a single platform for work/project requests, requirements, test plans, test cases, design, development, test execution is very useful. It reduces the potential for loss of data and redundancy as well as provides one source for metric reporting. Refactoring Refactoring is the concept of cleaning up, improving the inner working of a system. All system functionality and outcomes are to remain unaffected given the same inputs. The amount of refactoring is dependent on the “swamp” or “ball of spaghetti” inside the applications. Lack of architecture direction and strategic planning of application growth can often result in this situation. Refactoring neither fixes bugs nor adds new functionality, though it might precede either activity. Rather, it improves the understandability of the code, changes its internal structure and design, and removes dead code. The amount of refactoring is based on the technical team’s willingness and speed to technical debt repayment plan. Code Coverage Analysis/Tool The process of finding areas of a program not exercised by a set of test cases, creating additional test cases to increase coverage, and determining a quantitative measure of code coverage, which is an indirect measure of quality. An optional aspect of code coverage analysis is identifying redundant test cases that do not increase coverage. Analysis of code coverage (or test coverage analysis) is a technique whereby test cases are created to exercise all parts of the software.

20. Agenda Definition of “Best Practices” Software Quality Assurance as it relates to: Process People Technology Application to SCRUM Questions?

21. The Scrum Framework Potential Deployment ScrumPlanning Product & Team Backlog Formation Sprint2-4 Weeks Team Retrospective Sprint Planning2 Parts: Selection and Decomp Sprint Review Daily Scrum

22. Agenda Definition of “Best Practices” Software Quality Assurance as it relates to: Process People Technology Application to SCRUM Questions?

23. Questions?

24. Thank You For questions or more information, please contact: Suzette Hackl, Senior Project Manager shackl@skylinetechnologies.com