Effective Estimation Techniques for Test Managers

1. Top Down Viewof Estimation Test Managers Forum 25th April 2007

2. Estimating Credentials • Risk assessment on fixed price bespoke development projects • If test elements incorrect, the project lost money and my company would go out of business • Delivery of managed testing services • Contracted to meet specific and defined goals • Management of IBM Test Services • Included estimating testing for major programmes of work, fixed price contracts, and audits of global programmes when things not gone to plan

3. Topics • “Broad Brush” estimating • Specialist skills • Scope considerations • Building the team • Creating elapsed time from local factors • Metrics to consider

4. Broad Brush - Considerations • How much testing can we afford? • If you’re working to an existing budget, be sure and spend it wisely • What is our organisations standard percentage for test effort? • Many organisations have a standard view, even if it’s not documented, “we allocate (30%, 50% 10%) of our budget to testing • Do we have any specific time scale considerations? • Typically driven by market or legal demands • What are our quality objectives for this project? • Typically driven by business criticality and overall risk drivers

5. Broad Brush – Programme Parameters

6. Specialist Skills - Additional headcount • Performance • Not full time, but include planning early on and execution in several phases of testing • Security • Not full time, and maybe not as part of your testing project, but has to be included somewhere • Reliability • Typically part of operational acceptance, but consider value of including in integration testing on complex projects • Automation • Include in the Test Strategy, consider the value for each test phase and manage value contribution carefully

7. Scope Considerations • Environment set up • Who orders, receives and builds the environment? • Environment management • Who supports test activity with providing known state of all components? • Test data • Who will provide it, and manage it • Data migration • How will this be tested? • Operational acceptance and UAT • Even if it’s out of scope, what support is required from your team

8. Building the Team • Team structure adds in effort and elapsed time • How will the team be managed? • How many team leaders? • What tasks will Test Manager undertake to contribute to planning and execution effort – if any? • What is the scope of test support and who will do this? • Where will the skills come from? • Will they be available and at the right level?

9. Building the Team - Example

10. Elapsed Time – Local Factors • Test Data • Often difficult to provide data to support testing • Development resources available to fix bugs • Do your developers move onto other projects before testing is completed and signed off • Environment availability and support • Very few organisations get this right and it impacts test effectiveness greatly • Skills availability • Do you have all the skills internally, is training required • Meeting culture • Build meeting attendance into your plans • Frequency and amount of change • Build in time to review plans in line with the amount of change

11. Improving Estimating - Considerations • Actual against planned effort by phase • Track the lost time associated with local factors • Monitor level of quality achieved in production • Undertake causal analysis • Capture the effort required to find and then fix bugs – will require analysis but can be very enlightening

12. Example

13. Improving Estimation – E.G. • Cost • Effort expended on test activity by phase of testing – timesheets • Quality • Number of defects by severity registered during first 3 months of production • Number of released vs planned releases • Number of criteria met for each quality gate and impact of not having met specific criteria • Timescale • Actual against plan for project milestones • Ratio of test effort vs development effort against plan • Number and impact of scope changes by phase • Environment availability to support testing

14. In Summary • Estimating test effort is difficult – no right answer? • Can be more of an art than a science but there are standard industry metrics to help, what’s important is learn from your experience so capture the right metrics and report accurately • Test effort is a small part of where the time goes on a project • Remember team structure, how things work in your organisation and lessons learnt from previous projects • Testers love to discuss estimation and metrics experience • Network in the industry internet community, attend events, absorb information from all sources, then relate to your own business goals.

15. Discuss!