OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy

1. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Metrics in the context of the CMM/SPICE SPIN-UK, 29 September 1998

2. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Some views on measurement… • “What is not measurable, make measurable.”-Galileo • “When you can measure what you are speaking about and express it in numbers you know something about it; but when you cannot measure it, when you cannot express it in numbers your knowledge is of a meagre and unsatisfactory kind.” - Lord Kelvin • “…This remark [above] has been quoted with an approval it does not altogether deserve - it does not, for example, do justice to the work of Darwin or Pasteur, to name only two of Kelvin’s contemporaries.” - G.U.Yule (statistician) • “You can’t control what you can’t measure.” - DeMarco • “Metrics is crap.” - Dijkstra

3. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy What is measurement?

4. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy real world Empirical relational system

5. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy real world mathematical world Empirical relational system Formal relational system measurement

6. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy real world mathematical world Empirical relational system Formal relational system measurement mathematics and statistics Results

7. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy real world mathematical world Empirical relational system Formal relational system measurement mathematics and statistics Relevant empirical information Results interpretation

8. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy real world mathematical world Empirical relational system Formal relational system measurement decisions and actions mathematics and statistics Relevant empirical information Results interpretation

9. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Characteristics... • good measurement should not (in general) conflict with empirical insights • sound measurement systems reflect the level of understanding of those things that are measured • good measurement encourages improved understanding

10. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy real world mathematical world Empirical relational system Formal relational system refined measurement better decisions and actions mathematics and statistics more sophisticated interpretation Relevant empirical information Results

11. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Measurement - state of practice… • can be self serving, but increasing awareness of role • often not well founded, but effort is invested • data collected but often not valued (even when useful)

12. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Effective measurement require an appreciation of... • measurement fundamentals • how do we measure? • context • what do we need to measure? • what can we measure? • soft (human) factors • why should I measure/be measured?

13. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Measurement within the Capability Maturity Model:

14. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy The Capability Maturity Model Optimizing Managed Defined Repeatable Organizational abilities are described as five levels of increasing capability Initial

15. OPF OPD ISM IGC PR TP SPE OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy maturity level key process areas common features commitment Defined activities verification ability M & A measurement and analysis in each KPA from levels 2 to 5

16. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Measurement within the CMM... • is a common feature of every key process area at each described maturity level • measurement and analysis common feature is least well described • SEI recognize need for organizations to derive their own measures • SEI say nothing about how to measure • in practice we tend to find measurement and analysis is the last common feature to be satisfied for any given KPA • can use this as an informal assessment ‘short cut’.

17. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy ...Measurement within the CMM... • When investigating measurement capability look for... • a complementary view of quality throughout the organization • recognition of why data is collected, and understanding of definitions and analysis • data verification (accuracy known) and measures validated • information used to make decisions and direct actions

18. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy ...Measurement within the CMM • Level 4 • measure products and process quantitatively • Software Quality Management • Quantitative Process Management • aspires to production engineering (cf project engineering) • still controversial

19. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Measurement within SPICE (ISO 15504)

20. Proc 1 Practice 1 CUS Proc 2 Practice 2 Eng. 3 Develop S/W Design Practice 3 ENG Proc 4 Eng 3.4 Establish traceability PROJ Proc 4 Proc 5 SUPP ORG OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy The SPICE Architecture Process Capability 5 4 Well defined 2 1 0

21. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy A comparison of the CMM and SPICE • SPICE... • is more complex • this is a good thing and a bad thing • it may be unrealistic • but does require thought • is structured by process • ...not capability - so may make arbitrary target setting more difficult • allows all processes to be addressed to some extent (c.f. CMM’s SPE) • has process capability defined by generic processes

22. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Measurement within SPICE... • Measures required throughout (like CMM)... • Generic Practices (define process capability of every process) • level 2 - plan process (I.e. estimation) • level 2 - track with measurement • note - “use of measurement implies measures have been defined and selected and data has been collected” • level 3 - use well defined data • level 4 - measurement oriented (like CMM level 4)

23. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy ...Measurement within SPICE... • ...Measures required throughout (like CMM) • Processes • engineering 2.2 hints at ISO9126 • project 2.5 and 7.2 - size, effort,cost schedule resources risks • project 5.2 - define quality metrics • organization 2.9 - define process measures • organization 2.12 - set performance expectations • organization 3.6 - define measures of impact

24. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy ...Measurement within SPICE • introduced at low capability level (like CMM) • states expectation of what measurement requires (unlike CMM) • (even at low capability level) • minimal guidance (examples) on what to measure (less than CMM) • implies a distinct capability to define and assess measures

25. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy Summary • Measurement • within the Capability Maturity Model • within SPICE (ISO 15504)

26. OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy C.C.Shelley shelley@osel.netkonect.co.uk +44 (0) 1993 700878 O X F O R D S O F T W A R E E N G I N E E R I N G L I M I T E D www.osel.co.uk