COSYSMO 2.0: A Cost Model and Framework for Systems Engineering Reuse

1. COSYSMO 2.0: A Cost Model and Framework for Systems Engineering Reuse Jared Fortune University of Southern California Ricardo Valerdi Massachusetts Institute of Technology 2009 COCOMO Forum Massachusetts Institute of Technology

2. Outline • Introduction • Research Question and Hypotheses • Methodology • Research Approach • Research Products • COSYSMO 2.0 • Reuse Framework • Results • Model Validation • Framework Validation • Conclusion • Future Research Areas

3. Need to Estimate Effect of Systems Engineering Reuse • Practitioners noted large errors between model estimates and actuals • Variations discovered to be attributable to programs with reuse • Systems engineering reuse • The utilization of systems engineering products from previous efforts • Ex: architectures, requirements, test plans, interfaces • Means of reducing cost, schedule, and/or risk • Current version COSYSMO assumes all systems engineering products will be “built from scratch” • Unable to account for the impact of systems engineering reuse …COSYSMO 2.0

4. Reuse is a Universal Concept

5. Academic Perspective Literature Review Results • Reuse is done for the purpose of economic benefit, intending to shorten schedule, reduce cost, and/or increase performance. • Reuse is not free, upfront investment is required. • Hardware, software, processes, and knowledge are all reusable products. • Reuse needs to be planned from the conceptualization phase of system development. • Reuse is as much of an organizational issue as it is a technical one. • The benefits of reuse are limited to related domains and do not scale linearly.

6. Practitioner Perspective • Industry Survey Results • Requirements reuse is only performed occasionally, but has the largest benefit associated with it. • Reuse occurs more frequently early in the life cycle than later. • Cost savings is the most promoted benefit for reuse, but benefits also exist in risk, schedule, and performance. • Experienced personnel is a key factor for successful reuse.

7. Research Evolution Updated to six Categories Focused on capturing reuse in size drivers Previous Hypotheses • Each COSYSMO size driver can be further decomposed into New, Modified, Adopted, Managed, and Deleted categories of reuse each with corresponding rating scales and weights, and function as statistically significant predictors of equivalent size from the systems engineering standpoint. • Two additional COSYSMO cost drivers, Reuse Understanding and Artifact Unfamiliarity, capture the important organizational, personnel, process, and project factors that lead to the accurate estimation of systems engineering effort for systems that involve reuse.

8. Research Question What technical and organizational characteristics drive the amount of reduced or added systems engineering effort due to systems engineering product reuse?

9. Hypothesis #1 The requirements size driver can be further decomposed into New, Design for Reuse, Modified, Adopted, Managed, and Deleted categories of reuse, each with corresponding definitions and weights, and function as a good predictor of equivalent size from a systems engineering standpoint.

10. Hypothesis #2 By comparing the normative and descriptive models for managing systems engineering reuse, the systems engineering reuse framework provides a prescriptive model for enabling organizations to take advantage of reuse in systems engineering.

11. Hypothesis #3 The COSYSMO 2.0 model and the associated systems engineering reuse framework help organizations think differently about systems engineering reuse, and its associated costs and benefits.

12. Mixed-Methods Research Approach

13. COSYSMO 2.0 Operational Concept

14. Model Form

15. Reuse Categories

16. Reuse Category Weights

17. Example COSYSMO 2.0 Estimate Estimated as 129.1 Person-Months by COSYSMO (without reuse) …a 30.4% difference

18. COSYSMO Calibration Data Set Analysis (1) • Data set used to calibrate and validate the COSYSMO tool when it was published in 2005 • Immature form of reuse data was collected • Reported amount of reuse in the system • Ex: If a responder reported the project had 100 nominal requirements and that 20 of those 100 requirements where reused, they would report 20% of the nominal requirements were reused • No consistent definition of term • Results indicated potential for overestimation of effort • 42 projects in the data set • 54% reported some amount of reuse in one of the four size drivers • 13% reported some amount of reuse in all of the size drivers.

19. COSYSMO Calibration Data Set Analysis (2) • In most cases, COSYSMO is capable of estimating the effort of a systems engineering project within 30% of the actual, 50% of the time • PRED(.30)=50% • Experimental model created to test hypothesis that a single reuse category is inadequate • Experimental model utilized one reuse category instead of six • Category weights varied parametrically from 0.0 to 1.5 • Estimation power of one versus no reuse categories were compared • Experimental model (one category) consistently performed worse than standard COSYSMO (no reuse categories) • Supports the conclusion that a single reuse category is inadequate, multiple reuse categories are required

20. COSYSMO 2.0 Implementation Results • Across 44 projects at 1 diversified organization • Using COSYSMO: • PRED(.30) = 14% • PRED(.40) = 20% • PRED(.50) = 20% • R2 = 0.50 • Using COSYSMO 2.0: • PRED(.30) = 34% • PRED(.40) = 50% • PRED(.50) = 57% • R2 = 0.72 • Result: 36 of 44 (82%) estimates improved

21. Reuse Framework

22. Reuse Framework Interview Results • Conducted interviews and mini-case studies with five systems engineering organizations • BAE Systems U.S., BAE Systems U.K., Boeing, British Petroleum (BP), and Rolls-Royce “Applicable and valuable to organizations at multiple levels of reuse maturity.” Organization A “A valid and reasonable illustration of the reuse process.” Organization B

23. Research makes SE’s think differently about reuse (1) “Systems Engineers are frequently tasked to leverage previously developed systems engineering products for the benefit of future projects. The inclusion of reuse in COSYSMO has an inherent value by adding clarity to project characterization and measurement refinement to the estimation of systems engineering cost.” Marilee Wheaton, Systems Engineering Division, The Aerospace Corporation "Metrics-based estimates do account for the cost of systems engineering or systems engineering reuse very well. The parametric nature of COSYSMO 2.0, as COCOMO did for software reuse, improves fidelity and confidence in estimates of systems engineering reuse and provides systems engineers with valuable feedback on reuse decisions.” Allan McQuarrie, Leveraged Technology & Products, BAE Systems

24. Research makes SE’s think differently about reuse (2) “The benefits of the COSYSMO reuse model are significant, including improving the accuracy of COSYSMO and demonstrating its validity across a wider set of systems engineering projects. The reuse model fundamentally enables the development of the Total Engineering Estimation Model and Process (TEEMaP) estimating model, an implementation of COSYSMO, to achieve high degrees of estimation accuracy and robustness. TEEMaP is now widely used in different Lines of Businesses.” Gan Wang, Electronics and Integrated Systems, BAE Systems

25. Threats to Validity • Mostly Uncontrollable • Limited Visibility in Data Analysis • Performed using inputs and guidance from this research • Relied on Subjective Inputs • Utilized experts familiar with COSYSMO research • Increased Number of User Inputs • Created 3 prime & 3 sub-categories to address needs raised in discussions Mostly Controllable • Construct Validity • Minimized COSYSMO overlap • Identified products for reuse • Result Convergence • Utilized mixed-methods approach • Internal Validity • Iterated definitions at multiple COSYSMO workshops • Collected multiple Delphi results • Generalizability • Validated model with multiple projects • Validated framework with interviews at multiple organizations

26. Contributions • Identification of needs and gaps in the research of systems engineering reuse • Literature review • Definition and characterization of systems engineering reuse • Reuse categories • Improvement of the COSYSMO tool to account for systems engineering reuse • COSYSMO 2.0 • Documentation of heuristics for systems engineering reuse • Reuse framework • Systems engineers will think about their responsibilities differently • COSYSMO 2.0 and reuse framework

27. Academic COSYSMO 2.0 (without reuse)

28. Academic COSYSMO 2.0 (with reuse)

29. Future Areas of Research • Tailoring COSYSMO 2.0 to individual organizations • Developing industry and local calibrations • Exploring a range of weights for the reuse categories • Evaluating effect of variation within categories • Identifying reuse opportunities and mismatches • Extending development of Expert COSYSMO • Quantifying the effect of reuse on risk and schedule • Supporting development of new COSYSMO extensions • Harmonizing reuse across hardware, software, and systems engineering • Continuing efforts to integrate COCOMO and COSYSMO