COSYSMO Extension as a Proxy Systems Cost Estimation

1. COSYSMO Extension as a Proxy Systems Cost Estimation Reggie ColeLockheed Martin Senior Fellowreggie.cole@lmco.com Garry RoedlerLockheed Martin Fellowgarry.j.roedler@lmco.com April 30, 2014

2. Agenda COSYSMO as a Proxy for System Cost Estimation Deep Dive on the Proxy/Bias Function Key Use Cases Workshop Results and Recommendations

3. COSYSMO as a Proxy for System Cost Estimation

4. COSYSMO as a Proxy Estimator for System Cost • Need for a Top-Down System Cost Model • Better Buying Power 2.0 Mandate • Perform ongoing should-cost analysis • Better Buying Power 2.0 Implication • Perform design-to-cost analysis and design for affordability • There is Currently a Gap in Tools • Primarily in early-stage analysis – when directions can still be changed without significant repercussions • Extending COSYSMO for System Costing • Evaluate the Viability of Extending COSYSMO • COSYSMO seems to have the right parameters

5. Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy We Have a Good Selection of Tools for Late-Stage Cost Modeling Detailed Solution Description Cost Estimate ± 5% High-Level Solution Description Cost Estimate ± 10% Increasingly Refined Information About the Solution Increasingly Refined Cost Estimate Problem-Space Description Increasingly Refined Solution High-Level Solution Assumptions Cost Estimate ± 20% Cost Estimate ± 25% We Have Gaps in Early-Stage Cost Modeling Increasing Effort and Cost-Modeling Expertise

6. SE Effort as a Proxy Measure of Overall System Size and Complexity • Proxy Measures • Proxy measures are used when you cannot directly measure what you want to measure – and when an indirect measure provides sufficient insight • Proxy measures are often used in clinical studies since direct measurement is often infeasible or can even alter the outcome • It is not always possible to directly measure what you want to measure – or directly estimate what you want to estimate • System Engineering Effort is a Proxy Measure for System Cost • There is strong evidence for the link between systems engineering effort and program cost – dating back to a NASA study in the 1980s • The optimal relationship between systems engineering effort and overall program cost is 10% - 15% • Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems • Systems engineering effort can be an effective proxy measure for overall system cost

7. COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Sizeand Complexity Size Drivers Number of System Requirements Initial Estimate of System Size Number of Major System Interfaces Number of Critical Algorithms Number of Operational Scenarios Requirements Understanding Reuse Factors Cost Drivers Architecture Understanding Managed Elements Level of Service Requirements Adopted Elements Migration Complexity Deleted Elements Scaled Estimate of System Size Technology Risk Modified Elements Level of Documentation Required New Elements Diversity of Installed Platforms Consolidated Cost Driver Factor Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost Multisite Coordination Level of Tool Support

8. Relationship Between SE Effort and Total Effort NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

9. Putting It All Together Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort • Size Drivers (Problem Space) • Customer Requirements • System Interfaces • Major Algorithms • Operational Scenarios • Complexity Drivers (Problem/Solution) • Requirements Understanding • Architecture Understanding • Level of Service Requirements • Migration Complexity • Technology Risk • Documentation Needs • Installations/Platform Diversity • Levels of Recursion in the Design • Stakeholder Team Cohesion • Personnel/Team Capability • Personnel Experience/Continuity • Process Capability • Multisite Coordination • Tool Support • Reuse Factors (Solution Space) • New • Modified • Deleted • Adopted • Managed SE Effort is an estimator for total system cost…but it is a biased estimator Estimation of Total System Cost

10. Adjusting Our View of COSYSMO Parameters Our view of the size drivers can be preserved – their context doesn’t change under COSYSMO extension Our view of reuse requires the most extreme adjustment – we are not just talking about systems engineering artifacts Our view of the cost drivers needs to be broadened to include all aspects of the system, not just systems engineering

11. Expanding Our View of Cost Drivers Precedented systems and unprecedented systems are fundamentally different Cost Drivers Requirements Understanding Architecture Understanding System modification and reuse have a significant effect on some cost drivers Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Need to consider the entire team – including subcontractors Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Need to consider the all processes and tools Process Capability Multisite Coordination Level of Tool Support Expand to a view of all aspects of the system for the Cost Drivers

12. Expanding Our View of Reuse Factors New Elements – These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient. Reuse Factors Modified Elements – These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category. For Requirements… Think Functional Components New Elements Modified Elements For Interfaces… Think Connection Points Deleted Elements Adopted Elements Adopted Elements – These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category. For Algorithms… Think Functional Components Managed Elements For Scenarios… Think Implications to Both Managed Elements – These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category. Need to include the system elements, as well as the SE artifacts

13. Example • Consider the case of large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers. • Case 1 – Unprecedented System • Case 2 – Developed Replacement System • Case 3 – COST/GOTS-Based Replacement System Similar Cost Drivers – But Significantly Different Size Drivers Similar Size Drivers – But Significantly Different Cost Drivers

14. Part 1 Wrap-Up • The Approach Based on Well-Established Approaches • COSYSMO provides the basis for estimation of systems engineering effort – and a biased proxy estimator for overall system cost • There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort • The Approach Can Improve System Cost Modeling • It occupies an important niche – fully parametric system cost modeling in the early stages of system definition • It can serve as a powerful affordability analysis tool – supporting rapid-turnaround analysis of alternatives • But…it is not a replacement for existing models

15. Deep Dive on the Proxy/Bias Function

16. Context of the Bias Function Deeper Discussion of the Proxy/Bias Function is Necessary – As Well as a Technique for Generating Cumulative Distribution of System Costs

17. The Bias Function We are going to discuss each of these factors!

18. SE Effort (EffortSE) COSYSMO provides a Proxy Estimate of the system cost. We will not try to de-bias it right now….that is the next step. • Size Drivers (Problem Space) • Customer Requirements • System Interfaces • Major Algorithms • Operational Scenarios • Complexity Drivers (Problem/Solution) • Requirements Understanding • Architecture Understanding • Level of Service Requirements • Migration Complexity • Technology Risk • Documentation Needs • Installations/Platform Diversity • Levels of Recursion in the Design • Stakeholder Team Cohesion • Personnel/Team Capability • Personnel Experience/Continuity • Process Capability • Multisite Coordination • Tool Support • Reuse Factors (Solution Space) • New • Modified • Deleted • Adopted • Managed Since we want to perform Monte Carlo simulation of costs, we would like to generate a distribution of the proxy costs. Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for a sampling distribution. Pessimistic Optimistic Expected COSYSMO Estimate of Hours Becomes the Parameters for Either Triangular or Beta Pert Distribution Beta Pert Distribution Triangular Distribution

19. Effort Conversion Factor (FConv) This is probably the most important factor in the bias function! Studies provide some insight into what the value should be Heuristic approaches for determining SE costs for software intensive systems are also consistent with these studies All indications point to a range of 0.08 to 0.16 for FConv And this range is consistent with all the data we’ve collected to date…for relatively healthy programs And it provides the basis for our random variable parameters Expected Pessimistic Optimistic 0.08 0.12 0.16 Beta Pert Distribution Triangular Distribution

20. Other Stochastic “Adder” Factors • Labor Costs • Labor costs vary – especially in early stages – and needs to be treated as a random variable • Any number of distributions would probably be OK – Beta Pert would be a good default • If hours are the item of interest rather than cost, this factor can be omitted • Material Costs • Material costs vary – especially in early stages – and needs to be treated as a random variable • Any number of distributions would probably be OK – Beta Pert would be a good default but there is at least one study that looks at using a normal distribution • If hours are the item of interest rather than cost, this factor can be omitted • Travel Costs • Travel costs vary – especially in early stages – and needs to be treated as a random variable • Any number of distributions would probably be OK – Beta Pert would be a good default • If hours are the item of interest rather than cost, this factor can be omitted • Other • Any number of other “stochastic adders” can be treated similarly

21. Other “Deterministic Adders” • Some Factors Are Well Known • To the Point They Can Be Considered Deterministic • They are often set and apply across programs • Examples include: • G&A Costs • Other Direct costs • Management Reserve • Fee

22. COSYSMO Calibration Factor • Local Calibration is Important • Keeps us from over-tuning the Effort Conversion Factor • This Also Serves as a Type of Organizational Efficiency Factor • Can vary across organizations within an enterprise • It is a Simple Scalar Factor • Optimally, it should be 1.0

23. Calibration for Proxy Estimation • It is Not Necessary to Revalidate or Recalibrate COSYSMO • The strength of this approach is that it rests on the COSYSMO foundation • It is Necessary to Validate and Calibrate the Bias Function • Important to validate the relationship between system costs and systems engineering costs • Important to calibrate the COSYSMO Calibration Function

24. Our Current Approach for Validation • Use a Few Long-Running Programs • They tend to have good data collection and good process discipline – so the data is reliable • Treat Each Major Release as a Separate Entity • That really allows us to dig into reuse between releases • It is necessary to collect data on reuse – we found that to be a little challenging • Use some releases for validation and others for calibration

25. Revisiting the Overall Approach Construct the COSYSMO Scenarios 1 Define Parameters for Remaining Factors in Bias Function 2 Run Monte Carlo Simulations and Generate Cumulative Distribution of Costs 3

26. Revisiting the Overall Approach 1 1. Construct the COSYSMO Scenarios Pessimistic Optimistic Expected Requirements Requirements Baseline Interfaces Algorithms Architecture Baseline Scenarios 2. Define Parameters for Remaining Factors in Bias Function - Effort Conversion Factor - Stochastic Adder Factors - Deterministic Adder Factors 2 3. Run Monte Carlo Simulations and Generate Cumulative Distribution of Costs 3 Bias Function Target Cost Risky Range Target Reserve

27. Part 2 Wrap-Up • The Basic Bias Function • Concerns with the basic bias function • Suggested improvements on the basic bias function • The Approach for Validation and Calibration • Concerns with the basic approach • Suggested improvements for validation and calibration

28. The Key Use Cases

29. The Key Use Cases • Should-Cost Analysis • Establishing a should-cost for which cost estimates from bidders can be evaluated • Establishing a DTC target for performing DTC analysis • Design-to-Cost Analysis • Performing cost vs. capability trades as way to provide more affordable solutions • Analysis of Alternatives • Evaluating alternative solution strategies • It’s not just about the problem space – the solution space can be evaluated too

30. Should-Cost Analysis • Goal is to Establish a Target Cost • Can also be a cost range • Usually performed very early in the lifecycle • Approach • Given a requirements baseline, use the extended COSYSMO approach to estimate the cost • Use “plug” numbers for adders • e.g., labor, material, fee, etc.

31. Part 3 – Wrap-Up • There Are Three Very Important Use Cases • Should-Cost Analysis • DTC Analysis • Analysis of Alternatives • There Are At Least a Couple More • Change Evaluation for Operational Systems • Scope Creep Monitoring • Probably Others We Haven’t Thought Of • Discussion on Use Cases

32. Workshop Results and Recommendations

33. Conclusions and Recommendations (1 of 3) • Validity of overall approach • Unanimous support – approach is valid and should continue to be developed and refined for wider application • Feedback was all focused on ensuring all factors had been considered and areas for refinement – no discussion resulted in conclusion that the approach had major issues • Appropriately uses the concepts of COSYSMO and tailors the perspective for systems

34. Conclusions and Recommendations (2 of 3) • Improvement of Approach • Best distribution to use? Cumulative, frequency, or other? • Preference is to not change the Scale Factor • Want to retain as close to COSYSMO as possible – ready to leverage COSYSMO 3 • Review and refine the bias function and calibration of the bias function • May consider adding in other COCOMO factors, as applicable • May need to establish rules for when to leverage a HW model using the same approach to use as input for HW, when highly HW intensive • May need to consider calibration for different types/classes of systems • Bottom line – User needs to consider what tailoring/adaptation of the bias function is needed for the system application • Determine under what conditions it OK to eliminate a cost factor • Additional use cases? • Should include Impact Analysis / Change Evaluation

35. Conclusions and Recommendations (3 of 3) • Thoughts on moving forward • Form small, diverse working group • Periodic meetings (USC ARR, COCOMO Forum, INCOSE IW, …) • Validation through pilots • Use on completed programs • Access the more robust data points from COSYSMO data • Comparison of estimates • Use on non-LM programs • Incorporate feedback from validation and refine • Can this support the SERC project for COSATMO?

36. Back-up CHarts

37. COSYSMO Setup Optimistic Assume a mature supplier, experienced in the domain, minimal scope creep, and cooperative stakeholders Expected Assume an average supplier, with some experience in the domain, average scope creep, and generally cooperative stakeholders Pessimistic Assume a new supplier who will have some challenges, a fair amount of baseline volatility, and stakeholders who need to be corralled

38. Cost Analysis Approach & Results Pessimistic Optimistic Expected Requirements Requirements Baseline Interfaces Algorithms Architecture Baseline Scenarios Use a “Plug” Number for Adders Bias Function • Anticipated Distribution of Labor Rates • Anticipated Distribution of Material Costs • Anticipated Travel Costs • Anticipated Supplier Fees Target Cost Risky Range Target Reserve

39. Design-to-Cost Analysis • Goal • The goal is, given a target cost, design a solution to meet the target cost • Approach • Given a requirements baseline, identify and prioritize capabilities • Use the extended COSYSMO approach to estimate the cost for each capability • Evaluate the “cost for capability” against the capability priority

40. Capability Decomposition Major Capabilities 1 – Service Provisioning Problem: TELECOM Operations Support System Major Upgrade, Budget Limited to $40M 2 – Service Order Orchestration 3 – Pre-Provisioning Support 4 – Service Order Validation Requirements Baseline 5 – Service Activation 6 – Network Management 7 – Dashboards for Awareness & Reporting Architecture Baseline 8 – Service Catalog Management 9 – Service-to-Subscriber Mapping 10 – Auto-Discovery 11 – Service Reconciliation 12 – Billing - Service Order Creation Evaluate Each Capability with Respect to Cost and Enterprise Utility to Determine Best-Value Baseline 13 – Billing - Trouble Ticketing 14– Service Coverage Mapping 15– Resource Management

41. Mission Utility Analysis of Capabilities 1 – Service Provisioning 8 1 2 – Service Order Orchestration 3 – Pre-Provisioning Support 7 6 4 – Service Order Validation 5 – Service Activation 2 11 6 – Network Management 3 9 13 7 – Dashboards for Awareness & Reporting 5 4 10 14 12 8 – Service Catalog Management 9 – Service-to-Subscriber Mapping 10 – Auto-Discovery 15 11 – Service Reconciliation 12 – Billing - Service Order Creation 13 – Billing - Trouble Ticketing 14– Service Coverage Mapping Operational Tempo is a combination of the frequency with which the capability is used operationally and its criticality to the overall mission. Operational Burden is a combination of the skill level required for the capability and the time it takes to perform. 15– Resource Management

42. Cost Analysis Approach Optimistic Pessimistic Expected Requirements Requirements Baseline Interfaces Three COSYSMO Scenarios for Each Capability Algorithms Architecture Baseline Scenarios A Cost Curve is Produced for Each Capability Use a Common Number for Adders Bias Function • Anticipated Distribution of Labor Rates • Anticipated Distribution of Material Costs • Anticipated Travel Costs • Anticipated Supplier Fees Take the 80% Confidence Cost as the Capability Cost

43. Simplified Cost Analysis Approach A Simpler Approach is Slightly Less Robust…Bust Still Just as Valid Expected Requirements Requirements Baseline Interfaces A Single COSYSMO Scenario for Each Capability Algorithms Architecture Baseline Scenarios A Cost Curve is Produced for Each Capability Use a Common Number for Adders Bias Function • Anticipated Distribution of Labor Rates • Anticipated Distribution of Material Costs • Anticipated Travel Costs • Anticipated Supplier Fees Take the 80% Confidence Cost as the Capability Cost

44. Analysis Results Capabilities Sorted By Cost Total Cost = $81.3M • Colors Map Mission Utility Priorities • RED = High • YELLOW = Medium • Green = Low Capabilities Sorted By Mission Utility $34.4M (Cost for Priority 1 Capabilities Only) (Capabilities at DTC Target) $39.9M $64.1M (Cost for Priority 1&2 Capabilities) $81.3M (Cost for All Capabilities)

45. Analysis of Alternatives • Goal • The goal is, given a requirements baseline, determine the most affordable solution that meets requirements • Approach • Given a requirements baseline, identify possible solution alternatives • Use the extended COSYSMO approach to estimate the cost for each capability • Evaluate the cost to find the most affordable alternative that meets all requirements

46. Case Study Overview • 20-Year Old C2 System • The system was unprecedented at the time • Twenty years later, a replacement system is needed due to obsolescence and needed changes • Alternative Solutions • Full Replacement System • Develop and deploy a new replacement system • COTS/GOTS/NDI-Based Replacement System • Use a combination existing non-obsolete components and COTS components • Modify components as necessary to meet requirements

47. The Two Key Alternatives Newly Developed System • Ground-Up Development of System – Requirements Refinement, Architecture, Detailed Design – Soup-to-Nuts • But…It is No Longer an Unprecedented System – So Requirements/Architecture Understanding, etc. Are High Refurbished System Using Modified NDI • Use of NDI is Maximized – Additional Components Developed as Necessary to Meet Requirements • Reuse Considerations Drive This Alternative

48. COSYSMO Size and Cost Drivers • Size Drivers for the Two Alternatives Are Largely the Same • Cost Drivers for the Two Alternatives Are Very Similar– With a Few Notable Exceptions Cost Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity System modification and reuse have an effect on some cost drivers Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support

49. COSYSMO Reuse Factors New Elements – These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient. Most Elements Are New for the Development Alternative Modified Elements – These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category. Most Elements Are Modified for the NDI Alternative Reuse Factors New Elements Deleted Elements – For the NDI Alternative, Some Elements May Need to Be Deleted/Retired Elements That Need to Be Retired Should Be Treated as Deleted Elements Modified Elements Deleted Elements Adopted Elements Managed Elements Adopted Elements – These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category. Elements That Are “Wrapped” Can Be Treated as Adopted for the NDI Alternative Managed Elements – These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category. Elements That Stay in Place Without Modification (or Wrapping) Can Be Treated as Managed

50. Cost Analysis Approach Optimistic Pessimistic Expected Requirements Requirements Baseline Interfaces Three COSYSMO Scenarios for Each Alternative Algorithms Architecture Baseline Scenarios A Cost Curve is Produced for Each Capability Use a Common Number for Adders Bias Function • Anticipated Distribution of Labor Rates • Anticipated Distribution of Material Costs • Anticipated Travel Costs • Anticipated Supplier Fees Take the 80% Confidence Cost as the Capability Cost