1 / 26

USC CSSE Distinguished Lecture My Cost Estimation Journey Gary Thomas Raytheon Intelligence and Information Systems (

USC CSSE Distinguished Lecture My Cost Estimation Journey Gary Thomas Raytheon Intelligence and Information Systems (IIS) November 8, 2006 . In Memoriam . Stutzke, Richard David (Dick) Feb. 14, 1943 - Sept. 23, 2006

tory
Download Presentation

USC CSSE Distinguished Lecture My Cost Estimation Journey Gary Thomas Raytheon Intelligence and Information Systems (

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. USC CSSE Distinguished LectureMy Cost Estimation Journey Gary ThomasRaytheon Intelligence and Information Systems(IIS)November 8, 2006

  2. In Memoriam Stutzke, Richard David(Dick) Feb. 14, 1943 - Sept. 23, 2006 He attended Purdue University and the University of Illinois, majoring in physics…after graduating with a Ph.D. in nuclear physics from the University of Illinois, he served in the U.S. Air Force for four years… The remainder of his career was spent in the defense industry, working at Science Applications International Corp. in Colorado Springs in 1972 and in Huntsville, Alabama since 1992 (held title of Vice President and Senior Scientist)… For the last two decades of his life, he maintained an active interest in estimation and measurement for software-intensive systems. Dr. Stutzke authored a book on the subject in 2005, as well as dozens of papers on estimation and related topics… He was selected as the first recipient of the USC Lifetime Achievement award in 2005… He was selected as the 2006 Parametrician of the Year by the International Society of Parametric Analysts… He was an avid recreational ice skater… he competed in pairs ice dance events and skated in many ice shows (see the “14 step” at right)… Memorials may be made to the Richard David Stutzke Foundation, 138 Michli Road, Madison, AL 35758, which is an education fund for children in need or to Hospice Family Care, 3304 Westmill Drive, Huntsville, AL 35805

  3. Some Reactions to News of This Award • “Why would they do a thing like that?” • Source of quote: my current (and first) wife of 38 years  • “Congratulations, that's very cool. you took a dry subject and …. made a career out of it!” • “Congrats… but, let me get this straight… they gave you this award for GUESSING?” • Source of quote:aRaytheon colleague • However, as Mark Twain said… “Predictions are hard to make, especially when they are about the future”

  4. My Cost Estimation Journey • Yesterday and Today • 1985-1990 – Not Sure Where I Am Going, But Making Good Time! • 1989-1997 – A Roadmap! • 1995-2006 – USC Software Engineering Cost Estimation • 2000-2006 – USC Systems Engineering Cost Estimation • Tomorrow • Heard Along the Way • Rules of the Road • Acknowledgements and Closing

  5. 1985:1990 – Not Sure Where I Am Going, But Making Good Time! • After 20 years as a software developer / project leader… • Transitioned to a product line software metrics person / software cost estimation “tool guy” • SW Cost Estimation Approach • Developed wrapper around an IBM Mainframe tool that provided bottom up labor estimates • Embedded business rules for direct pricing system input • Effort estimates based on simple productivity factors • Generated a software estimate summary BOE document called a “software cookbook” • Noticed Significant Variation in Estimates v. Actuals • Inconsistent, ad hoc processes in program execution • Inconsistent data collection/definitions – sizing, reuse, labor scope, etc. • Using simple productivity factors, a “one size fits all” concept didn’t work so well snapshot was taken on one of my better days back then…

  6. 1989:1997 – A Roadmap! One size doesn’t fit all! • In early 90’s, developed a COCOMO 81 implementation on the IBM mainframe • Improved code counting tools, crisper size and scope definitions helped • Began to get better results • Transitioned this tool to be MS Excel™ based in 1997, improving capability and enabling portability across Raytheon Garland site • MS Office 97 version of Excel™ provided a huge potential for new capabilities • Eventually renamed tool SWCOST (and upgraded to COCOMO II) • Don’s Rosetta Stone was very useful to upgrade database to COCOMO II • Became the Raytheon Garland site standard • Began attending USC/CSE events in 1995…

  7. 1995-2006: USC Software Engineering Cost Estimation 2006: The Evolution Path Continues 2001: Chris Abts Dissertation 2000: The COCOMO II Book Published 1999: Sunita Chulani Dissertation 1997: Brad Clark Dissertation 1997-2001: COCOTS Formulation 1995-1999: COCOMO II Formulation 1995: Became USC Affiliate POC for Raytheon

  8. 2000-2006: USC Systems Engineering Cost Estimation 2006: The Evolution Path Continues 2005: Ricardo Valerdi Dissertation 2002-4: COSYSMO Formulation 2001: COSYSMO Project Initiated 2000: Taking a Test Drive…

  9. 2000: Taking a Test Drive… • May - “Can you build a Systems Engineering Cost Model?” • My answer was “No, but I know who can, and I will help…” • September - pulsed the USC staff on this topic at the SEI-CSE Spiral Workshop in D.C. • Positive feedback from Barry and staff • November - suggested topic as a breakout candidate at COCOMO II Forum • Sketched out some ideas and made copies available at registration desk • USC/CSE changed breakout agenda based upon affiliate interest

  10. 2001: COSYSMO Project Initiated • Barry Announced COSYSMO at the USC ARR • USC COSYSMO Project Development Team’s “Forming” • Ricardo Valerdi introduced as Ph.D. student to “kick start” it • Early on, considerable contractor community interest, but… • Lotsof customer community interest (government, FFRDC, consortia) • Continued customer interest / support has been critical to the project’s success! • MyCOSYSMO Developed • Allowed me to organize my thoughts on the model as it evolved • Shared with USC affiliates to allow their organizations to visualize what “could be” and get buy-in, so we could move the COSYSMO project forward • MyCOSYSMO is COCOMO II “under the hood”, a migration path follows

  11. Raytheon SWCOST MyCOSYSMO Raytheon SECOST Early 2001: Built experimental version of USC COSYSMO • MyCOSYSMO leveraged off Raytheon Garland’s SWCOST • COCOMO II based • Removed Raytheon company proprietary functionality and business rules Early 2004: SECOST developed as Raytheon’s proprietary version of MyCOSYSMO • Added back company proprietary functionality and business rules (removed earlier) • Added many new capabilities as well

  12. 2002-2004: COSYSMO Formulation • COSYSMO Project Team’s “Storming, Norming, and Performing” • Iterations were required as momentum built and new players came onboard • Many “rich discussions” occurred, which provided needed diversity of thought • About 9-10 face-to-face meetings; on balance, we made progress each time • INCOSE involvement and endorsement was critical! • Some Representative Issues We Dealt With… • Size factor or cost driver? • Requirement level to use? • How to reduce number of cost drivers? • Scope of model: development only or full lifecycle? • System of Interest? • USC Data Collection Instrument Developed • Affiliates began providing their historical data points for USC industrial database • “Meanwhile, Back at the Ranch in Texas …” • John Rieff was chairing a Raytheon Garland Six Sigma team to independently develop an SE cost model and associated process • Eventually, this team provided first round of data points for Industrial calibration

  13. 2005: Ricardo Valerdi Dissertation • Spring • First industrial data calibration complete with 42 data points, 6 companies • Ricardo feverishly working on dissertation • Summer • Dr. Ricardo Valerdi, a.k.a. “Dr. COSYSMO”… • married a lovely lady and… • moved from LA to Boston to… • join the Lean Aerospace Initiative (LAI) team at MIT • Fall • Academic COSYSMO version 1.0 made available • Highlights from 20th USC CSE Cost Modeling Forum • Excellent COSYSMO team meeting • COSOSIMO began to take shape • Received an HCP (Honorary COSYSMO Philosopher) degree  • Implicated in “theft” of Brad and Betsy Clark’s rental car (Gary Constantine drove) 

  14. 2006: The Evolution Path Continues • Ricardo’s publications • Academic COSYSMO User Guide (version 1.1 available) • COSYSMO book in work, for early 2007 review • Affiliate local data collection and calibration is gaining momentum to feed the next industry-level calibration baseline • Industrial calibration now has 58 data points (7 companies) and counting… • Three commercial COSYSMO implementations are emerging, either as separate tools or integrated into current suites • SoftStarSystems : SystemStar™ and COSYSMO Calico™ • Galorath, Inc. : SEER-SEM™ • Price Systems L.L.C. : True COSYSMO™ • A roadmap for future COSYSMO evolution is emerging • We don’t see “Adjourning” happening anytime soon… 

  15. Tomorrow • Unified Program Level Costing • Integrate discipline specific models into a program level framework • Use during early business planning for a “Should Cost” v. “Price to Win” comparison • Use during proposal for analysis prior to submission • Ensure no gaps and no overlaps in coverage • Focus on Component development • Across the enterprise (multi-site), • Between enterprises (multi-business teammates/subs) • Use during program execution to support Estimates At/To Completion, CAIV, etc. • Life Cycle Cost Modeling • More and more recent proposals are requiring this • While some tools exist, still many home grown solutions • Component Based Integration • During Development (have USC COCOTS and some commercial model support), but more work needed • During Operations & Maintenance • Need to extend/quantify COTS-LIMO concept

  16. Tomorrow (Cont.) • Total Cost Estimate Risk - 1 • For a proposal, the total cost estimate risk is greater than sum of discipline-specific cost estimate risks, take for example -- COSYSMO • Developed SECOST Expert COSYSMO • Leveraged from Ray Madachy’s Expert COCOMO II in late 1990’s) • Developed a set of pairwise risk rules using the most sensitive cost drivers • As the user rates each cost driver, flags are raised when a product pair exceeds a predefined threshold, for example… • Selecting a Very High Rating for Level of Service Requirements (1.74)… • while at the same time selecting a Low Rating for Team Cohesion (1.22)… • will result in a product (2.12) --- that might be considered “risky” • Also, a composite SECOST driver selection risk score is computed and compared to the local calibrated norm, this information feeds the Total Cost Estimate Risk - 2 (next chart)

  17. Tomorrow (Cont.) • Total Cost Estimate Risk - 2 • During a proposal, there are additional business/resource risks outside of the discipline-specific cost model risks (previous chart), all need to be considered, such as • Type and complexity of the contract • Multi-site, multi-business, multi-national • Availability of key domain people to bid effort • Availability of estimation support team • Special customer requirements related to costing/estimation • Extremely short bidding durations (schedule compression) • Etc. • Total proposal cost estimate risk prototype developed within IIS business • Combining all risks (both qualitative and quantitative) using a scorecard • Assessing risk (with mitigation / avoidance) throughout proposal cycle • Piloting and incorporating lessons learned

  18. Tomorrow (Cont.) • Parametric Cost Engineering will become a more widely recognized career path within the contractor community • ISPA’s Parametric Practitioner Certification is a good start! • USC CSSE will continue to be an “incubator” of new ideas and research in this field • Recent USC CSSE formation will provide even greater “diversity of thought” with the formal melding of their Systems and Software Centers • The COxxxMOs will mature and flourish

  19. Heard Along the Way… • We have to design it first, then we can cost it…come back later. • I can get any number I want with your tool, in a lot less time! • Can’t you take a 20% challenge? We have a cost target to meet! • Why can’t you guarantee an estimate within 10% of the actual cost? • After all, don’t you have a draft Statement of Work? • We think this new project is going to be just like the project we just completed, let’s simply use its productivity, then we can forgo this time-consuming, modeling stuff. • We need to forecast lots of junior grade engineers, have to keep costs down! • I’m an engineer, not an economist, let’s leave the pricing stuff to the experts. • I know this probably works for most folks… but, we are different!

  20. Heard Along the Way… (Cont.) • Schedule compression? • We really need an estimate in the next 2 hours for the customer’s budgetary wedge; their emailed objectives should suffice… after all, it’s only a rough “guesstimate” for planning purposes, not to worry… • I heard that Joe’s program was very productive… I want your modeling to mirror what they accomplished… dang, we are as smart as they are, and better managed, to boot! • We won’t use cost models around here anymore, model x gave the wrong answer! • The model made me do it…

  21. Rules of the Road • Simplify, Simplify, Simplify… • ”Instrument” the estimation and pricing process • Focus on need for speed, without loss of accuracy or quality • Ensure “no surprises” and minimal churn when pricing model generated effort • Embed business rules within your framework for pricing / approximating how it would be priced • Perform simple user input data integrity checks • Ensure your cost model driver selections tell a consistent, believable story • Translate the most important factor selections into sentences comprising a simple narrative to share with stakeholders, listen for risky contradictions • Carefully document rationale for all inputs • Use various constructs and methods to support multiple views of risk • Qualitative: exportable cost estimate risk register • Quantitative: • Expert COSYSMO results (referenced earlier) • Monte Carlo results based on input uncertainties • Ensure that you train/mentor stakeholders at all levels of enterprise

  22. Rules of the Road (Cont.) • Manage Expectations of All Stakeholders, per George E. P. Box… • ”All models are wrong, some are useful…”, along with its corollary… • ”Some models are more useful than others” • Focus Continually on Historical Program Data Collection • It must be supported by management due to need for program expertise • It costs money to do it right, and at the right time • “Local” historical calibrations are key • Develop a “calibration window” criteria to retire oldest project data • Validate “goodness” of each calibration (PRED, MMRE, etc.) • Strongly type/characterize each archived project to allow comparison to one under bid • Data can be very useful for non-estimation related initiatives • Ensure You Leave a Legacy • “I’ll never forget old what’s his name” • Now mentoring (as well as learning from!) our IIS cost estimation support team • “Live For The Journey, Not The Destination”… (have fun along the way) 

  23. Acknowledgements • USC Staff, Researchers, and Affiliates • Simply too numerous to mention! • Every attempt to develop a comprehensive list proved futile (and dangerous) • E-Systems / Raytheon • Gary Constantine, John Rieff, the Raytheon Measurement and Estimation Team, and many others • My management over the past 20 years that have not only supported my cost modeling pursuits but provided a risk taking environment • Family • Judy, my loving wife …  • My 5 grandchildren… oh yes, my 2 daughters (Gwen and Gretchen) that produced them! • Introducing the USC freshman classes of… 

  24. … 2017, 2023, 2019, 2021, …

  25. … and 2024!

  26. In Closing, Paul McCartney sums it up… “Will you still need me, will you still feed me, when I’m 64…” “I just turned 64 (along with Sir Paul)… Thank you for feeding me… and we all know what they say about feeding stray dogs…they don’t go away!”

More Related