1 / 34

USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items

USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items. Barry Boehm, USC-CSSE February 14, 2007 http://csse.usc.edu/BoehmsTop10/ boehm@usc.edu. Outline: Top-3 SIS Risks Workshop. Working group guidelines Risk survey results and survey update(?) The top three risks

fineen
Download Presentation

USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. USC CSSE Workshop Overview:Top 3 Software-Intensive Systems Risk Items Barry Boehm, USC-CSSE February 14, 2007 http://csse.usc.edu/BoehmsTop10/boehm@usc.edu

  2. Outline: Top-3 SIS Risks Workshop • Working group guidelines • Risk survey results and survey update(?) • The top three risks • Architecture complexity; system quality tradeoffs • Requirements volatility; rapid change • Acquisition and contracting process mismatches • Architecture complexity and system quality tradeoffs • Architecture complexity phenomenology • Nature of system quality • Quality tradeoff perspectives

  3. Working Group Guidelines • Product: briefing, preferably with notes • Topics should include: • Most critical success factors in each area • Current best practices for addressing them • Areas for further research • Rated 0-10 on value and difficulty of research

  4. Research Topics: Agile Methods • Relationship between plan driven and agility • For individuals • For organizations • Differences between agile and plan driven outcomes • Effect of Gurus • Mismatches between development approach and acquisition practices • How do you measure quality in an agile environment? • Data collection; agile experience base • Team of teams • Agile Development and Evolutionary Prototyping • Shared Code and/or module ownership • Architecture: when, how much, how to express • Lack of user consensus • Dynamic Homegrounds

  5. SIS Risk Survey 2006: Statistics • Number of Surveys: 25 • Average Experience: ~28 years (6 years – 51 years) • Area Distribution: • Software: 20 • Systems: 17 • Hardware: 0 • Business Domain Distribution: • Aerospace: 18 • Software Infrastructure: 5 • Business: 4 • Telecom: 3 • Others: Secure Apps (1); Safety Critical Apps (1); C4ISR (1)

  6. Risk Survey 2006: Nominees • Acquisition and contracting process mismatches • Architecture complexity; quality tradeoffs • Budget and schedule constraints • COTS and other independently evolving systems • Customer-developer-user team cohesion • Migration complexity • Personnel shortfalls • Process maturity • Requirements mismatch • Requirements volatility; rapid change • Technology maturity • User interface mismatch

  7. Risk Survey 2006 Results

  8. SIS Risk Grouping

  9. Survey 2007: Early Statistics • Number or Surveys: 41 • Average Experience: ~27 years (6 years – 51 years) • Area Distribution: • Software: 33 • Systems: 34 • Hardware: 0 • Business Domain Distribution: • Aerospace: 32 • Software Infrastructure: 7 • Business: 6 • Telecom: 5 • Others: Secure Apps (1); Safety Critical Apps (1); C4ISR (1); Network and Protocols (1); Defense (1); Program and Risk Management (1)

  10. Survey Results: 2006-2007

  11. SIS Risk Grouping 2006-2007

  12. Outline: Top-3 SIS Risks Workshop • Working group guidelines • Risk survey results and survey update(?) • The top three risks • Architecture complexity; system quality tradeoffs • Requirements volatility; rapid change • Acquisition and contracting process mismatches • Architecture complexity and system quality tradeoffs • Architecture complexity phenomenology • Nature of system quality • Quality tradeoff perspectives

  13. SIS Architecture Complexity: Future Combat Systems

  14. Platform N • • • Platform 1 Infra C4ISR Breadth DOTMLPF 1.0 2.0 3.0 4.0 5.0 … Length 2008 2010 2012 2014 2016 Command and Control Situation Assessment Info Fusion Sensor Data Management Sensor Data Integration Sensors Sensor Components : Depth Requirements Volatility: Ripple Effects of Changes- Breadth, Depth, and Length Legend: DOTMLPF Doctrine, Organization, Training, Materiel, Leadership, Personnel, Facilities C4ISR Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance

  15. Average Change Processing Time: 2 Systems of Systems • Average workdays to process changes

  16. Acquisition/Contracting Mismatches: Fitness Landscapes • Role of Fitness Landscapes in Complex Adaptive Systems (CAS) • S. Kauffman, At Home in the Universe, Oxford University Press, 1995 • CSoS Acquisition Challenges • B. Boehm, “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), 2006, pp. 1-19. • A Candidate Three-Agent Acquisition Fitness Landscape • D. Reifer and B. Boehm, “Providing Incentives for Spiral Development: An Award Fee Plan”, Defense Acquisition Review 13(1), 2006, pp. 63-79.

  17. Role of Fitness Landscapes in CAS • Incentive structures for local behavior • Induce global behavior via adaptation to change

  18. Complex Systems Acquisition Challenges

  19. Candidate Approach: 3-Agent Model

  20. Risk-Driven Scalable Spiral Model:Increment View

  21. Outline: Top-3 SIS Risks Workshop • Working group guidelines • Risk survey results and survey update(?) • The top three risks • Architecture complexity; system quality tradeoffs • Requirements volatility; rapid change • Acquisition and contracting process mismatches • Architecture complexity and system quality tradeoffs • Architecture complexity phenomenology • Nature of system quality • Quality tradeoff perspectives

  22. Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC Sweet Spot 100 KSLOC Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward 10 KSLOC Larger Systems Need More Architecting: COCOMO II Analysis

  23. TRW Project B 1005 SPR’s 100 90 80 TRW Project A 373 SPR’s 70 % of Cost to Fix SPR’s 60 50 Major Rework Sources: Off-Nominal Architecture-Breakers A - Network Failover B - Extra-Long Messages 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 % of Software Problem Reports (SPR’s) Architecture-Breakers are the Biggest Source of Rework

  24. $100M Arch. A: Custom many cache processors $50M Arch. B: Modified Client-Server Original Spec After Prototyping 5 3 1 2 4 Response Time (sec) Best Architecture is a Discontinuous Function of Quality Level

  25. The Nature of Quality: Participant Survey Which figure best symbolizes quality improvement?

  26. Holistic Approach

  27. Lean Approach

  28. Analytic Approach

  29. Preoccupation with Booze and Sex

  30. There is No Universal Quality-Value Metric • Different stakeholders rely on different value attributes • Protection: safety, security, privacy • Robustness: reliability, availability, survivability • Quality of Service: performance, accuracy, ease of use • Adaptability: evolvability, interoperability • Affordability: cost, schedule, reusability • Value attributes continue to tier down • Performance: response time, resource consumption (CPU, memory, comm.) • Value attributes are scenario-dependent • 5 seconds normal response time; 2 seconds in crisis • Value attributes often conflict • Most often with performance and affordability

  31. Quality of Service Adaptability Robustness Affordability Protection Attributes Stakeholders Info. Suppliers, Dependents ** * * Info. Brokers ** ** ** ** * Info. Consumers * ** * * Mission Controllers, Administrators ** * ** ** Developers, Acquirers * * ** ** Overview of Stakeholder/Value Dependencies • Strength of direct dependency on value attribute **- Critical ; *-Significant; blank-insignificant or indirect

  32. Implications for Quality Engineering • There is no universal quality metric to optimize • Need to identify system’s success-critical stakeholders • And their quality priorities • Need to balance satisfaction of stakeholder dependencies • Stakeholder win-win negotiation • Quality attribute tradeoff analysis • Need value-of-quality models, methods, and tools

  33. (RELY, MTBF (hours)) • For 100-KSLOC set of features • Can “pick all three” with 77-KSLOC set of features -- Cost/Schedule/RELY: “pick any two” points Tradeoffs Among Cost, Schedule, and Reliability: COCOMO IIWant 10K hour MTBF within $5.5M, 20 months

  34. Agenda : Wednesday, Feb 14 • 8:15 – 10:00 am: Architecture Complexity and Quality Tradeoffs; Elliot Axelband (RAND), Chair • Overview, Issues and Approaches; Barry Boehm (USC) • From Dependable Architectures To Dependable Systems; Nenad Medvidovic (USC) • Architecture Tradeoff Analysis: Towards a Disciplined Approach to Balancing Quality Requirements; Azad Madni (Intelligent Systems Technology) • 10:00 – 10:30 am: Break • 10:30 am – 12:30 pm: Requirements Volatility; George Friedman (USC), Chair • Process Synchronization and Stabilization; Rick Selby, Northrop Grumman • Disciplined Agility; Rich Turner (SSCI) • Using Anchor Point Milestones; Tom Schroeder, BAE Systems • 12:30 – 1:30 pm: Lunch • 1:30 – 3:30 pm Acquisition and Contracting Mismatches; Rick Selby (NGC), Chair • Acquisition Assessment Analyses; Kristen Baldwin (OSD/AT&T/S&SE) • Commercial Acquisition Practices; Stan Rifkin (Master Systems Inc.) • Space Program Acquisition: Systems Engineering & Programmatic Improvements; Marilee Wheaton (Aerospace Corporation) • 3:30 – 4:00 pm: Break • 4:00 – 5:00 pm: General Discussion: Working Group Formation; Barry Boehm, Chair

More Related