Process Principles for COTS-Based Systems

1. Process Principles for COTS-Based Systems • 5 September 2003 • Software Engineering Institute • Carnegie Mellon University • Pittsburgh PA 15213 • Sponsored by the U.S. Department of Defense Tricia Oberndorf

2. Agenda • COTS Background • A/APCS Framework • CMMI Background • Implications of Using CMMI for COTS-Based Systems

3. COTS Background • Building systems today: • Few entirely custom-built to order • Commercial products expected to play major role • Diverse things influence the shape and function of a COTS-based system (CBS): • Stakeholder needs • Product characteristics and marketplace dynamics • Degree of interaction with legacy systems • Together, these compel many changes in system development and maintenance methods for CBSs.

4. What is a COTS-based system? • A COTS product is one that is … • Sold, leased, or licensed to the general public • Offered by a vendor trying to profit from it • Is supported and evolved by the vendor, who retains intellectual property rights • Available in multiple, identical copies • Used without modification of its internals • A COTS-based system is one that … • Uses one or more off-the-shelf components from one or more suppliers plus any needed custom components • Is integrated to achieve new or expanded system functionality COTS products Legacy Custom Reuse assets

5. System that is created must accommodate competing sources of gravity Constraints come from stakeholder needs AND marketplace imperatives Conceptual Bases: Requirements1 • “Nail down the requirements first” will not work – the marketplace will not cooperate • Think of it as a new sphere of influence:

6. Conceptual Bases: Requirements2 • Requirements (cont.) • Must distinguish between negotiable and non-negotiable • Keep the non-negotiable set as small as possible • Understand how to prioritize and trade-off the negotiables • A risk-driven spiral approach is key: • Frequent use of prototypes • Considerable interaction with system’s end users • Gradual refinement of understanding of system

7. BoK Conceptual Bases: Knowledge • CBS development and maintenance is dependent on an evolving Body of Knowledge. Non-negotiables Prioritized negotiables Marketplace offerings Legacy system context Evolving System View Architecture and design constraints End-user business processes Acquisition constraints Risks

8. Traditional Engineering Approach Required CBS Approach Stakeholder Needs/ Business Processes Requirements Simultaneous Definition and Tradeoffs Architecture Design Marketplace Architecture & Design Programmatics/ Risk Implementation Marketplace Affects CBS Approach Requirements-driven Negotiation-driven

9. The Process Framework • Based on these realizations and concepts, we have defined a framework that sets out the basics of a process that would be appropriate for the creation of a COTS-based system: • A/APCS: The Assembly/Acquisition Process for COTS-based Systems

10. Process Overview • Three classes of activities: • Iterative: short-term, engineering-oriented • Discovery: gather and refine system knowledge • Assembly: construct a prototype • Assessment: determine success of iteration & plan • Pervasive: long-term, organizational scope • e.g., CM, license management, vendor relationship management, contract tracking and oversight • Executive: event-driven, deal with decision making • e.g., cost estimating, contract negotiation, project oversight • Today we focus on the Iterative.

11. Discovery • Gather and Refine activities occur simultaneously. • Gather: • Sources take many forms. • Stakeholders, business processes, legacy systems, COTS marketplace • Each is independent of the others. • There is no optimal order. • Refine: • Analysis and harmonization of the gathered knowledge • Yields the technical definition of the emerging system • Finds gaps, negotiates conflicts

12. Go on? Negotiate conflicts Analyze Gather data z Gather data y Gather data x Discovery Activities Knowledge is sufficient for constructing executable Continue Discovery Harmonized data Data from: stakeholders products … Mismatch Agreement Refine Gather Gap - need more data

13. Assembly • Produces a prototype that reflects what’s been discovered • Reflects a traditional sequential process • Guided by local “ candidate requirements” • Iteration Objectives, Detailed Iteration Plan plus hypotheses and desired behaviors expect to derive from prototype • Vets “candidate requirements” • Produces an executable version of full system • Likely to have less than complete functionality • Executable, not paper or specification or small piece • Does not preclude prototyping “in the small” throughout

14. Evaluation and Assessment • Occur at two levels: • Evaluation of the prototype in its own context • Measures actual outcome against expected outcome • Considers degree to which prototype satisfies its local “requirements” • Assessment of the iteration itself • Addresses issues at project (not iteration) level • Considers whether objectives were good ones • Determines whether iteration results indicate changes in project schedule, budget, etc.

15. Discovery Deployed System Discovery Assembly Assembly Body of Knowledge BoK Body of Knowledge On-going Iterations of A/APCS Discovery Assembly Prototype

16. Key Aspects of COTS Approach • Simultaneous Definition and Trades • Parallel Business Process Engineering • Negotiable Requirements • Flexible Architecture • Current Market Knowledge • Integral Programmatics and Risks • Continuous Stakeholder Involvement • Disciplined Spiral or Iterative Practices • Frequent Executables

17. CMMI • Guidance for improving an organization’s processes and ability to develop, acquire, and maintain products or services • Provides guidance for developing processes • is not processes or process descriptions • will not map one to one with the processes in your organization • Integrates four disciplines (bodies of knowledge) • System Engineering • Software Engineering • Integrated Product and Process Development • Supplier Sourcing • Expects all practices to be interpreted using knowledge of • CMMI • the organization • the business environment CMMI process areas must be interpreted for each targeted domain

18. Work Process Implications High-level guidance for selectedCMMI Process Areas or disciplines Highlight particular relevance Provide unique interpretation Suggest new process areas Correct misconceptions Guidance for additional CMMI Process Areas for later reference Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Presentation Format • Important process considerations for a COTS-based system approach Take-away message for each aspect

19. Selected Work Process Implications Decision Analysis and Resolution–continuous stakeholder negotiations require robust, agreed-upon decision processes Technical Solution–alternative solutions (including COTS product selection) are analyzed continuously to accommodate new information Risk Management – a key project risk is over-defining one sphere without adequate understanding of implications on other spheres Project Planning – project activities start concurrently with extensive interaction among them from project start until the system is retired Verification and Validation – continuous determination that information in each sphere is sufficient, complete, and meets operational needs Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Simultaneous Definition and Trades • Balanced consideration of four diverse spheres of influence • - Simultaneous discovery of information in each • - Decisions in one inform and constrain decisions in others • - Continuous identification and analysis of trades • - Trades negotiated as definition of the solution evolves Continuously reconcile what stakeholders want with what is available

20. Selected Work Process Implications CMMI does not address changes to end-user business processes; expand concepts in Organizational Process Focus to plan and implement end-user business process improvement Organizational Environment for Integration – a shared vision of success among stakeholders, with suitable incentives and leadership, is critical to aligning business processes with alternative solutions Project Planning/Integrated Project Management for IPPD – implementing agreed upon business processes must be integrated in system planning Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Parallel Business Process Engineering • End-users must be willing/able to change business processes to match those in COTS products • Business process changes must be explicitly managed and coordinated as part of the project • Business processes may continue to change with new COTS releases or new COTS products Align business process engineering with system engineering

21. Selected Work Process Implications Requirements Development– must prioritize requirements to aid tradeoffs stakeholders agree on minimum set of critical “must-have” requirements evolvability is a high-priority, required quality attribute Requirements Management–disciplined and controlled management of requirements begins at project start to track identified and negotiated trades Project Planning/Integrated Project Management for IPPD – managing the project to encourage and reinforce the continual discovery of requirements while establishing sufficient stability to deliver a solution is challenging Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Negotiable Requirements • Requirements must be flexible enough to leverage available and projected COTS products • Committing to a requirement is premature until COTS product behavior is understood • As marketplace continues to change, requirements must be renegotiated Requirements formation is a journey of discovery

22. Selected Work Process Implications Technical Solution – each COTS product release requires evaluation and analysis of alternatives to support any incorporation decision Describe behavior and linkage among system componentsfor each Product Integration – composing and evaluating executable representations from project start is critical to verify and validate architecturesuitability and evolvablity Project Planning/Integrated Project Management for IPPD – appropriate skills and resources are necessary to form, evaluate, and maintain system flexibility include effort to create and maintain “wrappers” or “glue” and re-integrate solution as COTS products change Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Flexible Architecture • Architecture is considered early; evolves and is maintained until the system retired • Potential drivers of change accommodated in architecture Architecture is created and maintained as a corporate asset

23. Selected Work Process Implications Supplier Agreement Management – license agreements with COTS vendors must be negotiated to meet project needs Integrated Supplier Management – partnering with key vendors is critical (not just supplier management) vendors seldom allow process monitoring; use hands-on evaluation relationships with vendors’ other customers helps to amplify leverage Technical Solution – modification of COTS products introduces long term maintenance considerations and sizeable risk to project; avoid if possible Project Planning– significant resources may be required to monitor the marketplace and conduct COTS product evaluation; including experimentation facilities Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Current Market Knowledge • Anticipate and track changes to relevant market segments until system retired • Anticipate and prototype system changes from updates to COTS products critical to the system • Influence (not direct) COTS product changes, technology investments, and standards development Marketplace is proactively monitored

24. Selected Work Process Implications Technical Solution – engineering trades should include risk, cost, schedule and other programmatic factors associated with each alternative solution Project Planning – estimates of work product and task attributes should be generated for each alternative Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Integral Programmatics and Risks • Analysis of alternative solutions includes team skills and expertise, cost, schedule, and associated risks for • building, fielding, and supporting the system • implementing any needed changes to business processes (functional, operational, support) Programmatic factors shape technical alternatives

25. Selected Work Process Implications IPPD discipline – integrated teaming among disparate stakeholders throughout the development and maintenance is essential Validation – end users must always be involved in validating solution suitability Integrated Project Management for IPPD – accommodates resources for stakeholder involvement necessary changes to end-user operational processes must be explicitly and continuously managed and coordinated with solution development Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Continuous Stakeholder Involvement • Significant commitment from all stakeholders required • identify, evaluate, and select alternative solutions • confirm results of any and all negotiations • agree evolving system meets their needs • Stakeholders must reflect full diversity of interests Required stakeholder commitment may be unprecedented

26. Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Disciplined Spiral or Iterative Practices • Continuously determine a compatible and feasible set of: business processes, requirements, plans, design, COTS products, and other components • Enterprise business objectives drive solution definition • Risk considerations drive degree of detail • Marketplace dynamics drive development and maintenance processes • Selected Work Process Implications • Project Planning – if not already implemented, extensive effort may be needed to revamp planning and engineering processes for a spiral development approach • Risk Management – tracking effectiveness of risk mitigation is key • highest priority remaining risks should be used to (re) direct and manage the project Spiral development facilitates developing a viable solution

27. Stakeholder Needs/ Stakeholder Needs/ Business Processes Business Processes Simultaneous Simultaneous Definition Definition Marketplace Marketplace Architecture Architecture and Tradeoffs and Tradeoffs Design Design Programmatics/ Programmatics/ Risk Risk Frequent Executables • Frequent executable representations reduce risk and reduce misunderstandings • Provide critical insight into the solution’s behavior • Explore critical system attributes • Validate end user business process • Verify technical viability • Selected Work Process Implications • Requirements Development, Technical Solution, and Product Integration – each iteration should produce an executable representation reflecting current understanding of requirements, COTS products, business processes, and alternative designs explored and negotiated Executable representations demonstrate stakeholder buy-in

28. Conclusion • CBS: • changes many things about your approach • requires even more discipline than custom development • Although it needs interpretation, CMMI is a sound basis for CBS process improvement. • Applying CMMI to CBSs is more than just Supplier Management: • All four disciplines • All process area categories • Process areas will need to be integrated and applied differently • Must add in your own activities to plan, design, and deploy changes to enterprise processes

29. For more information: • TR coming: CMU/SEI-2003-TR-022 • www.sei.cmu.edu/cbs • Tricia Oberndorf Lisa Brownsword • Director, Dynamic Systems Sr. MTS, CBS (ISIS) • 412-268-6138 703-908-8203 • po@sei.cmu.edullb@sei.cmu.edu