1 / 26

Engineering Quality Software

Engineering Quality Software . Amitabh Srivastava Corporate Vice President Microsoft Corporation. Talk Outline. Motivation Microsoft Experience Summary. Key Points. Automation drives efficiency Automate software development with smart tools

hume
Download Presentation

Engineering Quality Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Engineering Quality Software Amitabh Srivastava Corporate Vice President Microsoft Corporation

  2. Talk Outline • Motivation • Microsoft Experience • Summary

  3. Key Points • Automation drives efficiency • Automate software development with smart tools • Introduce new technologies into the development process • Seamless integration into development process is key • Collect information about the development process • Build process with technology and tools in mind

  4. Systems People Information Devices Software today Has increasing requirements for security, continuous operation, maintenance and change Is developed primarily with manual methods Evolves continually: Bugs, requirements, changes in environment, efficiency Lives in dynamic, heterogeneous and distributed environments

  5. Software Quality …. Bugs …. Software Bugs Cost U.S. Economy $59.6 Billion Annually, Commerce Department's National Institute of Standards and Technology (NIST) Numerous articles like: Why Software Is So Bad

  6. Why? Talk focus • Software development process lacks automation • Mostly an art: relies on experience • Lack of technology : primarily manual • Lack of generally accepted metrics and techniques • Formal Methods research should be more practically significant • Does it scale? • Are they usable by most software developers? • Are we attacking the right problems? • Software Engineering efforts are fragmented • Overloaded term

  7. Challenges • Software development primarily manual process • Complex development environment • Tens of millions lines of source code • Thousands of developers and testers • Large number of configurations • Legacy code • Contracts or interfaces are not required • Software development stages do not cooperate • Marketplace demands higher reliability and short release cycles

  8. Apply computing power with Smart Tools • Aid, not replace, the developer & tester • Amplify human effort • manage cumbersome details • find inconsistencies in complex environments • ensure quality by reducing errors • Good tools enable better processes

  9. PPRCProgrammer Productivity Research Center “Technology-based approach for software development” • Focus on defect prevention and early detection • Collect information about the development process • Achieve process automation through technology • Enable rapid research and tool development with rich infrastructures Approach Key Focus Areas: Performance and Reliability Products that Microsoft ships have been touched by at least one of PPRC tools

  10. Product Development Stages Coding SE …… Testing Debugging TOOLS Collection/ Analysis Management Transformations Distributed Repository Service API Models Source Control Test Behavior Binary Symbols Project Plans Specification Program Information Bug Database Crash Dumps Customer Feedback PPRC Approach Collect information about development process Analyze information to guide tools Need lots of tools throughout the development process

  11. SPA Scalable Program Analysis AST Source Code Analysis Infrastructure Rapid Research and Tool Development TOOLS Analysis TOOLS Testing TOOLS Performance TOOLS Correctness Higher Abstraction Infrastructure Technologies BMAT Binary Matching MaX Magellan Dependency Framework VULCAN Binary Editing Infrastructure Nirvana Run Time Infrastructure … using common infrastructures

  12. Example: PPRC Correctness Tools • PREfix • Detailed, path-by-path interprocedural analysis • Heuristic (unsound, incomplete) • Used at build time • PREfast • User-supplied bug finding plug-ins traverse AST • Desktop use, easily customized • Monitor code being checked-in • 12.5% of bugs fixed in Windows Server 2003 found by PREfix/PREfast • Every code that is checked in Windows has to be PREfast clean

  13. Eliminating Vulnerability:SAL – Function Header Annotation • Function annotations that describe contracts in header files • SAL annotations are important • Enables PREfix/PREfast to detect more Buffer Overruns • Reduces false positives in PREfix/PREfast tools • Example SAL annotation WINUSERAPIUINTWINAPIGetDlgItemTextW( __in HWND hDlg, __in int nIDDlgItem, __out_ecount(nMaxCount) LPWSTR lpString, __in int nMaxCount); • Annotations indicates parameter lpString is a NULL-terminated string buffer large enough to hold nMaxCount elements

  14. Eliminating Vulnerability:Restricting usage of certain APIs • Commonly used APIs that caused security class bugs • Examples of APIs that are restricted? • API that does unbounded buffer copying • E.g. strcpy, vsprintf, wcscat, wcscpy • API that does not guarantee NULL termination • E.g. wnsprintf • Makes tools more effective • Improves error detection • Reduces noise

  15. PPRC Correctness Tools : Sound Analysis • SDV (Static Driver Verifier) • Uses software model checking to verify specific rules • Can prove absence of bugs • Development with Windows Driver Quality Group • More information: Sriram Rajamani and Tom Ball http://research.microsoft.com/slam • ESP • Uses scalable program analysis to check specific rules • Scales to 10+ M lines of code • Can prove absence of bugs • More information: Manuvir Das http://research.microsoft.com/esp

  16. Lessons • Developers are rational • Usage based on cost-benefit tradeoff • Has to be integrated into the development process • Heuristics suffice • Effective in finding customer facing bugs – if not all • Soundness is additional benefit • User interfaces are crucial • Prioritizing, reducing noise, and error diagnosis are crucial

  17. Another ExampleTest Prioritization: Motivation • Full tests take weeks to run • Want to detect bugs on day 1 rather day 21 • Critical fix has to be released in a limited time frame • What should be tested • Developer needs to run tests before checkin • What tests should be run to exercise changed code Observation: New defects are introduced from changed code

  18. What is hard? • Detecting change and affected parts of the program • Scalability to handle complex systems • Tens of millions of tests • Thousands of developers and testers • Tens of millions lines of source code • Acceptable response times • Integrating seamlessly into a development process

  19. Binary Matching • BMAT – Binary Matching, [Wang, Pierce, and McFarling JILP 2000] • Vulcan, [Srivastava, Edwards, Vo]

  20. Scout: Test Prioritization System What changed? Detect impacted blocks (new + old changed) New Image Image Change Analysis Old Image What can be leveraged? Detect impacted blocks likely to be covered by existing tests Coverage Impact Analysis Old Image Coverage What order should tests be run? Detect minimal set of test cases likely to cover the impacted blocks Test Prioritization Output 1Prioritized list of test casesbased on coverage of the impacted blocks Output 2List of impacted blocks not covered by the existing tests

  21. Scout Results: ProductX.EXE Image Info Results Scout took about 210 seconds 1.6 million lines of source code

  22. Summary: Test Prioritization • Effectively being used in MS Windows, SQL, and Exchange development process • Scales to production environments - millions of tests and thousands of binaries • Combination of approximations and static analysis to eliminate manual methods • Collect information about development process • More information: Amitabh Srivastava and Jay Thiagarajan. Effectively Prioritizing Tests in Development Environment, ISSTA 2002. 

  23. Microsoft Experience • Can software technology produce more reliable software? • Answer is definitely YES • Revolution in developer attitude • Want more and better tools • Tools have become critical “must-have” in developing quality software • Tools have affected process changes • Windows is re-inventing software development with tools as an essential component

  24. PPRC has evolved into CSE • CSE : Center for Software Excellence in Windows division • Vision remains the same but charter has expanded • Integrated view of tools and development process • Tackling the entire development lifecycle from design to sustained engineering • Contacts: Todd Proebsting (toddpro@microsoft.com) GS Rana (gsrana@microsoft.com)

  25. Key Points • Automation drives efficiency • Automate software development with smart tools • Introduce new technologies into the development process • Seamless integration into development process is key • Collect intelligence of the development process • Build process with technology and tools in mind

  26. Questions?

More Related