The Cleanroom Approach to Quality Software Development

1. The Cleanroom Approach to Quality Software Development • (or ZeroDefict Software is really possible!) • References: • The Cleanroom Approach to Quality Software Development, Michael Dyer, Wiley 1992, • IEEE Software paper by Linger in March 1994 issue

2. An analogy • Compare sight typing with touch typing (In sight typing, the learner looks at the keys.) • Touch typing takes longer to learn but results in increased productivity with higher quality of typing in the long run. • Sight typing requires the operator to switch between the text and keys and mistakes and lapses are more common. • Compare this to: • The difference between Programming & Debugging and Specification & Refinement. • The latter takes longer to learn but the payoff is increased productivity and quality.

3. Key features of Cleanroom • Serious programming begins after formal specification, there is emphasis on more explicit design and verification from the specification. • There is no programmer testing. • The Cleanroom approach combines mathematical reasoning during specification and design refinement to code with statistical reasoning during test case generation and testing. • The aim is to produce zero defect software (or minimal defect software).

4. Cleanroom Successes - proven in use since the 1980s • The US 1980 Census System • Controlled by 25kloc program which operated its entire 10 months in production with no failures observed. • The IBM Wheelwriter Typewiter Systems (1984) • A 65 kloc program with millions of users and no failure ever detected. • The US Space Shuttle Software • Over 500 kloc while not completely zero defect, has been zero defect in flight.

5. Improved Specification and testing • Cleanroom is the first practical attempt • to place software development under statistical quality control, and • to deliver software with a known and certified meantime to failure (MTTF) • The key practices are: • to use formal specification and verification methods to create software of sufficient quality to forego programmer testing (ie unit test/debug) of code and • to require statistical based testing for evaluating software reliability • The pay off is that Cleanroom statistically based testing with random sampling driven from input probability distributions has been shown to be highly effective at finding errors with high failure rates. (It is better at finding the errors that occur most often.)

6. Cleanroom Process Flow (overview) Incremental software delivery Software requirements specification Software design and development Incremental statistical testing and regression testing Action Statistical Control Process Error Diagnosis and Correction Software reliability measurement Basis for FEEDBACK Basis for Level 5 Process Improvement Incremental approach based on independent specifications allows parallel development if required

7. The Cleanroom Process Model (in more detail) (stacked boxes indicate successive increments) Customer requirements Specification Functions Usage Incremental Development Planning Usage Specification Functional Specification Formal design correctness verification Statistical test-case generation Statistical testing Feedback of improvements Quality certification model MTTF Estimates

8. TABLE 1.1 Cleanroom Component Techniques Technology Cleanroom Focus Perspective Baseline Capability Defined process Starting point Design and inspection Early quality visibility Software Specification Software quality Focal point Formal description Software correctness Drive verification Usage/build data Customer acceptance Drive validation Software Verification Software quality Software quality In construction Error prevention Correct designs In inspection Confirmed correctness Zero defect No Developer Testing Software acceptance Software productivity Statistical Testing Customer acceptance Requirements validation MTTF Prediction Software reliability Certified MTTF Statistical Process Process improvement Software warranty

9. FIGURE 1.2 Roadmap for Introducing Cleanroom Component Techniques B A S E L I N E P R O C E S S Formal Specifications Correctness Verification No developer testing Process Control S/w Configuration Management Continuous inspection Statistical testing MTTF measurement

10. Trends in decreasing defect Rates based on improving development towards full use of Cleanroom methods Table 1.3 Trends in Software Quality Total Defect Rate Postdelivery Rate Traditional Development 50 to 60 5 to 10 Unstructured design Only testing for detection Baseline Development 20 to 40 1 to 4 Structured programming Formal inspections Advanced Development 0 to 20 0 to 1 Correctness verification Formal specification Statistical testing [IBM results cited in Dyer.]

11. IBM Cobol/SF Size: 85 Kloc of PL/l Testing Error Rate: 3.4 errors per kloc Productivity Rate: 740 Loc/person month NASA satellite-control project Size: 40 kloc of FORTRAN Testing Error Rate: 4.5 errors per kloc Productivity Rate: 780 loc/person month IBM 3090E tape drive Size: 86 kloc of C Testing Error Rate: 1.2 errors/kloc* (N.B. comparison with Unit Testing) Erisesson Telecom 0532 Operation System Size: 350 kloc Testing Error Rate: 1.0 error/kloc Finally more recently reported results from mid90s [in Linger 94] (sample)

12. Summary of Cleanroom Impacts on the SLC 1. Requirements Specification Function and Performance but with Usage Probabilities and Build Strategy 2. Software Design/Implementation Incremental Software Development but with Correctness verification not Unit Test 3. Independent Software Test Integration and Test of Released Increments but with Representative Statistical Usage Samples 4. Software Acceptance Demonstrated Function and Performance but with Certified Software MTTF