1 / 33

Software Quality: Testing and Verification II

Software Quality: Testing and Verification II. Software Flaws are identified at three levels. A failure is an unacceptable behaviour exhibited by a system The frequency of failures measures software reliability Low failure rate = high reliability

geneva
Download Presentation

Software Quality: Testing and Verification II

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Software Quality: Testing and Verification II

  2. Software Flaws are identified at three levels • A failure is an unacceptable behaviour exhibited by a system • The frequency of failures measures software reliability Low failure rate = high reliability • Failures result from violation of a requirement • A defect is a flaw that contributes to a failure • It might take several defects to cause one failure • An error is a software decision that leads to a defect

  3. Eliminating Failures: Testing vs Verification • Testing = running the program with a set of inputs to gain • confidence that the software has few defects • Goal: reduce the frequency of failures • When done: after the programming is complete • Methodology: develop test cases; run the program with • each test case • Verification = formally proving that the software has no defects • (in this case, the program is said to be “correct”) • Goal: eliminate failures • When done: before, during and after the programming is complete • Methodology: write separate specifications for the • code; prove that the code and the specifications • are mathematically equivalent

  4. Program Correctness • The correctness of a program is based on a specific standard. • That standard is called a specification. int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m; } • E.g., a specification for the above program might be that it “finds the maximum value of any two integers.”

  5. Formalizing a Specification • A Formal specification is written as a logical expression called an assertion. • An assertion describes the state of the program’s variables. • Two key assertions are the program’s precondition and its postcondition. int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m; } • A domain is a set of values over which a variable is well defined. • The primitive types (int, float, boolean, etc.) and standard Java classes (String, Vector, HashMap, etc.) provide domains for reasoning about programs. PreconditionP = what’s required for the program to begin its run. PostconditionQ = what’s ensured to be true when the program finishes

  6. Pre- and Postconditions • Postconditions describes what it will compute. • For max, a postcondition is Q = m = max(a, b), where max is a mathematical function that delivers the larger of two integers. • Preconditions describe minimum requirements for the program to run. • For max, a and b can be any integers, so the precondition is P = true. • Before proving a program’s correctness, we first write its specifications: {true} int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m; } {m = max(a, b)}

  7. Another Example – computing a factorial (n!) • {n > 1} int Factorial (int n) { int f = 1; int i = 1; while (i < n) { i = i + 1; f = f * i; } return f; } • {f = n!} • This example raises two issues. What happens if: • 1. the program has a loop that never terminates? • 2. The program terminates abnormally (e.g., an exception is raised)? Precondition P Postcondition Q

  8. Defining away the problem: “partial correctness” • Let’s assume that nothing bad ever happens… • programs always terminate after a finite number of steps, and • termination is always normal. • Then such a program is partially correct if, for every set of input values that satisfies precondition P, the program computes a result that satisfies postcondition Q. • E.g., Factorial is partially correct if for every value of n that satisfies n> 1, it computes f =n!

  9. (Partial) Correctness Proofs • Let’s think generally about any program or sequence of statements s, whose pre- and postconditions are P and Q. • A “Hoare triple” is a predicate of the form • {P} s {Q} • which asserts that “execution of statements s, beginning in a state that satisfies P, results in a state that satisfies Q.” • If we can prove that this Hoare triple is valid, (i.e., it is true for all assignments of values to variables in P, Q, and s) then the program s is said to be correct. • But how can we construct such a proof?

  10. Constructing a Correctness Proof • If the program is a series of statements: • s1; s2; …; sn • We start with the Hoare triple {P} s1; s2; …; sn{Q}, and • use inference rules for programs to • derive an intermediate triple {Pi} si {Pi+1} for every statement si in this sequence. • When done, we also ensure that • Note: This process is similar to a direct proof in logic, where we • use inference rules to • derive a series of assertions that logically link the premises to the conclusion. • So, what are the inference rules for programs?

  11. Inference Rules are tied to program statement types

  12. Formal methods and real programs • Major question: Where do formal methods fit in the software process? That is, • How do we integrate them into object oriented programs? • How do we write pre- and postconditions P and Q for • methods? • classes? • systems? • Once written, how are these used to ensure that software is correct?

  13. Recall the spiral model for software design Review Deployment (v 1.0) v 1.1 v 1.0 Testing and Verification Prototype Requirements Coding/ Integration Specifications (use cases) Design

  14. Formal methods in the object oriented design process • In OO design, we focus on classes and objects Methods and messages are subordinate • Collections of objects have state, which is the set of all active objects and the values of their variables at any moment of run time. • Formal specifications P and Q are thereforelogical expressions about each object’s state. • Tools for the formal design process • Specifications : Java Modeling Language (JML) • Design: Unified Modeling Language (UML and JML) • Coding: Java and JML • Verification: JML

  15. Correctness of OO systems • Where? • Method level: pre- and post-conditions, loop invariants • Class level: class invariant (class state) • System level: intra-class invariants (system state) • When? • Specification and design phases: Write specifications for all classes and methods (UML/JML) • Coding phase: Develop code from the specifications (UML/JML/Java) • Validation phase: Prove (mathematically) that specifications and code are logically equivalent (JML <==> Java)

  16. What is JML? (www.jmlspecs.org) • History • Emerged in early 2000s out of ESC/Java2 • Goals • Integration of formal methods throughout the software process • Formal specification accessible to programmers • Direct support for design by contract • Integration with a real language (Java) • JML allows us to mix specifications directly with the Java code • Preconditions • Postconditions • Loop invariants • Class invariants

  17. JML Basics • JML specifications are special comments in a Java program: • //@ for one-liners • /*@ …. @*/ for multiple-liners • The Hoare triple • {P} s1; s2; …; sn{Q} • is written in JML/Java as • (P and Q are written as Java • boolean expressions, and use • parameters, local, and class • variables as arguments.) /*@ requires P ; ensures Q ; @*/ type method (parameters) { local variables s1; s2; …; sn }

  18. JML Language Summary Note: p and e are also good old fashioned Java boolean expressions, possibly augmented by JML-specific operations.

  19. Here’s a simple example, first as a Hoare triple: • {n >1} P (precondition) int Factorial (int n) { int f = 1; int i = 1; • {1 < i  i < n f = i!}R (loop invariant) while (i < n) { i = i + 1; f = f * i; } return f; } • {f = n!}Q (postcondition)

  20. And again with JML expressions for P, Q, and R: P Q R • /*@ requires 1 <= n ; • ensures \result == (\product int i; 1<=i && i<=n; i) ; • @*/ • static int Factorial (int n) { • int f = 1; • int i = 1; • /*@ loop_invariant i <= n && f == (\product int j; 1 <= j && j <= i; j); • @*/ • while (i < n) { • i = i + 1; • f = f * i; • } • return f; • }

  21. JML-based software tools (1 and 3 are built into Eclipse) • 1. Compiling (usejmlcinstead ofjavac) • Does syntactic and type checking, and byte code generation for all JML assertions and Java code • 2. Static checking (ESC/Java2) • 3. Runtime assertion checking (usejmlracinstead ofjava) • Checks truth of precondition P at entry to every call • Checks truth of postcondition Q at exit from every call • Checks truth of loop invariant R before every iteration • Issues a Java Exception when any of these is not true • Note: this is not formal verification • (Checking truth for one instance of a call is not the same as checking truth for all instances. The latter is proof of correctness!) • 4. Proof assistance tools (Daikon, LOOP)

  22. JML Eclipse Environment 1. Compiling 3. Runtime assertion checking

  23. Continuing our JML example, let’s wrap Factorial inside a simple class: • public class myFactorial { • /*@ requires 1 <= n; • ensures \result == (\product int i; 1<=i && i<=n; i); • @*/ • static int Factorial (int n) { • … • } • public static void main(String[] args) { • int n = Integer.parseInt(args[0]); • System.out.println("Factorial of " + n + " = " + Factorial(n)); • } • }

  24. … compile it with jmlc, and run it twice with jmlrac: • % jmlc -Q myFactorial.java • % jmlrac myFactorial 3 • Factorial of 3 = 6 • % jmlrac myFactorial -5 • Exception in thread "main” • org.jmlspecs.jmlrac.runtime.JMLEntryPreconditionError: • by method myFactorial.Factorial regarding specifications at • File "myFactorial.java", line 3, character 15 when • 'n' is -5 • at myFactorial.checkPre$Factorial$myFactorial(myFactorial.java:240) • at myFactorial.Factorial(myFactorial.java:382) • at myFactorial.main(myFactorial.java:24) normal run abnormal run (throws a JML exception)

  25. JML Exceptions

  26. JML helps identify errors • Example 1: Suppose we change the while loop from • while (i < n) • towhile (i <= n) • so that n! will be computed incorrectly. • Here’s the result: • % jmlrac myFactorial 3 • Exception in thread "main" • org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT: • by method myFactorial.Factorial regarding specifications at • File "myFactorial.java", line 9, character 24 when • 'n' is 3 • at myFactorial.internal$Factorial(myFactorial.java:102) • at myFactorial.Factorial(myFactorial.java:575) • at myFactorial.main(myFactorial.java:211) invariant not satisfied

  27. JML Example 2 • Suppose we change the while loop from • while (i < n) to • while (i <=n ) • and also remove the JML loop invariant. Now we get: • % jmlrac myFactorial 3 • Exception in thread "main" • org.jmlspecs.jmlrac.runtime.JMLNormalPostconditionError: • by method myFactorial.Factorial regarding specifications at • File "myFactorial.java", line 4, character 23 when • 'n' is 3 • '\result' is 24 • at myFactorial.checkPost$Factorial$myFactorial(myFactorial.java:321) • at myFactorial.Factorial(myFactorial.java:392) • at myFactorial.main(myFactorial.java:24) postcondition not satisfied

  28. JML Example 3 • Disagreement between a JML specification and a program may signal an error in the specification. • E.g., if the loop invariant had specified j <= i rather than j < i the following outcome would occur: • % jmlrac myFactorial 3 • Exception in thread "main" • org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT: • by method myFactorial.Factorial regarding specifications at • File "myFactorial.java", line 9, character 24 when • 'n' is 3 • at myFactorial.internal$Factorial(myFactorial.java:101) • at myFactorial.Factorial(myFactorial.java:573) • at myFactorial.main(myFactorial.java:209) invariant not satisfied

  29. But beware… JML is no panacea • jmlrac doesn’t trap all errors… here are two “normal” runs: • % jmlrac myFactorial 21 • Factorial of 21 = -1195114496 • % jmlrac myFactorial 32 • Factorial of 32 = -2147483648 • Recall: (1) Java has no ArithmeticOverflow exception, but • (2) Factorial(n) for n > 12 should give a result > 231-1 • Note: jmlrac computes the same wrong result when it checks the postcondition as the Factorial method computes, so that this error goes undetected. • Conclusion: the program and its specifications are both wrong. wrong results!

  30. Exception Handling in JML We can throw Java Exceptions, and then validate their circumstances in JML whenever they occur. /*@ requires P ; ensures Q ; signals (exception) expression; @*/ type method (parameters) { locals s1; s2; …; sn (includes “throw new exception ;”) } JML executes this whenever this happens Two outcomes:1)expression is true and normal Java exception handling proceeds, or 2)expression is false and JMLExceptionalPostconditionError is reported.

  31. JML Example 4: Throwing and Checking Exceptions • /*@ requires 1 <= n; • ensures \result == (\product int i; 1<=i && i<=n; i); • signals (ArithmeticException) n > 12; • @*/ • static int Factorial (int n) { • if (n > 12) throw new ArithmeticException(); • else { … • % jmlrac myFactorial 22 • Exception in thread "main" java.lang.ArithmeticException • at myFactorial.internal$Factorial(myFactorial.java:9) • at myFactorial.Factorial(myFactorial.java:610) • at myFactorial.main(myFactorial.java:213) Normal Java exception handling occurs, since n > 12 is true.

  32. Additional Points about JML • 1. We can sometimes avoid a signals clause by strengthening the precondition. E.g., for Factorial, we could have said: requires 1 <= n && n < 13 ; • 2. Specifications are always declarative; they never affect the state of the program. • 3. Runtime assertion checking is not proof, but it does provide a rigorous framework for debugging. • 4. JML provides a language in which formal methods and Java programs can be integrated. • 5. There’s a lot more to JML. We are particularly interested in: a. class level specifications? b. Tools for static checking of specifications? c. Tools for proving correctness?

  33. Readings (at the course web site) • Hoare, An axiomatic basis for computer programming, Communications of the ACM 12(10):576-580. 
 • Leino, Hoare-Style Program Verification I • Leino, Hoare-Style Program Verification II • Gordon, Specification and Verification I • Dwyer et al., Software Specifications • Leavens and Cheon, Design by Contract with JML • Poll, Kiniry, and Cok, Introduction to JML • Burdy et al., Overview of JML Tools • JML Reference Manual

More Related