Generating High-Quality Tests for Boolean Circuits by Treating Tests as Proof Encoding

1. Generating High-Quality Tests for Boolean Circuits by Treating Tests as Proof Encoding Eugene Goldberg, Pete Manolios Northeastern University, USA TAP-2010, July 1-2, Malaga, Spain

2. Summary • Introduction • Test generation algorithm based on TTPE • Experimental results and conclusion

3. Motivation • Testing is easy (scalable) but incomplete • Formal verification is complete but hard • How do we get the best of both worlds (e.g. by generating a small test set that is complete or close to such)? • We study testing by the example of combinational circuits • We describe circuits by propositional logic

4. Main Idea TTPE: Treating Tests As Proof Encoding Q: Why does testing work at all? A: Short proof  small encoding test set Q: What is the best coverage metric? A: A formal proof is an ideal coverage metric (contains a complete set of corner cases) Q: When does one stop generating tests? A: When the test set encodes a proof

5. Description of the Setup N is a combinational circuit, Itis correct if N  0, It has a bug if N(x) = 1. z N Y Circuit N  CNF formula FN(X,Y,z), N  0  (FNz)  0 …. X Point p is (x,y,z) i.e. a complete assignment to X Y  {z} Test t extracted from p is the assignment to X, i.e. t = x

6. A Naive Testing Procedure Si = {p1,…,pi} z p1, t1, N(t1)=0, Enc(S1) = no N Y p2, t2, N(t2)=0, Enc(S2) = no …….. pi, ti, N(ti)=0, Enc(Si) = no …. X pi+1, ti+1, N(ti+1)=1, Enc(Si+1) = no pi+1, ti+1, N(ti+1)=0, Enc(Si+1) = yes

7. Summary • Introduction • Test generation algorithm based on TTPE • Experimental results and conclusion

8. High-Level Description • We use the resolution proof system • Checking if a set of points encodes a resolution proof is hard • Instead, we generate points (called boundary) that encode mandatory fragments of a proof • So, instead of trying to encode an entire proof we target essential parts of it • We stop when a counterexample is found or a resource is exceeded

9. Encoding a Resolution Proof C = x1 x2  ~x5, C =~x1x7  C = x2  ~x5 x7 p, p legalize the resolution of C and C on v iff a) p and p are different only in v b) C (p ) = 0, C (p ) = 0 c) C(p ) = 0, C(p ) = 0, where C is the resolvent S={p1,..,pk} encodes a proof if the resolutions legalized by points of S are sufficient to derive an empty clause. Tests T encode a proof if they are extracted from S encoding a proof

10. Small Complete Test Sets A proof of (FNz)  0 of k resolutions is encoded by 2k points (at most) A complete set of  2k tests for checking N  0. N with 1000inputs: 106 tests instead of 21000

11. Boundary Points • Given a CNF formula F, an unsatisfying assignment to Vars(F) is a v-boundary point p iff • If C  F and C(p)=0, then v  Vars(C) • If p is a v-boundary point of F and F is unsatisfiable, any proof has to have a resolution such that • it is a resolution on v, • the resolvent is falsified by p Adding the resolvent of this resolution to F eliminates p as a v-boundary point. If F is satisfiable, there is a boundary point of F that cannot be eliminated.

12. Extracting Tests from Boundary Points Circuit N , CNF formula F=FN  z, Is N(t)=1? Generate a v-boundary point pi of F (finding pirequires a SAT-check) Extract test ti from pi N(ti)=1? If so, stop. Add to F a resolvent on v (mandated by pi) Go to step 1 (to generatepi+1)

13. Summary • Introduction • Test generation algorithm based on TTPE • Experimental results and conclusion

14. Faults in Arithmetic Components G is a 24-input AND gate

15. Comparing SAT, Random Tests and Tests from Boundary Points • 17 faults. Random tests failed (106 per fault) • Reused tests generated for previous faults • Precosat is a winner of SAT-2009 competition • Used Precosat for finding boundary points too

16. Some Concluding Remarks • TTPE is not a trick. There is a deep relation between proofs and tests • Many ways to use TTPE (e.g. encoding mandatory parts of a proof) • TTPE for more expressive logics (describing sequential circuits and software)?