Nicole Lesperance, Peter Lisherness , and Tim Cheng Univ. of California, Santa Barbara

1. Coverage Discounting: Improved TestbenchQualification by Combining Mutation Analysis with Functional Coverage Nicole Lesperance, Peter Lisherness, and Tim Cheng Univ. of California, Santa Barbara September 9th, 2013

2. Verification Questions • Have the test vectors adequately stimulated the design? • Is the testbench able to identify errors?

3. Motivation Functional Coverage Functional Coverage + Functionally Meaningful + Evaluates design activation • Ignores propagation and • checker quality Coverage Discounting Coverage Discounting • + Functionally Meaningful • + Evaluates design activation, propagation and checkers Mutation Analysis Mutation Analysis • + Evaluates propagation and checker quality • - Mutants hard to analyze

4. Coverage Discounting: The Main Idea Use fault insertion to discount (revise downward) coverage scores Coverage Score Functional Coverage 100% Discounted Coverage Score Coverage Discounting 0% Mutation Analysis Detected + undetected mutants Undetected Mutants

5. An Existing Solution: Mutation Analysis • Benefits • Evaluates quality of testbench checkers • Indicates if tests fail to propagate potential errors to the checker or fail to activate mutants Add/Fix Tests Detected DUT Checker Fix Undetected

6. Mutation Analysis Drawbacks • Long runtime for simulation • Many man-hours required to analyze results: 2.1 Mutants are synthetic - difficult to identify testbench improvements to target mutant detection 2.2 Redundant mutants never detectable: Some undetected faults are redundant if (a >=0) b=1+a; else b=1-a; if (a >0) b=1+a; else b=1-a; Example:

7. Premise of Coverage Discounting 1) A mutant may change design functionality 2) A coverpoint, covered by testbench in the original design, may be suppressed (i.e. no longer be covered) in the mutated design 3) If the mutant suppressing the coverpoint is not detected, then the coverpoint is no longer considered covered in the original design

8. Coverage Discounting Flow Tests Detected DUT Checker Undetected Add more tests Fix Coverage Changed? Yes No

9. Experimental Results

10. OpenRISC CPU • Experiment #1: Measure test quality • Tests: Instruction-Set v. Random • Coverpoints: Opcodes • Experiment #2: Identify checker weaknesses • Tests: SoC Regression Suite • Coverpoints: CPU top-level signals

11. MeasureTest Quality Functional Tests Original Discounted Random Tests Original Discounted

12. Identify Checker Weaknesses Tests

13. Identify Checker Weaknesses

14. UART • Illustrates success of coverage discounting applied to a sophisticated testbench • Simulation Infrastructure: • Design RTL w/OVM Testbench • SystemVerilogcoverpoints (ModelSim) • Certitude fault insertion • Tests: OVM Sequences • Coverpoints: Hand-written spec-based

15. UART Results • Reduces debug effort from analysis of 146 mutants to examining 3 functional coverpoints • 1588 Mutants: • 7 not activated, 106 not propagated, 33 not detected • Total 146 mutants demand attention • 846 Coverpoints: • 4 uncovered • 3 discounted • All discounted relate to specific unchecked functions • Loopback, timeout interrupt identification register

16. Results Examined • Coverage discounting: • Identified "good" tests (propagation) • Diagnosed checker problems • Identified coverpoints vacuously covered • However: • Runtime of mutation analysis still an issue • Technique sensitive to the quantity and quality of faults

17. Confidence Metric • Aims to answer: • Is a coverpoint adequately challenged by the set of mutants? • When have enough faults been inserted (when can simulation be stopped)? • What is the optimal simulation ordering of mutants for coverage discounting?

18. Quicker Confidence CPU UART

19. Quicker Discounting CPU UART

20. Technology Transfer • Working closely with Freescale to apply discounting technique to industry design • JayantaBhadra and Vivekananda Vedula • Publications • Lisherness, P., Lesperance, N., and K. T. Cheng, “Mutation Analysis with Coverage Discounting”, Design Automation and Test Europe (DATE), March 2013. • Lisherness, P., and K-T. Cheng, "Improving Validation Coverage Metrics to Account for Limited Observability", Asia and South Pacific Design Automation Conference, Feb. 2012. 

21. Summary • Analyzed information gained from coverage discounting for two designs • Developed a confidence metric to gauge mutant effectiveness and an ordering heuristic to reduce runtime