Scaling Formal Methods Toward Hierarchical Protocols in Shared Memory Processors

1. Scaling Formal Methods Toward Hierarchical Protocols in Shared Memory Processors GRC CADTS Review, Berkeley, March 18, 2008 Presenters: Ganesh Gopalakrishnan and Xiaofang Chen School of Computing , University of Utah, Salt Lake City, UT 84112 {ganesh, xiachen}@cs.utah.edu http://www.cs.utah.edu/formal_verification Supported by SRC Contract TJ-1318 (Intel Customization)

2. Multicores are the future! Their caches are visibly central… > 80% of chips shipped will be multi-core (photo courtesy of Intel Corporation.)

3. Hierarchical Cache Coherence Protocols will play a major role in multi-core processors Chip-level protocols Intra-cluster protocols … mem mem dir dir Inter-cluster protocols State Space grows multiplicatively across the hierarchy! Verification will become harder

4. Protocol design happens in “the thick of things” (many interfaces, constraints of performance, power, testability). From “High-throughput coherence control and hardware messaging in Everest,” by Nanda et.al., IBM J.R&D 45(2), 2001.

5. Future Coherence Protocols • Cache coherence protocols that are tuned for the contexts in which they are operating can significantly increase performance and reduce power consumption [Liqun Cheng] • Producer-consumer sharing pattern-aware protocol [Cheng et.al, HPCA07] • 21% speedup and 15% reduction in network traffic • Interconnect-aware coherence protocols [Cheng et.al., ISCA06] • Heterogeneous Interconnect • Improve performance AND reduce power • 11% speedup and 22% wire power savings • Bottom-line:Protocols are going to get more complex!

6. Main Result #1 : Hierarchical • Developed way to reduce verification complexity of hierarchical (CMP) protocols using A/G Intra-cluster Remote Cluster 1 Home Cluster Remote Cluster 2 L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache L2 Cache+Local Dir L2 Cache+Local Dir L2 Cache+Local Dir RAC RAC RAC Global Dir Inter-cluster Main Mem

7. Main Result #2 : Refinement Developed way to Verifya Proposed Refinement of ONE unit into its low level (RTL) implementation

8. Main Result #2 : Refinement Developed way to Verifya Proposed Refinement of ONE unit into its low level (RTL) implementation Murphi

9. Main Result #2 : Refinement Developed way to Verifya Proposed Refinement of ONE unit into its low level (RTL) implementation Murphi

10. Main Result #2 : Refinement Developed way to Verifya Proposed Refinement of ONE unit into its low level (RTL) implementation Murphi HMurphi

11. Differences in Modeling: Specs vs. Impls Multiple steps in low-level One step in high-level home remote home remote router buf an atomic guarded/command

12. Our Refinement Check Guard for Spec transition must hold Spec transition Spec(I) Spec(I’) Observable vars changed by either must match Multi-step Impl transaction I I’ I is a reachable Impl state

13. Workflow of Our Refinement Check Murphi Spec model Muv Property check Product model in Hardware Murphi Product model in VHDL Hardware Murphi Impl model Check implementation meets specification

14. Anticipated Future Result Developed way to Verifya Proposed Refinement of the ENTIRE hierarchy

15. Anticipated Future Result Deal with pipelining Pipelined Interaction Sequential Interaction

16. Anticipated Future Result Develop ways to “tease apart” protocols that are “blended in” e.g. for power-down or post-si observability enhancement More protocols… .. do they interfere?

17. Basics • PI : Ganesh Gopalakrishnan • Industrial Liaisons : Ching Tsun Chou (Intel), Steven M. Geman (IBM), John W. O’Leary (Intel), Jayanta Bhadra (Freescale), Alper Sen (Freescale), Aseem Maheshwari (TI) • Primary Student : Xiaofang Chen • Graduation Date : Writing PhD Dissertation; in the market • Other Students :Yu Yang (PhD), Guodong Li (PhD), Michael DeLisi (BS/MS) • Anticipated Results: • Hierarchical : Methodology for Hierarchical (Cache Coherence) Protocol Verification, with Emphasis on Complexity Reduction (was in original SRC proposal) • Refinement : Methodology for Expressing and Verifying Refinement of Higher Level Protocol Descriptions (not in original SRC proposal)

18. Basics • Deliverables (Papers, Software, Xiaofang’s Dissertation) • Hierarchical: • Methodology for Applying A/G Reasoning for Complexity Reduction • Verified Protocol Benchmarks – Inclusive, Non-Inclusive, Snoopy (Large Benchmarks) • Automatic Abstraction Tool in support of A/G Reasoning • Refinement: • Muv Language Design (for expressing Designs) • Refinement Checking Theory and Methodology • Complete Muv tool implementation

19. What’s Going On • Accomplishments during the past year • Hierarchical: • Finishing Non-inclusive Hierarchical Protocol Verif • Developing and Verifying a Hier. Protocol with a Snoopy First Level

20. Experimental Results on One Hierarchical Protocol

21. A Snoopy Multicore Protocol • Motivation: • Snoop protocols commonly used in 1st level of caches • Have applied our approach on directory protocols • How about snoop protocols?

22. Applying Our Approach • Abstracted protocols • Experimental results

23. What’s Going On • Accomplishments during the past year (contd.) • Refinement: • HMurphi was fleshed out in great detail • Most of Muv was implemented (a large portion during IBM T.J. Watson Internship) – joint work with Steven German and Geert Janssen

24. What’s Going On • Future directions • Hierarchical + Refinement • Develop ways to verify hierarchies of HMurphi modules interacting • Pipelining • Teasing out protocols supporting non-functional aspects • Power-down protocols • Protocols to enhance Post-si Observability • Architectural Characterization • How do we describe the “ISA” of future multi-core machines? • How do we make sure that this ISA has no hidden inconsistencies

25. What’s Going On • Technology Transfer & Industrial Interactions • With Liaisons • Publications • FMCAD 06, FMCAD 07, HLDVT 07 • TECHCON 07 (best session paper award) • Journal paper and Dissertation (under prep) • Request to IBM for Open-sourcing Muv has been placed

26. Overview of “Hierarchical” Given a protocol to verify, create a verification model that models a small number of clusters acting on a single cache line Verification Model Home Remote Global directory Inv P

27. 2. Exploit Symmetries Model “home” and the two “remote”s (one remote, in case of symmetry) Verification Model Inv P

28. 3. Initial abstraction will be extreme; slowly back-off from this extreme… • P1 fails • Diagnose failure • Bug • report to user • False Alarm • Diagnose where guard is overly weak • Add Strengthening Guard • Introduce Lemma to ensure Soundness of Strengthening Inv P2 Inv P1 Inv P3

29. Overview of Theory Involved 

30. 3. Create Abstract Models (three models in this example) Inv P1 Inv P2 Inv P Inv P3

31. Step 1 of Refinement Inv P2 Inv P1 Inv P2 Inv P1 Inv P3 Inv P3’

32. Step 2 of Refinement Inv P2 Inv P1 Inv P2 Inv P1 Inv P3 Inv P3’ Inv P2’ Inv P1 Inv P3’

33. Final Step of Refinement Inv P2 Inv P1 Inv P2 Inv P1 Inv P3 Inv P3’ Inv P2’ Inv P1’ Inv P2’ Inv P1 Inv P3’ Inv P3’’

34. Detailed Presentation of RefinementNote: Three examples have been presented in full detail at http://www.cs.utah.edu/formal_verification/muv

35. Our Approach of Refinement Check Murphi Spec model Muv Property check Product model in Hardware Murphi Product model in VHDL Hardware Murphi Impl model Check implementation meets specification

36. Basic Features of Hardware Murphi vs Murphi … signal s1, s2 … s1 <= … chooserule rules; end; … firstrule rules; end; … transaction rule-1; rule-2; … end; …

37. Language Extensions to Hardware Murphi (I) • Directives --include spec.m • Joint variables correspondence correspondence u1[0..7] :: v1[1..8]; u1 :: v2; end;

38. Language Extensions to Hardware Murphi (II) • Transactionset transactionsetp1:T1; p2:T2 do transaction … end; • Rules with IDs rule:id guard ==> action; ruleset p1:T1; p2:T2 do rule:id … end;

39. Language Extensions to Hardware Murphi (III) • Execute a rule by ID << id.guard() >>; << id.action() >>; << id[v1][v2].guard() >>; … • Fine-grained assignments for write-write conflicts var[i]<:=data;

42. Our Extensions to Muv • Language extensions support • Automatic assertion generation for refinement • Ensure exclusive write to a var in each clock cycle • Serializability check for spec rules • Enableness for spec rules • Joint vars equivalence when inactive • Many done with static analysis

43. Refinement Extensions to Muv (I) • No write-write conflicts v := d; for i: s1..s2 do assert (update_bits[i] = false); end; v := d; for i: s1..s2 do update_bits[i] := true; end;

44. Refinement Extensions to Muv (II) • Serializability for specification rules t1 t3 t2 t1 t2 S1 S0 S0 S’2 S1 S’1 t3 • Obtain read and write sets of variables of each rule • Analyze read-write dependency • Check for cycles

45. Check for Dependency Cycles t1 t3 t2 t1 t2 S1 S0 S0 S’2 S1 S’1 t3 t1 read v1, write v3 t2 write v1, read v2 t3 write v2, read v3

46. Refinement Extensions to Muv (III) • Enableness of specification rules rule:id guard  action; bool function id_guard() {…} void procedure id_action(…) {…} << id.guard() >>; << id.action() >>; assert id_guard(); id_action();

47. Refinement Extensions to Muv (IV) • Joint variables equivalence when inactive • For each joint variable v • When all transactions that write to v are inactive • v must be equivalent in Impl and Spec assert inactive(T1) & inactive(T2) => v = v’; … transaction T1 … transaction T2 … …

48. The Cache Coherence Protocol Benchmark Dir Cache Mem Local Buf Home Buf Remote Buf Router Dir Cache Mem Local Buf Home Buf Remote Buf S. German and G. Janssen, IBM Research Tech Report 2006

49. Details of the Cache Example • Hardware Murphi model • ~2500 LOC • 15 transactionsets • Generated VHDL • ~1000 assertions, of which ~800 are write-write conflicts check assertions • Took ~16min with SixthSense for all assertions • Took ~13min w/o write-write conflicts check

50. Bugs Found with Refinement Check • Benchmark satisfies cache coherence already • Bugs still found • Bug 1: router unit loses messages • Bug 2: home unit replies twice for one request • Bug 3: cache unit gets updated twice from 1 reply • Refinement check is an automatic way of constructing such checks