1 / 28

Practical Model-Checking Method For Verifying Correctness of MPI Programs

Practical Model-Checking Method For Verifying Correctness of MPI Programs. Salman Pervez, Ganesh Gopalakrishnan, Robert M. Kirby, Robert Palmer School of Computing University of Utah. Rajeev Thakur, William Gropp Mathematics and Computer Science Division Argonne National Laboratory.

triage
Download Presentation

Practical Model-Checking Method For Verifying Correctness of MPI Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Practical Model-Checking Method For Verifying Correctness of MPI Programs Salman Pervez, Ganesh Gopalakrishnan, Robert M. Kirby, Robert Palmer School of Computing University of Utah Rajeev Thakur, William Gropp Mathematics and Computer Science Division Argonne National Laboratory Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  2. Thesis of the Talk • Concurrent algorithms are notoriously hard to design and verify. • Formal methods, and in particular finite-state model checking, • provide a means of reasoning about concurrent algorithms. • Principle advantages of modeling checking approach: • Provides formal framework for reasoning • Allows coverage – examination of all possible process interleavings • Principle challenges of modeling checking approach: • Requires modeling step • Can lead to “state explosion” Thesis: In-Situ modeling checking with dynamic partial-order reduction provides the advantages of the model checking approach while ameliorating the challenges. 2/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  3. Why MPI is Complex: Collision of Features • Rendezvous mode • Blocking mode • Non-blocking mode • Reliance on system buffering • User-attached buffering • Restarts/Cancels of MPI Operations • Send • Receive • Send / Receive • Send / Receive / Replace • Broadcast • Barrier • Reduce An MPI program is an interesting (and legal) combination of elements from these spaces • Non Wildcard receives • Wildcard receives • Tag matching • Communication spaces 3/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  4. Conventional Debugging of MPI • Inspection • Difficult to carry out on MPI programs (low level notation) • Simulation Based • Run given program with manually selected inputs • Can give poor coverage in practice • Simulation with runtime heuristics to find bugs • Marmot: Timeout based deadlocks, random executions • Intel Trace Collector: Similar checks with data checking • TotalView: Better trace viewing – still no “model checking”(?) • We don’t know if any formal coverage metrics are offered 4/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  5. What is Model Checking? Navier-Stokes Equations are a mathematical model of fluid flow physics “V&V” – Validation and Verification “Validate Models, Verify Codes” “Formal models” can be generated either automatically or by a modeler which translate and abstract algorithms and implementations. 5/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  6. Related work on FV for MPI programs • Main related work is that by Siegel and Avrunin • Provide synchronous channel theorems for blocking and non-blocking MPI constructs • Deadlocks caught iff caught using synchronous channels • Provide a state-machine model for MPI calls • Have built a tool called MPI_Spin that uses C extensions to Promela to encode MPI state-machine • Provide a symbolic execution approach to check computational results of MPI programs • Define a static POR algorithm which ameliorates challenge 2. • Schedules processes in a canonical order • Schedules sends when receives posted – sync channel effect • Wildcard receives handled through over-approximation 6/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  7. Traditional Execution Checking Versus Model Checking “Execution Checking” In current practice, concrete executions on a few diverse platforms are often used to verify algorithms/codes. Consequence: Many feasible executions might not be manifested. “Model Checking” Model checking forces all executions of a judiciously down-scaled model to be examined. Current focus of our research: minimize modeling effort and error. 7/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  8. “In Situ” Model Checking • Fundamental challenges of model checking: • Model creation (and validation) • Managing state explosion Ameliorate first challenge by running instrumented versions of the code. • Solution – Runtime (i.e. “In Situ”) Model Checking • Pioneered by Patrice Godefroid (at Bell labs) • Developed in the context of his Verisoft project. He called it Runtime model checking. • Godefroid created the dynamic partial-order reduction algorithm in 2005 Ameliorate second challenge by pruning the state-space based upon independence of operations. 8/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  9. Our Contribution: In Situ Model Checker For MPI Scheduler Consider Wildcard Receives and Their Interleaving Socket Communication Process 0 Process 1 Process 2 Process 3 9/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  10. Code to handle MPI_Win_unlock (in general, this is how every MPI_SomeFunc is structured…) MPI_Win_unlock(arg1, arg2...argN) { sendToSocket(pID, Win_unlock, arg1,...,argN); while(recvFromSocket(pID) != go-ahead) MPI_Iprobe(MPI_ANY_SOURCE, 0, MPI_COMM_WORLD...); return PMPI_Win_unlock(arg1, arg2...argN); } An innocuous Progress-Engine “Poker” Introduced for handling one-sided MPI 10/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  11. Current MPI Constructs Examined • MPI Constructs Examined: • MPI_Init • MPI_Send • MPI_Ssend • MPI_Recv • MPI_Barrier • MPI_Finalize • MPI_Win_lock • MPI_Win_unlock • MPI_Put • MPI_Get • MPI_Accumulate Required creating code which communicated with scheduler. Required understanding how the progress engine worked with MPICH (with adjustments to the scheduler to employ this information judiciously). 11/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  12. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / NULL Scheduler Options: P0:0 and P1:0 Scheduler Choice: 12/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  13. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / NULL Scheduler Options: P0:0 and P1:0 Scheduler Choice: P1:0 13/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  14. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:0 Scheduler Options: P0:0 and P1:1 Scheduler Choice: 14/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  15. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:0 Scheduler Options: P0:0 and P1:1 Scheduler Choice: P1:1 15/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  16. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:1 Scheduler Options: P0:0 and P1:2 Scheduler Choice: 16/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  17. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:1 Scheduler Options: P0:0 and P1:2 Scheduler Choice: P1:2 17/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  18. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:2 Scheduler Options: P0:0 and P1:3 Scheduler Choice: 18/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  19. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:2 Scheduler Options: P0:0 and P1:3 Scheduler Choice: P1:3 19/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  20. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:3 Scheduler Options: P0:0 and P1:4 Scheduler Choice: 20/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  21. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: NULL / P1:4 Scheduler Options: P0:0 Scheduler Choice: 21/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  22. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: P0:0 / P1:4 Scheduler Options: P0:1 Scheduler Choice: P0:1 – P0:4 22/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  23. MPI One-Sided Example Process P0 Process P1 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize 0: MPI_Init 1: MPI_Win_lock 2: MPI_Accumulate 3: MPI_Win_unlock 4: MPI_Barrier 5: MPI_Finalize Current Position: P0:4 / P1:4 Scheduler Options: P0:5 and P1:5 Scheduler Choice: Does it matter which choice It makes? Are these independent? 23/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  24. Partial-Order Reduction • With 3 processes, the size of an interleaved state space is p3=27 • Partial-order reduction explores representative sequences from each equivalence class • Delays the execution of independent transitions • In this example, it is possible to “get away” with 7 states (one interleaving) 24/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  25. Dynamic Partial-Order Reduction Full = { … } Enabled = {…} Backtrack = {…} Run the “instrumented” program to populate the full set of transitions and the enabled set of transitions at each state. Transition 1 Full = { … } Enabled = {…} Backtrack = {…} Given enabled sets E, we want to find backset sets B such that B is a proper subset of E and such that B captures representatives of all equivalent executions (under the notion of Independence) Transition 2 Full = { … } Enabled = {…} Backtrack = {…} Transition 3 25/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  26. Defining Dependence 26/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  27. Example Benefits: One-Sided Byte-Range Protocol 27/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

  28. Summary • Formal methods, and in particular finite-state model checking, • provide a means of reasoning about concurrent algorithms. • Principle challenges of modeling checking approach: • Requires modeling step • Can lead to “state explosion” • Both of which can be ameliorated by In-Situ Model Checking • Future Work: • Expand number of MPI Primitives (and the corresponding dependence table) • Exploit code-slicing to remove ancillary operations • Funding Acknowledgements: • NSF (CSR–SMA: Toward Reliable and Efficient Message Passing Software Through Formal Analysis) • Microsoft (Formal Analysis and Code Generation Support for MPI) • Office of Science – Department of Energy 28/28 Argonne National Laboratory School of Computing and SCI Institute, University of Utah

More Related