1 / 42

Dynamic Verification of Multicore Communication Applications in MCAPI Subodh Sharma, UofU

HLDVT 2009. Dynamic Verification of Multicore Communication Applications in MCAPI Subodh Sharma, UofU Ganesh Gopalakrishnan, UofU Eric Mercer, BYU. Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994. 1. Concurrency Space in Multicore Era. Concurrency Space in Multicore Era.

clyde
Download Presentation

Dynamic Verification of Multicore Communication Applications in MCAPI Subodh Sharma, UofU

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HLDVT 2009 Dynamic Verification of Multicore Communication Applications in MCAPI Subodh Sharma, UofU Ganesh Gopalakrishnan, UofUEric Mercer, BYU Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994 1

  2. Concurrency Space in Multicore Era

  3. Concurrency Space in Multicore Era Requires Standardization Multicore Association (MCA) Effort

  4. What is MCAPI (Multicore Communication API)? • An API specification from MCA (Multicore Association) • To program embedded systems like mobile phones, PDAs, routers, servers, etc. • Not restricted to SPMD (like MPI) or multi threaded style of programming.

  5. Inter API interactions Taxonomy for MCA APIs MTAPI – Task Management API • Specification work yet to begin • Thread Pooling, Work Stealing queues e.g. CILK, TBB, TPL etc. MCAPI – Communication API • Message based • Packet/Scalar Channel based MRAPI – Resource Management API • Semaphores, mutexes • Shared memory segment allocation, deallocation

  6. State of MCA Efforts? • MCAPI standard is released. • MCAPI reference implementation exists • A commercial implementation – Polycore, Inc. • In-house implementation on FPGA in progress (Utah). • MRAPI standard work in progress. • MTAPI efforts yet to begin.

  7. State of MCA Efforts? • Standardization efforts take a long time to mature • Need for FV in MCA API development. • Our effort: Early Engagement of FV in the MCA APIs space • To promote early adoption of the API • To promote better programming practices • To help avoid early pit-falls – e.g. unspecified behaviors

  8. Achievements so far! • Formal Specification work (at BYU) • Permits symbolic analysis • Murphi -> CVC • Query Oracle for scenario outcome calculation. • Platform testing • Dynamic Verification of MCAPI user apps (at Utah) • MCC tool created. • Current capabilities of MCC • Dynamic search of dependencies • Exerting a deterministic runtime control • Algorithmic Advances • Symmetry reduction in MCAPI domain space.

  9. Focus of the talk! • Formal Specification work (at BYU) • Permits execution based symbolic analysis • Murphi -> CVC • Query Oracle for scenario outcome calculation. • Platform testing • Dynamic Verification of MCAPI user apps (at Utah) • MCC tool created. • Current capabilities of MCC • Dynamic search of dependencies • Exerting a deterministic runtime control • Algorithmic Advances • Symmetry reduction in MCAPI domain space.

  10. Related Work • CHESS tool from MSR • Uses preemption bounded search • Cannot be used for message passing programs • ISP tool from Utah • Used for MPI verification • MCC differs in the way determinism is enforced at runtime. • Inspect tool from Utah • Used for shared memory programs • Implements the classical DPOR algorithm. • JPF • A model checker for Java bytecode.

  11. MCAPI Preliminaries • MCAPI Node – • Processes, Threads, OS instance, Processor core, etc. • MCAPI Endpoint – • Socket-like communication termination points • Identified by a tuple <node_id, port_id> • MCAPI Channels • Point to point unidirectional FIFO connections between a pair of endpoints. • A single communication universe

  12. Message received at FIFO receive queue MCAPI Messages Node 1 Node 2 Connectionless Message Port 1 <1,1> Port 1 <2,1> To <2,1> Port 2 <1,2>

  13. Sent over a connected channel • <1,1> can communicate on ONE channel only MCAPI Connection Oriented Communication Node 1 Node 2 Port 1 <1,1> Port 1 <2,1> To <2,1> Port 2 <1,2> ERROR!! To <3,1> Node 3 Port 1 <3,1>

  14. MCAPI API generic + message calls

  15. Illustration of MCAPI C program int main () { … for(t=0; t<NUM_THREADS; t++){ rc = pthread_create(&threads[t], NULL, run_thread, (void*)&thread_data_array[t]); } for (t = 0; t < NUM_THREADS; t++) { pthread_join(threads[t],NULL); } } void* run_thread (void *t) { … mcapi_initialize(tid,&version,&status); if (tid == 1) { recv_endpt = mcapi_create_endpoint (PORT_NUM, &status); mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE,&recv_size, &status); mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE, &recv_size, &status); } else { send_endpt = mcapi_create_endpoint (PORT_NUM,&status); recv_endpt = mcapi_get_endpoint (1,PORT_NUM,&status); mcapi_msg_send(send_endpt,recv_endpt, msg,strlen(msg), 1,&status); } mcapi_finalize(&status); return NULL; } Potential deadlock

  16. instrumentation compile request/permit request/permit MCC – MCAPI Checker MCAPI C Program Executable Scheduler Instrumented Program thread 1 thread n MCAPI Library Wrapper Workflow of MCC.

  17. Bug Classes in MCAPI User Applications • Non determinism due to communication race. • The recv() call is passed with only receiving endpoint as a parameter. • The receiving endpoint extracts message from the FIFO receive queue. T0 T1 T2 --- --- --- The send that matches the recv may decide the correctness !! S(e1,e2) R(e2) S(e1,e2) R(e2) POTENTIAL BUG

  18. Bug Classes in MCAPI User Applications • Mismatched Calls Does not exist T0 T1 --- --- R(1) Get_endpoint(2) S(1,2) “get_endpoint” invoked for a non-existent endpoint -- ERROR

  19. Challenge: exponential interleavings T0 T1 T2 T3 T4 T5 Possible Interleaving A B Only these 2 are RELEVANT!!! Dependent MCAPI calls 19

  20. Relevant interleaving exploration by MCC scheduler T0 T1 T2 Intra happens-Before Edge S (ep1,ep3) S(ep2,ep3) R(ep3) R(ep3)

  21. Relevant interleaving exploration by MCC scheduler Set of matching transitions No runnable thread Interleaving 1 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Issue to the runtime R (ep3) Match sets are computed at “decision points” Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3) Match Set: <S(ep1, ep3), R (ep3)> <S(ep2, ep3), R (ep3)>

  22. Relevant interleaving exploration by MCC scheduler Interleaving 1 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Issue to the runtime R (ep3) Enabled Transitions: S(ep2, ep3), R (ep3) Match Set: <S(ep2, ep3), R (ep3)>

  23. Relevant interleaving exploration by MCC scheduler Interleaving 2 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Issue to the runtime R (ep3) Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3) Match Set: <S(ep2, ep3), R (ep3)>

  24. Relevant interleaving exploration by MCC scheduler Interleaving 2 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) R (ep3) Issue to the runtime Enabled Transitions: S(ep2, ep3), R (ep3) Match Set: <S(ep1, ep3), R (ep3)>

  25. Relevant interleaving exploration by MCC scheduler Interleaving 2 T0 T1 T2 S (ep1,ep3) S (ep2,ep3) R (ep3) Error Caught R (ep3) Issue to the runtime Enabled Transitions: S(ep2, ep3), R (ep3) Match Set: <S(ep1, ep3), R (ep3)>

  26. Enforcing a deterministic runtime match • For non blocking requests, there is an additional problem • Runtime communication race even after a scheduler decides deterministically. T0 T1 T2 Match-set 1 IS (ep2,ep3) IR (ep3) IS (ep1,ep3) IR (ep3) Match-set 2

  27. Enforcing a deterministic runtime match T0 T1 T2 Match-set 1 RACE IS (ep2,ep3) IR (ep3) IS (ep1,ep3) Not finished yet!! IR (ep3) Match-set 2

  28. Enforcing a deterministic runtime match Solution 1 to enforce a deterministic match. • Addition of Probing function to the MCAPI library • Probes an endpoint’s receive queue for a message • Returns TRUE if the message is found • Scheduler issues the next match-set only when the probe function returns TRUE.

  29. Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) matched IR (ep3)

  30. Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) IR (ep3) matched Probes Runtime MCC Scheduler

  31. T1 Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) IR (ep3) Probes Runtime MCC Scheduler

  32. T0 T1 Enforcing a deterministic runtime match Probing Solution T0 T1 T2 Receive Queue IS (ep1,ep3) IS (ep2,ep3) IR (ep3) IR (ep3)

  33. Enforcing a deterministic runtime match Solution 2 to enforce a deterministic match. • Scheduler induces a “test” call • Scheduler spin-loops on the request handle until the successful completion of the “test” call. We opt Solution 2 in our work as it is non-intrusive.

  34. Enforcing a deterministic runtime match Dummy “wait” Solution T0 T1 T2 IS (ep1,ep3) IS (ep2,ep3) IR (ep3) matched IR (ep3) Enabled Transitions: Send(ep1, ep3), Send(ep2, ep3), Recv (ep3) Match Set: <Send(ep2, ep3), Recv (ep3)> <Send(ep1, ep3), Recv (ep3)>

  35. Enforcing a deterministic runtime match Dummy “wait” Solution T0 T1 T2 IS (ep1,ep3) IS (ep2,ep3) IR (ep3) Test () IR (ep3) Runtime MCC Scheduler

  36. Enforcing a deterministic runtime match Dummy “wait” Solution T0 T1 T2 IS (ep1,ep3) IS (ep2,ep3) IR (ep3) Test() IR (ep3) matched Runtime MCC Scheduler

  37. Concluding remarks and future work • MCC currently supports connectionless API constructs • Checks for safety assertion violations and Deadlocks • Steadily improve MCC over this year • Support for connection-oriented API calls, sanity checks etc. • Release MCC and assess experience of industrial app writers

  38. www.cs.utah.edu/formal_verification

  39. Happens-Before Relationship – Computed Dynamically T0 T1 T2 IS (ep2,ep3, r3) IR (ep3, r4) IS (ep1,ep3, r1) IS (ep1,ep4, r2) IR (ep3, r5) W(r2) R (ep3)

  40. Review of POR • Dynamic POR Dependencies based on concrete values x=56, y=753 (e.g.) at runtime! t1: a[x] := … t2: … := a[y] t2: … := a[y] t1: a[x] := … t2: … := a[y] Equal ??

  41. Happens-Before Relationship – Example T0 T1 T2 IS (ep2,ep3, r3) IR (ep3, r4) IS (ep1,ep3, r1) IS (ep1,ep4, r2) IR (ep4, r5) W(r2) R (ep3)

  42. OOO completion: Why construct Happens-Before relationship? T0 IS (1GB to T1) Must calls complete in PROGRAM ORDER? IS (1 byte to T2) NO!!! 42

More Related