How Fast and Fat Is Your Probabilistic Model Checker?

1. How Fast and Fat IsYour Probabilistic Model Checker? an experimental performance comparison David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,Mariëlle Stoelinga2, Ivan Zapreev1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede 3 ICIS, Radboud University, Nijmegen

2. ProbabilisticModel Checking Probabilistic System Probabilistic Requirement PRISM (hybrid) PRISM (sparse) MRMC VESTA ETMCC Probabilistic Model Probabilistic Formula YMER ≤ Probabilistic Model Checker Yes No Probability

3. Why Are Probabilities Useful? • system performance • uncertainty in the environment • randomized (networking) algorithms • abstract from large populations

4. ProbabilisticModel Checking... • What is inside? • temporal logics + model checking • numerical and optimisation techniquesfrom performance and operations research • Where is it used? • powerful tools • applications: distributed systems, security, biology, quantum computing... • Problem: Which tool to choose?

5. Probabilistic Models Discrete time Markov chains Continuous time Markov chains automata transitions are probabilistic timing for CTMC: Prob(wait time ≤ t) = 1 – e–t Probabilistic System Probabilistic Requirement Probabilistic Model Probabilistic Formula Probabilistic Model Checker Yes No Probability

6. SynchronousLeader Election • nodes in a ring elect a leader • each node selects random number as id • passes it around the ring (synchronously) • if  unique id,node with maximum unique id is leader • [Itai & Rodeh 1990]

7. 1 4 2 2 5 3 1 5 SynchronousLeader Election

8. 1 4 4 5 1 5 2 5 1 4 5 1 2 2 1 1 3 5 3 5 2 2 2 3 SynchronousLeader Election 1 4 2 2 5 3 1 5

9. Probabilistic Formulas Reachability Bounded Reachability Steady-State Property extensions of CTL Probabilistic System Probabilistic Requirement Probabilistic Model Probabilistic Formula Probabilistic Model Checker Yes No Probability

10. ProbabilisticModel Checkers Probabilistic System Probabilistic Requirement Choices made Three examples Probabilistic Model Probabilistic Formula Overall evaluation Probabilistic Model Checker Yes No Probability

11. Tools

12. Modelling informal description PRISM model adapt syntax VESTA model .tra format model YMER model ETMCC MRMC PRISM YMER VESTA

13. Selected Benchmarks

14. Experiment Relevance • Repeatable • Verifiable • Significant • Encapsulated

15. Experiment 1Reachability • Cyclic Polling Server:server cycles over n stationsand serves each one in turn • e.g. teacher walks through class,each pupil may ask a question • busy1P≥1(trueUpoll1)If station 1 is busy,the server will poll it eventually

17. Analysis

18. PRISM: MTBDD Size • Multi-Terminal BDD =data structure for transition matrix • size heavily depends on model • large MTBDD  slow

19. CPS versus SLE runtime 458.847 states 1.131.806 MTBDD nodes 7.077.888 states 2.745 MTBDD nodes

20. VESTA:simulation problem • actual probability close to bound P≥p(...) • estimate is almost always in [p–,p+] • some irregularity stops the simulation • 0.95 Prob(yes  actual Prob≥p) Prob(actual Prob≥p  yes)

21. Result Overview: Timing depends heavily on MTBDD size depends heavily on MTBDD size depends heavily on MTBDD size

22. Result Overview: Memory MTBDD size varies heavily almost independent from model size

23. Experiment 2Bounded Reachability • Tandem Queueing Network • two queues after each other • P<0.01(trueU ≤2full)Is the probabilitythat the system gets full in 2 time unitssmall? checkin counter security check

25. Analysis

26. Result Overview: Timing

27. Result Overview: Memory

28. Experiment 3Steady State Property • Tandem Queuing Network • S>0.2( P>0.1(X2nd queue full) )In equilibrium,the probability to satisfy is > 0.2 P>0.1(X2nd queue full) P>0.1(X ...)

30. Analysis

31. Simulating Steady State? • simulation of bounded reachabilityhas clear stopping criterion • simulation of unbounded reachability reachability with very large bound • simulation of steady state? never stops

32. Result Overview: Timing

33. Result Overview: Memory

34. Nested Formulas • we also checked nested propertiesP≥0.8(P≥0.9(trueU ≤100n70) Un50) • not detailed here

35. Result Overview: Timing based on a single property only: did not terminate

36. Result Overview: Memory

37. Conclusions