Download Document

How Fast and Fat Is
Your Probabilistic Model Checker?
an experimental performance comparison
David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,
Mariëlle Stoelinga2, Ivan Zapreev1,2
1
MOVES Group, RWTH Aachen University
2 FMT Group, University of Twente, Enschede
3 ICIS, Radboud University, Nijmegen
Probabilistic
Model Checking
Probabilistic System
Probabilistic Requirement
PRISM PRISM
MRMC (hybrid) (sparse)
VESTA
Probabilistic
Model
Probabilistic
Formula
ETMCC


Probabilistic
YMER
Model Checker
Yes
No
Probability
Why This Work?
• Used more often
– applications: distributed systems, security,
biology, quantum computing...
• Powerful tools
• Problem: Which tool to choose?
Probabilistic
Model Checkers
Probabilistic System
Probabilistic Requirement
Probabilistic Model
Probabilistic Formula
Probabilistic
Model Checker
Yes
No
Probability
Tools
statist
PRISM
sparse
VESTA
YMER
numerical
ETMC
C
MRMC
PRISM
hybrid
CTMC
Java
DTMC + CTMC
DTMC + CTMC
MTBDD for transition
matrix
DTMC + CTMC
sparse transition matrix
CTMC, no S
CTMC, only U ≤t
C
C(++
) and
Java
Java
C(++
Experiment Relevance
•
•
•
•
Repeatable
Verifiable
Significant
Encapsulated
Selected Benchmarks
Synchronous Leader Election
discrete time
Randomized Dining Philosophers discrete time
Birth–Death Process
discrete time
Tandem Queuing Network
continuous time
Cyclic Server Polling
continuous time
Synchronous
Leader Election
• nodes in a ring elect a leader
– each node selects random number as id
– passes it around the ring (synchronously)
– if  unique id,
node with maximum unique id is leader
• [Itai & Rodeh, 1990]
Synchronous
Leader Election
1
4
2
2
5
3
1
5
Synchronous
Leader Election
1
2
5
4
1
1
5
4
4
4
1
2
1
5
2
5
1
2
3
5
1
3
2
3
5
2
3
5
1
Randomized
Dining Philosophers
• Dining Philosophers
– pick up chopsticks in random order
• Deadlocks resolved
– if there is no second chopstick,
give up eating
• [Pnueli & Zuck, 1986]
Birth–Death Process
• Models a waiting queue
• Standard model
in performance evaluation
• Limit queue size to get finite model
Tandem Queueing Network
• Two queues after each other
checkin
counter
security
check
two-phase
exponential
• [Hermanns, Meyer-Kayser & Siegle, 1999]
Cyclic Polling System
• server cycles over n stations
and serves each one in turn
– e.g. teacher walks through class,
each pupil may ask a question
• [Ibe & Trivedi, 1990]
Modelling
informal
description
PRISM
model
.tra format
model
ETMCC
MRMC
adapt
syntax
VESTA
model
YMER
model
PRISM
YMER
VESTA
Experiment 1
Qualitative Properties
• unbounded reachability with prob 1
• Cyclic Polling System:
busy1  P≥1(true U poll1)
If station 1 is busy,
the server will poll it eventually
log.scale model check time (sec.)
10000
1000
100
10
1
0,1
0,01
0,001
0,0001
log.scale VSZ memory (Kbytes)
0,00001
3
10000000
6
9
12
15
18
number of stations n
1000000
100000
mrmc
prism hybrid
prism sparse
etmcc
ymer
vesta
10000
1000
100
10
1
3
6
9
12
15
18
number of stations n
PRISM: MTBDD Size
• Multi-Terminal BDD =
data structure for transition matrix
• size heavily depends on model
• large MTBDD  slow
Model
# states # MTBDD nodes
Synchronous
500.000
1.000.000 .
leader election
Cyclic polling 7.000.000
< 3.000 .
log.scale model check time (sec.)
CPS versus SLE runtime
10000
1000
100
10
1
0,1
0,01
100000
10000
1000
100
10
1
0,1
0,01
0,001
0,001
0,0001
0,0001
4_4
,00001
3
6
9
12
15
18
number of stations n
7.077.888 states
2.745 MTBDD nodes
4_8
4_12
4_16
8_2
8_4
number of processes_random set size n_k
mrmc
prism hybrid
prism sparse
etmcc
ymer
vesta
458.847 states
1.131.806
MTBDD nodes
VESTA:
simulation problem
•
•
•
•
actual probability close to bound P≥p(...)
estimate is almost always in [p–,p+]
some irregularity stops the simulation
0.95  Prob(yes  actual Prob≥p)
 Prob(actual Prob≥p  yes)
P≥1(... U ...)
Timing Overview
1E+3
time [sec]
1E+0
1E-3
ETMCC
MRMC
PRISM hybrid
PRISM sparse
VESTA
1E-6
1E+0
1E+3
1E+6
number of states
1E+9
Analysis
ETMCC
slow, out of memory
MRMC
fastest tool for small models
PRISM
only symbolic  sparse=hybrid
depends on MTBDD size
VESTA
excessive number of samples, slow
YMER
not implemented
Result Overview: Timing
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
–
+
+/++ +/++
depends heavily
on MTBDD size
–/0
N/A
Result Overview: Memory
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
–
+
+/++ +/++
0/+
N/A
MTBDD size almost independent
varies heavily
from model size
Experiment 2
Bounded Reachability
• Tandem Queueing Network:
P<0.01(true U ≤2 full )
Is the probability
that the system gets full in 2 time units
small?
log.scale model check time (sec.)
10000
1000
100
10
1
0,1
0,01
0,001
0,0001
log.scale VSZ memory (Kbytes)
0,00001
10
50
10000000
100
255
511
1023
queue size n
1000000
100000
mrmc
prism hybrid
prism sparse
etmcc
ymer
vesta
10000
1000
100
10
1
10
50
100
255
511
1023
queue size n
Analysis
ETMCC
slow, out of memory
PRISM
sparse=faster, hybrid=smaller
MRMC
fast for small models
VESTA
YMER
ok
if you can afford statistical errors
best choice
if you can afford statistical errors
Result Overview: Timing
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
bounded U
–
–
+
+
+/++ +/++
0/+ +/++
–/0
+
N/A
++
Result Overview: Memory
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
bounded U
–
–
+
+
+/++ +/++
+/++ +
0/+
+
N/A
++
Experiment 3
Steady State Property
• Tandem Queuing Network
S>0.2( P
P>0.1
(X 2nd
2nd queue
queue full
full )) )
>0.1(X
In equilibrium,
the probability to satisfy P>0.1(X ... )
is > 0.2
log.scale model check time (sec.)
10000
1000
100
10
1
0,1
0,01
0,001
0,0001
log.scale VSZ memory (Kbytes)
0,00001
10
10000000
50
100
255
511
1023
queue size n
1000000
100000
mrmc
prism hybrid
prism sparse
etmcc
ymer
vesta
10000
1000
100
10
1
10
50
100
255
511
1023
queue size n
Simulating Steady State?
• simulation of bounded reachability
has clear stopping criterion
• simulation of unbounded reachability
 reachability with very large bound
• simulation of steady state?
 never stops
Analysis
ETMCC
slow, out of memory
PRISM
sparse=faster
hybrid=slightly smaller
MRMC
fastest
VESTA
not implemented
YMER
not implemented
Result Overview: Timing
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
bounded U
steady state
–
–
–
+
+
++
+/++ +/++
0/+ +/++
0/+
+
–/0
+
N/A
N/A
N/A
++
Result Overview: Memory
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
bounded U
steady state
–
–
–
+
+
+
+/++ +/++
+/++ +
+/++ +
0/+
+
N/A
N/A
N/A
++
Nested Formulas
• Birth–Death Process:
P≥0.8(P≥0.9(true U ≤100 n70) U n50)
The probability to reach n50
(while the probability to reach n70
in 100 steps never drops <0.9)
is ≥0.8
log.scale model check time (sec.)
10000
1000
100
10
1
0.1
0.01
0.001
0.0001
log.scale VSZ memory (Kbytes)
0.00001
100
10000000
1000
10000
100000
maximum population size m
1000000
100000
mrmc
prism hybrid
prism sparse
etmcc
ymer
vesta
10000
1000
100
10
1
100
1000
10000
100000
maximum population size m
Result Overview: Timing
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
bounded U
steady state
nested
–
–
–
–
+
+
++
++
+/++ +/++
0/+ +/++
0/+
+
0/+
+
–/0
+
N/A
N/A
N/A
––
N/A
++
did not terminate
Result Overview: Memory
ETMCC MRMC PRISM PRISM VESTA YMER
hybrid sparse
unb’nded U
bounded U
steady state
nested
–
–
–
–
+
+
+
+
+/++ +/++
+/++ +
+/++ +
+/++ +
0/+
+
N/A
N/A
N/A
N/A
N/A
++
Conclusions
ETMCC
MRMC
PRISM
hybrid
PRISM
sparse
VESTA
YMER
worst, only small models
fastest for small models
fast if MTBDD is small
fast
rather slow, no S , statistical errors
slim & very fast, only U ≤t,
few statistical errors