Download Automated Test Generation

Korat: Automated Testing Based on
Java Predicates
Chandrasekhar Boyapati1, Sarfraz Khurshid2, and Darko Marinov3
1University
of Michigan Ann Arbor
2University of Texas at Austin
3University of Illinois at Urbana-Champaign
Examples of Structurally Complex Data
red-black tree
1
0
3
2
XML document
<library>
<book year=2001>
<title>T1</title>
<author>A1</author>
</book>
<book year=2002>
<title>T2</title>
<author>A2</author>
</book>
<book year=2003>
<title>T3</title>
<author>A3</author>
</book>
</library>
/library/book[@year<2003]/title
Testing Setup
inputs
outputs
1
2
0
3
0
3
2
test
generator
testing
oracle
code
3
2
0
3
• examples of code under test
– abstract data type
• input/output: data structure
– XML processing program
• input/output: XML document
0
pass
fail
Manual Test Generation
inputs
outputs
1
2
0
3
0
3
2
test
generator
• manual
testing
oracle
code
3
2
0
3
0
• drawbacks of manual generation
– labor-intensive and expensive
– vulnerable to systematic bias that eliminates
certain kinds of inputs
pass
fail
Automated Test Generation
inputs
outputs
1
2
0
3
0
3
2
test
generator
• automated
testing
oracle
code
3
2
0
3
0
• challenges of automated test generation
– describing test inputs
– (efficient) test generation
– checking output
pass
fail
Running Example
class BST {
Node root;
int size;
static class Node {
Node left, right;
int value;
}
void remove(int i) { … }
…
}
B0: 3
root
N 0: 2
left
N 1: 1
right
N 2: 3
Example Valid Inputs
• trees with exactly 3 nodes
B0: 3
B0: 3
root
root
N 0: 2
left
N1: 1
B0: 3
root
N 0: 1
right
N2: 3
B0: 3
root
N 0: 1
right
N 1: 2
N 1: 3
right
N 2: 3
left
N 2: 2
root
N 0: 3
right
B0: 3
N 0: 3
left
N 1: 1
left
N 1: 2
right
N 2: 2
left
N 2: 1
Example Invalid Inputs
• object graphs violating some validity property
B0: 3
root
B0: 2
B0: 3
root
N 0: 2
left
right
N 1: 1
N2: 3
root
N 0: 2
left
N1: 1
N 0: 3
right
N2: 3
left
N1: 1
right
N2: 2
Validity Properties for Example
• underlying graph is a tree
(no sharing between subtrees)
B0: 3
root
• correct number of nodes
reachable from root
N0: 2
left
N 1: 1
• node values ordered
for binary search
right
N 2: 3
Describing Validity Properties
• Korat uses a predicate written in standard
implementation language (Java, C#…)
– takes an input that can be valid or invalid
– returns a boolean indicating validity
• advantages
– familiar language
– existing development tools
– predicates often already present
• challenge: generate tests from predicates
Example Predicate
boolean repOk(BST t) {
return isTree(t) && hasCorrectSize(t) && isOrdered(t);
}
boolean isTree(BST t) {
if (t.root == null) return true;
// empty tree
Set visited = new HashSet(); visited.add(t.root);
List workList = new LinkedList(); workList.add(t.root);
while (!workList.isEmpty()) {
Node current = (Node)workList.removeFirst();
if (current.left != null) {
if (!visited.add(current.left)) return false;
// sharing
workList.add(current.left); }
if (current.right != null) {
if (!visited.add(current.right)) return false;
// sharing
workList.add(current.right); }
}
return true;
// no sharing
}
Input Space
• all possible object graphs with a BST root
(obeying type declarations)
B0: 2
B0: 3
root
root
N0: 2
N0: 2
left
N1: 1
B0: 3
right
right
left
N1: 1
N2: 3
B0: 3
N2: 3
root
B0: 1
N0: 3
N0: 1
N0: 2
root
right
left
root
N1: 1
left
N1: 1
left
N2: 2
N2: 3
right
B0: 1
B0: 1
B0: 0
N 0: 1
root
N0: 1
root
left
right
B0: 3
B0: 3
B0: 1
N0: 3
root
N0: 1
root
N0: 1
root
right
N1: 2
N1: 2
left
right
B0: 3
N2: 3
B0: 3
N2: 1
right
N0: 1
root
N1: 3
N2: 2
N0: 3
root
right
left
N1: 1
N2: 2
left
right
left
Bounded-Exhaustive Generation
• finitization bounds input space
– number of objects
– values of fields
• generate all valid inputs up to given bound
– eliminates systematic bias
– finds all errors detectable within bound
• avoid isomorphic inputs
– reduces the number of inputs
– preserves capability to find all errors
Example Finitization
• specifies number of objects for each class
– 1 object for BST: { B0 }
– 3 objects for Node: { N0, N1, N2 }
• specifies set of values for each field
–
–
–
–
sets consist of objects and literals
for root, left, right: { null, N0, N1, N2 }
for value: { 1, 2, 3 }
for size: { 3 }
Example Input Space
• 1 BST object, 3 Node objects: total 11 fields
B0
root size
N0
null
3
3
N0
left right value
N1
left right value
N2
left right value
N1
null null
null null
3
N1
2
1
null null
1
null null
1
null null
1
N0
N0
N0
2
N0
N0
2
N0
N0
2
N1
N1
N1
3
N1
N1
3
N1
N1
3
N2
N2
N2
N2
N2
N2
N2
4 * 1 * (4 * 4 * 3)3 > 218 inputs, only 5 valid
Example Input
• each input is a valuation of fields
B0
root size
N0
3
N0
left right value
N1
left right value
N2
left right value
N1
null null
null null
N1
2
B0: 3
root
N 0: 2
left
right
N 1: 1
N 2: 3
1
3
Generation Problem
• given
– predicate
– finitization
• generate
– all nonisomorphic valid inputs within finitization
• simple “solution”
– enumerate entire input space
– run predicate on each input
– generate input if predicate returns true
– infeasible for sparse input spaces (#valid << #total)
– must reason about behavior of predicate
Example Execution
boolean repOk(BST t) {
return isTree(t) && …;
B0: 3
}
root
boolean isTree(BST t) {
if (t.root == null) return true;
N0 : 2
Set visited = new HashSet(); visited.add(t.root);
right
left
List workList = new LinkedList(); workList.add(t.root);
while (!workList.isEmpty()) {
N 1: 1
N 2: 3
Node current = (Node)workList.removeFirst();
if (current.left != null) {
if (!visited.add(current.left)) return false;
workList.add(current.left); }
if (current.right != null) {
if (!visited.add(current.right)) return false;
workList.add(current.right); }
field accesses:
}
return true;
] N0.left
[ B0.root,
N] 0.right
[NB00[.left,
.root
[ B0.root,
]
}
]
Failed Execution
• failed after few accesses for a concrete input
B0
root size
N0
3
N0
left right value
N1
left right value
N2
left right value
N1
null null
null null
N1
2
1
3
• would fail for all inputs with partial valuation
B0
root size
N0
-
1 *
N0
left right value
N1
N1
-
N1
left right value
-
-
-
N2
left right value
-
-
-
3 * 4 * 4 * 3 * 4 * 4 * 3 > 212
Key Idea
• observe execution of predicate
• record field accesses
• prune large chunks of input space for each
failed execution
• use backtracking to efficiently enumerate valid
inputs
Search Step
• backtracking on [ B0.root, N0.left, N0.right ]
B0
root size
N0
3
N0
left right value
N1
left right value
N2
left right value
N1
null null
null null
N1
2
1
3
• produces next candidate input
B0
root size
N0
3
N0
left right value
N1
left right value
N2
left right value
N1
null null
null null
N2
2
1
3
Next Input
• execution returns true, generate this input
B0
root size
N0
3
N0
left right value
N1
left right value
N2
left right value
N1
null null
null null
N2
2
B0: 3
root
N 0: 2
left
N 1: 1
right
N 2: 3
1
3
Korat
• solver for executable predicates
repOk
finitization
Korat
all
nonisomorphic
valid inputs
• Korat systematically searches input space
–
–
–
–
–
encodes inputs as vectors of integers
creates candidate object-graph inputs
monitors execution of predicate
prunes search by backtracking on field accesses
avoids isomorphic inputs
Isomorphic Inputs
• equivalent for all code and all properties
• example for trees: permutation of nodes
B0: 3
B0: 3
root
root

N 0: 2
left
N 1: 1
right
N 2: 3
N 1: 2
left
N 2: 1
right
N 0: 3
• n! isomorphic trees => generation infeasible
• removing isomorphic inputs from test suites
– significantly reduces the number of tests
– does not reduce quality
Isomorphism Definition
• inputs: rooted and labeled (object) graphs
– edges have fields
– nodes have object identities
• isomorphism with respect to object identitites
• definition: inputs I and I’ consisting of objects
from a set O are isomorphic iff
: O->O. o,o’ reachable in I. ffields(o).
o.f = o’ in I => (o).f = (o’) in I’ and
o.f = p in I => (o).f = p in I’
Example Nonisomorphic Trees
• different edges or primitives, not only nodes
B0: 3

B0: 3
root
N 0: 2
right
root
left
N 1: 1
N 2: 3
N0: 2
left
N 1: 1
right
N 2: 3
B0: 3

root
N 0: 3
left
N 1: 1
right
N 2: 2
Nonisomorphic Generation
• simple “solution”
– generate all inputs
– filter out isomorphic inputs
• Korat does not require filtering
– generate only one (candidate) input from each
isomorphism class
– only the lexicographically smallest input
– search increments some field values for >1
Correctness
• Korat’s input generation
– sound
• no invalid input
– complete
• at least one valid input from each isomorphism class
– optimal
• at most one (valid) input from each isomorphism class
Example Specification
class BST {
Node root; int size;
static class Node {
Node left, right; int value;
}
//@ invariant repOk();
//@ precondition true;
//@ postcondition !contains(i) && …;
void remove(int i) { … }
boolean contains(int i) { … }
…
}
Testing Scenario
invariant
repOk()
postcondition
!contains(i) && …
precondition
true
and
Korat
B
0
:
root N
3
0
:
1 N right
1
:
2 Nright
2
:
3
,3
remove
B
0
:
rootN
3
0
:
1 Nright
1
:
2
an
d
pass
fail
finitization
Korat Structure Generation
linked data structures and array-based data structures
benchmark
BST
HeapArray
java.util.LinkedList
java.util.TreeMap
java.util.HashSet
IntentionalName
Korat Structure Generation
very large input spaces
benchmark
size input
space
BST
8
12
253
292
HeapArray
6
8
220
229
java.util.LinkedList
8
12
291
2150
java.util.TreeMap
7
9
292
2130
java.util.HashSet
7
11
2119
2215
IntentionalName
5
250
Korat Structure Generation
pruning based on filed accesses very effective
benchmark
size input candidate
space inputs
BST
8
12
253
292
54418
12284830
HeapArray
6
8
220
229
64533
5231385
java.util.LinkedList
8
12
291
2150
5455
5034894
java.util.TreeMap
7
9
292
2130
256763
50209400
java.util.HashSet
7
11
2119
2215
193200
39075006
IntentionalName
5
250
1330628
Korat Structure Generation
correct number of nonisomorphic structures (Sloane’s)
benchmark
size input candidate valid
space inputs
inputs
BST
8
12
253
292
54418
12284830
1430
208012
HeapArray
6
8
220
229
64533
13139
5231385 1005075
java.util.LinkedList
8
12
291
2150
5455
4140
5034894 4213597
java.util.TreeMap
7
9
292
2130
256763
50209400
35
122
java.util.HashSet
7
11
2119
2215
193200
39075006
2386
277387
IntentionalName
5
250
1330628
598358
Korat Structure Generation
800Mhz Pentium III Sun’s Java 2 SDK 1.3.1 JVM
benchmark
size input candidate valid time
space inputs
inputs [sec]
BST
8
12
253
292
HeapArray
6
8
java.util.LinkedList
54418
12284830
1430
208012
2
234
220
229
64533
13139
5231385 1005075
2
43
8
12
291
2150
5455
4140
5034894 4213597
2
690
java.util.TreeMap
7
9
292
2130
256763
50209400
35
122
9
2149
java.util.HashSet
7
11
2119
2215
193200
39075006
2386
277387
4
927
IntentionalName
5
250
1330628
598358
63
Korat Method Testing
several operations for each data structure
benchmark bound
inputs
gen. test
generated [sec] [sec]
BST
7
41300
9
1
HeapArray
7
1175620
7
18
LinkedList
7
58175
1
2
TreeMap
7
12754
3
1
HashSet
7
54844
3
2
IntentionalName
5
417878
87
135
SortedList
7
1047608
23
38
DisjSet
5
1246380
11
20
BinomialHeap
7
2577984
36
76
FibonacciHeap
5
941058
14
23
Conclusions
• Korat automates specification-based
testing
– uses method precondition to generate all
nonisomorphic test inputs
• prunes search space using field accesses
– invokes the method on each input and uses
method postcondition as a test oracle
• Korat prototype uses JML specifications
• Korat efficiently generates complex data
structures including some from JCF