Download Evolutionary Testing

Evolutionary Testing
Metaheuristic search techniques applied to test problems
Stella Levin
Advanced Software Tools Seminar
Tel-Aviv University
11.2004
1
Contents
1.
2.
3.
4.
5.
6.
7.
Introduction
Metaheuristic Search Techniques
White-Box Testing
Black-Box Testing
Object-Oriented Testing
Non-Functional Testing
Search Based Software Engineering
2
Introduction - Why testing?
“The biggest part of software cost is the
cost of bugs: the cost of detecting them,
the cost of correcting them, the cost of
designing tests and the cost of running
those tests” - Beizer
“In embedded systems errors could result
in high risk,endanger human life, big cost”
- Wegener
3
Successful EA applications
NASA evolvable
antenna
Equal to 12 years
working of
experienced designer
There is no guarantee
that a design would
be as good [8]
4
1. Metaheuristic Search
Problem characteristics
Large solution space
No precise algorithm, no “best” solution
Classification of “better” solution
“Due to non-linearity of software (if,
loops…) test problems are converted to
complex, discontinuous, non-linear
search spaces”- Baresel
5
Transform a problem to
optimization problem
Candidate solution representation –
individual
Fitness function for individual
Movement from one individual to
another
6
Hill Climbing - “local” search
1. Select a point in the search space
2. Investigate neighboring points
3. If there is a better neighbor solution
(with fitness function), jump to it
4. Repeat steps 2-3 until current position
has no better neighbors
Require definition of neighboring points
7
Hill Climbing - Problem
Is there a bigger hill?
8
Simulated Annealing
Analogy of the chemical process of
cooling of a material in a heat bath
If F(X2)<F(X1) then move to neighbor X2
Else move to X2 with probability
P=e^(-ΔF/T)
Initially T is high; T decreases more like
hill climbing
Require definition of neighbor and cooling
9
function
Simulated Annealing
10
Evolutionary Algorithms
• Genetic algorithms
Developed by J. Holland in the 70s
• Evolution strategies
Developed in Germany at about the
same time
• Analogy with Darwin’s evolution
theory, survival of the fittest
11
Initialize Random
Generation
Evaluate each
individual with
fitness function
YES
Results
Solution?
or exceed limit of
generations
NO
Next generation
Selection Crossover
Mutation
12
Evolutionary Algorithms
Selection: roulette wheel with fitness
Crossover: 01101
01000
11000
11101
Cross at random point
Mutation:
11101 10101
Random bit change
13
Genetic Programming
Program is an individual
Crossover and mutation of program’s
abstract syntax tree
Particular use – to find functions
which describe data
14
Genetic Programming - Crossover
+
2
*
*
7
3
6
4
+
2
*
7
*
3
6
4
15
Flow Graph
2. White-Box Testing
Statement coverage
Branch coverage
Specific path(statement)
selection
1
2
T
3
F
4
5
6
7
T
F
8
9
16
White-Box Testing
Input variables (x1, x2, … Xk)
Program Domain D1•D2•…Dk
Individuals decode input of the program
Goal: to find input data that satisfies
coverage criteria (statement / branch /
path)
17
Fitness Function = AL + D
AL: Approximation Level acc. McMinn [3]
Critical branch: branch missing the Target
AL =(Number of critical branches between
Target and diverging point) - 1
AL=1
AL=2
3
T
1
F
2
F
4
T
AL=0
F
5
T
Target
18
Fitness Function = AL + D
D: branch distance
If (x==y) then …
if x!=y then D=abs(x-y) else D=0
D normalize to [0,1]
Goal: min fitness (min D)
if (x<y) then …
If x>=y then D=x-y else D=0
If (flag) then …
If flag==false then D=K else D=0
19
Example: Triangle Classification
Input: int a,b,c from [0,15]
1. Sort a,b,c that a<=b<=c
2. If a+b<=c then NOT A TRIANGLE
3. If a==b or b==c then EQUILATERAL
4. If a==b and b==c then ISOSCELES
5. Else REGULAR
20
Optimization problem
Program domain: I•I•I
Individual string
5
7
9
0101 0111 1001
a
b
c
Goal: branch coverage

Sub-goals: NOT A TRIANGLE,
EQUILATERAL, ISOSCELES, REGULAR
21
Simulation by hand
Goal: EQUILATERAL
Generation: <9,13,5> REGULAR
<10,4,2> NOT A TRIANGLE
<3,11,7> NOT A TRIANGLE
1010 01 00 0010
1010 0111 0111
0011 10 11 0111
0011 1000 0010
<10,7,7> EQUILATERAL
22
GA vs. Random Testing
Schatz[11]
Comp(x,y) =
x nesting “if”
complexity;
y condition
complexity
23
Real Example
Schatz[11]
Autopilot
system
2046 LOC
75 conditions
Branch cov.
Performance
GA vs. random
24
Real Example
Performance of
gradient descent
algorithm
25
White Box Testing Summary
Variety of problem mapping and fitness
functions
Flag and state problems
26
3. Black Box Testing
- Tracey [4]
Specification with pre/post-conditions
Search for test input data that satisfy
pre-condition && ! post-condition
Good fitness for data that is near to satisfy
bool
If TRUE then 0 else K
a==b
If abs(a-b)==0 then 0 else abs(a-b)+K
a<b
If a-b<0 then 0 else (a-b)+K
a&&b
fit(a) + fit(b)
a||b
min(fit(a), fit(b))
27
int wrap_counter(int n)
{//pre (n>=0 && n<=10)
if (n>=10) i=0;
else i=n+1;
return i;
//post (n<10 -> i=n+1)
//post (n=10 -> i=0)}
Goal1: n>=0 && n<=10 && (n<10 && i!=n+1)
Goal2: n>=0 && n<=10 && (n=10 && i!=0)
28
Example
Insert Error: if (n>10) i=0
Goal2: n>=0 && n<=10 &&
(n=10 && i!=0)
n=2 i=3 : 0+0+(8+K)+0=8+K
n=7 i=8 : 0+0+(3+K)+0=3+K
n=10 i=11 : 0+0+0+0=0 FOUND!!!
29
Application
Applied to safety-critical nuclear
protection system






Use simulated annealing and GA for search
Use mutation testing to insert errors
About 2000 lines of executable code
733 different disjunctive goals
100% error detection
The code was simple
30
3. Black-Box Testing
Automated Parking System [10]
Individual: geometry data of parking
space and vehicle – 6 parameters
Fitness: minimum distance to collision area
Results: 880 scenarios-25 incorrect
31
Parking System - Results
32
Parking System - Results
33
4. Object-Oriented Testing
Tonella [5]: Unit Testing of Classes
1. Create object of class under test
2. Put the object to proper state
Repeat 1 and 2 for all required objects
3. Invoke method under test
4. Examine final state
34
Individual String and Fitness
$a=A():$b=B():$a.m(int,$b)@3
That means
A a = new A();
B b = new B();
a.m(3,b);
Goal: branch coverage
Fitness: proportion of exercised decision
nodes that lead to the target
35
Crossover
Crossover at random point after
constructor and before tested method
Repair the individual string
$a=A():$b=B(int):$a.m(int,$b)@1,5
$a=A(int,int):$b=B():$b.g():$a.m(int,$b)@0,3,4
$a=A():$b=B(int):$b.g():$a.m(int,$b) @1,4
$a=A(int,int):$b=B():$a.m(int,$b)@0,3,5
36
Mutations
Mutation of input value
Constructor change
Insertion of method call
Removal of method call
$a=A():$b=B(int):$a.m(int,$b)@1,5
$a=A():$b=B(int):$a.m(int,$b)@1,7
$a=A():$c=C():$b=B($c):$a.m(int,$b)@1,7
$a=A():$c=C():$b=B($c):$b.f():$a.m(int,$b)@1,7
37
Test cases
Branch
Coverage
Public
methods
Time
Execs (sec)
Class
LOC
String
Tokenizer
313
6
11/11
3
820
2
BitSet
HashMap1
HashMap2
LinkedList
Stack
TreeSet
1046
982
982
704
118
482
26
13
13
23
5
15
172/177
39/41
39/41
56/57
10/10
20/21
28
8
7
9
2
4
38700
4380
4310
25690
230
2480
4930-1.5h
1338
697
2034
1
60
38
5. Non-Functional Testing
Execution Time Testing
Real-time systems: WC/BC exe time
Fitness: execution time for specified input

Problem: no sufficient guidance for search
Wegener[6] on real systems:

GA is better than random and hand-made
Problem with low probability branches
No guarantee to find WC/BC execution
time
39
6. SBSE
Search Based Software Engineering
Module Clustering using simulated
annealing, genetic algorithms
Cost/Time Estimation using genetic
programming
Re-engineering using Program
Transformation

Ryan[7]: Automatic parallelization using
genetic programming
40
References
1.
2.
3.
4.
Goldberg “Genetic Algorithms”
McMinn “SBS Test Data Generation: A Survey”
McMinn “Hybridizing ET with the Chaining Approach”
Tracey “A Search Based Automated Test-Data Generation
Framework for Safety-Critical Systems”
5. Tonella “Evolutionary Testing of Classes”
6. Wegener,Pitschinetz,Sthamer “Automated testing of realtime tasks”
7. Ryan “Automatic re-engineering of software using genetic
programming”
8. Nasa “Intelligence report”
9. “Reformulating Software Engineering as a Search
Problem” Clarke,Jones…(11 authors)
10. Buehler,Wegener “Evolutionary Functional Testing of an
Automated Parking System”
11. McGraw, G., Michael, C., Schatz, M. “Generating Software
41
Test Data by Evolution”