Download cmp

Anomaly-Based Bug Prediction, Isolation, and
Validation: An Automated Approach for
Software Debugging
Martin Dimitrov
Huiyang Zhou
School of Electrical Engineering and Computer Science
University of Central Florida
Background: Terminology
•
Defect: Erroneous piece of code (a bug)
•
Infection: The defect is triggered => the program state
differs from what the programmer intended.
•
Failure: An observable error (crash, hang, wrong
results) in program behavior.
Terminologies are based on the book “Why Programs Fail” by Andreas Zeller.
University of Central Florida
1
Background: From Defects to Failures
Variable and input values
Erroneous code
Observer sees failure
Sane state
Figure from the book
“Why Programs Fail” by A. Zeller
Infected state
Program execution
University of Central Florida
2
Motivation
• The typical process of software debugging involves:
1) Examine the point of program failure and reason
backwards about the possible causes.
2) Create a hypothesis of what could be the root cause.
3) Modify the program to verify the hypothesis.
4) If the failure is still there, the search resumes.
• Software debugging is tedious and time consuming !
• In this work we propose an approach to automate the
debugging effort and pinpoint the failure root cause.
University of Central Florida
3
Presentation Outline
•
•
•
•
•
•
•
•
Motivation
Proposed approach
Detecting anomalies (step 1)
Isolating relevant anomalies (step 2)
Validating anomalies (step 3)
Experimental methodology
Experimental results
Conclusions
University of Central Florida
4
Proposed Approach
Dynamic Instruction Stream
mov
cmp
jge
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
...
...
...
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
cmp
jge
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
jr
...
...
...
...
...
...
...
...
...
...
...
...
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
jge
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
...
...
jr
...
movl
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
cmp
jl
movl
inc
cmp
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
jl
movl
inc
cmp
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
Current instruction
time
University of Central Florida
5
Proposed Approach
Dynamic Instruction Stream
movl
inc
cmp
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
mov
...
time
•
...
...
...
A program failure is observed:
–
–
–
•
•
Failure point
Crash
Hang
Incorrect results, etc.
Start the automated
debugging process
The output of our approach is
a ranked list of instructions
(the possible root-cause of
failure)
Failure
University of Central Florida
5
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
mov
cmp
jge
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
...
...
...
...
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
cmp
jge
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
...
...
...
jr
...
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
jge
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
jr
...
...
...
...
...
...
...
...
...
...
...
anomaly
anomaly
...
movl
...
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
mov
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
...
anomaly
anomaly
jr
...
movl
...
anomaly
inc
Current instruction
...
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
mov
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
anomaly
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
lea
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
anomaly
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
movl
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
anomaly
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
inc
cmp
jl
movl
inc
cmp
...
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne
...
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
cmp
jl
movl
inc
cmp
...
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
jl
movl
inc
cmp
...
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
anomaly
anomaly
anomaly
anomaly
Current instruction
time
University of Central Florida
6
Proposed Approach
Step 1: Detect anomalies in program execution
Dynamic Instruction Stream
movl
inc
cmp
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
mov
...
time
anomaly
anomaly
• Each anomaly constitutes a
hypothesis for the root
cause of program failure.
anomaly
anomaly
Failure point
Failure
University of Central Florida
6
Proposed Approach
Step 2: Isolate the relevant anomalies
Dynamic Instruction Stream
movl
inc
cmp
...
...
...
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
mov
...
time
anomaly
anomaly
anomaly
• Create dynamic forward
slices from the anomalies
to the failure point.
• Discard anomalies which
do not lead to the failure
point.
anomaly
Failure point
Failure
University of Central Florida
7
Proposed Approach
Step 3: Validate the isolated anomalies
Dynamic Instruction Stream
movl
inc
cmp
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
mov
...
time
• Automatically “fix” the
anomaly and observe if
the program still fails.
...
...
...
anomaly
anomaly
Failure point
Failure
University of Central Florida
8
Proposed Approach
Step 3: Validate the isolated anomalies
Dynamic Instruction Stream
movl
inc
cmp
jr
...
movl
...
inc
...
cmp
...
jl
...
test
...
jne ...
mov
...
call
...
mov
...
time
• If the failure disappears
we have a high confidence
the root cause have been
pinpointed.
...
...
...
No failure
Success
University of Central Florida
8
Detecting Program Anomalies (Step 1)
• When infected by a software bug the program is likely to
misbehave:
–
–
–
–
Out-of-bounds addresses and values
Unusual control paths
Page faults
Redundant computations, etc.
• Anomaly detection: Infer program specifications from
passing runs and turn them into ‘soft’ assertions.
– Learn program invariants during passing runs
(e.g. variable “i” is always between 0 and 100)
– Flag violated invariants during the failing run
(e.g. Report anomaly if variable “i” is 101)
University of Central Florida
9
Detecting Program Anomalies
• We use several anomaly detectors to monitor a large
spectrum of program invariants and catch more bugs.
• DIDUCE [S. Hangal et al. , ICSE 2002]
– Instructions tent to produce values/addresses within a
certain range (e.g. 0 <= i <= 100). Detect violations of these
invariants.
• AccMon [P. Zhou et al. , MICRO-37 2004]
– Only a few static instructions access a given memory
location (load/store set locality). Signal an anomaly if
memory access does not belong to the load/store set.
• LoopCount
– Detect abnormal number of loop iterations.
University of Central Florida
10
Detecting Program Anomalies
void more_arrays () {
...
a_count += STORE_INCR;
Heap Memory
/* Copy the old arrays. */
for (indx = 1; indx < old_count; indx++)
arrays[indx] = old_ary[indx];
/* Initialize the new elements. */
for (; indx < v_count; indx++){/* defect */
arrays[indx] = NULL; /* infection: */
/* Free the old elements. */
if (old_count != 0){
free (old_ary); /* crash */
}
a_count 
v_count 
data
data
data
data
data
size
data
data
data
data
0x80bfe6c
0x80bfe70
0x80bfe74
0x80bfe78
0x80bfe7c
0x80bfe80
}
University of Central Florida
11
Detecting Program Anomalies
void more_arrays () {
...
a_count += STORE_INCR;
/* Copy the old arrays. */
for (indx = 1; indx < old_count; indx++)
arrays[indx] = old_ary[indx];
DIDUCE
24 anomalies
AccMon
68 anomalies
LoopCount
36 anomalies
LoopCount:
/* Initialize the new elements. */
for (; indx < v_count; indx++){/* defect */
arrays[indx] = NULL; /* infection: */
/* Free the old elements. */
if (old_count != 0){
free (old_ary); /* crash */
}
Loop iterates
more
AccMon:
times than
store
usual.
instruction
(false –is not
in store set of this
positive)
memory location.
(false – positive)
DIDUCE:
Address of store
instruction is out
of normal range.
(false – positive)
}
University of Central Florida
12
Detecting Program Anomalies: Architectural Support
• DIDUCE and AccMon capture invariants using limited
size caches structures, as proposed in previous work
• LoopCount utilizes the existing loop-branch predictor to
detect anomalies.
• Advantages and disadvantages of hardware support:
+ Performance efficiency
+ Portability
+ Efficient ways to change or invalidate dynamic instructions
- Limited hardware resource may become a concern
University of Central Florida
13
Isolating Relevant Anomalies (Step 2)
• Anomaly detectors alone are NOT effective for debugging:
– May signal too many anomalies / false positives
– Tradeoff between bug coverage and number of false positives
• Our solution:
– Allow aggressive anomaly detection for maximum bug
coverage
– Automatically isolate only the relevant anomalies by
constructing dynamic forwards slices from the anomaly to the
failure point
University of Central Florida
14
Isolating Relevant Anomalies: Architectural Support
• Add token(s) to each register and memory word.
• Detected anomalies set a token associated with the
destination memory word or register.
• Tokens propagate based data dependencies.
• When the program fails, examine the point of failure for
token.
University of Central Florida
15
Isolating Relevant Anomalies: Architectural Support
void more_arrays () {
...
Token
Memory
0x80bfdf8
/* Copy the old arrays. */
for (indx = 1; indx < old_count; indx++)
arrays[indx] = old_ary[indx];
0x80bfdfc
0x80bfe00
0x80bfe04
/* Initialize the new elements. */
for (; indx < v_count; indx++){/* defect */
arrays[indx] = NULL; /* infection */
}
/* Free the old elements. */
if (old_count != 0){
free (old_ary); /* crash */
Failure
}
mov %ebx,0xc(%edx)
University of Central Florida
...
0x80bfe08
0x80bfe7c
0x80bfe80
0x80bfe84
0x80bfe88
16
Isolating Relevant Anomalies: Architectural Support
• Problem: Many tokens for each memory location/ register
• Solution:
– We leverage tagged architectures for information flow
tracking.
– Use only one token (1 bit) (i.e., shared by all anomalies )
– We leverage delta debugging [A. Zeller, FSE 1999] to isolate
the relevant anomalies automatically.
Number of Initial Anomalies
Number of Isolated Anomalies
DIDUCE
24 anomalies
DIDUCE
3 anomalies
AccMon
68 anomalies
AccMon
2 anomalies
LoopCount
36 anomalies
LoopCount
4 anomalies
University of Central Florida
17
Delta-Debugging
University of Central Florida
18
Validating Isolated Anomalies (Step 3)
• Validate the remaining anomalies by applying a “fix” and
observing if the program failure disappears.
• Our “fix” is to nullify the anomalous instruction (turn it
into no-op)
• If the program succeeds, we have a high confidence we
have found the root cause (or at least broken the infection
chain)
University of Central Florida
19
Validating Isolated Anomalies
void more_arrays () {
...
Token Memory
/* Initialize the new elements. */
for (; indx < v_count; indx++){/* defect */
arrays[indx] = NULL; /* infection */ 0x0
0x0
/* Free the old elements. */
if (old_count != 0){
free (old_ary); /* crash */
}
}
Success
• The “size” information is not corrupted
and the program terminates successfully.
University of Central Florida
data
data
data
data
data
size
data
data
data
data
0x80bfe6c
0x80bfe70
0x80bfe74
0x80bfe78
0x80bfe7c
0x80bfe80
20
Validating Isolated Anomalies
• Four possible outcomes of our validation step:
> Hello World !
Success
Program produces
expected output.
> Hello Martin 
No crash
Program does not
crash, but produces
unexpected results.
> ...
(hang)
Unknown
Program fails in an
unexpected manner.
> Segmentation
Fault
Failure
Program fails the
same way as before.
• Rank isolated anomalies based on the outcome:
succeed (highest) , no crash, unknown, failure (lowest)
• In our running example the root-cause is ranked #1.
University of Central Florida
21
Experimental Methodology
• Implemented a working debugging tool using binary
instrumentation (PIN).
• Evaluated applications from BugBench [S. Lu et al., Bugs
2005] and gcc compiler.
Application
Lines of Defect Location
Code
Defect Description
bc-1.06
17,042
storage.c: 176
Incorrect bounds checking causes heap buffer overflow
gzip-1.2.4
8,163
gzip.c: 1009
Buffer overflow due to misuse of library call strcpy
ncompress-4.2.4
2 defects
polymorph-0.4.0
1,922
716
compress42.c: 886 Buffer overflow due to misuse of library call strcpy
and 1740
Incorrect bounds checking causes stack buffer underflow
polymorph.c: 200
Incorrect bounds checking causes stack buffer overflow
man-1.5h1
4675
man.c:998
Incorrect loop exit condition causes stack buffer overflow
gcc-2.95.2
338,000
combine.c: 4013
Incorrect call to apply_distributive_law causes a loop in
the RTL tree
University of Central Florida
22
Experimental Results
Application
Initial
Anomalies
Isolated
Anomalies
Validated
(Application
Succeeds)
Defect
rank
D
A
L
D
A
L
D
A
L
bc-1.06
24
68
36
3
2
4
1
1
1
1
gzip-1.2.4
21
40
19
1
1
1
1
1
1
2
ncompress-4.2.4
(strcpy defect)
6
7
6
2
2
1
0
1
1
1
ncompress-4.2.4
(stack underflow)
2
4
n/a
1
1
0
0
0
0
1
polymorph-0.4.0
21
10
20
3
1
0
0
0
0
3
man-1.5h1
15
114
46
2
2
0
1
1
0
1
gcc-2.95.2
768
1062
666
84
130
47
2
4
3
9
University of Central Florida
23
Case Study: GCC
University of Central Florida
24
GCC Defect
University of Central Florida
25
GCC Fix
University of Central Florida
26
Experimental Results
Compared to Failure-Inducing Chops
Application
Failure-Inducing Proposed
Chops
Approach
bc-1.06
167
1
gzip-1.2.4
6
2
ncompress-4.2.4
(strcpy defect)
ncompress-4.2.4
(stack underflow)
polymorph-0.4.0
4
1
11
1
8
3
man-1.5h1
n/a
1
gcc-2.95.2
1335
9
University of Central Florida
27
Limitations
• No failure, no bug detection
– Un-triggered bugs
– Bugs are triggered but output is correct
• Target at bugs in sequential programs
University of Central Florida
28
Conclusions
• We present a novel automated approach to pinpoint the
root causes of software failures:
– Detect anomalies during program execution.
– Isolate only the relevant anomalies.
– Validate isolated anomalies by nullifying execution results
• Our experimental results demonstrate that we accurately
pinpoint the defect even for large programs such as gcc.
University of Central Florida
29
Questions
• The tool is available for download at:
http://csl.cs.ucf.edu/debugging
University of Central Florida
30