Download presentation

Automated Test Reuse via Code Transplantation
Tianyi Zhang, Miryung Kim
University of California, Los Angeles
1
Problem Statement
 Code clones are common in modern software systems
 Developers often find it difficult to examine the runtime behavior of clones
 This problem is exacerbated by a lack of tests. 46% of clone pairs are only
partially covered by existing test suites.
 We present Grafter to reuse tests between clones and examine behavior differences
2
A programmer applies similar edits to update the use of StringTokenizer to StringUtils.split
in the Copy and Delete classes.
public class Copy extends Task{
private IncludePatternSet includes;
public void setIncludes(String patterns){
…
- StringTokenizer tok = new StringTokenizer(patterns, “,”);
- while(tok.hasMoreTokens()){
includes.addPattern(tok.next);
- }
+ String[] tokens = StringUtils.split(patterns, “,”);
+ for(String tok : tokens){
+
includes.addPattern(tok);
+ }
…
}
}
public class Delete extends Task{
private ExcludePatternSet excludes;
public void setExcludes(String patterns){
…
- StringTokenizer tok = new StringTokenizer(patterns, “,”);
- while(tok.hasMoreTokens()){
excludes.addPattern(tok.next);
- }
+ String[] tokens = StringUtils.split(patterns, “.”);
+ for(String tok : tokens){
+
excludes.addPattern(tok);
+ }
…
}
}
Copy.java
* The example is adapted from Apache Ant 1.9.6 for presentation purposes .
Delete.java
3
Limitation of Existing Techniques
 Existing clone inconsistency detection techniques do not detect behavioral
differences between clones [Jiang et al., CBCD, SPA]
 Existing differential testing or random testing techniques are not geared
towards examining behavioral differences of intra method clones without
well-defined input argument and output types [Geno, Diffut, Randoop]
4
Grafter: Automated Test Reuse and Differential Testing
Copy.java
testCopy
Delete.java
public class Copy extends Task{
private IncludePatternSet includes;
…
String[] tokens = StringUtils.split(patterns, “,”);
for(String tok : tokens){
includes.addPattern(tok);
}
…
Success
Failure
public class Delete extends Task{
private ExcludePatternSet excludes;
…
String[]tokens
tokens==StringUtils.split(patterns,
StringUtils.split(patterns,“.”);
“.”);
String[]
for(String
for(String tok
tok :: tokens){
tokens){
excludes.addPattern(tok);
excludes.addPattern(tok);
}}
…
?
5
Grafter: Automated Test Reuse and Differential Testing
Copy.java
testCopy
Delete.java
public class Copy extends Task{
private IncludePatternSet includes;
…
String[] tokens = StringUtils.split(patterns, “,”);
for(String tok : tokens){
includes.addPattern(tok);
}
…
Success
public class Delete extends Task{
private ExcludePatternSet excludes;
…
String[]tokens
tokens==StringUtils.split(patterns,
StringUtils.split(patterns,“.”);
“.”);
String[]
for(String
for(String tok
tok :: tokens){
tokens){
excludes.addPattern(tok);
excludes.addPattern(tok);
}}
…
Failure
6
 Clones may have variations in variable names, types, method call targets, etc.
(Type II clones)
If (cert instanceof X509Certificate) {
XMLX509IssuerSerial certSerial =
new XMLX509IssuerSerial(
x509Serial.getDocument(),
(X509Certificate)cert);
…
If (cert instanceof X509Certificate) {
XMLX509SubjectName certSN =
new XMLX509SubjectName(
x509SubjectName.getDocument(),
(X509Certificate)cert);
…
Apache XML Security
 Clones may include additional code insertions and deletions. (Type III clones)
if (f.isDirectory()) {
deleteDir(f);
} else if (deleteFiles && ! getFileUtils.tryHardToDelete) {
throw new BuildException (“Unable to delete file” …);
} else {
throw new BuildException (“Unexpected Error. File
should not exist!”);
}
if (f.isDirectory()) {
deleteDir(f);
} else {
log(“Deleting ” + f.getPath());
if (!delete(f)) {
handle(“Unable to delete file” + …);
}
}
Apache Ant
7
Grafter Approach Overview
Clone Pair
Test Suite
A
Variation
Identification
B
Code
Transplantation
Data
Propagation
Differential
Testing
Test
Clone A
Clone B
State
Clone A
Clone B
T1
pass
fail
S1
“string”
“string”
T2
pass
pass
S2
true
false
Test-level Comparison
State-level Comparison
8
Grafter Approach Overview
Variation
Identification
• Name Variation
• Type Variation
• Method Call
Variation
Code
Transplantation
•
•
•
•
Declare Variables
Transform Types
Declare Methods
Recursive Calls
Data
Propagation
• Populate intermediate
input data to clone
• Transfer intermediate
output back to test
Differential
Testing
• Test Comparison
• State Comparison
9
Grafter Approach Overview
Variation
Identification
• Name Variation
• Type Variation
• Method Call
Variation
Code
Transplantation
•
•
•
•
Declare Variables
Transform Types
Declare Methods
Recursive Calls
Data
Propagation
• Populate intermediate
input data to clone
• Transfer intermediate
output back to test
Differential
Testing
• Test Comparison
• State Comparison
10
Grafter Approach Overview
Variation
Identification
• Name Variation
• Type Variation
• Method Call
Variation
Code
Transplantation
•
•
•
•
Declare Variables
Transform Types
Declare Methods
Recursive Calls
Data
Propagation
• Populate intermediate
input data to clone
• Transfer intermediate
output back to test
Differential
Testing
• Test Comparison
• State Comparison
11
Grafter Approach Overview
Variation
Identification
• Name Variation
• Type Variation
• Method Call
Variation
Code
Transplantation
•
•
•
•
Declare Variables
Transform Types
Declare Methods
Recursive Calls
Data
Propagation
• Populate intermediate
input data to clone
• Transfer intermediate
output back to test
Differential
Testing
• Test Comparison
• State Comparison
12
Step 1: Variation Identification
Variation
Identification
Code
Transplantation
public class Copy extends Task{
private IncludePatternSet includes;
public void setIncludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “,”);
for(String tok : tokens){
includes.addPattern(tok);
}
}
…
Data
Propagation
Differential
Testing
public class Delete extends Task{
private ExcludePatternSet excludes;
public void setExcludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “.”);
for(String tok : tokens){
excludes.addPattern(tok);
}
}
…
13
Step 1: Variation Identification
Variation
Identification
Code
Transplantation
public class Copy extends Task{
private IncludePatternSet includes;
public void setIncludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “,”);
for(String tok : tokens){
includes.addPattern(tok);
}
}
…
Data
Propagation
Differential
Testing
public class Delete extends Task{
private ExcludePatternSet excludes;
public void setExcludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “.”);
for(String tok : tokens){
excludes.addPattern(tok);
}
}
…
14
Step 2: Code Transplantation
Variation
Identification
Code
Transplantation
public class Copy extends Task{
private IncludePatternSet includes;
public void setIncludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “,”);
for(String tok : tokens){
includes.addPattern(tok);
}
}
…
Data
Propagation
Differential
Testing
public class Delete extends Task{
private ExcludePatternSet excludes;
public void setExcludes(String patterns){
…
String[]
tokens
= StringUtils.split(patterns,
String[]
tokens
= StringUtils.split(patterns,
“.”);“.”);
for(String
for(String tok
tok :: tokens){
tokens){
excludes.addPattern(tok);
excludes.addPattern(tok);
}}
}
…
15
Step 2: Code Transplantation
Variation
Identification
Code
Transplantation
public class Copy extends Task{
private IncludePatternSet includes;
+ private ExcludePatternSet excludes;
public void setIncludes(String patterns){
…
+ excludes = new ExcludePatternSet();
String[] tokens = StringUtils.split(patterns, “.”);
for(String tok : tokens){
excludes.addPattern(tok);
}
}
…
Data
Propagation
Differential
Testing
public class Delete extends Task{
private ExcludePatternSet excludes;
public void setExcludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “.”);
for(String tok : tokens){
excludes.addPattern(tok);
}
}
…
16
Step 3: Data Propagation
Variation
Identification
Code
Transplantation
public class Copy extends Task{
private IncludePatternSet includes;
+ private ExcludePatternSet excludes;
public void setIncludes(String patterns){
…
+ excludes = new ExcludePatternSet();
+ excludes.set = includes.set;
String[] tokens = StringUtils.split(patterns, “.”);
for(String tok : tokens){
excludes.addPattern(tok);
}
+ includes.set = excludes.set;
}
…
Data
Propagation
Differential
Testing
public class Delete extends Task{
private ExcludePatternSet excludes;
public void setExcludes(String patterns){
…
String[] tokens = StringUtils.split(patterns, “.”);
for(String tok : tokens){
excludes.addPattern(tok);
}
}
…
17
Step 4: Differential Testing
Variation
Identification
Code
Transplantation
State
patterns
Test
Copy.java
Delete.java
testCopy
pass
fail
tokens
in(ex)cludes
Test-level Comparison
Data
Propagation
Differential
Testing
Copy.java
Delete.java
“src/*.java, test/*.java”
“src/*.java, test/*.java”
[“src/*.java”, “test/*.java”]
[“src/*”, “java, test/*”, “java”]
<IncludePatternSet>
<set>
[“src/*.java”, “test/*.java”]
</set>
</IncludePatternSet>
<ExcludePatternSet>
<set>
[“src/*”, “java, test/*”, “java”]
</set>
</ExcludePatternSet>
State-level Comparison
18
 Our work is accepted to ICSE 2017 and our tool & dataset are now
publicly available.
Tool & Dataset: http://web.cs.ucla.edu/~tianyi.zhang/grafter.html
19
 Behavioral differences are represented in tables and highlighted for
ease of investigation.
20
Evaluation Dataset
 Our dataset contains 52 pairs of non-identical clones from 3 open source
projects.
 In particular, our dataset includes 38 Type II clones and 14 Type III
clones, based on a well-known clone taxonomy [Roy et al.].
Subject
Version
Description
Apache Ant
1.9.6
A software build framework
Java APNS
1.0.0
A Java client for Apple Push Notification
service (APNs)
XML Security
2.0.5
A XML signature and encryption library
LOC
Test#
Branch
Stmt
267,048
1,864
45%
50%
8,362
103
59%
67%
121,594
396
59%
65%
Clone Pair
18
7
27
21
Research Questions
 RQ1. What is Grafter’s transplantation capability?
 RQ2. How does Grafter compare with a static approach by Jiang et al. in
its ability to detect differences in clones?
 RQ3. How sensitive is Grafter in detecting behavioral differences caused
by mutants?
22
 Grafter successfully grafts 49 of 52
Clone Pair#
RQ1. Transplantation Success and Test Reuse Capability
pairs of clones
 Grafter inserts 6 lines of stub code on
average to ensure type safety
18
16
14
12
10
8
6
4
2
0
17
14
Failure
7
4
0
4
0
pairs
0
1
0
Type Type
II
III
Type Type
II
III
Type Type
II
III
Ant
APNS
XMLSec
90%
60%
40%
30%
82%
77%
75%
48%
Before
After
65%
60%
70%
50%
coverage for partially tested clone
3
0
80%
 Grafter roughly doubles the test
Success
43%
28%
20%
10%
0%
Branch
Stmt
Partially
Branch
Fully
Stmt
23
Jiang et al.’s Clone Inconsistency Detection
 Jiang et al. detect three types of syntactic inconsistencies between clones.
if (cmd_type == READ_M2){
msgbuf[xa_count*3] = 0;
msg(DBG_XA1, “xa head: %s\n”, msgbuf);
}
for (i = 0; i < response_count; i++) {
msgbuf[i*3] = 0;
msg(DBG_SQ1, “cc_ReadSubQ: %s\n”, msgbuf);
}
Example 1. Control Construct Inconsistency
if (length >=9 && strncmp (buffer, “EESOXSCSI”,
9) == 0) {
buffer += 9;
length += 9;
…}
if (length >=11 && strcmp (buffer, “CUMANNASCSI2”)
== 0) {
buffer += 9;
length += 9;
…}
Example 2. Control Predicate Inconsistency
if (l_stride != NULL) {
mps_cdiv_q (X1, X1, l_stride->value);
Example 3. Renaming Mistake
if (l_stride != NULL) {
mps_cdiv_q (X1, X1, r_stride->value);
24
RQ2. Behavioral Difference Detection
 Grafter exposes behavioral differences in 84% clone pairs while Jiang et
al. detect syntactic inconsistency in 33% clone pairs only.
No Cloning Bug
No Differences
16%
Inconsistent Conditional Predicate
State Differences Only
Test and State Differences
Renaming Mistake
31%
41%
43%
Grafter
2%
67%
Jiang et al.
25
RQ3. Robustness
 We systematically injected 361 mutants on 30 pairs of clones with no
existing test behavioral differences
 We compare its behavioral differencing capability with a static approach
by Jiang et al.
Operator
Description
Example
AOR
Arithmetic operator replacement
a b  a b
LOR
Logical operator replacement
ab  a |b
COR
Conditional operator replacement
a  b  a & &b
ROR
Relational operator replacement
SOR
Shift operator replacement
ORU
Operator replacement unary
STD
Statement deletion operator: delete (omit) a single statement
LVR
Literal value replacement: replace by a positive value, a negative value or zero
a  b  a  b
a ba b
a  a
foo(a, b)  / / foo( a, b)
0 1
26
RQ3. Robustness
public void setType(String type) {
if ( type
type==
!= null && type.length() == 0){
this.type = null;
} else {
URI tmpType = null;
try {
tmpType = new URI(type);
} catch (URISyntaxException ex) {
…
}
this.type = tmpType.toString();
}
}
public void setEncoding(String encoding) {
if (encoding == null && encoding.length() == 0){
this.encoding = null;
} else {
URI tmpEncoding = null;
try {
tmpEncoding = new URI(encoding);
} catch (URISyntaxException ex) {
…
}
this.encoding = tmpEncoding.toString();
}
}
Mutation Example from Apache XML Security
27
RQ3. Robustness
 Grafter detects 36% more mutants using the test-level comparison and
almost 2X more mutants using the state-level comparison
 Grafter is less biased to mutant types than Jiang et al.
400
100%
350
90%
300
250
80%
130
194
234
60%
200
50%
150
100
70%
40%
231
167
30%
127
50
20%
10%
0
Test-level
Comparison
State-level
Comparison
Detected Mutants
Jiang et al.
Undetected Mutants
0%
ROR
LVR
Test-level Comparison
COR
STD
ORU
State-level Comparison
AOR
Jiang et al.
28
Q&A
29