Download A Safety-Critical Java Technology Compatibility Kit

A Safety-Critical Java
Technology Compatibility Kit
Hans Søndergaard
Stephan Korsholm
VIA University College, Horsens, Denmark
&
Anders P. Ravn
Aalborg University
JTRES 2014
October 2014
Test Suite – why?
• Java Community Process (JCP)
The formal process for developing or revising Java
technology Specifications.
• Final Release
The final stage in a JSR when the Specification, RI,
and TCK have been completed.
• Technology Compatibility Kit (TCK)
The suite of tests, tools, and documentation that allows
an organization to determine if its implementation is
compliant with the Specification.
2
Ingredients in a TCK
• Tests
Test cases and test datasets
• Tools for test execution
Execute the tests and generate results
• Test report documents
Classification of results
3
Test Case Development
Checks conformance with a specification
 Must be implementation independent:
- data representations must be hidden
 Black-box testing using public entities only
Must check specification of classes
 Invariants and Pre- and Post-conditions
? Intuition or formalization in development ?
! JML (Java Modeling Language)
4
Background
• Anders P. Ravn and Hans Søndergaard. A Test Suite for SafetyCritical Java using JML. 2013.
• Yoonsik Cheon and Gary T. Leavens. A Simple and Practical
Approach to Unit Testing: The JML and JUnit Way. 2003.
• A. Sarcar and Y. Cheon. A new Eclipse-based JML compiler
built using AST merging. 2010.
• JML4c
http://www.cs.utep.edu/cheon/download/jml4c/index.php
• L. Zhao, D. Tang, and J. Vitek. A Technology Compatibility Kit
for Safety Critical Java. 2003.
5
Specification of Invariants
Class:
public abstract class HighResolutionTime
Specification:
” A time object in normalized form represents negative time if both
components are nonzero and negative, or one is nonzero and negative and
the other is zero.”
Invariant:
public invariant
(this.getMilliSeconds() >= 0 &&
(0 <= this.getNanoSeconds() && this.getNanoSeconds() < 1000000))
||
(this.getMilliSeconds() <= 0 &&
(-1000000 < this.getNanoSeconds() && this.getNanoSeconds() <= 0));
6
Pre/Post-conditions Normal behaviour
Class:
public abstract class HighResolutionTime
Specification:
public void set(HighResolutionTime time)
“Change the value represented by this to that of the given time.
…
Throws ClassCastException if the time parameter is not of the same class as this.
Throws IllegalArgumentException if the time parameter is null. ...”
public normal_behaviour
requires time != null && this.getClass() == time.getClass();
ensures this.getMilliseconds() == time.getMilliseconds();
ensures this.getNanoseconds() == time.getNanoseconds();
ensures this.getClock() == time.getClock();
7
Exceptions
Class:
public abstract class HighResolutionTime
Specification:
public void set(HighResolutionTime time)
“Change the value represented by this to that of the given time.
…
Throws ClassCastException if the time parameter is not of the same class as this.
Throws IllegalArgumentException if the time parameter is null. ...”
also
public exceptional_behaviour
requires time == null;
signals (IllegalArgumentException) true;
public exceptional_behaviour
requires time != null && this.getClass() != time.getClass();
signals (ClassCastException) true;
8
JML annotations compiled and executed
• Compilation of JML annotations
– Uses a JML compiler
– Compiled to Java bytecode
– Translated to runtime assertion checks
• Execution of the extended Java bytecode
– Constructor:
Runtime check of Precondition
Constructor execution
Runtime check of Postcondition and Invariant
– Method:
Runtime check of Invariant and Precondition
Method execution
Runtime check of Postcondition and Invariant
• Violation of runtime checks
–
Throws a JMLAssertionError
Test Case
Using class TestCase from JUnit
public class TestAbsoluteTime extends TestCase {
...
public void test(int i) {
AbsoluteTime abs;
switch (i) {
...
// AbsoluteTime(long millis, int nanos)
case 32: new AbsoluteTime(0,0); break;
case 33: new AbsoluteTime(0,1000001); break;
...
// wrap around value
case 37: abs = new AbsoluteTime(Long.MAX_VALUE,1000001);
assert abs.getNanoseconds() == -999999; break;
...
Test Cases - exceptions
// AbsoluteTime(AbsoluteTime time)
case 39: abs = new AbsoluteTime();
new AbsoluteTime(abs); break;
case 40: abs = null;
try{ new AbsoluteTime(abs);
assert false;
} catch (IllegalArgumentException e){};
break;
...
case 91: ...
default: break;
}
public static final int testCount = 91;
Beyond Unit Testing
Class:
public abstract class Clock
Specification:
public abstract RelativeTime getResolution();
“Gets the resolution of the clock defined as the nominal interval between ticks. “
public behaviour
requires true;
ensures \result != null;
ensures (\result.getMilliSeconds() > 0 ||
\result.getMilliSeconds() == 0 && \result.getNanoSeconds() > 0);
Is unit testing sufficient for a Clock?
12
Specification of clock test method
case 14: clockTest(clk); break;
Specification:
behaviour
requires true;
ensures !failure;
ensures
(\forall int i; 0 < i && i < SIZE;
sample[i-1].compareTo(sample[i]) < 0); // time is moving forward
ensures
(\forall int i; 0 < i && i < SIZE;
(sample[i].subtract(sample[i-1])).compareTo(c.getResolution()) >= 0 );
// distance between two samples >= resolution
Use JML to specify test methods!
13
Implementation of the clock test method
boolean failure; AbsoluteTime[] sample; RelativeTime resolution;
behaviour ... // JML specification here
void clockTest(Clock c) {
resolution = c.getResolution();
sample = new AbsoluteTime[SIZE];
failure = false;
sample[0] = c.getTime(sample[0]);
for (int i = 1; i < SIZE; i++){
int j = 0;
do {
sample[i] = c.getTime(sample[i]); j++;
} while (sample[i].subtract(sample[i-1]).
compareTo (resolution) < 0 && j < MAXLOOPS );
if (j == MAXLOOPS) { failure = true; break;}
}
14
Tools for test execution
• Requirements
– Executable on different VMs,
•
e.g. VMs for resource constrained platforms
•
HVM has been extended with reflection
• JML compilers
– jmlc works with Java 1.4 source files
– jml4c works with Java 1.5, including generics
• built on the Eclipse Java compiler
• JUnit
– a subset of junit-3.8.2 is used
• package framework
– class TestResult has been extended
• add JML error to a list of errors
15
TestSuite assembly and execution
public class AllTests
{
public static final TestResult result = new TestResult();
public static void main (String[] args) {
TestSuite suite = new TestSuite();
suite.addTest(test_AbsoluteTime);
suite.addTest(test_RelativeTime);
...
suite.run(result);
...
// TestCase objects defined here (next slide)
}
16
TestCase object
public class AllTests {
public static final TestResult result = new TestResult();
...
public static TestCase test_AbsoluteTime =
new TestAbsoluteTime(”AbsoluteTime”) {
public void runTest () {
try {
for (int i = 1; i <= TestAbsoluteTime.testCount; i++)
test(i);
}
catch (JMLAssertionError e) {result.addJMLError(this, e);}
catch (Throwable e) { result.addError(this, e); }
}
};
}
17
Test report
• No errors
Test cases:
Test errors:
JML errors:
2
0
0
• If e.g. an JML error
Test number is 67,
By method AbsoluteTime.add
Regarding specifications at
File "./src/javax/realtime/AbsoluteTime.java",
line 216, character 15
With values
nanos: 1000001
millis: 0
Where are we?
javax.realtime
47 classes and interfaces
24 finished
15 are Empty
Where are we?
javax.safetycritical
35 classes and interfaces
12 (nearly) finished
3 are Empty
Conclusion
• Tests
JML is an efficient vehicle for generating test conditions
and may even improve the informal specifications.
• Concrete Test cases
Guided by specifications
and also additional cases based on “intuition”.
• Tools for test execution
The tools involved are able to run on a resource
constrained VM (HVM).
• Test report documents
A simple test report.
21
Conformance tests of other SCJ implementations
• Extract the JML specifications from our SCJ
implementation (simple)
• Merge this extract with another SCJ implementation
(more complicated)
• Use the test cases (no change)
• Compile classes with JML annotations into Java
bytecode, using jml4c (? depends of the VM)
• Run the test suites, using (part of) JUnit (simple).
22