Download EEC 521: Software Engineering Software Faults Types of Faults

Software Faults
• Quite inevitable
• Many reasons
EEC 521: Software
Engineering
–
–
–
–
–
–
Verification and Validation
Software Testing
11/10/09
EEC 521: Software Engineering
Types of Faults
• Algorithmic
• Syntax
• Computation/
Precision
• Documentation
• Stress/Overload
• Capacity/Boundary
1
Logic is wrong
Code reviews
Software systems with large number of states
Complex formulas, activities, algorithms
Customer is often unclear of needs
Size of software
Number of people involved
…
11/10/09
Types of Faults
Wrong syntax; typos
Compiler
Not enough accuracy
Misleading
documentation
•
•
•
•
•
EEC 521: Software Engineering
2
Synchronization issues
Very hard to
replicate
System performs
below expectations
Timing/Coordination
Throughput/Performance
System restarted from
abnormal state
Recovery
Hardware and related software
Standards
Maximum load
violated
Compatibility issues
Makes for difficult
maintenance
Boundary cases are
usually special cases
11/10/09
EEC 521: Software Engineering
3
11/10/09
EEC 521: Software Engineering
4
Inspections
Manual Inspections
• Group reviews
–
–
–
–
Owner presents artifact for review
Facilitator convenes meeting
Reviewers study artifact prior to meeting
Group reviews artifact in meeting to provide
feedback to owner
– Multiple iterations, if necessary
Process of analyzing and reviewing
artifacts - code, documentation, user
interfaces, etc. with the intent of
reducing possibilities of faults.
11/10/09
EEC 521: Software Engineering
5
Automated Inspections
6
Testing is the process of exercising a
program with the specific intent of
finding errors prior to delivery to the
end user.
– Buffer overruns
– Data races
– Dead code (control flow)
EEC 521: Software Engineering
EEC 521: Software Engineering
Software Testing
• Tools for static analysis
• Can identify several common errors
11/10/09
11/10/09
7
11/10/09
EEC 521: Software Engineering
8
Who Tests the Software?
How to Test?
• No point in testing without plan
– Too much wasted time and effort
Developer
Independent Tester
Understands the system
but, will test "gently"
and, is driven by "delivery"
Must learn about the system,
but, will attempt to break it
and, is driven by quality
11/10/09
• Need a strategy
– Dev team needs to work with Test team
– “Egoless Programming”
EEC 521: Software Engineering
9
Component
code
Unit Test
Component
code
When to Test?
Unit Test
Design Specs
Tested
Other
software
requirements
User
environment
nt
compone
Tested co
mponent
Function
Test
Integrate
d modules
Performance
Test
Functioning
system
Verified,
validated
software
Unit Test
11/10/09
Customer
requirements
specification
EEC 521: Software Engineering
10
Verification and
Validation
Verification:
Are we building the product right?
Integration
Test
Component
code
System
functional
requirements
11/10/09
Acceptance
Test
Installation
Test
Accepted
system
Validation:
Are we building the right product?
System
in use!
EEC 521: Software Engineering
11
11/10/09
EEC 521: Software Engineering
12
Validation and
Verification
SW
Specs
Actual
Requirements
Validation
Includes usability
testing, user
feedback
11/10/09
V&V Activities
System
Verification
validation
Includes testing,
inspections, static
analysis
EEC 521: Software Engineering
verification
13
11/10/09
EEC 521: Software Engineering
14
ever
You can’t always get what
you want
Decidability
• A problem is decidable, if
Property
Program
Decision
Procedure
– it has a solution,
– and the solution can be found in a finite
amount of time
Pass/Fail
• Otherwise the problem is undecidable
Correctness properties are undecidable
the halting problem can be embedded in
almost every property of interest
11/10/09
EEC 521: Software Engineering
15
11/10/09
EEC 521: Software Engineering
16
HP is Undecidable
[Turing, 1936]
Turing’s Proposition
• “There must be problems that are
undecidable”
• Example: The Halting Problem (HP)
• Suppose that there is a solution, called H
Program P
Input I
– “Given a program and an input to the
program, determine if the program will
eventually stop when it is given that input.”
EEC 521: Software Engineering
“Halts”
or
“Loops”
• Program P is just a string of characters,
and so P can be provided as input to H
• Simply running the program with the
given input is not a valid approach. Why?
11/10/09
H
Program P
H
Program P
17
11/10/09
“Halts”
or
“Loops”
EEC 521: Software Engineering
18
HP is undecidable
HP is Undecidable
• Construct a new program K:
• Since K is a program, we can provide K as
input to itself
– if H outputs “loops”, then halt
– Else loop forever
– If H says K halts, then K itself would loop
– If H says K loops, then K will halt
• K does the opposite of H
Program P
H
Loop?
K
11/10/09
• In either case, H gives the wrong answer
for K
• Thus, H cannot work for all cases!
EEC 521: Software Engineering
loop
19
11/10/09
EEC 521: Software Engineering
20
Getting what you need ...
• optimistic inaccuracy: we may
accept some programs that do
not possess the property (i.e.,
it may not detect all
violations).
– testing
• pessimistic inaccuracy: it is not
guaranteed to accept a
program even if the program
does possess the property
being analyzed
– automated program analysis
techniques
• simplified properties: reduce
the degree of freedom for
simplifying the property to
check
11/10/09
EEC 521: Software Engineering
21
Example: Unmatched
Semaphore Operations
original problem
if ( .... ) {
...
lock(S);
}
...
if ( ... ) {
...
unlock(S);
}
11/10/09
simplified property
Static
checking
for match is
necessarily
inaccurate
...
Java prescribes a
more restrictive,
but statically
checkable
construct.
synchronized(S) {
...
...
}
EEC 521: Software Engineering
Terminology
Views of Test Objects
• Safe: A safe analysis has no optimistic
inaccuracy, i.e., it accepts only correct
programs.
• Sound: An analysis of a program P with respect
to a formula F is sound if the analysis returns
true only when the program does satisfy the
formula.
• Complete: An analysis of a program P with
respect to a formula F is complete if the
analysis always returns true when the program
actually does satisfy the formula.
• Black box or Closed box
11/10/09
11/10/09
EEC 521: Software Engineering
23
22
– Testing based only on spec
• White box or Open box
– Testing based on actual source code
• Grey box
– Partial knowledge of source code
EEC 521: Software Engineering
24