Download Architectural-Level Risk Analysis for UML Dynamic Specifications

West
Virginia
University
Architectural-Level Risk Analysis for
UML Dynamic Specifications
Dr. Sherif M. Yacoub
[email protected]
Hewlett-Packard Laboratories
Palo Alto, CA
Alaa Ibrahim, and Hany H. Ammar
{ibrahim,ammar}@csee.wvu.edu
Department of Computer Science and
Electrical Engineering
West Virginia University
9th International Conference on Software Quality
Management, SQM2001
18th-20th April, 2001
Loughborough University, Loughborough, England
Outline
 Research Objectives
 Methodology
 Towards an Automated Methodology
 Process
 Case Study: The Pacemaker example
 Conclusions
West
Virginia
University
Automated Risk Assessment
Research Objectives
West
Virginia
University
 Architectural-Level Risk Assessment
Methodology at the early stages of
development(S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October, 2000)
 Automated Environment
Automated Risk Assessment
(continued)
Architectural-Level Risk Assessment
Methodology
West
Virginia
University
(S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October, 2000)
Utilizes:
• Dynamic Metrics: Component Complexity cpxi Connector Complexity
cpxij (S. Yacoub, H. Ammar, and T. Robinson. Metrics'99, November 1999)
• Failure Mode Effect Analysis FMEA (MIL_STD 1629A to define Component
Severity svrtyi Connector Severity svrtyij)
• Component Dependency Graphs CDG (adopted from: S. Yacoub, B. Cukic,
and H. Ammar. ISSRE'99 November 1999)
Defines:
• Heuristic Component Risk Factor hrfi = cpxi x svrtyi
• Heuristic Connector Risk Factor hrfij = cpxij x svrtyij
• Risk Aggregation Algorithm that produces HRFappl
Automated Risk Assessment
Architectural-Level Risk Assessment
Methodology
6 Steps
West
Virginia
University
(continued)
• Model the architecture of the system using simulation models
(UML-RT).
• Perform complexity analysis using simulation traces.
• Perform severity analysis using FMEA and simulation runs.
• Develop heuristic risk factors for components and connectors.
• Develop Components Dependency Graph for risk assessment
purposes. (System/Subsystems)
• Aggregate the risk factors using the graph traversal algorithm.
Automated Risk Assessment
(continued)
Automated Environment
West
Virginia
University
Severity Analysis (Failure/Effect
analysis)
Analyst
Simulation
Settings
Severity
Ranking
CARA Tool
Inspection
Viewing Macro
UML Simulation Environment
Sub Run
Settings
UML Model
Observer
Rose Real Time tool
Simulation
Log and
Violation
Report
Analysis
Tool
Text File
MS Excel
Processing
Macro
Timing Diag.
Violation Table
Excel sheets
Component
Complexity
Factors
Connector
complexity Factors
CDG “hrfi and hrfij
unidentified”
Formatted Excel
charts
Violation Tables
Analysis
Tool
MS Excel
Risk
Macro
HRF
Automated Risk Assessment
Automated Environment
Process
(continued)
West
Virginia
University
 Model the architecture of the system together with
the risk logging capability using Rose RealTime.
 Adjust the simulation runs in the observer as
desired.
 Run the simulation and get two log files containing:
• Component complexities.
• Component Execution Time.
• A log of all the messages exchanged.
Automated Risk Assessment
Automated Environment
Process
West
Virginia
University
(continued)
 Process the log with Excel Risk Macro and get:
• Transition Probabilities.
• Connector complexities.
• CDG “where Risk Factors = Severity Factors *
Complexity Factors (hrfi = cpxi x svrtyi )”
 Perform severity analysis using FMEA and simulation
runs.
 Traverse the CDG using the Excel traversal macro.
West
Virginia
University
Example: Pacemaker
Main Use Case Diagram
DoctorsProgramer
1
1
Programming
Mode
Programming
«extend»
«extend»
Operational
Modes
1
Operating_in_AVI
«extend»
«extend»
«extend»
1
Operating_in_ AAT
Operating_in_ AAI
1 Operating_in_ VVI
1
1
1
1
1
PatientsHeart
1
1
Operating_in_ VVT
Example: Pacemaker
1) Develop a Simulation Model
Capsule Diagram
West
Virginia
University
Case Study: Pacemaker
West
Virginia
University
(continued)
Atrial statechart
ToOn
ToOn
ToOff
A_Self_inhibited
Idle
ToInhibited
ToAVI
ToTriggered
A_AVI
A_Self_triggered
Case Study: Pacemaker
West
Virginia
University
(continued)
Atrial statechart
ToAVI
ini tial ize
Refractory
A_Pace_Pul se_Done
Paci ng
V_Refract_Done_Received
Time_Out
Wait
V_Sense_Recei ved
A sequence diagram for the AVI scenario
Communication
Gnome
Atrial
Ventricular
Heart
ToON
ToON
ToAVI
Refactoring
ToAVI
Refactoring
RefTimeOut
V Refract Done
Waiting
Waiting
V Sense
Got V Sense
SensTimeOut
Pacing
A Pace Start
Pacing
A Pace Start
Pace
PaceTimeOut
A Pace Done
Refactoring
Refactoring
A sequence diagram for the Programming scenario
Programmer
ReedSwitch
CoilDriver
Communication
Gnome
Atrial
ApplyMagnet
EnableComm
IDLE
IDLE
EnableComm
ToON
ToON
Pulse
Count = 1, SetTimer
Receiving
Pulse
Count++,
ResetTimer
BitTimeout
Decode(Count)
Store Bit in Byte
Byte Full?
Yes enqueue(byte)
Waiting For
Byte
Waiting for
Bit
Pulse
Count =0
Receiving
OR
ByteTimeOut
ByteTimeOut
Validating
IDLE
IsValid?
ToAVI
HerezaByte(ACK)
Yes
Processing
HerezaByte(NAK)
Waiting to
Transmit
No
Waiting to Send
Next Byte
ToAVI
Ventricular
2) Perform Complexity
Analysis
West
Virginia
University
A Transition between Composite States in a component’s Statechart
s2
init
I
init
I
s1
s21
t12
t11
t13
s11
s22
VGx(s11) + VGa(t11) + VGx(s1)+ VGa(t12) + VGe(s2) + VGa(t13) +VGe(s22)
Operational Complexity of a component using the scenario profile
and its complexity per scenario.
|X |
OCPX (oi )   PSx  ocpxx (oi )
x 1
West
Virginia
University
2) Perform Complexity
Analysis (cont’d)
A) Quantify Component Complexity Factors
using dynamic complexity metrics.
Programming ( 0.01)
AVI (0.29)
AAT (0.15)
AAI (0.20)
VVI (0.15)
VVT (0.20)
% of architecture complexity
Normalized to max. complexity
RS
8.3
.083
0.002
CD
67.4
0.674
0.013
CG
24.3
0.243
0.005
AR
VT
53.2
100
100
46.8
50.428
1
100
100
48.572
0.963
2) Perform Complexity
Analysis (cont’d)
West
Virginia
University
Export Object Coupling
Export Object Coupling
(EOC)
| {Mx (oi, oj ) | oi, oj  O  oi  oj} |
EOCx (oi, oj ) 
 100
MTx
the export coupling for component Ci with respect to component
Cj, is the percentage of the number of messages sent from Ci to
Cj with respect to the total number of messages exchanged
during the execution of the scenario x
|X |
EOC with scenario profiles
EOC (oi, oj )   PSx  EOCx (oi, oj )
x 1
OQFS with scenario profiles
|X |
OQFS (oi )   PSx  OQFSx (oi )
x 1
2) Perform Complexity
Analysis (cont’d)
West
Virginia
University
B) Quantify Connector Complexity Factors
using dynamic coupling metrics.
RS
RS
CD
CG
AR
VT
Programmer
Heart
CD
0.0014
0.002
CG
0.0014
0.003
AR
VT
Heart
0.011
0.0014
0.0014
0.25
0.27
0.0014
Programmer
0.006
0.123
0.307
1
0.873
3) Perform Severity
Analysis
 In performing severity analysis, each potential
failure mode is ranked according to the
consequences of that failure mode.
 Steps:
• Identifying Failure Modes


Failure modes of individual components.
(Functional faults and state-based faults)
Failure modes of individual connectors.
(Interface fault analysis)
West
Virginia
University
3) Perform Severity
Analysis (cont’d)
West
Virginia
University
 Steps (cont’d):
• Conducting Effect Analysis

Inject the fault.

Simulate the faulty model.

Monitor output and compare to expected output.

Identify the effect of the fault.
• Rank Severity


Identify category: Minor, Marginal, Critical, or Catastrophic.
Assign severity index to each component i as (svrtyi), which
takes a value of 0.25, 0.50, 0.75, and 0.95
FMEA table for the Pacemaker components
Connector Name
RS
Failure Mode
Failed to enable
communication
Cause of Failure
Error in translating
magnet command
CD
Failed to generate
good command
Fault in developing
the command
CG
Failed to validate
command
Fault in the
validation
procedure
Fault in processing
command routine
Mis-interpreting a
VVT command for
VVI
VT
AR
No heart pluses are
sensed though heart is
working fine.
Refract timer does not
generate a timeout in
an AVI mode
Wait timer does not
generate a timeout in
AAI mode
Heart sensor is
malfunctioning.
Timer not set
correctly.
Timer not set
correctly.
Effect of Failure
Unable to program the
pacemaker, schedule
maintenance task.
Unable to program the
pacemaker, schedule
maintenance task.
Cannot program the
pacemaker, schedule
maintenance task.
Heart is continuously triggered
but device is still monitored by
physician, need immediate fix
or disable.
Heart is incorrectly paced,
patient could be harmed by
continuous pulses.
AR and VT are in refactoring
state, no pace is generated for
the heart, patient could die.
AR stuck at the wait state, no
pacing is done to the heart
West
Virginia
University
Criticality of effects
Minor
Minor
Minor
Marginal
Critical
Catastrophic
Catastrophic
 Worst case severity found for the RS, CD, CG, VT, and AR are
Minor(0.25), Minor(0.25), Marginal(0.50), Catastrophic(0.95)
and Catastrophic (0.95), respectively
FMEA table for the Pacemaker connectors
Connector Name
RS-CG
Failure Mode
Failure to enable
communication of the
CG
Unable to disable
communication of the
CD with the
programmer
Failed to acknowledge
programming
Failed to send bytes of
program data to CG
Send incorrect
command (ex ToOff
instead of ToIdle)
Cause of Failure
Magnet malfunctioning.
RS failed to generate
message.
Magnet malfunctioning.
RS failed to generate
correct disable message.
CG-VT
Send incorrect
command (ex ToOff
instead of ToIdle
Incorrect interpretation
of program bytes
AR-Heart
Failed to sense heart in
AAI mode
Sensor error.
Failed to pace the heart
in AVI mode
VT failed to inform
AR of finishing
refractoring in AVI
mode
Pacing hardware device
malfunctioning
Timing mismatches
between AR and VT
operation.
RS-CD
CD-Programmer
CD-CG
CG-AR
VT-AR
Fault in coding the
sending message
Inappropriate count of
number of bits in a byte.
Incorrect interpretation
of program bytes
West
Virginia
University
Effect of Failure
Pacemaker is not programmed,
schedule maintenance task
Criticality of effects
Minor
Pacemaker receive bits accidentally
from hazards but device is never
programmed because CG is disabled,
schedule maintenance task.
Pacemaker is not programmed,
schedule maintenance task.
Pacemaker is not programmed,
schedule maintenance task.
Incorrect operation mode and
incorrect rate of pacing the heart.
Device is still monitored by the
physician, immediate maintenance or
disable is required.
Incorrect operation mode and
incorrect rate of pacing the heart.
Device is still monitored by the
physician, immediate maintenance or
disable is required.
Heart is always paced while patient
condition requires only pacing the
heart when no pulse is detected
Heart could be in serious problem
because of no pacing.
Failure to pace the heart.
Minor
Minor
Minor
Marginal
Marginal
Critical
Catastrophic
Catastrophic
4) Develop Risk Factors
West
Virginia
University
hrfi = cpxi x svrtyi
where:
0 <= cpxi <= 1, is the normalized complexity level (dynamic
complexity for components or dynamic coupling for connectors),
and
0<= svrtyi < 1 , is the severity level for the architecture element.
Dynamic
Complexity
Severity
Risk Factors
RS
0.002
CD
0.013
CG
0.005
AR
1
VT
0.963
0.25
0.0005
0.25
0.00325
0.5
0.0025
0.95
0.95
0.95
0.91485
Risk Factors for the components in the example
4) Develop Risk Factors
West
Virginia
University
(cont’d)
1
0.9
Risk Factors
0.8
0.7
0.6
Dynamic
0.5
CBO
0.4
NAS
0.3
0.2
0.1
0
RS
CD
CG
AR
VT
Comparison between risk factors based on static and dynamic metrics
Connector Risk Factors
RS
CD
CG
AR
VT
Programmer
Heart
RS
CD
0.00035
0.0005
CG
0.00035
0.00075
AR
VT
Programmer
0.00275
0.0007
0.0007
0.2375
0.2565
0.00035
Heart
.0015
0.11685
0.29165
Risk Factors for the connectors in the pacemaker example
0.95
0.82935
5) Constructing the CDG
West
Virginia
University
s
<, 0, .35>
<, 0, .01>
t
<, 0, .64>
<, 0, .99>
<Prog., 0,5>
<, 0, .36>
<, 0, .34>
t
<VT,0.9,40>
<,.26,.29>
<,3.5x10-4, .002>
<AR,0.95,40>
<, 0, .99>
<,.24,.19>
-4
<RS,5x10 ,5>
<,2.7x10-3,.008>
<,.26,.29>
<,.12,.35>
<,1.5x10-3,.008>
<,3.5x10-3,.005>
<,.29,.64>
-4
<,7x10 ,.0025>
<,3.5x10-4,.005>
<,.95,.47>
-4
<,7x10 ,.0025>
-4
<,7.5x10 ,.002>
<CG, 2.5x10-2,5>
<CD, 3x10-3,5>
<Heart,0,5>
<,5x10-4,.005>
<, 0, .99>
<, 0, .01>
<, 0, .99>
t
6) Risk Aggregation Algorithm
West
Virginia
University
 The algorithm expands all branches of the CDG starting from
the start node.
 The breadth expansions of the graph represent logical "OR"
paths.
• translated as the summation of aggregated
risk factors weighted by the transition
probability along each path.
 The depth of each path represents the sequential execution
of components:
• is given by the aggregate: HRF = 1 - i(1hrfi)
Risk Aggregation Algorithm
Procedure AssessRisk
Parameters
consumes CDG, AEappl,(average execution time for the application)
produces Riskappl
Initialization:
Rappl = Rtemp = 1 (temporary variables for (1-RiskFactor) )
Time = 0
Algorithm
push tuple <C1, hrf1, EC1 >, Time, Rtemp
while Stack not EMPTY do
pop < Ci, hrfi , ECi >, Time, Rtemp
if Time > AEappl or Ci = t; (terminating node)
Rappl += Rtemp
;(an OR path)
else
 < Cj ,hrfj , ECj >  children(Ci)
push (<Cj, hrfj ,ECj>, Time += ECi ,
Rtemp = Rtemp*(1-hrfi)*(1-hrfij )*PTij ) ( AND path)
end
end while
Riskappl = 1- Rappl
end Procedure AssessRisk
West
Virginia
University
Risk Aggregation Algorithm
West
Virginia
University
 The algorithm can be used for
• System-level Risk Assessment

The risk of the pacemaker that is found to be ~ 0.9
• Subsystem-level Risk Comparison



Complex systems are composed of many subsystems.
The algorithm can be used to obtain a risk factor for a subsystem
using risk factors of its individual components.
Compare risk factors of individual subsystems.
• Sensitivity Analysis


Sensitivity to Uncertainties in Component Risk Factors
Sensitivity to Uncertainties in Connector Risk Factors
Overall Risk Factor of the System
Sensitivity Analysis
West
Virginia
University
1.0
0.8
R(AR)
R(VT)
0.6
R(CG)
0.4
R(CD)
R(RS)
0.2
0.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Risk Factor of Individual Components
The Pacemaker risk factor as function of component risk factors (one at a time)
Overall System Risk Value
1.0
0.8
R(RS-CD)
R(CG-CD)
0.6
R(AR-Heart)
0.4
R(VT-AR)
R(VT-Heart)
0.2
0.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Risk Factor of Individual Connectors
The Pacemaker risk factor as function of connector risk factors (one at a time)
Benefits
West
Virginia
University
 The approach helps in:
• Deciding which components in the architecture require
more development resources.
• Deciding which connectors in the architecture are of
highest risk. A high risk connector indicates that the
interfaces between the corresponding components and the
messaging protocol should be carefully designed.
• Studying how uncertainties in component risk factors
affect the overall risk value of the system.
• Studying how uncertainties in connector risk factors affect
the overall risk value of the system.
Conclusion : Benefits
West
Virginia
University
 The methodology is applicable early at the
architectural level.
 The methodology is based on dynamic metrics. We
use dynamic metrics to account for the fact that a
fault in a frequently executed component will
frequently manifest itself into a failure.
 The methodology is based on simulation of
architecture models. Simulation helps in:
• Performing FMEA procedures .
• Calculating the CDG parameters such as probability of
transitions.
• Obtaining dynamic metrics.
Conclusion : Issues
West
Virginia
University
 Using ordinal scale for measuring severity.
 Effect of uncertainties in the scenario probabilities
and the estimated average execution times.
 Scalability issues, applying the methodology to a
larger case study.
 Methodology is limited to systems with statechart
and sequence diagram specifications.
Questions...
West
Virginia
University
Main Use Case Diagram
DoctorsProgramer
1
1
Programming
Mode
Programming
«extend»
«extend»
Operational
Modes
1
Operating_in_AVI
«extend»
«extend»
«extend»
1
Operating_in_ AAT
Operating_in_ AAI
1 Operating_in_ VVI
1
1
1
1
1
PatientsHeart
1
1
Operating_in_ VVT