Download Extracting Software Failure Knowledge from Anomaly Databases at

Extracting Software Failure
Knowledge from Anomaly
Databases at NASA
Lucas Layman, Davide Falessi
Fraunhofer Center for Experimental Software Engineering
NASA OSMA Software Assurance Research Program
© 2014 National Aeronautics and Space Administration
1
Overview
We want to learn more about how software fails in
NASA missions in order to improve software
assurance
This is a work in progress
– Will provide some early analyses
– The planned path forward with data mining and
machine learning
– Comments, concerns, criticisms are welcome
How could this research help you?
© 2014 National Aeronautics and Space Administration
2
Outline
• Background and motivation
• Current uses of anomaly information
• Early analyses and data exploration
• Machine learning
© 2014 National Aeronautics and Space Administration
3
BACKGROUND AND
MOTIVATION
© 2014 National Aeronautics and Space Administration
4
Background: Project Lifecycle
Program approval
Pre- Phase A
Concept
Studies
Phase A
Concept
Development
Spacecraft launch
Phase B
Preliminary
Design
Phase C
Final Design &
Build
Phase D
Assembly,
Integration, and Test
© 2014 National Aeronautics and Space Administration
5
Phase E
Deployment,
Operations
Background: Anomaly Definition
!!!
Anomaly: A deviation from expected,
nominal behavior in the Operations phase
Anomaly report: Artifact documenting the
anomaly occurrence and investigation
© 2014 National Aeronautics and Space Administration
6
Background: Anomaly Reports
• Date/time of event
• Criticality
• Description
• Investigation
• Cause
• Corrective Action
Anomaly reports are a rich source of information on how
software fails in operation
© 2014 National Aeronautics and Space Administration
7
Anomalies are Institutional Memory
Goal: To learn across missions and centers to improve
software development
http://phonedirectory.ksc.nasa.gov/assets/docs/NASA_centers1_700.jpg
© 2014 National Aeronautics and Space Administration
8
HOW ARE ANOMALIES
REPORTS USED TODAY?
© 2014 National Aeronautics and Space Administration
9
Current uses – tracking and reporting
Current Status
Submitted
Project
0
Age of open anomalies
#
Mission Impact Open anomalies
< 30 days
3
Critical
0
20
Major
2
Substantial
5
Minor
3
Open
23
30 days – 90 days
Rejected
20
> 90 days
Closed
0
562
Mishaps
No Effect
13
Index
Anomaly
Date
Days
Mission Impact Mishap Open
0595
08/13/2011
Major
13
Title
Y
338 Reaction Wheel 3 Dragged down to zero
0686
03/22/2012
Substantial
Y
DOY 2012-082 Processor Reset / Sun Safe
116 Incident
0651
11/22/2011
Substantial
N
237 Overlapping LROC File Handle Assignments
0095
08/06/2009
Substantial
0674
02/25/2012
Major
Plethora of Open Files Prevents New File
1075 from Opening, Causes LROC Handle Errors
N
© 2014 National Aeronautics and Space Administration
N
10
142 OPSMPS1 Database Corruption
Current uses - Investigating
“How many times has this star tracker
problem caused an anomaly?”
“What problems can we expect from the
Microwave Sounding Unit based on previous
missions?”
“How many of these erroneous tracks are due
to an iridium flash vs. an error in the software?
© 2014 National Aeronautics and Space Administration
11
Problems in the current usage
Anomaly data is rarely shared across
centers or missions
Anomaly reports tend to be “write once, read
never”
Meta-data is unreliable
© 2014 National Aeronautics and Space Administration
12
EXPLORING SOFTWARE
ANOMALY DATA
© 2014 National Aeronautics and Space Administration
13
Analysis approach
Finding software-related anomalies is not trivial
1. Search for search for “software”ish text strings)
2. Manually root out false positives
–
False negatives (estimated %)
3. Label the anomaly with some tag we wish to count
Anomalies
Subset
analyzed
Software
anom.
Missions
Launch
dates
Software
related*
JPL
13696
4621
1031
9
‘89-’05
257
GSFC
25320
9921
2328
29
‘97-’10
634+
Location
© 2014 National Aeronautics and Space Administration
14
How severe are the software
anomalies?
Mission Impact of Software Anomalies
70%
60%
50%
40%
Project
30%
Astrophysics
All projects
20%
10%
0%
Critical
Major
Substantial
Minor
No Effect
© 2014 National Aeronautics and Space Administration
15
How are we fixing software
anomalies?
Software Anomaly Repair Actions
60%
50%
40%
30%
Project
20%
Astrophysics
10%
All projects
0%
© 2014 National Aeronautics and Space Administration
16
What is the software anomaly growth
rate?
50%
© 2014 National Aeronautics and Space Administration
17
Data drawn from anomaly samples of Goddard spaceflight projects
WHAT CAN WE LEARN
ACROSS MISSIONS?
© 2014 National Aeronautics and Space Administration
18
“Why do I need Software Assurance?”
JPL anomaly risk rating distributions
All
% caused by
anomalies
software
Risk rating
1 - Unacceptable Risk
5.7%
26.8%
2 - Accepted Risk
34.4%
25.2%
3 - No Significant Risk
53.7%
34.7%
4 - No Risk
6.2%
25.1%
Total
100%
30.4%
Taken from a sample of JPL missions between 1989 and 2007.
Sample size > 1000.
© 2014 National Aeronautics and Space Administration
19
Where are the software
anomalies?
Flight vs. Ground Software Anomalies
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Astrophysics
Heliophysics
Earth
Planetary
Flight software
Ground software
© 2014 National Aeronautics and Space Administration
20
ONGOING RESEARCH
© 2014 National Aeronautics and Space Administration
21
Our goal: Lower the price of
knowledge
To do this analysis, you must know which anomalies
below to a category, e.g.
– Software cause
– Attitude control system
– Cruise phase
Currently, determining these categories is an intense
manual process
Research Objective: Automatically label anomalies with high
accuracy to enable anomaly “analytics”
© 2014 National Aeronautics and Space Administration
22
The possibilities: Predicting causes and fixes
!!!
System
Power subsystem
Mission impact
Low
Likely cause
Software logic error
Likely corrective actions
Procedure restart
Responsible party
L. Layman
© 2014 National Aeronautics and Space Administration
23
Our automatic labeling approach
Compute similarity using Natural Language Processing techniques
0.45
Compute NLP
similarity score
0.72
0.93
Root cause: Operations error
Root cause: Operations error
Corrective action: Software fix
Corrective action: Software fix
© 2014 National Aeronautics and Space Administration
24
Finding the best NLP technique
Input Variables
Title
Class Variables
Description
Root Cause
Corrective Action
480 anomalies
Class(X) ?= Class(most similar)
NLP technique
% correct
Kappa
LSA BINARY COSINE Porter stemmer
76.3
0.49
LSA RAW COSINE Porter stemmer
76.0
0.49
LSA TF_IDF COSINE Stanford (noun and verb))
75.0
0.46
© 2014 National Aeronautics and Space Administration
25
Using NLP output for prediction
Title
Description
Title
Description
Title
Description
Both
NLP
NLP
#1 NLP
#1 NLP
NLP
NLPNLP
NLP
NLP
NLP
NLPNLP
NLP
NLP
NLP
NLPNLP
NLP
NLP
Data
miner
Data
miner
Data
miner
Data
miner
Anomaly
class
Anomaly
class
Anomaly
class
Anomaly
class
Anomaly
class
Anomaly
class
© 2014 National Aeronautics and Space Administration
26
Evaluation effort
NLP techniques: 41 million comparisons
– Over 1 month of computer runtime
Data miners: 40,000 models evaluated
© 2014 National Aeronautics and Space Administration
27
Predicting “Root Cause” types
Title + Description
Anomaly Title and/or Description
NLP
NLPNLP
NLP
NLP
to predict
Data
miner
Root Cause classes
Software
Hardware
Operations
Environmental
Anomaly
class
Average
Correctness
Kappa
75.8%
0.21
Min
66.6% 85.4%
-0.09
Promising, but need to test on a more diverse sample
© 2014 National Aeronautics and Space Administration
28
Max
0.50
Labeling “Corrective Action” types
Description
Anomaly Title and/or Description
NLP
NLPNLP
NLP
NLP
to predict
Data
miner
Corrective Action classes
Software
Hardware
Operations
Ignore / none / unspecified
Anomaly
class
Average
Correctness
Kappa
77.3%
0.53
Min
60.0% 93.3%
0.14
Very promising. Significant performance gains.
© 2014 National Aeronautics and Space Administration
29
Max
0.86
Looking ahead
Need to categorize anomalies in useful ways – how
do engineers think about spacecraft?
© 2014 National Aeronautics and Space Administration
30
Acknowledgements
This work sponsored by NASA OSMA Software Assurance
Research Program grants NNX11AQ40G and NNX08A260G
Goddard Space Flight Center
Lucas Layman
Jet Propulsion Lab
Allen Nikora
NASA IV&V Facility
Koorosh Mirfakhraie
Dan Painter
Keenen Bowens
Wes Deadrick
Ricky Forquer
NASA Headquarters
Martha Wetherholt
Wallops Flight Facility
Donna Smith
Marshall Space Flight Center
John McPherson
http://commons.wikimedia.org/wiki/File:Nasa-logo.gif
© 2014 National Aeronautics and Space Administration
31
Thank you
Contact
Lucas Layman
[email protected]
or
[email protected]
© 2014 National Aeronautics and Space Administration
32
BACKUP SLIDES
© 2014 National Aeronautics and Space Administration
33
Does mission type affect SW
anomaly discovery rate?
% of anomalies discovered
Normalized Software Anomaly Growth
100.0
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
0
Earth
20
40
60
80
Normalized time scale
Heliophysics
Planetary
Astrophysics
© 2014 National Aeronautics and Space Administration
34
100
120
all sw anomalies
Who patches their software?
Software anomaly repair category
80%
70%
60%
50%
Astrophysics
40%
Heliophysics
30%
Earth
20%
Planetary
10%
0%
S/W patch or
reconfigure
Ops procedure or
workaround
None/Ignore
© 2014 National Aeronautics and Space Administration
35
60 NLP techniques
Each technique is a combination of:
1. Term extraction: {simple tokenization, Part of
Speech tagging, stemming}
2. Term weighting: {TF, IDF, TF-IDF, Simple, Binary}
3. Model type: {Vector Space Model, WordNet}
4. Similarity computation: {Cosine, Jaccard, Dice,
Resnik, Lin, Jiang, Pirro and Seco}
© 2014 National Aeronautics and Space Administration
36
Acronyms
• IV&V – Independent Verification and Validation
• JPL – Jet Propulsion Laboratory
• LSA – Latent Semantic Analysis
• NASA – National Aeronautics and Space Administration
• SW, S/W – software
• TF – term frequency
• IDF – inverse document frequency
© 2014 National Aeronautics and Space Administration
37