Download Innovation via the joint evolution of people, process, and Knowledge

National Aeronautics and Space Administration
Engineer’s Perceptions of Risk
+ A Real World Example
Lynne P. Cooper
[email protected]
Jet Propulsion Laboratory
California Institute of Technology
and
Daniel J. Epstein Department of Industrial and Systems
Engineering
University of Southern California
March 8, 2013
©2013 L.P. Cooper and Jet Propulsion Laboratory, California Institute of Technology
Hello…
• …I’m Lynne Cooper
• BS Electrical & Computer Engineering
• MS Computer Engineering
• PhD Industrial & Systems Engineering
• Project team dynamics relative to perceptions of risk
• Former Captain, USAF
• 25 Years at JPL
• Built over 30 “Knowledge Based” Systems
• Worked on Multiple Mars missions
• Currently Lead, JPL Proposal Center
• Industry Faculty, Daniel J. Epstein Department of Industrial &
Systems Engineering
• Teams, Collaborative Engineering
2
Agenda
• Pre-Quantitative Risk
• Research on how real world project teams conceive of and
manage risk – long before quantitative tools are practical
• Case Story
• Building the Proposal Authorization/Bid & Proposal Tool
• How this all relates to software engineering
• Questions Welcome at Any Time
3
“Risk” from a different perspective:
One that includes:
• Emotions
• Intuition
• Relationships
• Collective belief
“Pre-Quantitative Risk”
4
Project Risk Scenario
What is this judgment based on?
System Design
That doesn’t
make sense
Experts
x1
x2
x3
Y
x4
xn
Inputs
Team
Model
Output
5
Research Approach
• Studied real world teams
• Working on high risk, high technology projects
• Analyzed:
• What they said
• What they wrote
• Team processes
• Actions they took
• Over time
6
Language of Risk
• Rarely talked about risk explicitly
• But language of risk and uncertainty
permeated team discussions
Neg Outcome
Negative Outcome - Uncertainty
Uncertainty - Opportunity
Neg Outcome Trend
Opportunity
Uncertainty Trend
Opportunity Trend
Normalized Frequency
80
70
60
50
40
30
20
10
7
0
0
20
40
60
80
Day of Project
100
120
140
Components of Risk Discussions
• Goals
• Design
• Environment
• How things interact
• Bad things that could happen
• What they don’t know
• What they could influence/control
• Acceptable levels of risk
8
Relationships of Components
Goal
Uncertainty
Design
Interactions
Ability to
Influence
Bad things
that could
happen
Judgment of
Acceptability
Risk
Action
Environment
9
= interactions
What increases risk?
• Inability to predict behavior
• Interactions (complexity)
• Lack of intuition about interactions
• Competing/Multiple goals (scope)
• Interactions (complexity)
• Uncertainty
• Changes in the environment
• Harshness of the environment
• Constrained resources
10
What decreases risk?
• Many potential solutions to problems
• Previous success
• Decoupling
• Predictability
• Ability to model performance and behavior
• Increasing intuition (e.g., by testing,
simulation)
• Well characterized, stable environment
• Access to resources
• “Acceptable” risk:
• Target level of risk
11
Visualizing Risk: Sliders
Degree of Risk
Too Little
Low
Acceptable Risk
Medium
Too Much
High
Acceptable Risk Thresholds
12
Aggregating Risk
Site Selection
Thermal Environment
Energy Demands
Power System
Batteries
O&M Costs
Solar Arrays
• Not linear
• Highly dynamic
• Strange couplings
13
Results & Practical Implications
• Sheds light on risk-related team
dysfunctions
• Circular meetings  Intuition building
• Quantifying to justify decisions
• Building a shared mental model of risk
• Identifying the components that contribute
to risk and their interactions
• Rather than just “Quantifying” risk
• Managing based on relative risk
• What makes it “less risky” vs. “more risky”
• Developing intuition for understanding
complex interactions
14
Proposal Authorization/Bid & Proposal (PABP) Tool
CASE STORY
15
General Proposal Process
Concurrence
form on
Proposals
Website
PA/B&P Form
Fill out & submit
Proposal
Authorization
(PA) form
Write
proposal
Develop
cost
estimate
Authorize
proposal and
provide B&P
funds
Finalize
proposal
Get
Concurrence
Review
JPL Officially submits
16
Sample B&P Form
17
PA/B&P and Concurrence
Processes
Process to get permission to write the proposal, request B&P funding
Process to get permission to submit the proposal
18
Current State
DPM
Send
B&P
$ $
$ $ $
$
19
Benefits for Enhanced Process
• Replaces hardcopy/hand-carry process with electronic
submission, routing & approval
• Proposal Authorizations/Bid & Proposal Requests
• Concurrence Packages
• Primary Result:
• Reduce (eliminate) hand-carrying of paper forms
• Side Benefits
• Ability to track status – visibility into proposal pipeline from individual, line, program
office, and service provider perspectives
• Single tool for all Science & Technology proposals
• New visibility into non-NASA submissions
• Access via computer and mobile devices
• Centralized source of proposal data
• Including B&P allocated, status of HQ evaluation, end result
20
Key Development Activities
• Requirements Definition
• Replace two existing, less capable systems  Backwards
Compatibility
• Process Flow
• Focus on single driving issue  electronic routing/sign-off
• Reliability  Tool now supports critical business process with
hard time constraints
• Beta Test
• Find bugs
• Expose process issues
• Assess ease of use
• Roll-Out
• Multi-tiered support for users
• Broad communications strategy
21
Reliability/Risk
• System Failure vs. Process Failure
• System Recovery vs. Process Recovery
• Transition
• Data Migration
• Website Transition
• Process Transition
• Handling of in-process activities
22
PABP Beta Test Prelim Results
• Two days, simulated end-to-end process with volunteer
participants and key stakeholders
• 52 Participants, provided constructive feedback
• From all 3x Tech Divisions
• From 4x, 6x, 7x, and 8x (including NSTA) program offices
• Enthusiasm for the tool
• Broad spectrum of platforms
• Mac, PC, Linux, running different versions of operating systems
• Multiple browsers: Firefox, Safari, Chrome, IE
• Results: No Major Issues
• Identified one system bug, fix implemented and deployed
• Identified two areas that users found confusing – requires minor
modifications to the user interface
• Identified process issues – assessing best way to address
23
PABP vs. Classic Mistakes
•
•
•
•
•
•
•
•
•
•
•
•
1. Poor estimation and/or scheduling
2. Ineffective stakeholder management
3. Insufficient risk management
4. Insufficient planning
5. Short-changed quality assurance
6. Weak personnel and/or team issues
7. Insufficient project sponsorship
8. Poor requirements determination
9. Inattention to politics
10. Lack of user involvement
11. Unrealistic expectations
12. Undermined motivation
Let’s
Discuss
24
“The revolutionary idea that defines the
boundary between modern times and the
past is the mastery of risk: the notion that
the future is more than a whim of the gods
and that men and women are not passive
before nature”
Peter L. Bernstein, 1996, p.1
25
Thank You
26