Download Measuring Control Effectiveness

Dr Maureen
Hassall
2015 NSW Mining - Health, Safety, Environment and Community Conference
What is a control?
What are critical controls?
Maureen Hassall – UQ
Acknowledgements
ACARP
• C23007 Selection and optimisation of risk controls
ICMM
• Health and Safety Critical Control Management –Good Practice Guide
NSW Mining 2015 HSEC conference
Maureen Hassall
Why are we interested in controls
NSW Mining 2015 HSEC conference
Maureen Hassall
Why are we interested in controls?
“The top factors for … incidents are people
not properly identifying risks, controls not
being in place, or the controls not being
effectively implemented or maintained”.
(ICMM 2013)
NSW Mining 2015 HSEC conference
Maureen Hassall
Why are we interested in controls?
Low Risk
High Risk
Risk is determined by the effectiveness of controls
NSW Mining 2015 HSEC conference
Maureen Hassall
Why are we interested in controls?
Analysis by ICMM that the top factors for why incidents occur are due to:
Controls not being in place and controls not being effectively
implemented or maintained.
!
Threat
NSW Mining 2015 HSEC conference
Unwanted
Event
Maureen Hassall
Why are we interested in controls?
Analysis by ICMM that the top factors for why incidents occur are due to:
Controls not being in place and controls not being effectively
implemented or maintained.
!
Threat
NSW Mining 2015 HSEC conference
Unwanted
Event
Maureen Hassall
Why are we interested in controls?
Effective controls prevent unwanted events . . . .
!
Threat
NSW Mining 2015 HSEC conference
Unwanted
Event
Maureen Hassall
Why are we interested in controls?
Effective controls prevent unwanted events . . . .
. . . . or mitigate consequences of unwanted events
NSW Mining 2015 HSEC conference
Maureen Hassall
What is a control?
A control is an act, object or
system (combination of act
and object) that directly
arrests or mitigates an
unwanted event.
A control is specifiable,
measureable and auditable.
NSW Mining 2015 HSEC conference
Maureen Hassall
What is a control?
A control is an act, object or
system (combination of act
and object) that directly
arrests or mitigates an
unwanted event.
A control is specifiable,
measureable and auditable.
Human actions
Control Objects
Adapted from:
Rasmussen & Svedung, 2000
Leveson, 2004
NSW Mining 2015 HSEC conference
Operational
Hazards and Threats
Maureen Hassall
What is a control?
A control for mitigating child pedestrian fatalities outside schools.
The control is drivers driving vehicles at a safe speed that is specified as 40km/hr
or less and is measureable via speed cameras and auditable with speed versus
fatality data.
NSW Mining 2015 HSEC conference
Maureen Hassall
What is a control?
A control is an act, object or
system (combination of act
and object) that directly
arrests or mitigates an
unwanted event.
A control is specifiable,
measureable and auditable.
NSW Mining 2015 HSEC conference
Maureen Hassall
Impact of changing definition
NSW Mining 2015 HSEC conference
Maureen Hassall
Impact of changing definition
Results in more succinct bowties - easier checking for adequacy/coverage of
controls
Controls that Controls that
minimise
detect and
exposure deflect threats
Last
chance
intervention
controls
Protection
controls
Isolation/
containment
controls
Recovery/.
Restoration
controls
Hazard
Control
Control
Control
Control
Control
Control
Control
Decreasing response time for ARRESTING CONTROLS
NSW Mining 2015 HSEC conference
Control
Control
Increasing response time of MITIGATION CONTROLS
Maureen Hassall
Impact of changing definition
NSW Mining 2015 HSEC conference
Maureen Hassall
Impact of changing definition
Because we said controls need to be
specifiable, measurable and auditable
We should now be able to determine
the effectiveness of controls
NSW Mining 2015 HSEC conference
Maureen Hassall
Measuring
Control
Effectiveness:
Measuring control
effectiveness should
be done for
important controls to
allow for the tracking
and actioning of
control performance
over time.
What is a critical control?
NSW Mining 2015 HSEC conference
Maureen Hassall
What is a critical control?
From ICMM guide
NSW Mining 2015 HSEC conference
Maureen Hassall
Hazard
Monitoring, maintaining and improving controls
CONTROL ASSURANCE
MANAGEMENT SYSTEM (CAMS)
Operations activities
Maintenance activities
Engineering activities
Management activities
CAMS: Activities that ensure people and
equipment are ready and able to
perform the control activities as required
when required (i.e. activities that ensure
controls sustain effectiveness over time)
What is a critical control?
From ICMM guide
NSW Mining 2015 HSEC conference
Maureen Hassall
What is a critical control?
The following questions can help to determine if a control is critical:
•
•
Is the control crucial to preventing the event or minimizing the
consequences of the event?
Is it the only control, or is it backed up by another control in the event
the first fails?
•
Would its absence or failure significantly increase the risk despite the
existence of the other controls?
•
Does it address multiple causes or of mitigate multiple consequences
the MUE? (In other words, if it appears in a number of places on the
bowtie or on a number of bowties, this may indicate that it is critical.)
NSW Mining 2015 HSEC conference
Maureen Hassall
What is a critical control?
Identify controls
Critical
control?
Y
Determine erosion
factors
N
Control Assurance
Management Plan
Control Assurance
Management Plan
Determine
verification
requirements
Audit processes
NSW Mining 2015 HSEC conference
Maureen Hassall
Conclusion
• Improvement of risk controls and their effectiveness will positively impact the
safety, operational efficiency and cost performance
• Critical control focus will concentrate efforts on the selection, implementation
and management of those controls that directly address fatality risks
• Improvements will only be achieved if systems, tools and people drive a shift in
mindset to the importance of managing controls.
• Potential impacts and improvements include the following:
• Development of quality of bowtie analysis and quality control assurance
management systems.
• Further work on effectiveness and dependency analysis for control sets
• Development of knowledge management systems to help quantify control
effectiveness e.g. by upgrading and expanding RISKGATE.
NSW Mining 2015 HSEC conference
Maureen Hassall
Acknowledgements
ACARP
• C23007 Selection and optimisation of risk controls
ICMM
• Health and Safety Critical Control Management –Good Practice Guide
NSW Mining 2015 HSEC conference
Maureen Hassall
More on Measuring Control Effectiveness:
QUANTATIVE ANALYSIS:
Uses actual data to determine the 3 components of control effectiveness:
• The ability of the control to function as required.
• The availability and use of the control when required.
• The extent to which the control a) minimises exposure, detects/deflects
or intervenes to prevent threats becoming unwanted events or b)
mitigates the severity of the consequence.
For example the effectiveness of alcohol testing =
Measuring Control Effectiveness:
TREE (SEMI-QUANTATIVE) ANALYSIS:
Measuring Control Effectiveness:
ILLUSTRATION DATA ONLY
Numbers have not be derived from real data
Proximity detection
systems (humans
respond to alarms)
Verification data
Q1. Proximity detection system reliably
picks up objects as required when
required?
Q2. Operator competent to respond to
proximity detection system
feedback?
Q3. Operator aware and does respond
correctly to proximity detection
feedback?
Q4. When object in proximity detected
and correct actions taken were they
effective in avoiding accident?
ESTIMATED EFFECTIVENESS
OF CONTROL
0.25
N
Y
0.75
0.01
N
% fleet fitted x routine
reliability test information
Y
0.99
0.40
N
Driver training/competency
testing, driver interviews
Y
0.60
0.00
N
Vehicle data reports, incident
reports,
Y
1.00
45%
Analysis of incidents reports
and vehicle data records
Measuring Control Effectiveness:
CATEGORISATION (QUALITATIVE) ANALYSIS: Based on peoples judgements
Examples of
categories used to
assess control
effectiveness
Three category control
effectiveness scale
Four category control
effectiveness scale
Five category control
effectiveness scale
Good
effectiveness
High Effectiveness
Excessive – too many controls
[that adversely effectiveness]
Satisfactory but could be
improved
Satisfactory Effectiveness
Acceptable – no action
Inadequate action required
Unsatisfactory
Acceptable but current
controls could be enhanced
Replace
Not acceptable – refinement
to current controls needed
Inadequate – more/better
controls needed
NSW MINING
Health, Safety,
Environment
& Community