Download Model Risk Management

Model Risk
Management
Nav Vaidhyanathan
Director, Head of Model Risk Management
Wintrust Financial Corporation
Global Association of Risk Professionals
August 2014
These materials reflect the views and opinions of the speaker and are not
intended to reflect the views, policies or practices of Wintrust Financial or its
affiliated companies. They are for informational purposes only.
The views expressed in the following material are the author’s and do not
necessarily represent the views of the Global Association of Risk
Professionals (GARP), its Membership or its Management.
2
Agenda
Model Risk Management is a broad, evolving, and an expanding topic. This presentation
will cover only select components at a high level.
 Inherent risk (model tier) and residual risk of model
 A framework for model monitoring
 Model risk aggregation framework
 Model validation for stress test (e.g., DFAST) models
3 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Model Risk Axioms*
The following can potentially be considered as self-evident truths for Model Risk. Given
everything else equal,

All models have limitations

Model Risk is related to model limitations

Model Risk increases with more models (this might fall in the grey area)
−
E.g., two Tier 1 models potentially pose higher risk than 1 Tier 1 model (everything else being the
same, i.e., similar limitations, similar validation status, similar materiality, similar findings, etc.)

Model Risk for Tier 1 model > Model Risk for Tier 2 model

Model Risk for validated model ≤ Model Risk for unvalidated model (for the same model)

More findings ⇒ more Model Risk (same category and level of findings)

Certain types of findings will have higher Model Risk than others

Models with deteriorating performance pose higher Model Risk

Model Risk is exhibited by the variance between model output and observed actual
(this is just one of the ways that Model Risk exhibits itself)
* may not be axioms in the purest sense of the term
4 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Residual Risk Rating could be
driven by:
•
Model validation findings:
•
Category
•
Level
•
# of Open Findings
•
Model performance /
monitoring
Residual
Risk Rating
Inherent Risk and Residual Risk of Model
Model owners can reduce residual
risk – by improving models,
improving controls, and/or
remediating findings
High
Moderate
Low
Low (Tier 3)
Moderate
Inherent Risk Rating (Model Tier)
Inherent Risk Rating could be driven by:
• Materiality
• Type of model
• Used for regulatory / financial reporting or critical business
decisions
• Feeds into another Tier 1 model?
Model owners cannot (usually) reduce inherent risk
5 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
High (Tier 1)
Residual Risk Rating Framework
 Residual Risk Rating (High,
Moderate, Low) is based on
the Residual Risk Score range
 E.g., RRS > 200 ⇒ High, RRS ≤
100 ⇒ Low
 If the model is deemed unfit for
use, the RRR is High
 The Residual Risk Score
Range to Residual Risk
Ratings mapping needs to be
monitored periodically
 The ranges can be determined
by typical observations from
model validation and model
monitoring exercises
 If a model is unvalidated, the
residual risk rating is HIGH
6 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Model Monitoring
 Model monitoring should evaluate whether changes (including anticipated) in
products, exposures, activities, clients, or market conditions necessitate
adjustment, redevelopment, or replacement of the model
 Ongoing activities include – review of input data and assumptions, outcome
analysis, model stability checks, review of overlays and expert judgments,
benchmarking, etc.
 Key considerations
−
Model performance monitoring
›
Pros and cons of model performance measures
›
Model type
›
Model use
›
Type of measure
−
Periodicity of monitoring
−
Thresholds for model recalibration or rebuild
−
From performance measure to model performance rating
7 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
A Bottom Up Framework For Model Risk Aggregation
Impact of Failure (e.g., on Capital)
Yes
High Residual
Risk
The end nodes in the
decision tree can be
used for aggregating
model risk
Tier 1
Tier 2
Tier 3
Validated
…
…
No
Moderate
Residual Risk
Low Residual
Risk
As part of an RCSA
process, the impact of
the model failure is
estimated by model
owners
For DFAST projection models, for Severely Adverse Scenario, the impact on capital can be assessed
using backtesting during Great Recession period for all models (adjust the variance by severity of the
scenario compared to the Great Recession)
By looking at the distribution, a conservative starting point for capital can be chosen for each model
Tier
8 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Some Considerations in Validation of Stress Test Models

Are estimates conditioned on macroeconomic scenarios?

Do the models produce outcomes that show separation and order between the baseline,
adverse, and severely adverse scenarios?

Are the scenario and assumptions consistent across all models?

Are exposures modeled using appropriate levels of segmentation?

Qualitative elements of the model

Relevance of third party vendor models to company specific characteristics, especially when
using proxy portfolio

Sensitivity analysis around assumptions

Use of weak models vs. expert judgment

For models not validated, compensating controls and conservativeness

Documentation

Capital buffer for model risk
9 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
C r e a t i n g a c u l t u r e
r i s k a w a r e n e s s ®
o f
Global Association of
Risk Professionals
111 Town Square Place
14th Floor
Jersey City, New Jersey 07310
U.S.A.
+ 1 201.719.7210
2nd Floor
Bengal Wing
9A Devonshire Square
London, EC2M 4YN
U.K.
+ 44 (0) 20 7397 9630
www.garp.org
About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make
better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,
academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)
Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for
professionals of all levels. www.garp.org.
10 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Model Risk
Management for
Non Banks
Michelle McCarthy
Managing Director, Risk Management
Nuveen Investments
Global Association of Risk Professionals
August 2014
Disclaimers
These materials reflect the views and opinions of the speaker and are
not intended to reflect the views, policies or practices of Nuveen
Investments or its affiliated companies. They are not intended to
provide investment advice and are for informational purposes only.
The views expressed in the following material are the author’s and
do not necessarily represent the views of the Global Association of
Risk Professionals (GARP), its Membership or its Management.
2
Contents
 Differences between models and their uses at banks vs. the buy side
 What should be in scope for model risk management activity in the asset
management business?
 Potential tiering of asset management models, and effect on model risk
management activities
3 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Most literature on model risk management has concerned banks
 OCC 2000-16
 OCC 2011-12, FRB SR 11-7
 The case of AXA Rosenberg brought the conversation to the buy side, without
much guidance on how bank practices can and should differ from asset managers’
practices
4 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Different uses of models in asset managers vs. banks
 Asset managers are not computing regulatory capital

Bank-owned asset managers may need to provide operational risk capital
calculations, but this is not in service of the asset management business
 We rarely use valuation models as the sole source of valuation; we frequently
use external sources
 Many asset management models are an input to a decision, but not the sole
driver of that decision
5 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Potential scope of model management activity for asset managers
 Includes models that are used for investing or hedging
Drive investing activity to some extent
Value key assets
 Does not include calculators; model must provide estimates and involve
uncertainty to require model risk management
Good change controls always an important operational control, but calibrations and
validations should be reserved for models that estimate uncertainty
Would not include compliance systems
Would not include price-to-yield calculations
Would not include simple screens for fundamental stock characteristics
Would not include pure index replication models without meaningful uncertainty
 Does not include analyses that help develop an investment thesis—only the
tools used to execute the thesis, or to value assets once purchased
6
6 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Tiering for materiality
 A potential tiering of asset management models:
Tier 1
Tier 2
Models that:
• have high potential direct
impact on company results
• are broadly applied across the
entire company
• are understood and
maintained by a relatively
narrow range of experts at the
company
• Result in instant electronic
trading with little or no ability
to intervene, and/or affecting
a large portion of the portfolio
•
Such as:
• Statistical arbitrage model
• High frequency trading model
• A model that is rigorously
adhered to, is key to a broad
set of strategies and is
disclosed as a critical feature
of the investment process
• Significant valuation models
where third party sources are
unavailable
•
•
•
•
•
•
•
•
Tier 3
are tightly coupled to trading but permit
human intervention, and/or affect a
small portion of the business
result in trading with little other input
than direction from a model (quant
strategies)
provide hedge ratios for using
derivatives, or to balance proportions
of fixed income instruments, in
portfolios
value smaller, less significant balance
sheet items than those captured in Tier
1
•
A strategy that strongly adheres to
rebalancing if Barra or Yieldbook
measures exceed a threshold
An economic indicator model that is
the basis of an investment strategy
High frequency trading models
Models used for delta hedging or asset
allocation rebalancing
Minor valuation models where third
party sources are unavailable
•
7 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
•
•
•
•
•
Tier 4
apply a screen to a universe of
stocks, using some measure of
uncertainty, such as a measurement
of the dispersion of some
fundamental characteristic (low priceto-earnings ratios, for example)
compare characteristics of securities
to assist the portfolio manager in
finding securities that he or she
deems desirable
•
Using Barra or Yieldbook to see
where a portfolio is overweight, in
order to select trades that will balance
the portfolio better
Economic indicator screens to
provide general investment signals
Index replication models with tilts
Risk oversight models
Valuation models where third party
valuations are available
•
•
•
•
may support a certain
amount of decision
making, but with many
other inputs to these
decisions outside the
model.
are outside the
company’s areas of key
business risk
are understood by a
broad range of business
people
Measures of portfolios
that are not used to
drive action
Index replication
models that approach
100% replication
Slotting typical asset management models into tiers
Tier I
Investment Models
Fundamental
Business Models
Stock Selection Screen
Quantitative
Stock Selection
Portfolio Construction
Index Replication with Tilts
Index Replication
Statistical Arbitrage
High Frequency Trading
Leverage
Hedging
Economic Models
Growth
Inflation
Industry
Credit
Liquidity
Risk Models
Capital Loss
Liquidity
Ex Ante Tracking Error
Ex Post Tracking Error
Duration, Beta, Factor Exposure Limits
Valuation Models
Third Party Pricing Services
Pricing Model - Third Party Developed
Pricing Model - In-House Developed
8 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
Tier II
x
x
Tier III
Tier IV
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
What steps may make sense for model management of the
different tiers?
 Inventory
 For material models (Tiers 1 to 2)
Initial independent validation
─ Where independence may or may not require the different reporting lines set forth for
banks: effective challenge
Documentation (model theory, assumptions, limitations, code, user instructions)
Change validation
Periodic formal, documented calibration
Periodic re-validation given results of calibration
 Model management for Tier 3 models could be lighter in terms of frequency,
intensity, formality of independence
 The challenge of third party vendor models
9 | 
© 2014 Global Association of Risk Professionals. All rights reserved.
C r e a t i n g a c u l t u r e
r i s k a w a r e n e s s ®
o f
Global Association of
Risk Professionals
111 Town Square Place
14th Floor
Jersey City, New Jersey 07310
U.S.A.
+ 1 201.719.7210
2nd Floor
Bengal Wing
9A Devonshire Square
London, EC2M 4YN
U.K.
+ 44 (0) 20 7397 9630
www.garp.org
About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make
better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies,
academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®)
Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for
professionals of all levels. www.garp.org.
10 | 
© 2014 Global Association of Risk Professionals. All rights reserved.