Download Ragveer Brar

Pricing Model Risk & Control
Ragveer Brar
Valuation & Controls Team
September 2013
The views expressed in this presentation are personal and do not necessarily reflect those of BoE or PRA.
Pricing Model & Controls
Contents
•
The Role of Models in Derivative Pricing
•
Front to Back Valuation of Derivatives
•
Model Risk Management Principles
•
Model Development
•
Model Implementation
•
Model Usage
•
Model Inventory
•
Valuation Uncertainty
•
Model Risk and IPV
•
Background Documents / References
Pricing Model & Controls
The Role of Models in Derivative Pricing
What Are Models for
•Banks rely heavily on models in most aspects of financial decision making. They routinely
use models for a broad range of activities, including underwriting credits; valuing exposures,
instruments, and positions; measuring risk; managing and safeguarding client assets;
determining capital and reserve adequacy; and many other activities.
•Our presentation today will focus on models used in derivative pricing.
•Black-Scholes Formula changed the game, by making it faster and more efficient to price
risk. They also tied pricing and risk management to a specific, simplistic mathematical
framework, which does not well represent real-world dynamics for most instruments.
•More and more complex models developed over time:
- Hull-White
- SABR
- Gaussian Copula
- and many more
Pricing Model & Controls
The Role of Models in Derivative Pricing
Model Inputs and Outputs
•Derivative pricing models – Mathematical formulae that transform a number of input
parameters into prices for financial derivatives, often based on some assumptions.
P
a
r
a
m
e
t
e
r
s
•Inputs
Assumptions
underlying
volatility
strike
Models
Prices
risk free rate
–
Quantification of Risks
Parameters: visible vs. invisible; e.g. volatility of underlying, mean reversion, correlation etc.
Assumptions: explicit vs. implicit; e.g. risk free discount rate is LIBOR, efficient market hypothesis
•Outputs
–
Prices: pricing and trading; fair value in financial statements
Risks: risk management; hedging
•“All models are wrong but some are useful.” The use of derivative pricing models inevitably
lead to valuation uncertainty.
Pricing Model & Controls
Front to Back Valuation of Derivatives
What Are Models for
•The valuation process for derivatives is complicated, with various roles and responsibilities
shared by different areas of a bank. Each step in the process brings its own risks, and
requires specific controls to ensure accurate balance sheet values.
•The steps can be broken down into 3 main areas; Model Development, Model
Implementation and Model Usage. For each of these areas specific elements need to be
well executed, and strongly controlled by an independent function:
Process
Elements
Economic Assumptions
Model Development
Mathematical Process
Product Payoff Specification
Engine builds
Model Implementation Parameter Sets
Pricers
Model Choice
Model Parameter Setting
Model Usage
Market Calibration
Adjustments and Reserves
Executor
Quantitative Development
Quantitative Development
Quantitative Development
Quantitative Development / IT
Quantitative Development / IT
Quantitative Development / IT
Quantiative Development / Front Office
Quantiative Development / Front Office
Front Office
Quantiative Development / Front Office
Controller
Model Control
Model Control
Model Control
Model Control
Model Control
Model Control
Model Control
Valuation Control
Valuation Control
Valuation Control
Pricing Model & Controls
Model Risk Management Principles
FED / OCC Guidance
•The Supervisory Guidance on Model Risk
Management issued in April 2011 by Board
of Governors of the Federal Reserve
System (SR 11-7) and the Office of
Comptroller of the Currency (OCC 2011-12)
provides updated regulatory guidance
governing the use of models at all US
national banks.
•The graph summarises the framework
outlined in the Guidance.
•The principles contained therein are broadly
applicable to banks outside the US and
should be considered by other regulators
and market participants.
* Graph taken from The evolution of model risk management, May
2013
Pricing Model & Controls
Model Risk Management Principles
FED/OCC Guidance – Key Messages
•Firms should adopt a holistic model risk management framework, spanning all three lines
of defence.
•Model validation should be seen as a risk management function, and not as a compliance
function.
•Model risk should be primarily mitigated at source through increased formality in Front
Office control frameworks for model development, implementation and usage.
•Residual model risk should be mitigated by independent control functions, through more
real-time model validation and on-going model monitoring.
•Developing and maintaining strong governance, policies and controls over the model risk
framework is fundamentally important to its effectiveness.
•A bank’s internal audit function should assess the overall effectiveness of the model risk
management framework.
Pricing Model & Controls
Model Development
Control Environment
OCC/FED Guidance
•“Model design, theory and logic should be well documented and supported by sound practice, with
special attention to a models weaknesses and limitations”.
•“Comparison with alternative theories and approaches is fundamental to a sound modelling process”.
•Rigorous data analysis should be performed to demonstrate that models are suitable for any given
product-underlying mix.
•Model assumptions should be explicitly stated, tracked and analysed so that users are aware of potential
limitations.
•Model testing should be performed to assess a model’s accuracy, stability and behaviour in a range of
scenarios, including those outside of normal expectations, such as negative interest rates, or extremely
high volatility. Extreme values should be used to test limiting assumptions.
•Model validation should be performed by an independent function, with the requisite knowledge, skills
and expertise to assess the models, as well as the explicit authority to challenge developers.
Additional Comments
•The validation process should test every element of the model development process.
Pricing Model & Controls
Model Development
Common Deficiencies
•Model weaknesses and limitation are often largely ignored when documenting new models.
•Alternative economic theories are very rarely actively explored. Those theories that are
explored are generally just minor tweaks on the same theme. Alternative models are often
not used to compare model results.
•The quality of data analysis is patchy and inconsistent, e.g. very few model validation
teams consider a model’s expected P&L performance.
•Model assumptions are often buried in economic theory or mathematical derivation, and
are not stated clearly, considered by validation, or tracked post-development.
•Model testing is often limited to ‘reasonable’ scenarios, and does not test a model’s ability
to hand stressed market conditions.
•Reporting lines for model validation teams are often unclear due to join/multiple reporting
lines. The reporting line of model validation is inconsistent across industry.
•Model validation often do not test significant parts of the model development process.
Pricing Model & Controls
Model Implementation
Control Environment
OCC/FED Guidance
•Sound systems should be maintained “to ensure data and reporting integrity, together with controls and
testing to ensure proper implementation of models, effective systems integration and appropriate use”.
•Implementation of vendor or third party systems should also be subject to formal model validation.
Additional Comments
•Specific validation should be performed for each implementation of each element of a model, with
special connection to the interactivity between the elements, e.g. implementation of a new engine should
require testing of the impact on all pricers which use that engine.
•Production code should be checked to ensure that the model that has been implemented matches the
model that has been proposed and validated.
•Model libraries should be controlled by independent control functions, and any changes to models
should require formal change protocols. Model developers should not have write access to production
systems.
Pricing Model & Controls
Model Implementation
Common Deficiencies
•Often no consideration is given to data integrity, or the ability to generate accurate reports
when implementing new models, resulting in problems analysing and controlling model
inventory, or errors in model outputs.
•Validation is often restricted to specific elements of a model, with other elements taken for
granted, e.g. where a pricer has worked with one engine or system, it is assumed that it will
work with another.
•Surprisingly often, lack of sufficient checks results in models being implemented which are
erroneous, or different from that proposed by model developers.
•Model libraries are often controlled by the Front Office, with changes not being subject to
formal controls. In some cases, developers have access to production systems as well as
development systems.
•Third party systems are often taken for granted, with very little formal validation performed
by the bank implementing the system.
Pricing Model & Controls
Model Usage
Control Environment
•For each product, there should be one officially approved model. Model Users (Front
Office) should be explicitly restricted in their ability to select alternative models.
•“Scripts” (highly bespoke, low volume models) should be individually validated, and subject
to tight restrictions and limitations.
•“Model parameters” (model specific pricing inputs, typically updated infrequently) such as
mean reversion speeds or barrier shifts should be subject to monitoring, and independent
validation in the same way as other pricing inputs.
•Calibration sets, such as the structure of yield curves and volatility surface should be
locked down, with changes to the sets following model change protocols.
•Calibration results should be independently validated (IPV) as part of the valuation control
process.
•Firms should take a broad view of what counts as a model and requires formal model
validation. This should include “model adjustments” required to match market pricing, and
reserve calculations (such as bid-offer reserves).
Pricing Model & Controls
Model Usage
Common Deficiencies
•Flexibility is often given to Front Office in model choice, which can result in cherry picking of models in
order to maximize valuation / minimize capital.
•“Scripts” are often over-used by traders, don’t exist in core pricing systems, and are not well validated or
controlled.
•Model Parameters are sometimes excluded from formal controls, unmonitored by independent functions
and not subject to regular validation.
•We have seen cases of calibration sets being completely under the control of front office, with the impact
of changes not monitored or controlled independently. This has been used by Front Office as a tool to
mismark their portfolios.
•Model adjustments are often completely controlled by Front Office, and outside of the formal oversight of
independent controllers. These adjustments have built up to be highly material (>£1b) without meaningful
challenge by controllers.
•Reserve methodologies are often set by Front Office, without any validation by independent control,
despite their material impact on balance sheet values of derivatives. Alternatively, they are set by Product
Control, but without any of the model development or validation standards applied to other valuation
models used in the firm.
•Various weaknesses exist in IPV processes across all firms, but they are too many to cover in this
session.
Pricing Model & Controls
Model Inventory
Control Environment
OCC/FED Guidance
•Market dynamics change over time, sometimes dramatically (as seen in interest rate markets in 2008 /
2009). All models should be subject to on-going monitoring, and periodic revalidation to ensure they
maintain acceptable levels of performance, accuracy and stability.
Additional Comments
•Firms should have a model risk framework that attempts to model risk and ensures that senior
management are aware of the materiality of the uncertainty this creates.
•Restrictions and limitations raised during the model validation process should be centrally monitored,
and used to set threshold conditions for formal model reviews, and to ensure Front Office compliance
with model control policies.
•Excessive model adjustments suggest that a model does not accurately represent the market dynamic
for a particular product set. Model adjustments should form part of the formal monitoring and reporting on
model risk. At a minimum model adjustments need to be calculated by an independent control function
each month end, and ideally there should be strict thresholds at which adjustments trigger automatic
model reviews, cuts in risk limits or trading curbs.
Pricing Model & Controls
Model Inventory
Common Deficiencies
•Many firms lack any framework for modelling model risk. Where such frameworks exist,
reporting is often poor and little management attention is given to the risks inherent in
models.
•Periodic review of models is often of a very low standard, with a large proportion of models
going many years without any formal review.
•Some firms lack any framework for monitoring model restrictions and limitations. Amongst
those firms that do monitor restrictions, breaches are often ignored such that there are no
repercussions for trading breaching restrictions.
•Model adjustments are often allowed to build up in an uncontrolled manner, without any
challenge from controllers, or consequences for front office activity booked on those
models.
Pricing Model & Controls
Valuation Uncertainty
Valuation Uncertainty due to Pricing Models
•Valuation Uncertainty is the existence of a range of plausible values for a position or
portfolio of positions, at the reporting date and time. The recognition of the inherent
valuation uncertainty fundamentally changes the derivative valuation from a point estimate
to a range estimate.
•Different pricing models and calibration methods can be used for valuing the same
derivative contract; and different netting methodologies can be used for the same portfolio
of derivatives, leading to different valuations and valuation adjustments. Valuation
uncertainty assessment should include uncertainty due to pricing models.
•In practice, building alternative models and setting up the full range of possible models for
the full range of possible calibration methods and netting methodologies may be too
intensive an approach to be adopted across all model risk quantification.
•Where quantification of model risk by model comparison/re-calibration is not possible an
institution should establish a framework for quantifying model risk with an associated
uncertainty charging structure.
Pricing Model & Controls
Valuation Uncertainty
Regulatory Focus (1)
•In the UK, GenPru 1.3 requires firms to use prudent valuation principles when valuing
trading books (and other assets and liabilities held at fair value) and report their Prudent
Valuation Adjustment (PVA) to the FSA/PRA through the Prudent Valuation Return (Policy
Statement PS12/07).
•One of the considerations for assessing Prudent Valuation Adjustment should be the
assessment of model risk, which include but not limited to –
Impact of using different models
Impact of using different calibration methods
Impact of using different maturity/strike buckets
Impact of using different interpolation/extrapolation methods
Pricing Model & Controls
Valuation Uncertainty
Regulatory Focus (2)
•In Europe, EBA Consultation Paper relating to Draft Regulatory Technical Standards on
prudent valuation under Article 100 of the draft Capital Requirements Regulation was
published in Jul 2013.
•In the CP, model risk is a specific category of Additional Valuation Adjustment that need to
be assessed by firms in the core approach. Institutions shall estimate a model risk AVA for
each valuation model by considering valuation model risk which arises due to the potential
existence of a range of different models or model calibrations.
•In addition, the CP requires at least an annual review of valuation model performance.
There is also a requirement to validate the use of netting schemes for reducing calculations
for derivative exposures. This requirement arises from the concerns on the lack of model
calibration for reserve requirements and risk netting.
•Globally, regulators from other countries are also starting to focus on valuation risk.
Pricing Model & Controls
Model Risk and IPV
Control Environment
•IPV should not just focus on input testing. Output testing should be performed, where
possible, for complex derivatives and even for vanilla derivatives. Output testing should be
used to calibrate model reserves and quantify model limitations.
•Where output testing is not possible, input testing completeness should be performed
thoroughly to ensure no “hidden” parameter remains untested.
•IPV reporting should incorporate valuation uncertainty. Model reserves and their
uncertainty should also be reported. Where products are only input tested, this should be
explicitly shown in management packs to heighten awareness as to the scope of potential
model risk.
Pricing Model & Controls
Model Risk and IPV
Common Deficiencies
Independent Price Verification (IPV) is performed by firms to independently validate the
valuations by Front Office. IPV and model control are intricately linked, through model
inputs and outputs and model reserves as fair value adjustments;
•Input vs. outputs IPV:
- Many banks test input parameters ONLY, e.g. forward points, volatilities, dividend yields etc.
- An effective completeness check is often lacking. All parameters tested? All assumptions tested?
- There can be “hidden” input parameters not tested, e.g. mean reversion, volatility “shift” etc.
- There can be implicit assumptions not tested, e.g. some CMS swap pricing models assume perfect replication by vanilla
swaptions and therefore no other risks, such as a volatility shift factor used by traders.
- Output testing often not performed. Where output testing is not feasible, there is often no reporting and no consideration
for the implication on valuation uncertainty.
•Incorporating model uncertainty in IPV reporting:
- Lack of framework for effective management reporting that links the IPV and valuation uncertainty, including model
uncertainty.
- Lack of framework for linking model reserves to output IPV testing in management reporting.
Pricing Model & Controls
Background Documents / References
•
•
Volatility and Correlation 2nd Edition, Rebonato
Supervisory Guidance on Model Risk Management, Office of the Comptroller of the Currency
(http://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf)
•
Guidance on Model Risk Management, Federal Reserve Board
(http://www.federalreserve.gov/bankinforeg/srletters/sr1107.pdf)
•
Global Model Practice Survey 2011, Deloitte
(http://www.deloitte.com/view/en_GB/uk/industries/financial-services/f2640dfa4d9a5310VgnVCM1000001a56f00aRCRD.htm)
•
•
•
•
•
EBA Consultation Paper - Draft Regulatory Technical Standards
(http://www.eba.europa.eu/-/eba-consults-on-draft-technical-standards-on-prudent-valuation)
FSA Policy Statement 12/7 - Regulatory Prudent Valuation Return, FSA
(http://www.fsa.gov.uk/static/pubs/policy/ps12-07.pdf)
Product Control Findings and Prudent Valuation Presentation, FSA
(http://www.fsa.gov.uk/pubs/other/pcfindings.pdf)
Model Risk Management Survey 2012, Pwc
The evolution of model risk management, Pwc
(http://www.pwc.com/en_US/us/financial-services/regulatory-services/publications/assets/pwc-closer-look-model-risk-management.pdf)