Download document 8688801

Outline
1. Challenges for early warning models in the policy
process
2. The evaluation approach: What is a good early
warning model?
3. Types of early warning models: some examples
4. Caveats on thresholds to activate instruments
5. Conclusions
2
www.ecb.europa.eu ©
Challenges for
Early Warning Models in the
Policy Process
3
www.ecb.europa.eu ©
Challenges to overcome for EWMs…
• Eichengreen (2002): “Economic forecasting is like weather
forecasting except that our knowledge of the underlying science is
less complete”,
and with respect to early warning models even more scepticism:
• Structural relationships interact in nonlinear and state-contingent
way
• Complex systems often have multiple equilibria sensitive to small
perturbations
• Circularity, as forecast might affect outcome (Goodhart’s Law: “Any
observed statistical regularity will tend to collapse once pressure is
placed upon it for control purposes.” ( ~ Lukas critique)
=> The role of fundamentals might be difficult to detect
4
www.ecb.europa.eu ©
Further challenges to overcome…
• Out-of-sample failures due to in- sample over-fitting and variable
selection bias
• Too few crises events / degrees of freedom
• After 1997 Asian crisis wave of optimism regarding new EWMs
(data, methodologies, policy makers’ attention) very much like
today…
So what is different today to increase our level of confidence in this
type of models?
5
www.ecb.europa.eu ©
Reasons for optimism…
• Out of sample methods for model selection and performance (e.g.
cross-validation in random forests or recursive validation in
discrete choice models, Lasso)
• (Pseudo) real time concepts
• Predict vulnerable states, not crisis date (e.g. “crisis within next 3
years”)
• Focused EWMs for specific policy instruments (e.g. CCB analysis)
• Data availability; soon e.g. Credit registers, Corep/Finrep,
Securities holdings statistics, Liquidity indicators (LCR, NSFR)
• Evaluation methods more refined: AUROC, Loss function,
Usefulness measures [forget noise to signal ratio]
• Some key studies establishing that “not every time is different”.
Reinhart/Rogoff (2009) and Schularick/Taylor (2009, 2014): It’s
credit and it’s housing. See also:6 ESRB OP No. 5.
www.ecb.europa.eu ©
The Evaluation Approach:
What is a good early warning
model?
7
www.ecb.europa.eu ©
Types of Early Warning Models:
Some examples
10
www.ecb.europa.eu ©
Types of EWMs
• Three (operating) approaches to extract signals from indicators
to predict crises:
– Signalling approach: Signal is extracted directly from the indicator
when breaching a threshold. Both uni-variate and multi-variate
approaches possible.
– Discrete-choice approach: A probit or logit mapping function
transforms the variables into a continuous indicator variable. If the
predicted probability exceeds a specified threshold a signal is
issued.
– Decision trees / Random Forest: A partitioning algorithm that
recursively identifies the indicators and the (optimal) thresholds that
best split the sample into tranquil and pre-crises periods / then do
some “bragging”.
– Work in progress: artificial neural networks, support vector
machines
11
www.ecb.europa.eu ©
Discrete choice example: The bank early warning
model
Note: The aggregation is done by weighing the bank-specific distress probabilities by the respective bank shares in total
euro area bank assets. The decomposition of individual distress probabilities into the different factors is done by using the
(relative) distress probabilities that would prevail if all other variable blocks were set to their mean values.
Source: Lang, Peltonen, Sarlin (2015), A Framework for Early-Warning Modeling, ECB, mimeo.
13
www.ecb.europa.eu ©
Real-time prediction of the Bank Early Warning Model
• Probability above threshold in 2007Q2 and below in 2014Q2
Note: Based on a recursive out-of-sample exercise starting in 2006Q1. A bank is signalled to be vulnerable if the predicted probability
exceeds the optimal signalling threshold based on a policymaker with preference parameter of µ=0.9. The aggregation is done by
weighing the bank-specific distress probabilities by the respective bank shares in total euro area bank assets.
Source: Lang, Peltonen, Sarlin (2015), A Framework for Early-Warning Modeling, ECB, mimeo.
14
www.ecb.europa.eu ©
Caveats on thresholds to
activate instruments
16
www.ecb.europa.eu ©
Problems in using models for activating
instruments
• Model derived thresholds depend on policy makers preferences
• Indicators/models derived from pooled analysis might not “work”
for individual countries
• Even if they do, pooled thresholds might not be reasonable for
individual countries
• No simple solution for country specific thresholds: A) apply pooled
optimal percentile to country series B) group countries =>
FSC/MPAG and ESRB CCB groups working on this (and other
methods)
⇒Guided discretion (and suite of models) approaches
Enter presentation title by changing the footer.
17
www.ecb.europa.eu ©
Pooled thresholds with different country groups
Indicator
Euro area
Rank
T
Advanced
Economies
AUC Rank
T
"Property"
AUC Rank
T
AUC
Bank credit/GDP gap
1
2.9 0.86
1
2.7 0.83
3
3.0 0.82
Total credit to HH/GDP gap
3
1.0 0.78
2
1.8 0.81
1
2.7 0.82
Basel gap
2
2.9 0.81
3
2.7 0.80
9
2.8 0.77
Real total credit growth HH (3year)
4
17.5 0.77
5
17.5 0.78
4
22.9 0.79
“Euro area”: EMU 18
“Advanced economies” according to the ESRB OP definition: AT, BE, DE, DK, ES, FI,
FR, GR, GB, IE, IT, NL, PT, SE
“Property”: Countries with significant historical property sector developments: BE,
DK, ES, FR, GB, IE, NL, PT, SE
Source: Welz (2014), ECB mimeo, July.
19
www.ecb.europa.eu ©
Conclusions
20
www.ecb.europa.eu ©
Conclusions
• There are simple Early Warning Models out there which would
have predicted the financial crisis like many other past crises
[caveat: also significant number of false alarms (“below but close
to” 30%))]
• Policy makers preferences can and should be explicitly considered
in the evaluation of EWMs (loss function, partial AUROC).
• A suite of models approach seems necessary (like for monetary
policy)
• The guided discretion principle to trigger policy instruments is
recommendable.
• But importantly: EWMs have the potential to support overcoming
the “this time is different”-syndrom by shifting the burden of proof
to authorities with inaction bias (just in case the latter exist…)
21
www.ecb.europa.eu ©