Download BBA April 2008 - RePEc: Research Papers in Economics

Automated stress tests for
econometric models
Presentation to
Stata 2008 Fall North American Users Group Meeting
by
Roy J. Epstein, PhD
Adjunct Professor of Finance, Boston College
November 14, 2008
Need for Greater Quality Control and
Transparency in Applied Econometrics

Econometric models now part of mainstream in
policy analysis, legal proceedings, and other
advocacy situations

But assessing reliability of estimation results often
difficult


Lack of transparencyRisk of bad decisions
Enhanced regression software—a virtual referee—is
key to improving the practice of econometrics
2
Roy J. Epstein PhD
Regression Software as Virtual Referee
for Econometric “Best Practices”


Journal referee process and professional dialogues
routinely probe effects of different:

Explanatory variables

Data

Statistical procedures
Similar quality control process can/should be built
into regression software

Promotes neutral and transparent assessment of reliability
3
Roy J. Epstein PhD
Automating Regression Stress Tests



Software can anticipate reasonable
perturbations of a model
Program then can assess sensitivity of original
regression to the perturbations
Can be standard part of regression output

Much as we now expect to see t-statistics
4
Roy J. Epstein PhD
Extending Stata for Stress Tests




Macro facility allows design of highly flexible
testing strategy
Can implement as stand-alone post-estimation
routine
Probably more advantageous to incorporate as
part of standard output
Can reflect ongoing professional discussion of
“best practice” stress tests in econometrics
5
Roy J. Epstein PhD
A Taxonomy of Stress Tests

Algebraic sign of
regression coefficient

Magnitude

Statistical significance

Panel models:



Validity of pooling
assumptions

Temporal aggregation

Lagged variables

Dropping variables

Outliers

Functional form

Weighting
Cross-sectional error
correlation
Autocorrelation
6
Roy J. Epstein PhD
Stata Command (Prototype)
stresstest price focuscoef cost realgdp i.firm
i.density i.month, timeseries(year month)
panel(firm density) w(pounds);
Based on “regress” command syntax
 Parameters identify time, panel, and weighting
variables as appropriate
 Implemented as conventional Stata 10 ado file

7
Roy J. Epstein PhD
Case Studies
Conwood v. US Tobacco

Econometric analysis of tobacco sales

Logic of plaintiff model was such that the more
negative the estimated slope, the greater the
damages.

$1 billion claimed damages, after trebling

Sustained after review by Supreme Court

Timely use of stress tests probably would
have vindicated defendant
9
Roy J. Epstein PhD
The Conwood Data and Regression
Line
Washington, DC
10
Roy J. Epstein PhD
Stress Test Easily Identifies Effect of
Outlier
Potential Outliers Stress Tests
.1
.2
Estimated Coefficient for Conwood 1984 Share
0
Negative estimated slope,
$1 billion damages
-.2
-.1
Positive estimated slope,
(excludes Washington, DC)
Zero damages
Base Regression
Conwood_84 (trimmed)
Note: Stress test trims each indicated explanatory variable at 5th and 95th percentile
11
Roy J. Epstein PhD
Confidential Antitrust Case




Alleged price-fixing conspiracy
Panel model of average prices (by firm,
product, month)
Nearly 100 control variables
Claimed damages of $330 million based on
“statistically significant” coefficient of
dummy variable for conspiracy period
12
Roy J. Epstein PhD
Why the Virtual Referee Would Have
Helped

Plaintiff expert:





Did not test validity of pooling data for 4 different firms
Did not report results from aggregated model but admitted
having estimated it
Ignored serial correlation in the residuals
Did not report effect of excluding control variables with
dubious justification
Stress tests quickly cut through complexity of model
to identify serious reliability problems
13
Roy J. Epstein PhD
Stress Tests Reveal High Sensitivity to
Pooling Data from Different Firms
Pooling Stress Tests
0
.05
.1
.15
.2
Estimated Coefficient for Alleged Overcharge
Base Regression
Goodman
Sosa
Dakota
Nano
Note: Stress test performs separate regression for each indicated panel unit
14
Roy J. Epstein PhD
Stress Tests Reveal High Sensitivity to
Aggregating Across the Panel
Cross-Sectional Aggregation Stress Tests
0
.02
.04
.06
.08
Estimated Coefficient for Alleged Overcharge
Base Regression
firm and density
density
Note: Stress test aggregates over indicated levels of the panel
15
Roy J. Epstein PhD
Stress Tests Reveal High Sensitivity to
Autocorrelation
Autocorrelation and WLS Stress Tests
0
.02
.04
.06
.08
Estimated Coefficient for Alleged Overcharge
Base Regression
WLS Correction
AR(1) Correction, rho=.757
16
Roy J. Epstein PhD
Stress Tests Reveal High Sensitivity to
Presence of Certain Variables
Dropping Variables Stress Tests
0
.02
.04
.06
.08
.1
Estimated Coefficient for Alleged Overcharge
Base Regression
Housing Prices
Materials Cost
GDP, 1994 Dummy
Asian Prices, Materials Cost
GDP
Capacity Util.
Apparel Prices, Housing Prices
Nylon Prices, Materials Cost
Note: Stress test performs separate regressions omitting indicated variables
17
Roy J. Epstein PhD
Summary




Software designers have important role in
improving the practice of econometrics
Regression results not reliable until
sensitivities identified and explained
Software can and should automatically invoke
appropriate best practice stress tests
Software will act as virtual referee
18
Roy J. Epstein PhD
For Further Information…
Roy J. Epstein, PhD
Expert economic analysis for complex litigation
1280 Massachusetts Ave., 2nd Fl.
Cambridge, MA 02138
[email protected]
(617) 489-3818
Adjunct Professor of Finance, Boston College