Download Linear Models of Judgment

Linear Models of Judgment
 Judgment
vs. choice
 Multiattribute model of judgment
 Actuarial model of the environment
 Experts and computers
 Bootstrapping models
Desirability
Cost
Mileage
Safety
Price
Repairs
Crash Test
ABS
Airbags
Fun
Size
Looks
Handling
External
Marque
Features
Trunk
Cabin
Judgment vs. Choice
 Judgment
= assign a score or category
e.g., How much would you pay for a
one-week trip to Aspen?
How much do you like Bill Clinton?
What will the price of Intel be in 6
months?
 Choice = pick from a set of alternatives
e.g., which car? investment? job?
Multiattribute Choice Model
 Choice
= select most desirable
 Desirability is judged from attributes
 Attributes can be FACTS (e.g., price),
COMPOSITES (e.g., safety), or
subjective VALUES (e.g., prestige)
 There is often a hierarchical structure to
attributes and judgments
 How to make tradeoffs, e.g., weight &
add
Desirability of a Car
Desirability
Cost
Mileage
Safety
Price
Repairs
Crash Test
ABS
Airbags
Fun
Size
Looks
Handling
External
Marque
Features
Trunk
Cabin
Lens Model
 Judgment
(Ys) is an attempt to represent
or predict the environment from cues
 There is a criterion (Ye) that allows us to
estimate correctness of the judgment
c
u
Ye
e
s
Ys
Models of Decision Makers
 Slovic’s
study of two stockbrokers:
A: near term prospects, P/E, earnings qtly trend
B: earnings yearly trend, P/E, profit margin trend
 Ratings
of business schools:
USNWR: reputation with academics, reputation with
CEOs, selectivity, placement
BusWeek: recruiters rate analytics, teaming, global;
graduates rate teaching, curriculum, placement
Actuarial Environmental Models
Major Retailer’s Credit Scoring Table


Occupation
clergy
executive
professional
student
teacher
unemployed
no answer

46
62
62
46
46
33
47
Job Tenure
< .5 years
.5 - 5.5
5.5 - 8.5
8.5 - 15.5
> 15.5 years
31
24
26
31
39
Capon, J. Marketing, 1982, 46,
82-91.
Older economists make more extreme forecasts
Comparisons Using Models
 How
consistent are individuals?
 How consensual are experts?
 How accurate are judges? (Ye vs. Ys)
 What are judges doing? (Ysm)
 What predicts the criterion? (Yem)
 How good are our models?
 Do judges understand the
environment? (Yem vs. Ysm)
Graduate Admissions Example
 Ys
= judgment of admissions committee
(1 to 5 scale)
 Ye = faculty ratings of performance
 Ysm = prediction model of judgments
= -4.17 +.0032*GRE +1.02*GPA +.0791*QI
 Yem= actuarial model of performance
= -.71 +.0006*GRE +.76*GPA +.2518*QI
Admission and Job Interviews
 Harvard
Business School stopped
conducting interviews
– Are interviews accurate?
– Are interviews overweighted?
– What is their proper role?
 HBS
no longer uses the GMAT
 HBS criteria: academics and character
Advantages of Models
 Makes
strategy explicit
 Can see how experts vary
 Train new judges
 Learn about environments
 Enhance or replace experts
 Can use the model when expert gone
Judges vs. Environment
 Which
should be more accurate, expert
judges or actuarial models?
 Judges have their experience, ability to
use cues in complex ways
 Actuarial models are simple, typically
linear in form, consistent
Judges vs. Actuarial Model
Actuarial
Model
.95
Task
Credit scoring
Judge
.80
Stock analysis
.23
.80
Personnel
.35
.57
Cancer survival
-.01
.35
Graduate GPA
.33
.69
Why Don’t Experts Do Better?
 They
have the wrong rules
 They don’t use their rules
- distractions
- fatigue, boredom
- “exceptions”
- unable to make tradeoffs
Bootstrapping Models
 If
intuitive decision makers have good
rules but fail to use them consistently,
can we separate signal from noise?
 Consensus of judges (see groups later)
 Model of a judge (bootstrapping)
Judgment = Linear + Nonlinear + Noise
 What wins: Judge vs. linear model?
A Tale of Three Models
Bootstrap Actuarial
Model
Model
.85
.95
Task
Credit scoring
Judge
.80
Stock analysis
.23
.29
.80
Personnel
.35
.46
.57
Cancer survival -.01
.13
.35
Graduate GPA
.50
.69
.33
Some Typical Results
Some tasks are much harder than others
 Actuarial models almost always win
 Bootstrapping works!
 Linear models correlate with any monotonic
function, work well when there is noise,
positively correlated cues, work with random
or unit weights
 To improve on linear models, you need lots of
data

Experts and Models
 What
do experts do best?
 What do computers do best?
 How can they be combined?
 Should we give the model to the expert
or give the expert to the model?
Batterymarch Example
 Stock
portfolio company
 Manage $12 Billion with 37 employees
 Experts identify variables, suggest
rules, design tests, deal with clients
 Computer keeps databases, runs tests of
rules, buys and sells stocks
 10-12 rules identify attractive stocks
Working With the Political Lens:
Separating Facts and Values
 Selecting
a bullet for Denver Police
- police want to immobilize suspects
- community concerned about injuries
- experts testify on each side
 What kinds of information are needed?
 How should this decision be made?
A Frame for Conflict Resolution
Facts
weight
speed
shape
etc.
Values
Desirability
Injury potential
Stopping power
Threat to
bystanders
Denver Bullet Resolution
 Experts
combine facts into judgments
on each value
 Constituencies compromise on how to
weight the values into overall worth
Injury
Potential
proposed
..............
by Police
. . . . . . P. . . . . . .
............
proposed by
. . . C. . . . . . . . .
Community
...........
Stopping Power
Role of Technical Experts
 Executive
whose daughter had a hip
deformity
 One doctor said, “Wait”
 A second said, “Brace for 6 months”
 The third said, “Operate”
 How would you make this decision?
Your Exercise #1: Job Selection
 What
were the attributes or objectives
of jobs that mattered to you?
 How different were the rankings due to
intuition, weighted linear model, unit
weighted model?
 If the rankings differ, which do you
trust? Why?
 Value-added in the process, not the
numbers