Download Logistic regression

Advanced Models and Methods
in Behavioral Research
• Chris Snijders
• [email protected]
ToDo:
Studyweb!
Enroll in 0a611
• 3 ects
• http://www.chrissnijders.com/ammbr (=studyguide)
• literature: Field book + separate course material
• laptop exam (+ assignments)
Advanced Methods and Models in Behavioral Research – 2011/2012
The methods package
• MMBR (6 ects)
– Blumberg: algemeen: vraagstelling, betrouwbaarheid,
validiteit etc
– Field: SPSS: factor analyse, multiple regressie, ANcOVA,
sample size etc
• AMMBR (3 ects)
- Field (deels): logistische regressie
- literatuur via website:
conjoint analysis  multi-level regression
Advanced Methods and Models in Behavioral Research – 2011/2012
Models and methods: topics
• t-test, Cronbach's alpha, etc
• multiple regression, analysis of (co)variance and
factor analysis
• logistic regression
• conjoint analysis / repeated measures
– Stata next to SPSS
– “Finding new questions”
– Practice data collection (a bit)
In the background:
“now you should be able to do it on your own”
Advanced Methods and Models in Behavioral Research – 2011/2012
Methods in brief (1)
• Logistic regression: target Y, predictors Xi.
Y is a binary variable (0/1).
-
Why not just multiple regression?
Interpretation is more difficult
goodness of fit is non-standard
...
Advanced Methods and Models in Behavioral Research – 2011/2012
Methods in brief (2)
• Conjoint analysis
Underlying assumption: for
each user, the "utility" of a
product can be written as
-10 Euro p/m
- 2 years fixed
- free phone
- ...
How attractive is this
offer to you?
U(x1,x2, ... , xn) = c0 + c1 x1 + ... + cn xn
Advanced Methods and Models in Behavioral Research – 2011/2012
Conjoint analysis as an “in between method”
Between
Which phone do you like and why?
What would your favorite phone be?
And:
Let’s keep track of what people buy.
Advanced Methods and Models in Behavioral Research – 2011/2012
Coming up with new ideas (3)
“More research is necessary”
But on what?
YOU: come up with sensible new
ideas, given previous research
Advanced Methods and Models in Behavioral Research – 2011/2012
Stata next to SPSS
•
It’s just better
•
Multi-level regression
is much easier than in
SPSS
•
It’s good to be
exposed to more than
just a single statistics
package (your knowledge
(faster,
better written, more
possibilities, better
programmable …)
should not be based on
“where to click” arguments)
•
More stable
•
Supports OSX as
well… (anybody?)
Advanced Methods and Models in Behavioral Research – 2011/2012
(I think)
But …
• Output less “polished”
• It takes some extra work
to get you started
• The Logistic Regression
chapter in the Field book
uses SPSS (but still readable
for the larger part)
• (and it’s not campus
software, but subfaculty
software)
• Installation …
Advanced Methods and Models in Behavioral Research – 2011/2012
Advanced Methods and Models
in Behavioral Research
Make sure to
• enroll in studyweb (0a611)
• Read the Field chapter on logistic regression
Advanced
Advanced
Methods
Methods
and Models
and Models
in Behavioral
in Behavioral
Research
Research
– 2008/2009
– 2011/2012
10
Logistic Regression Analysis
That is: your Y variable is 0/1: now what?
The main points
1.
Why do we have to know and sometimes use logistic
regression?
2.
What is the underlying model? What is maximum
likelihood estimation?
3.
Logistics of logistic regression analysis
1.
2.
3.
4.
4.
Estimate coefficients
Assess model fit
Interpret coefficients
Check residuals
An SPSS example
Advanced Methods and Models in Behavioral Research – 2011/2012
Suppose we have 100 observations with information
about an individuals age and wether or not this indivual
had some kind of a heart disease (CHD)
ID
age
CHD
1
2
3
4
…
98
99
100
20
23
24
25
0
0
0
1
64
65
69
0
1
1
A graphic representation of the data
CHD
Age
Let’s just try regression analysis
pr(CHD|age) = -.54 +.0218107*Age
... linear regression is not a suitable model for probabilities
pr(CHD|age) = -.54 +.0218107*Age
In this graph for 8 age groups, I plotted the probability of
having a heart disease (proportion)
A nonlinear model is probably better here
Something like this
This is the logistic regression model
Pr(Y | X ) 
1
1  e (b0 b1 X1 1 )
Predicted probabilities are always between 0 and 1
Pr(Y | X ) 
1
1  e (b0 b1 X1 1 )
similar to classic regression
analysis
Side note: this is similar to MMBR …
Suppose Y is a percentage (so between 0 and 1).
Then consider
…which will ensure that the estimated Y will vary between 0 and 1
and after some rearranging this is the same as
Advanced Methods and Models in Behavioral Research – 2011/2012
… (continued)
And one “solution” might be:
- Change all Y values that are 0 to 0.001
- Change all Y values that are 1 to 0.999
Now run regression on log(Y/(1-Y)) …
… but that doesn’t work so well …
Advanced Methods and Models in Behavioral Research – 2011/2012
Logistics of logistic regression
1.
2.
3.
4.
How do we estimate the coefficients?
How do we assess model fit?
How do we interpret coefficients?
How do we check regression assumptions?
Kinds of estimation in regression
• Ordinary Least Squares (we fit a line through a cloud
of dots)
• Maximum likelihood (we find the parameters that are
the most likely, given our data)
We never bothered to consider maximum likelihood in
standard multiple regression, because you can show
that they lead to exactly the same estimator.
OLS does not work well in logistic regression, but
maximum likelihood estimation does …
Advanced Methods and Models in Behavioral Research – 2011/2012
Maximum likelihood estimation
• Method of maximum likelihood yields values
for the unknown parameters which maximize
the probability of obtaining the observed set
of data.
Pr(Y | X ) 
1
1 e
 ( b0  b1 X 1 1 )
Unknown parameters
Maximum likelihood estimation
• First we have to construct the likelihood
function (probability of obtaining the
observed set of data).
Likelihood = pr(obs1)*pr(obs2)*pr(obs3)…*pr(obsn)
Assuming that observations are independent
Log-likelihood
• For technical reasons the likelihood is
transformed in the log-likelihood (then you
just maximize the sum of the logged
probabilities)
LL= ln[pr(obs1)]+ln[pr(obs2)]+ln[pr(obs3)]…+ln[pr(obsn)]
Note: optimizing log-likelihoods is difficult
• It’s iterative (“searching the landscape”)
•  it might not converge
•  it might converge to the wrong answer
Advanced Methods and Models in Behavioral Research – 2011/2012
Estimation of coefficients: SPSS Results
Pr(Y | X ) 
1
1 e
 ( 5.3.11 X 1 )
Variables in the Equation
B
Step 1a
age
Constant
S.E.
Wald
df
Sig.
Exp(B)
,111
,024
21,254
1
,000
1,117
-5,309
1,134
21,935
1
,000
,005
a. Variable(s) entered on step 1: age.
Pr(Y | X ) 
1
1 e
 ( 5.3.11 X 1 )
This function fits very well, other values of b0 and b1 give worse results
Pr(Y | X ) 
1
1  e ( 5.3.11X 1 )
Illustration 1: suppose we chose .05X instead of .11X
Pr(Y | X ) 
1
1  e ( 5.3.05 X 1 )
Illustration 2: suppose we chose .40X instead of .11X
Pr(Y | X ) 
1
1  e ( 5.3.40 X1 )
Logistics of logistic regression
• Estimate the coefficients
• Assess model fit
– Between model comparisons
– Pseudo R2 (similar to multiple regression)
– Predictive accuracy
• Interpret coefficients
• Check regression assumptions
Model fit:
comparisons between models
The log-likelihood ratio test statistic can be used to
test the fit of a model
  2[ LL( New)  LL(baseline )]
2
The test statistic has a
chi-square distribution
full model
reduced model
37
Between model comparisons:
likelihood ratio test
  2[ LL( New)  LL(baseline )]
2
full model
P(Y ) 
1
1 e
 ( b0  b1 X 1 )
reduced model
1
P(Y ) 
1  e (b0 )
The model including only an intercept
Is often called the empty model. SPSS uses this
model as a default.
Between model comparison: SPSS output
 2  2LL( New)  2LL(baseline )]
Omnibus Tests of Model Coefficients
Chi-square
Step 1
df
Sig.
Step
29,310
1
,000
Block
29,310
1
,000
Model
29,310
1
,000
Model Summary
Step
1
-2 Log likelihood
107,353a
Cox & Snell R
Nagelkerke R
Square
Square
,254
,341
a. Estimation terminated at iteration number 5 because
parameter estimates changed by less than ,001.

This is the test statistic,
and it’s associated
significance
Overall model fit
pseudo R2
log-likelihood of the model
that you want to test
 2 LL( Model )
R LOGIT 
 2 LL( Empty)
2
Just like in multiple
regression, pseudo
R2 ranges 0.0 to 1.0
– Cox and Snell
• cannot theoretically
reach 1
– Nagelkerke
log-likelihood of model
before any predictors were
entered
• adjusted so that it
can reach 1
NOTE: R2 in logistic regression tends to be (even) smaller than in multiple regression
40
Overall model fit: Classification table
Classification Table
a
Predicted
chd
Percentage
Observed
Step 1
chd
0
1
Correct
0
45
12
78,9
1
14
29
67,4
Overall Percentage
74,0
a. The cut value is ,500
We correctly predict 74% of our observations
41
Overall model fit: Classification table
Classification Table
a
Predicted
chd
Percentage
Observed
Step 1
chd
0
1
Correct
0
45
12
78,9
1
14
29
67,4
Overall Percentage
74,0
a. The cut value is ,500
14 cases had a CHD while according to our model
this shouldnt have happened
42
Overall model fit: Classification table
Classification Table
a
Predicted
chd
Percentage
Observed
Step 1
chd
0
1
Correct
0
45
12
78,9
1
14
29
67,4
Overall Percentage
74,0
a. The cut value is ,500
12 cases didn’t have a CHD while according to our model
this should have happened
43
Logistics of logistic regression
• Estimate the coefficients
• Assess model fit
• Interpret coefficients
– Direction
– Significance
– Magnitude
• Check regression assumptions
Interpreting coefficients: direction
We can rewrite our model as follows:
p(Y ) =
1
1+ e-(b0 +b1X1 +...+bn Xn )
e(b0 +b1X1 +...+bn Xn )
=
1+ e(b0 +b1X1 +...+bn Xn )

45
Interpreting coefficients: direction
• original b reflects changes in logit: b>0 implies positive relationship
p( y )
logit  ln
 b0  b1 x1  b2 x2  ...  bn xn
1  p( y )
• exponentiated b reflects the changes in odds: exp(b) > 1 implies a
positive relationship
46
3. Interpreting coefficients: magnitude
• The slope coefficient (b) is interpreted as the rate of change in
the "log odds" as X changes … not very useful.
p( y )
logit  ln
 b0  b1 x1  b2 x2  ...  bn xn
1  p( y )
• exp(b) is the effect of the independent variable on the odds,
more useful for calculating the size of an effect
p( y )
b0
bn xn
b1 x1
b2 x2
Odds 
 e  e  e  ...  e
1  p( y )
47
Magnitude of association: Percentage change in odds
 prob event 

Oddsi  
 1  prob event 
Probability
Odds
25%
0.33
50%
1
75%
3
Magnitude of association
Variables in the Equation
B
Step 1a
age
Constant
S.E.
Wald
df
Sig.
Exp(B)
,111
,024
21,254
1
,000
1,117
-5,309
1,134
21,935
1
,000
,005
a. Variable(s) entered on step 1: age.
• For the age variable:
– Percentage change in odds = (exponentiated coefficient – 1) * 100 = 12%, or “the
odds times 1,117”
– A one unit increase in age will result in 12% increase in the odds that the person will
have a CHD
– So if a soccer player is one year older, the odds that (s)he will have CHD is 12%
higher
Another way to get an idea of the size of effects:
Calculating predicted probabilities
Pr(Y | X ) 
1
1 e
 ( 5.3.11 X 1 )
For somebody of 20 years old, the predicted probability is .04
For somebody of 70 years old, the predicted probability is .91
But this gets more complicated
when you have more than a single X-variable
Pr(Y | X) =
1
1+ e
-(-5.3+.11X1+1*X2 )
(see blackboard)
Conclusion: if you consider the effect of a variable on
the predicted probability, the size of the effect of X1
depends on the value of X2!
Advanced Methods and Models in Behavioral Research – 2011/2012
Testing significance of coefficients
•
In linear regression
analysis this statistic is
used to test
significance
•
In logistic regression
something similar
exists
•
however, when b is
large, standard error
tends to become
inflated, hence
underestimation (Type
II errors are more
likely)
estimate
b
Wald 
SE b
t-distribution
standard error of estimate
Note: This is not the Wald Statistic SPSS presents!!!
Interpreting coefficients: significance
• SPSS presents
2
b
Wald 
2
SE b
• While Andy Field thinks SPSS presents this (at least in the 2nd
version of the book):
b
Wald 
SE b
Advanced Methods and Models in Behavioral Research – 2011/2012
Logistic regression
• Y = 0/1
• Multiple regression (or ANcOVA) is not right
• You consider either the odds or the log(odds)
• It is estimated through “maximum likelihood”
• Interpretation is a bit more complicated than normal
Advanced Methods and Models in Behavioral Research – 2011/2012
Advanced Methods and Models
in Behavioral Research
Make sure to
• enroll in studyweb (0a611)
• Read the Field chapter on logistic regression
Advanced
Advanced
Methods
Methods
and Models
and Models
in Behavioral
in Behavioral
Research
Research
– 2008/2009
– 2011/2012
56
Advanced Methods and Models in Behavioral Research – 2011/2012
Logistics of logistic regression
•
•
•
•
Estimate the coefficients
Assess model fit
Interpret coefficients
Check regression assumptions
Checking assumptions
• Influential data points & Residuals
– Follow Samanthas tips
• Hosmer & Lemeshow
– Divides sample in subgroups
– Checks whether there are differences between observed and
predicted between subgroups
– Test should not be significant, if so: indication of lack of fit
Hosmer & Lemeshow
Test divides sample in subgroups, checks whether
difference between observed and predicted is about
equal in these groups
Test should not be significant (indicating no difference)
Examining residuals in lR
1. Isolate points for which the model fits poorly
2. Isolate influential data points
Residual statistics
Cooks distance
Prediction for j from all
observations
Number of parameter
Prediction for j for
observations excluding
observation i
Means square error
Illustration with SPSS
• Penalty kicks data, variables:
– Scored: outcome variable,
• 0 = penalty missed, and 1 = penalty scored
– Pswq: degree to which a player worries
– Previous: percentage of penalties scored by a particulare
player in their career
64
SPSS OUTPUT Logistic Regression
Case Processing Summary
a
Unweighted Cas es
Selected Cases
Included in Analysis
Mis sing Cas es
Total
Uns elected Cas es
Total
N
75
0
75
0
75
Percent
100,0
,0
100,0
,0
100,0
a. If weight is in effect, s ee class ification table for the total
number of cas es .
Dependent Variable Encoding
Original Value
Internal Value
Mis sed Penalty
0
Scored Penalty
1
Tells you something
about the number of
observations and
missings
65
this table is based on
the empty model, i.e. only
the constant in the model
Block 0: Beginning Block
Classification Tablea, b
Predicted
Step 0
Obs erved
Res ult of Penalty
Kick
Mis sed Penalty
Scored Penalty
Res ult of Penalty Kick
Mis sed
Scored
Penalty
Penalty
0
35
0
40
Overall Percentage
Percentage
Correct
,0
100,0
53,3
1
P(Y ) 
1  e (b0 )
a. Cons tant is included in the model.
b. The cut value is ,500
Variables in the Equation
Step 0
Cons tant
B
,134
S.E.
,231
Wald
,333
df
1
Sig.
,564
Variables not in the Equation
Step
0
Variables
Overall Statistics
previous
pswq
Score
34,109
34,193
41,558
df
1
1
2
Sig.
,000
,000
,000
66
Exp(B)
1,143
these variables
will be entered
in the model
later on
Block is useful to check significance of
individual coefficients, see Field
Block 1: Method = Enter
Omnibus Tests of Model Coefficients
Step 1
Step
Block
Model
Chi-s quare
54,977
54,977
54,977
df
2
2
2
Sig.
,000
,000
,000
this is the test statistic
 2  2[ LL( New)  LL(baseline )]
Note: Nagelkerke
is larger than Cox
after dividing by -2
Model Summary
New
model
Step
1
-2 Log
Cox & Snell
likelihood
R Square
a
48,662
,520
Nagelkerke
R Square
,694
a. Estimation terminated at iteration number 6 because
parameter estimates changed by less than ,001.
67
Block 1: Method = Enter (Continued)
Classification Tablea
Predicted
Step 1
Obs erved
Res ult of Penalty
Kick
Res ult of Penalty Kick
Mis sed
Scored
Penalty
Penalty
30
5
7
33
Mis sed Penalty
Scored Penalty
Overall Percentage
Percentage
Correct
85,7
82,5
84,0
a. The cut value is ,500
Predictive accuracy has
improved (was 53%)
Variables in the Equation
Step
a
1
previous
pswq
Cons tant
B
,065
-,230
1,280
S.E.
,022
,080
1,670
Wald
8,609
8,309
,588
df
1
1
1
Sig.
,003
,004
,443
Exp(B)
1,067
,794
3,598
a. Variable(s ) entered on step 1: previous , ps wq.
estimates
standard error
estimates
significance
based on
Wald statistic
change in odds
68
How is the classification table constructed?
# cases not predicted
corrrectly
Classification Tablea
Predicted
Step 1
Obs erved
Res ult of Penalty
Kick
Mis sed Penalty
Scored Penalty
Res ult of Penalty Kick
Mis sed
Scored
Penalty
Penalty
30
5
7
33
Overall Percentage
a. The cut value is ,500
# cases not predicted
corrrectly
Variables in the Equation
Step
a
1
previous
pswq
Cons tant
B
,065
-,230
1,280
S.E.
,022
,080
1,670
Wald
8,609
8,309
,588
Percentage
Correct
85,7
82,5
84,0
df
1
1
1
Sig.
,003
,004
,443
Exp(B)
1,067
,794
3,598
a. Variable(s ) entered on step 1: previous , ps wq.
Pred. P(Y ) 
1
1  e (1, 28 0, 065* previous0, 230* pswq)
69
How is the classification table constructed?
Pred. P(Y ) 
1
1  e (1, 28 0, 065* previous0, 230* pswq)
pswq
previous
scored
18
56
1
Predict.
prob.
.68
17
35
1
.41
20
45
0
.40
10
42
0
.85
70
How is the classification table constructed?
pswq
previo
us
scored
18
17
20
10
56
35
45
42
1
1
0
0
Predict predict
. prob.
ed
.68
.41
.40
.85
1
0
0
1
Classification Tablea
Predicted
Step 1
Obs erved
Res ult of Penalty
Kick
Mis sed Penalty
Scored Penalty
Res ult of Penalty Kick
Mis sed
Scored
Penalty
Penalty
30
5
7
33
Overall Percentage
a. The cut value is ,500
71
Percentage
Correct
85,7
82,5
84,0