Download Lecture3-Logistic Regression 6-5

HSRP 734:
Advanced Statistical
Methods
June 5, 2008
Introduction
• Categorical data analysis
– multinomial
– 2x2 and RxC analysis
– 2x2xK, RxCxK analysis
• Stratified analysis (CMH) considers the
problem of controlling for other variable
Introduction
• Need to extend to scientific questions of higher
dimension.
• When the number of potential covariates increases,
traditional methods of contingency table analysis
become limited
• One alternative approach to stratified analyses is
the development of regression models that
incorporate covariates and interactions among
variables.
Introduction
• Logistic regression is a form of regression
analysis in which the outcome variable is binary
or dichotomous
• General theory: analysis of variance (ANOVA)
and logistic regression all are special cases of
General Linear Model (GLM)
OBJECTIVES
• To describe what simple and multiple logistic
regression is and how to perform
• To describe maximum likelihood techniques to
fit logistic regression models
• To describe Likelihood ratio and Wald tests
OBJECTIVES
• To describe how to interpret odds ratios for logistic
regression with categorical and continuous
predictors
• To describe how to estimate and interpret predicted
probabilities from logistic models
• To describe how to do the above 5 using SAS
Enterprise
What is Logistic Regression?
• In a nutshell:
A statistical method used to model dichotomous or
binary outcomes (but not limited to) using predictor
variables.
Used when the research method is focused on
whether or not an event occurred, rather than when
it occurred (time course information is not used).
What is Logistic Regression?
• What is the “Logistic” component?
Instead of modeling the outcome, Y, directly, the
method models the log odds(Y) using the logistic
function.
What is Logistic Regression?
• What is the “Regression” component?
Methods used to quantify association between an
outcome and predictor variables. Could be used to
build predictive models as a function of predictors.
What is Logistic Regression?
100
Length of Stay (days)
80
60
40
20
0
0
20
40
60
Age (yrs.)
80
What is Logistic Regression?
100 day Mortality
CHD (Died=1, Alive=0)
1.0
0.8
0.6
0.4
0.2
0.0
20
30
40
50
Age
Age
(yrs.)
60
70
Fig 1. Logistic regression curves for the three drug combinations. The dashed reference line represents the probability of DLT of
.33. The estimated MTD can be obtained as the value on the horizontal axis that coincides with a vertical line drawn through the
point where the dashed line intersects the logistic curve. Taken from “Parallel Phase I Studies of Daunorubicin Given With
Cytarabine and Etoposide With or Without the Multidrug Resistance Modulator PSC-833 in Previously Untreated
Patients 60 Years of Age or Older With Acute Myeloid Leukemia: Results of Cancer and Leukemia Group B Study 9420”
Journal of Clinical Oncology, Vol 17, Issue 9 (September), 1999: 283. http://www.jco.org/cgi/content/full/17/9/2831
What can we use Logistic Regression for?
• To estimate adjusted prevalence rates, adjusted for
potential confounders (sociodemographic or clinical
characteristics)
• To estimate the effect of a treatment on a
dichotomous outcome, adjusted for other covariates
• Explore how well characteristics predict a categorical
outcome
History of Logistic Regression
• Logistic function was invented in the 19th century to
describe the growth of populations and the course of
autocatalytic chemical reactions.
• Quetelet and Verhulst
• Population growth was described easiest by exponential
growth but led to impossible values
History of Logistic Regression
• Logistic function was the solution to a
differential equation that was examined from
trying to dampen exponential population growth
models.
History of Logistic Regression
• Published in 3 different papers around the
1840’s. The first paper showed how the logistic
models agreed very well with the actual course of
the populations of France, Belgium, Essex, and
Russia for periods up to the early 1830’s.
The Logistic Curve
p (probability)
exp  z 
 p 
LOGIT ( p)  ln 

z

p


1  exp  z 
 (1  p) 
p
LOGIT ( p)  z  log
p
(1  p)
z (log odds)
exp( z )
1  exp( z )
Logistic Regression
• Simple logistic regression = logistic regression with
1 predictor variable
• Multiple logistic regression = logistic regression with
multiple predictor variables
•
Multiple logistic regression =
Multivariable logistic regression =
Multivariate logistic regression
The Logistic Regression Model
Logistic Regression:
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


 K X K
Linear Regression:
Y   0  1 X 1   2 X 2 
 K X K  
The Logistic Regression Model
predictor variables
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


 K X K
dichotomous outcome
 PY  
 is the log(odds) of the outcome.
ln 
 1  PY  
The Logistic Regression Model
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


intercept
 K X K
model coefficients
 PY  
 is the log(odds) of the outcome.
ln 
 1  PY  
Logistic Regression uses Odds Ratios
• Does not model the outcome directly, which leads
to effect estimates quantified by means (i.e.,
differences in means)
• Estimates of effect are instead quantified by
“Odds Ratios”
Relationship between
Odds & Probability
Odds  event  =
Probability  event 
1-Probability  event 
Probability  event  
Odds  event 
1+Odds  event 
The Odds Ratio
Definition of Odds Ratio: Ratio of two odds estimates.
So, if Pr(response | trt) = 0.40 and Pr(response | placebo) = 0.20
Then:

Odds  response| trt group  
0.40
 0.667
1  0.40

0.20
Oddsresponse | placebo group  
 0.25
1  0.20

0.667
 OR  Trt vs. Placebo  
 2.67
0.25
Interpretation of the Odds Ratio
•Example cont’d:

Outcome = response, OR trt vs.plb  2.67
Then, the odds of a response in the treatment
group were estimated to be 2.67 times the odds
of having a response in the placebo group.
Alternatively, the odds of having a response
were 167% higher in the treatment group than in
the placebo group.
Odds Ratio vs. Relative Risk
• An Odds Ratio of 2.67 for trt. vs. placebo does
NOT mean that the outcome is 2.67 times as
LIKELY to occur.
• It DOES mean that the ODDS of the outcome
occurring are 2.67 times as high for trt. vs. placebo.
Odds Ratio vs. Relative Risk
• The Odds Ratio is NOT mathematically equivalent to the
Relative Risk (Risk Ratio)
• However, for “rare” events, the Odds ratio can
approximate the Relative risk (RR)
 1-P  response | trt  
OR=RR 
 1-P  response | plb  


Maximum Likelihood
Idea of Maximum Likelihood
• Flipped a fair coin 10 times:
T, H, H, T, T, H, H, T, H, H
• What is the Pr(Heads) given the data?
1/100?
1/5?
1/2?
6/10?
• Did you do the home experiment?
T, H, H, T, T, H, H, T, H, H
• What is the Pr(Heads) given the data?
• Most reasonable data-based estimate would be
6/10.
X
# of heads
• In fact, pˆ 

N total # of flips
is the ML estimator of p.
Maximum Likelihood
• The method of maximum likelihood estimation
chooses values for parameter estimates
(regression coefficients) which make the observed
data “maximally likely.”
• Standard errors are obtained as a by-product of
the maximization process
The Logistic Regression Model
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


intercept
 K X K
model coefficients
 PY  
 is the log(odds) of the outcome.
ln 
 1  PY  
Maximum Likelihood
• We want to choose β’s that maximizes the
probability of observing the data we have:
N
L  Pr  y1 , y2 , , y N   Pr  y1  Pr  y2  Pr  y N    Pr  yi 
i 1
Assumption: independent y’s
Maximum Likelihood
• Define p = Pr(y=1). Then for dichotomous
outcome => Pr(y=0) = 1-Pr(y=1) = 1-p.
Then:
Pr  y   p 1  p 
1 y
y
For y  1, Pr(1)  p 1  p   p
0
1
For y  0, Pr(0)  p 1  p   1  p
0
1
• So, given that Pr(y) = py(1-p)1-y :
N
N
i 1
i 1
L   Pr  yi    pi 1  pi 
yi
yi
 1 
 1  pi 
  pi 
i 1
 1  pi 
N
yi
yi
 pi 
 1  pi 
  
i 1  1  pi 
N
1 yi
• Taking the logarithm of both sides:
 pi 
Can you
   ln 1  pi  see why?
ln L   yi ln 
i
 1  pi  i
Remember that:
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


 xi
 K X K
• Substituting in using logistic regression model:
ln L   yi xi   ln 1  exp xi 
i
i
• Now we choose values of β that make this equation
as large as possible.
• Maximizing the lnL => maximizes L
• Maximizing involves derivatives & iteration
Maximum Likelihood
• The method of maximum likelihood estimation
chooses values for parameter estimates which make
the observed data “maximally likely.”
• ML estimators have great properties:
– Unbiased (estimate true β’s)
– Asymptotically efficient (narrow CI’s)
– Asymptotically Normally distributed
(can calculate CI’s and Test Statistics using familiar Z
formulas)
Estimating a Logistic Regression Model
Steps:
•
Observe data on outcome, Y, and
charactersitiscs X1, X2, …, XK
•
Estimate model coefficients using ML
•
Perform inference: calculate confidence
intervals, odds ratios, etc.
The Logistic Regression Model
predictor variables
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


 K X K
dichotomous outcome
 PY  
 is the log(odds) of the outcome.
ln 
 1  PY  
The Logistic Regression Model
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


intercept
 K X K
model coefficients
 PY  
 is the log(odds) of the outcome.
ln 
 1  PY  
Form for Predicted Probabilities
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


P Y 
exp   0  1 X 1   2 X 2 
1  exp   0  1 X 1   2 X 2 
 K X K
 K X K 
 K X K 
In this latter form, the logistic regression model directly
relates the probability of Y to the predictor variables.
The Logistic Regression Model
Logistic Regression:
 P Y 
ln 
  0  1 X 1   2 X 2 

 1-P  Y  


 K X K
Linear Regression:
Y   0  1 X 1   2 X 2 
 K X K  
Why not use linear regression for
dichotomous outcomes?
• If we model Y directly and Y is dichotomous, this
necessarily violates the linear regression
assumptions (homoscedasticity)
• One of the more intuitive reasons not to is that will
end up with predicted Y’s other than 0 or 1 (possibly
more extreme than 0 or 1).
Assumptions in logistic regression
• Assumptions in logistic regression
– Yi are from Bernoulli or binomial (ni, mi) distribution
– Yi are independent
– Log odds P(Yi = 1) or logit P(Yi = 1) is a linear
function of covariates
• Relationships among probability, odds and
log odds
Measure
Min
Max
Name
Pr(Y=1)
0
1
prob
Pr(Y  1)
1  Pr(Y  1)
0
∞
odds
 Pr(Y  1) 

log 
 1  Pr(Y  1) 
-∞
∞
log odds
Commonality between
linear and logistic regression
• Operating on the logit scale allows a linear model
that is similar to linear regression to be applied
• Both linear and logistic regression are apart of the
family of Generalized Linear Models (GLM)
Logistic Regresion is a
General Linear Model (GLM)
• Family of regression models that use the same general
framework
• Outcome variable determines choice of model
Outcome
Continuous
GLM Model
Linear regression
Dichotomous
Logistic regression
Counts
Poisson regression
Logistic Regression Models are
estimated by Maximum Likelihood
• Using this estimation gives model coefficient
estimates that are asymptotically consistent,
efficient, and normally distributed.
• Thus, a 95% Confidence Interval for  K is given
by:


 K  z 2  SE  
 K 

 L , U 
Logistic Regression Models are
estimated by Maximum Likelihood
• The Odds Ratio for the kth model coefficient is:
 
OR = exp   K 
 

• We can also get a 95% CI for the OR from:

 e ,e
L
U

where  L , U  is a 95% CI for  K
The Logistic Regression Model
Example:
In Assisted Reproduction Technology (ART) clinics, one of the
main outcomes is clinical pregnancy.
There is much empirical evidence that the candidate mother’s
age is a significant factor that affects the chances of pregnancy
success.
A recent study examined the effect of the mother’s age, along
with clinical characteristics, on the odds of pregnancy success on
the first ART attempt.
The Logistic Regression Model
 Pr pregnancy  
  2.67  0.13  Age
ln 
 1  Pr pregnancy  

exp 2.67  0.13  Age 
Pr pregnancy  
1  exp 2.67  0.13  Age 
The Logistic Regression Model
 Pr pregnancy  
  2.67  0.13  Age
ln 
 1  Pr pregnancy  
Q1. What is the effect of Age on Pregnancy?

A. The OR Age  exp  0.13  0.88
This implies that for every 1 yr. increase in age, the
odds of pregnancy decrease by 12%.
The Logistic Regression Model
Q2. What is the predicted probability of a 25 yr. old
having pregnancy success with first ART attempt?
The Logistic Regression Model
 Pr pregnancy  
  2.67  0.13  Age
ln 
 1  Pr pregnancy  

exp 2.67  0.13  Age 
Pr pregnancy  
1  exp 2.67  0.13  Age 
The Logistic Regression Model
Q2. What is the predicted probability of a 25 yr. old
having pregnancy success with first ART attempt?
exp 2.67  0.13  25
Pr pregnancy  
 0.359
1  exp 2.67  0.13  25

A. From this model, a 25 yr. old has about a
36% chance of pregnancy success.
Hypothesis testing
•
Usually interested in testing H 0 :  K  0
•
Two types of tests we’ll discuss:
1. Likelihood Ratio test
2. Wald test
Likelihood Ratio test
•
Idea is to compare the (log) Likelihood of two
models to test H 0 :  K  0
•
Two models:
1. Full model = with predictor included
2. Reduced model = without predictor
•
Then,
 LˆReduced 
  2 ln LˆReduced   2 ln LˆFull
 2 ln 

ˆ
L
 Full 
~  2 with df  # of extra parameters in full model


(here df  1; Critical 1  3.84 for   0.05)
2
Wald test
•
Idea is to use large sample Z statistic from a single
model to test: H 0 :  K  0
Here, Z 
•
ˆK
SEˆ
where Z ~ N 0, 1
K
Critical Z value for =0.05 is 1.96 (two-sided)
Hypothesis testing
•
As the sample size gets larger and larger, the
Wald test will approximate the Likelihood ratio test.
•
The LR test is preferred but Wald test is common
•
Why? Not to scald the Wald but…
Predictive ability of Logistic regression
•
Generalized R-squared statistics controversial
•
ROC curve plots Sensitivity vs. 1-Specificity based
on fitted model
•
c statistic = Area under ROC curve commonly
used to summarize predictive ability of model
SAS Enterprise:
chd.sas7bdat
Logistic Regression
• Motivating example
• Consider the problem of exploring how the risk for
coronary heart disease (CHD) changes as a function of
age.
• How would you test whether age is associated with
CHD?
• What does a scatter plot of CHD and age look like?
0.8
Proportion of CHD
0.6
0.4
0.2
0.0
20
30
40
50
Age Group
60
Logistic regression
Taking the exp of β1 gives the odds ratio:
odds of CHD taking Drug
exp 1  
odds of CHD taking Placebo
1
e  odds ratio of CHD for persons taking
the " New Drug" vs. " Placebo"
Logistic Regression
• We can add multiple predictor variables in modeling
the log odds of getting CHD:
 Pr(CHDi ) 
   0  1 Drug   2 ( Agei  45)
log 
 Pr( no CHDi ) 
Agei = Person i’s age in years
 1, if Drug
Drug i  
for Person i
0, if Placebo
Interpretation for Drug
• Interpretation of coefficient 1 when there are more than
one variable
– log odds ratio when the other variables are held
constant
– e.g., log odds ratio between having CHD with and
without drug, adjusting for age
Interpretation for Age
• Interpretation of coefficient 1 when there are more than
one variable
• Interpretation of coefficient 1 for continuous covariate
– e.g., log odds ratio for 1 year of change in age (unit
difference in covariate), adjusting for drug
• Maximum Likelihood Estimates:
ˆ1  ?
ˆ1
e ?
SEˆ  ?
1
z
ˆ1
SEˆ
?
1
95% CI for 1  ?
95% CI for e 1  ?
• Conclusion:
Strong evidence that the odds of CHD is associated
with the drug.
Likelihood Ratio test
•
Idea is to compare the (log) Likelihood of two
models to test H 0 :  K  0
•
Two models:
1. Full model = with predictor included
2. Reduced model = without predictor
•
Then,
 LˆReduced 
  2 ln LˆReduced   2 ln LˆFull
 2 ln 

ˆ
L
 Full 
~  2 with df  # of extra parameters in full model


(here df  1; Critical 1  3.84 for   0.05)
2
Suggested exercises
• Read Kleinbaum Chapter 1,2,3
Detailed Outline
• Chapter 1 in Kleinbaum & Klein
Practice Exercises (can check
answers)
• No need to hand in
Looking ahead
• HW 3: Due June 12th
• Next 2 classes: Model Building,
Diagnostics & Extensions
• Review & Exam 1