Download cash transfer - World Bank Internet Error Page AutoRedirect

Impact Evaluation Methods
Randomization and Causal Inference
Slides by Paul J. Gertler & Sebastian Martinez
Motivation
• “Traditional” M&E:
• Is the program being implemented as designed?
• Could the operations be more efficient?
• Are the benefits getting to those intended?
– Monitoring trends
• Are indicators moving in the right direction?
–  NO inherent Causality
• Impact Evaluation:
– What was the effect of the program on outcomes?
– Because of the program, are people better off?
– What would happen if we changed the program?
–  Causality
2
Monitoring vs. Impact Evaluation
Policy
Increase Access and
Quality in Early Child
Education
Improve learning in
Science and Math in
high school
Improve quality of
instruction in higher
education
3
Intervention
Monitoring
Impact Evaluation
-Construction
-New classrooms
-Increased attendance
-Feeding
-SES of students
-health/growth
-Quality
- # of Meals
-Cognitive
Development
-Upgrade science
laboratories
-Training of instructors
-Teacher training
-Online courses
-Use of curriculum
- # equipped labs
-Learning
-# trained instructors
-Labor market
-Lab attendance &
use
- University enrollment
- # of training
sessions
-Learning
- # of internet
terminals
-Attendance/drop out
-Labor market
Motivation
• Objective in evaluation is to estimate the
CAUSAL effect of intervention X on outcome Y
– What is the effect of a cash transfer on
household consumption?
• For causal inference we must understand the
data generation process
– For impact evaluation, this means
understanding the behavioral process that
generates the data
• how benefits are assigned
4
Causation versus Correlation
• Recall: correlation is NOT causation
– Necessary but not sufficient condition
– Correlation: X and Y are related
• Change in X is related to a change in Y
• And….
• A change in Y is related to a change in X
– Causation – if we change X how much does Y
change
• A change in X is related to a change in Y
• Not necessarily the other way around
5
Causation versus Correlation
• Three criteria for causation:
– Independent variable precedes the dependent
variable.
– Independent variable is related to the dependent
variable.
– There are no third variables that could explain
why the independent variable is related to the
dependent variable
• External validity
– Generalizability: causal inference to generalize
outside the sample population or setting
6
Motivation
• The word cause is not in the vocabulary of standard
probability theory.
– Probability theory: two events are mutually
correlated, or dependent  if we find one, we
can expect to encounter the other.
• Example age and income
• For impact evaluation, we supplement the language
of probability with a vocabulary for causality.
7
Statistical Analysis & Impact
Evaluation
• Statistical analysis: Typically involves inferring the
causal relationship between X and Y from
observational data
– Many challenges & complex statistics
• Impact Evaluation:
– Retrospectively:
• same challenges as statistical analysis
– Prospectively:
• we generate the data ourselves through the
program’s design  evaluation design
• makes things much easier!
8
How to assess impact
• What is the effect of a cash transfer on
household consumption?
•
• Formally, program impact is:
•
α = (Y | P=1) - (Y | P=0)
• Compare same individual with & without programs
at same point in time
• So what’s the Problem?
9
Solving the evaluation problem
• Problem: we never observe the same individual with
and without program at same point in time
• Need to estimate what would have happened to the
beneficiary if he or she had not received benefits
• Counterfactual: what would have happened without
the program
• Difference between treated observation and
counterfactual is the estimated impact
10
Estimate effect of X on Y
• Compare same individual with & without treatment
at same point in time (counterfactual):
•
sick 2 days
sick 10 days
Impact = 2 - 10 = - 8 days sick!
• Program impact is outcome with program minus
outcome without program
11
Finding a good counterfactual
• The treated observation and the counterfactual:
– have identical factors/characteristics, except for
benefiting from the intervention
– No other explanations for differences in outcomes
between the treated observation and
counterfactual
• The only reason for the difference in
outcomes is due to the intervention
12
Measuring Impact
• Tool belt of Impact Evaluation Design Options:
• Randomized Experiments
• Quasi-experiments
– Regression Discontinuity
– Difference in difference – panel data
– Other (using Instrumental Variables, matching,
etc)
• In all cases, these will involve knowing the rule for
assigning treatment
13
Choosing your design
• For impact evaluation, we will identify the “best”
possible design given the operational context
• Best possible design is the one that has the fewest
risks for contamination
– Omitted Variables (biased estimates)
– Selection (results not generalizable)
14
Case Study
• Effect of cash transfers on consumption
• Estimate impact of cash transfer on consumption
per capita
– Make sure:
• Cash transfer comes before change in
consumption
• Cash transfer is correlated with consumption
• Cash transfer is the only thing changing
consumption
• Example based on Oportunidades
15
Oportunidades
• National anti-poverty program in Mexico (1997)
• Cash transfers and in-kind benefits conditional on
school attendance and health care visits.
• Transfer given preferably to mother of beneficiary
children.
• Large program with large transfers:
– 5 million beneficiary households in 2004
– Large transfers, capped at:
• $95 USD for HH with children through junior high
• $159 USD for HH with children in high school
16
Oportunidades Evaluation
• Phasing in of intervention
– 50,000 eligible rural communities
– Random sample of of 506 eligible communities
in 7 states - evaluation sample
• Random assignment of benefits by community:
– 320 treatment communities (14,446
households)
• First transfers distributed April 1998
– 186 control communities (9,630 households)
• First transfers November 1999
17
Oportunidades Example
18
Common Counterfeit
Counterfactuals
• 1. Before and After:
2005
Sick 2 days
2007
Sick 15 days
Impact = 15 - 2 = 13 more days sick?
• 2. Enrolled /
Not Enrolled:
Sick 2 days
19
Sick 1 day
Impact = 2 - 1 = + 1 day sick?
“Counterfeit” Counterfactual No. 1
• Before and after:
– Assume we have data on
• Treatment households before the
cash transfer
• Treatment households after the cash
transfer
– Estimate “impact” of cash transfer on household
consumption:
• Compare consumption per capita
before the intervention to consumption
20
Case 1: Before and After
• Compare Y before and after
intervention
CPC
• αi = (CPCit | T=1) - (CPCi,t-1| T=0)
Before
After
A
• Estimate of counterfactual
• (CPCi,t| T=0) = (CPCi,t-1| T=0)
B
• “Impact” = A-B
t-1
21
t
Time
Case 1: Before and After
Mean
Case 1 - Before and After
Control - Before Treatment - After t-stat
233.48
268.75
16.3
Case 1 - Before and After
Linear Regression
Multivariate Linear Regression
Estimated Impact on CPC
** Significant at 1% level
22
35.27**
34.28**
(2.16)
(2.11)
Case 1: Before and After
• Compare Y before and after CPC
intervention
• αi = (CPCit | T=1) - (CPCi,t-1|
T=0)
Before
After
A
• Estimate of counterfactual
23
• (CPCi,t| T=0) = (CPCi,t-1| T=0)
D?
• “Impact” = A-B
B
• Does not control for time
varying factors
– Recession: Impact = A-C
– Boom: Impact = A-D
C?
t-1
t
Time
“Counterfeit” Counterfactual No. 2
• Enrolled/Not Enrolled
– Voluntary Inscription to the program
– Assume we have a cross-section of postintervention data on:
• Households that did not enroll
• Households that enrolled
– Estimate “impact” of cash transfer on household
consumption:
• Compare consumption per capita of those who did
not enroll to consumption per capita of those who
enrolled
• Difference in consumption per capita between the
two groups is “treatment”
24
Case 2: Enrolled/Not Enrolled
Mean CPC
Case 2 - Enrolled/Not Enrolled
Not Enrolled
Enrolled
t-stat
290.16
268.7541
5.6
Case 2 - Enrolled/Not Enrolled
Linear Regression
Multivariate Linear Regression
Estimated Impact on CPC
** Significant at 1% level
25
-22.7**
-4.15
(3.78)
(4.05)
Those who did not enroll….
• Impact estimate: αi = (Yit | P=1) - (Yj,t| P=0) ,
• Counterfactual:
(Yj,t| P=0) ≠ (Yi,t| P=0)
• Examples:
– Those who choose not to enroll in program
– Those who were not offered the program
• Conditional Cash Transfer
• Job Training program
• Cannot control for all reasons why some choose to sign up &
other didn’t
• Reasons could be correlated with outcomes
• We can control for observables…..
26
• But are still left with the unobservables
Impact Evaluation Example:
Two Counterfeit Counterfactuals
• What is going on??
Estimated Impact
on CPC
Case 1 - Before and After
Linear
Multivariate Linear
Regression
Regression
Case 2 - Enrolled/Not Enrolled
Linear
Multivariate Linear
Regression
Regression
35.27**
34.28**
-22.7**
-4.15
(2.16)
(2.11)
(3.78)
(4.05)
** Significant at 1% level
• Which of these do we believe?
• Problem with Before-After:
– Can not control for other time-varying factors
• Problem with Enrolled-Not Enrolled:
– Do no know why the treated are treated and the others not
27
Solution to the
Couterfeit Counterfactual
Sick 2 days
Observe Y with treatment
ESTIMATE Y without treatment
Impact = 2 - 10 = - 8 days sick!
On AVERAGE,
28
Sick 10 days
is a good counterfactual for
Possible Solutions…
• We need to understand the data generation process
– How beneficiaries are selected and how benefits
are assigned
• Guarantee comparability of treatment and control
groups, so ONLY difference is the intervention
29
Measuring Impact
• Experimental design/randomization
• Quasi-experiments
– Regression Discontinuity
– Double differences (diff in diff)
– Other options
30
Choosing the methodology…
• Choose the most robust strategy that fits the
operational context
• Use program budget and capacity constraints to
choose a design, i.e. pipeline:
– Universe of eligible individuals typically larger
than available resources at a single point in time
– Fairest and most transparent way to assign
benefit may be to give all an equal chance of
participating  randomization
31
Randomization
• The “gold standard” in impact evaluation
• Give each eligible unit the same chance of receiving
treatment
– Lottery for who receives benefit
– Lottery for who receives benefit first
32
Randomization
Randomization
External Validity
(sample)
Randomization
Internal Validity
(identification)
33
External & Internal Validity
• The purpose of the first-stage is to ensure that the
results in the sample will represent the results in
the population within a defined level of sampling
error (external validity).
• The purpose of the second-stage is to ensure that
the observed effect on the dependent variable is
due to some aspect of the treatment rather than
other confounding factors (internal validity).
34
Case 3: Randomization
• Randomized treatment/controls
– Community level randomization
• 320 treatment communities
• 186 control communities
• Pre-intervention characteristics
well balanced
35
Baseline characteristics
RANDOMIZATION
Variables
Consumption per
capita
Control
(2727)
t-stats
233.47
233.4
-0.04
1.02
1.3
41.94
42.35
0.2
0.27
2.95
2.81
0.04
0.05
37.02
36.96
0.7
0.22
2.76
2.76
0.03
0.04
Speaks an indigenous
language
41.69
41.95
0.007
0.009
Head is female
0.073
0.078
0.003
0.005
5.76
5.7
0.02
0.038
0.57
0.56
0.007
0.009
1.63
1.72
0.03
0.05
109.28
106.59
0.6
0.81
Head's age
Head's education
Spouse's age
Spouse's education
Household at
baseline
Bathroom at baseline
Total hectareas of
land
Min. Distance locurban
36
Treatment
(4,670)
1.2
-2.16
-0.38
-0.006
0.21
0.66
-1.21
-1.04
1.35
-1.02
Case 3: Randomization
Case 3 - Randomization
Control
Treatment
t-stat
Mean CPC
Baseline
Mean CPC
Followup
233.40
233.47
0.04
239.5
268.75
9.6
Case 3 - Randomization
Linear Regression
Multivariate Linear Regression
Estimated Impact on CPC
** Significant at 1% level
37
29.25**
(3.03)
29.79**
(3.00)
Impact Evaluation Example:
No Design vs. Randomization
Estimated Impact on
CPC
** Significant at 1% level
38
Case 1 - Before and
After
Multivariate Linear
Regression
Case 2 - Enrolled/Not
Enrolled
Multivariate Linear
Regression
Case 3 Randomization
Multivariate Linear
Regression
34.28**
-4.15
29.79**
(2.11)
(4.05)
(3.00)