Download Impact Evaluation Session VII Sampling and

Impact Evaluation
Session VII
Sampling and Power
Jishnu Das
November 2006
Sample Selection in Evaluation

Population based representative surveys:




Sampling for Impact evaluation




Sample representative of whole population
Good for learning about the population
Not always most efficient for impact evaluation
Balance between treatment and control groups
Power  statistical inference for groups of interest
Concentrate sample strategically
Survey budget as major consideration


In practice, sample size is many times set by budget
Concentrate sample on key populations to increase
power
HDN SAR WBI
2
Purposive Sampling:
Risk: We will systematically bias our
sample, so results don’t generalize to the
rest of the population or other sub-groups
 Trade off between power within population
of interest and population representation
 Results are internally valid, but not
generalizable.

HDN SAR WBI
3
Survey - Sampling
Population: all cases of interest
 Sampling frame: list of all potential
cases
 Sample: cases selected for analysis
 Sampling method: technique for
selecting cases from sampling frame
 Sampling fraction: proportion of cases
from population selected for sample (n/N)

HDN SAR WBI
4
Sampling Frame
Simple Sampling
 Stratified Sampling
 Cluster Sampling

HDN SAR WBI
5
Sampling Methods
Random Sampling
 Systematic Sampling

HDN SAR WBI
6
The Design Effect in Clustering

Necessary to take into account when
samples are clustered
Var
cluster
deff 
Var srs
HDN SAR WBI
7
Correlación intracluster ()

DEFF depends on the size of the cluster and the
intra-cluster correlation
Deff  [ 1   ( k  1 )]
where   intracluster correlation
k  cluster size

deff  1

k 1
deff  design effect
k  cluster size
 is the degree of homogeneity in the cluster,
and is called the “intra-cluster” correlation
HDN SAR WBI
8
Tamaño de muestra

The necessary sample size will increase
in clustered samples
n  nsrs  deff
where nsrs  size required with simple randomsampling

But, you have to have some idea of the
intra-cluster coefficient to get at this
number!
HDN SAR WBI
9
Power Calculations
Test significance of a null hypothesis.
 For example, whether two means are
different.

HDN SAR WBI
10
Type I and Type II errors
Type I error
=
Power
= 1-
Type II error
= 
-4
-2
0
2
4
6
x
HDN SAR WBI
Significance Level
11
Type I and type II errors

Type I error: Reject the null hypothesis
when it is true


Type II error: Accept (fail to reject) the
null hypothesis when it is false


Significance level  probability of rejecting the
null when it is true (Type I error)
Power  probability of rejecting the null when
an alternative null is true (1-probability of Type
II)
We want to minimize both types of errors

Increase sample size
HDN SAR WBI
12
Type I and Type II errors

Type I error = 


Type II error = 


Probability you conclude that intervention had
no effect when it actually did
Power = 1 - 


Probability that you conclude the intervention
had an effect if actually it did not
Probabilty of correctly conluding that the
intervention had an effect
Fix the type I error and use sample size to
increase the power
HDN SAR WBI
13
Power Calculations for sample size

Fix the confidence level and as you
increase the size of the sample:


Rejection region gets larger
The power increases
n↑
-4
-2
0
2
x
HDN SAR WBI
4
6
-4
-2
0
2
4
6
x
14
What we have so far
Clustering increases the required sample
size
 As does the need for statistical testing: if
we know



The estimated size of the treatment
The variance of the distribution

HDN SAR WBI
We can start making power calculations for
evaluations
15
In Practice
Many, many analytical statistical results
 May be simpler to use simulations in Stata
or similar package


Easily accounts for complicated designs
HDN SAR WBI
16
In Practice: An Example
Does Information improve child
performance in schools? (Pakistan)
 Randomized Design



Interested in villages where there are private
schooling options
What Villages should we work in?


Stratification: North, Central, South
Random Sample: Villages chosen randomly
from list of all villages with a private school
HDN SAR WBI
17
In Practice: An Example
How many villages should we choose?
 Depends on:




How many children in every village
How big do we think the treatment effect will
be
What the overall variability in the outcome
variable will be
HDN SAR WBI
18
In Practice: An Example

Simulation Tables

Table 1 assumes very high variability in testscores.
Number
of
children
in every
School

5
10
15
20
Number of Villages (assuming 3 schools per village)
21
27
36
42
60
N,n
N,n
N,s
N,s
N,a
N,s
N,m
N,a
N,a
S,a
N,s
N,m
N,a
N,a
M,a
N,m
S,m
M,a
M,a
M,a
X,Y: X is for intervention with small effect
size; Y for larger effect size



N: Significant < 1% of simulations
S: Significant < 10% of simulations
A: Significant > 99% of simulations
HDN SAR WBI
19
In Practice: An Example

Simulation Tables

Table 1 assumes lower variability in testscores.
Number
of
children
in every
School

5
10
15
20
Number of Villages (assuming 3 schools per village)
21
27
36
42
60
N,s
N,s
s,m
s,m
S,a
N,m
S,m
m,a
M,a
M,a
N,m
s,m
M,a
M,a
M,a
S,a
S,a
M,a
M,a
M,a
X,Y: X is for intervention with small effect
size; Y for larger effect size



N: Significant < 1% of simulations
S: Significant < 10% of simulations
A: Significant > 99% of simulations
HDN SAR WBI
20
A smorgasbord of topics
Probability proportional to size sampling to
pick clusters
 Using weights



Estimating means vs.
Estimating regressions
Increasing efficiency using matched
randomizations
 Using evaluations to say something about
baseline populations


Age targeted programs
HDN SAR WBI
21
When do we really worry about this?

IF


Very small samples at unit of treatment!
Suppose treatment in 20 schools and control in
20 schools



This is still a small sample
IF



But there are 400 children in every school
Interested in sub-groups (blocks)
Sample size requirements increase
exponentially
IF

Using Regression Discontinuity Designs
HDN SAR WBI
22