Download A Specification Test for Linear Regressions that use as Dependent Variables

A Specification Test for Linear Regressions that use
King–Based Ecological Inference Point Estimates
as Dependent Variables1
Michael C. Herron2
Department of Political Science
Northwestern University
Kenneth W. Shotts3
Department of Political Science
Northwestern University
March 8, 2002
1
The authors thank Wendy Tam Cho for comments. An earlier draft of this paper was presented
at the 2000 Annual Meeting of the American Political Science Association.
2
601 University Place, Evanston, IL 60208–1006 (m–[email protected]).
3
601 University Place, Evanston, IL 60208–1006 (k–[email protected]).
Abstract
Many researchers use point estimates produced by the King (1997) ecological inference
technique as dependent variables in second stage linear regressions. We show, however, that
this two stage statistical procedure is at risk of logical inconsistency. Namely, the assumptions
necessary to support the procedure’s first stage (ecological inference via King’s method) can
be incompatible with the assumptions supporting the second stage (linear regression). We
derive a specification test for logical consistency of the two stage procedure, describe options
available to a researcher whose ecological dataset fails the test, and illustrate the problems
that can arise if logical inconsistency is ignored.
1
Introduction
Many researchers use point estimates produced by the King (1997) ecological inference
technique as dependent variables in second stage linear regressions. King advocates this two
stage statistical procedure, which Herron and Shotts (2001) call EI–R, saying that “[it] has
enormous potential for uncovering new information” (p. 279). Burden and Kimball (1998),
for example, use King–based estimates of district–level ticket splitting rates as dependent
variables in second stage regressions which examine why individuals cast split ticket ballots.
Similarly, Gay (2001) employs estimates of black turnout rates, produced by King’s ecological
inference technique, in regressions that seek to determine whether black citizens vote at higher
rates in elections that include black Congressional candidates. Other examples of EI–R can
be found in Kimball and Burden (1998), Voss and Lublin (1998), Cohen, Kousser and Sides
(1999), Wolbrecht and Corder (1999), Lublin and Voss (2000), and Voss and Miller (2000).
Clearly, EI–R has achieved widespread usage despite the fact that it was only recently
proposed. We show, however, that applications of this two stage procedure can be logically
inconsistent in the sense that the assumptions necessary to support the procedure’s first
stage (ecological inference via King’s method) can be incompatible with the assumptions
supporting the second stage (linear regression). In contrast, we say that an application of
EI–R is logically consistent when its first stage and second stage assumptions are not mutually
contradictory.
The matter of logical consistency, as distinct from statistical consistency, of EI–R is an
important issue because this statistical procedure consists of two distinct stages. When
a researcher employs the procedure, she must therefore simultaneously adopt assumptions
that support both of EI–R’s stages. Previous work on EI–R has not sought to determine
whether this is even possible. To the best of our knowledge, all published and working–paper
implementations of EI–R simply assume that the procedure’s first and second stages are
inherently compatible, i.e., that EI–R is logically consistent.
We show that this may not be warranted and that, in fact, the assumptions necessary
to support EI–R can be self–contradictory. In particular, our analysis of EI–R is based on
1
the fact that standard implementations of King’s technique to an ecological dataset (the first
stage of EI–R) assume that there is no aggregation bias in the dataset. Nonetheless, we show
that a linear regression which uses King–based point estimates as dependent variables (the
second stage of EI–R) can imply the existence of aggregation bias.
To illustrate this clash of assumptions, suppose that a researcher wants to estimate and
explain disaggregated turnout rates of African–American and white voters but for this task
has access only to precinct–level racial composition and turnout data. To apply King’s
ecological inference technique to her turnout data and estimate African–American and white
turnout rates by precinct (first stage of EI–R), the researcher must assume that there is no
aggregation bias in her dataset, i.e., that black turnout is uncorrelated with the fraction of a
precinct’s population that is African–American.
Now, suppose that the researcher estimates a regression (second stage of EI–R) in which
King–based estimates of black turnout rates are assumed to be a linear function of precinct–
level income and education levels. If any of these right hand side variables are correlated with
the fraction in a precinct’s population that is African-American, then the researcher’s second
stage regression model virtually guarantees (in a sense that we make precise later) that black
turnout is correlated with the fraction of a precinct’s population that is African-American,
i.e., there is aggregation bias. Thus, in this hypothetical example the assumptions behind
the second stage of EI–R contradict those in its first stage.
This is problematic at two levels. At a fundamental level, any standard philosophy of science requires researchers to adopt internally consistent sets of assumptions. To do otherwise,
as is possible with EI–R, is to state simultaneously “A is true” and “A is false.” Once two
mutually contradictory premises are adopted, any implication can be logically derived, i.e.,
all subsequent analysis is meaningless.
At a more pragmatic level, logical inconsistency of EI–R is problematic because, in the
presence of aggregation bias, King’s ecological inference technique produces statistically inconsistent estimates (Tam 1998). And when this occurs, EI–R’s final results, i.e., second stage
regression results that depend on first stage ecological inference estimates, will be statistically
2
inconsistent in ways that cannot be readily rectified.
Given the problems that result if EI–R is logically inconsistent, it is important for
researchers to determine whether a particular application of the technique is problematic in
this way. Fortunately, it turns out that not all implementations of EI–R are logically inconsistent. Rather, we show that logical consistency of EI–R is application–dependent insofar as
EI–R is logically consistent in some empirical applications yet logically inconsistent in others.
We therefore propose a simple specification test for logical consistency of EI–R which can
be implemented with a single linear regression. Interpreting the results of the specification
test requires consideration of only a single F statistic, the well–known statistic which tests
the null hypothesis that all slope coefficients in a linear regression are zero. If the F statistic
is significant when applied to an ecological dataset, then an application of EI–R to the said
dataset would be logically inconsistent. If, on the other hand, the F value is not significant,
then logical consistency cannot be rejected.
It is worth noting that, even in an application where EI–R is logically consistent, second
stage regression results are statistically inconsistent. Fortunately, in some cases it is possible
to correct for this statistical inconsistency so long as EI–R is logically consistent (Herron and
Shotts 2001).
In contrast, when the test for logical consistency of EI–R rejects in a given ecological
dataset, a researcher who wants to study the dataset has only two options. One, she can
abandon EI–R entirely and choose a different method of analysis. Or, two, she can explicitly
incorporate covariates into the first stage of her EI–R procedure and use what King calls the
extended ecological inference model. Later in the paper we discuss, and indeed recommend,
this option.
We emphasize that our critique of EI–R and King’s ecological inference technique is
fundamentally different from the arguments in Tam (1998) and Cho and Gaines (2000), both
of which claim that aggregation bias is a common phenomenon in aggregate data and that
diagnostics advocated by King (1997, ch. 9) provide little help in identifying and rectifying
it. Our analysis is not based on a claim that the assumptions supporting King’s ecological
3
inference technique are unwarranted, either in general or in a particular empirical application.
Rather, our critique applies only to the use of King’s technique within the context of the
combined EI–R procedure, and what we argue is that the assumptions which support EI–R
must be compatible across EI–R’s two stages.
Furthermore, our test for logical consistency of EI–R should not be thought of as a general
test for aggregation bias in ecological data. Rather, the test asks whether the second stage
of EI–R implies that there is aggregation bias in the first stage. The test does not explore
whether there are other sources of aggregation bias that could cause first stage problems.
In Section 2 we provide notation and details on EI–R. Then, in Section 3 we present our
specification test for logical consistency of EI–R and explain how to interpret it. Section
4 considers the options faced by a researcher whose ecological dataset fails the test, and
in Section 5 we apply the test to a House election dataset used in Burden and Kimball’s
(1998) study of ticket splitting. This dataset fails the specification test, and thus we apply
King’s extended model to it. The results of our extended model analysis of Burden and
Kimball’s data are dramatically different from Burden and Kimball’s published findings on
the causes of ticket splitting in House elections, and this illustrates the costs of ignoring
logical inconsistency in EI–R. Section 6 offers general comments on EI–R and concludes.
2
Overview of EI–R
This section provides some details on ecological inference and King’s technique in particu-
lar and explains how the output of the technique is used in second stage regressions. Suppose
we have precinct–level data for an election: let Ti ∈ [0, 1], i = 1, . . . , p, denote precinct i’s
turnout rate in the election and Xi ∈ [0, 1] the fraction of Blacks in precinct i. The goal of
ecological inference is to use Ti and Xi to estimate βib , the fraction of Blacks in precinct i
who voted in the election, and βiw , the corresponding fraction of Whites who voted. Achen
and Shively (1995) discuss the history of the ecological inference problem and review various
solution attempts.
King develops two different implementations of his ecological inference technique standard
4
and extended. The precise details of these implementations are found in King (1997, ch. 6–8).
Critiques and discussions of King’s approach to ecological inference can be found in Freedman,
Klein, Ostland and Roberts (1998), Tam (1998), Cho and Gaines (2000), and McCue (2001).
Until further notice our comments on King’s technique apply to the standard model.
¡
¢
This model assumes that βib , βiw have a truncated bivariate normal distribution on the
¡
¢
unit square with means Bb , Bw , standard deviations (σb , σw ), and correlation ρ. These
©
ª
parameters are collected in a 5–vector ψ = Bb , Bw , σb , σw , ρ , and, notably, the two means
Bb and Bw are fixed across i.1 The importance of this assumption will be clear later when
¢
¡
we discuss the extended model. Furthermore, the standard model assumes Cov βib , Xi = 0.
This is equivalent to assuming there exists no aggregation bias, i.e., there is no correlation
between βib , the fraction of blacks in precinct i who voted, and Xi , the fraction of blacks in
precinct i.2
Let B̂b , B̂w , σ̂b , σ̂w , and ρ̂ denote the point estimates of Bb , Bw , σb , σw , and ρ, respectively, based on applying King’s standard model to an ecological dataset.3 Corresponding
to the vector ψ, the five point estimates are collected in a 5–vector ψ̂. Under the standard
P
model’s assumptions (recall, one of which is that there is no aggregation bias) B̂b −
→ Bb
P
as p tends to infinity where −
→ denotes convergence in probability. In other words, B̂b is a
consistent estimate of Bb , and a similar statement applies to the other four point estimates
P
in ψ̂. In general, then, ψ̂ −
→ ψ.
Let β̂ib be the point estimate of βib produced by applying King’s technique to an ecological
dataset Xi and Ti , i = 1, . . . , p; see McCue (2001) for a discussion of this estimator, which is
a function of ψ̂, Xi , and Ti . In EI–R, the β̂ib point estimates are used as dependent variables
1
As described in King (1997), the elements of ψ correspond to elements in a related 5–vector ψ̆ where ψ is a
function of ψ̆. We ignore ψ̆ and this is without loss of generality because King’s motivation for parameterizing
the elements of ψ with the elements of ψ̆ is numerical rather than substantive.
2
In addition, King’s technique assumes that there is no spatial autocorrelation, i.e., that T i and Tj are
independent for i 6= j.
3
King’s ecological inference technique can be estimated using standard maximum likelihood theory. Or,
optional priors can be used in which case King’s model is based on a Bayesian framework. Both of these
implementations of King’s technique are compatible our analysis.
5
in a second stage regression. Namely, King suggests that a researcher who wants to estimate
βib = α + γ 0 Zi + νi ,
(1)
β̂ib = α + γ 0 Zi + νi ,
(2)
instead estimate
where the difference between equation (1) and (2) is that β̂ib (observed) substitutes for βib
(unobserved). Here, α is a constant, γ is a k–vector of parameters to be estimated, Z i is
a k–vector of exogenous variables, and νi is a disturbance term assumed to be uncorrelated
with βib and β̂ib . Let α̂ and γ̂ denote the least squares estimates of α and γ, respectively. 4
Thus, EI–R consists of, first, estimating β̂ib using King’s ecological inference technique and,
second, estimating α̂ and γ̂ with least squares applied to equation (2).5 The final products
of EI–R, therefore, are the estimates α̂ and γ̂, and, ultimately, whether EI–R is statistically
P
P
consistent depends on whether α̂ −
→ α and γ̂ −
→ γ (henceforth we ignore the intercept α̂ and
focus only on the slope estimates in γ̂). On the other hand, first stage statistical consistency
P
depends on whether ψ̂ −
→ ψ where the 5–vector ψ̂, as noted previously, is the first stage
estimate produced by applying King’s standard model to an ecological dataset.
Suppose that EI–R is logically consistent so that its first and second stage assumptions
P
do not contradict one another. This implies that ψ̂ −
→ ψ, meaning that EI–R’s first stage
P
P
is statistically consistent. Nonetheless, even in this sanguine situation, γ̂ −
6 → γ, where −
6→
denotes a lack of convergence in probability, except in highly unusual circumstances (Herron
and Shotts 2001). In other words, EI–R is in general statistically inconsistent when it is
logically consistent; this is because β̂ib is not a perfect measure of βib even when the first
stage of EI–R is statistically consistent. However, Herron and Shotts describe manipulations
of γ̂ that, in the best of cases, can be used to produce consistent estimates of γ based on
4
Equations (1) and (2) could just as easily have had βiw and β̂iw on their left hand sides. All of our results
apply in a symmetric fashion to such a case.
5
King suggests a weighted least squares approach when estimating equation (2). Weighting, however, is
not germane to our discussion of logical inconsistency.
6
inconsistent estimates produced by equation (2).6
P
These manipulations are feasible only when ψ̂ −
→ ψ, i.e., only when first stage ecological
inference estimates of the vector ψ are statistically consistent. This is a key point for the
following reason: when EI–R is logically inconsistent, its second stage invalidates an assumption that supports its first stage. Hence, logical inconsistency of EI–R implies first stage
P
statistical inconsistency (ψ̂ −
6 → ψ) and EI–R completely falls apart.7 In particular, as far as
we are aware, no useful manipulations of γ̂ are possible when EI–R is logically inconsistent.
In this latter situation, these point estimates are hopelessly statistically inconsistent.
3
The Logical Consistency Test for EI–R
Recall that EI–R’s first stage assumption of no aggregation bias is mathematically equiv¡
¢
alent to Cov βib , Xi = 0. However, second stage regression implies βib = α + γ 0 Zi + νi from
equation (1). Hence, for EI–R to be logically consistent across its first and second stages, it
must be true that
³
´
0 = Cov βib , Xi
¢
¡
= Cov α + γ 0 Zi + νi , Xi
¡
¢
= Cov γ 0 Zi + νi , Xi
=
k
X
γj Cov (Zi,j , Xi ) + Cov (νi , Xi )
j=1
where γj is the jth element of γ and where Zi,j is the jth element of the vector Zi , j = 1, . . . , k.
This derivation forms the basis of the following result:
Proposition 1 Suppose that an ecological dataset, consisting of Xi , Ti , and Zi , i = 1, . . . , p,
6
At the time of this paper’s writing, it is not known how EI–R estimates can be interpreted when Herron
and Shotts’s manipulations are not applicable.
7
King (1997) argues in ch. 11 that his technique sometimes produces accurate estimates of ψ even in the
presence of aggregation bias. However, Tam (1998) argues that, in such situations, the technique produces
inaccurate estimates. In our opinion, the fact that an inconsistent estimator happens to be on target in some
applications does not constitute adequate justification for adopting it. Namely, inconsistent estimators are
guilty until proven innocent, and, at the time of this writing, the value of King’s technique in the presence of
aggregation bias remains uncertain.
7
is given. Let γ be the true second stage parameter vector from equation (1). If
k
X
γj Cov (Zi,j , Xi ) + Cov (νi , Xi ) 6= 0,
(3)
j=1
then EI–R as applied to this dataset is logically inconsistent.
Proposition 1 places a restriction on the second stage parameter vector γ that must be
satisfied for EI–R to be logically consistent. When the restriction is violated, EI–R is logically
inconsistent.
Suppose that, for a given ecological dataset, the k + 1 covariances in equation (3) all
happen to be exactly zero. Then, any k–vector γ, regardless of its elements, is consistent
with the restriction in Proposition 1. This is because the elements γj , when substituted into
equation (3), will be multiplied by zero and hence have no impact on whether the left hand
side of the equation is zero. In other words, when all covariances in equation (3) are zero it
is not possible for the elements of the vector γ to violate the restriction in Proposition 1 and
thereby cause rejection of logical consistency of EI–R.
On the other hand, if some of the Cov (Zi,j , Xi ) terms are not zero, then only certain γ
vectors satisfy the restriction in Proposition 1. For example, suppose that k = 2 so that γ
is a 2-vector. Also suppose that Cov (νi , Xi ) = 0, and Cov (Zi,j , Xi ) = 1, j = 1, 2. Then, for
EI–R to be logically consistent it must be the case that γ1 + γ2 = 0.
In sum, Proposition 1 has two cases. In one case – when all covariances in equation (3) are
zero – there is no restriction on γ. In the other case – when at least one of the Cov (Z i,j , Xi )
terms is non–zero – the restriction in equation (3) is meaningful and logical consistency of
EI–R depends on whether γ satisfies it.
When Proposition 1 imposes a meaningful restriction on γ, we can enquire as to the
nature of the restriction. Is it meaningful yet weak – for example, requiring γ j < 100 for all
j – or is it tight – for example, requiring γj = 1 for all j? If the former, then we would say
that the restriction is relatively easy to satisfy and hence there are no compelling reasons to
reject logical consistency of EI–R based solely on the fact that γ must satisfy the restriction.
8
On the other hand, if the restriction on the second stage parameter vector γ is very tight,
then most γ vectors will fail to satisfy it. From this it would follow that EI–R is unlikely to
be logically consistent. Similarly, if given an ecological dataset the associated restriction on
γ is so tight that is seems unimaginable that it can be satisfied, then in all likelihood EI–R
is logically inconsistent when applied to that dataset.
We now develop this idea of a very tight restriction on γ, a restriction that seems virtually
impossible to satisfy. In particular, our conceptualization of such a restriction is one that
will be satisfied with probability zero if γ is drawn randomly from a continuous distribution.
Consider the following example. Let k = 2 and suppose as before that logical consistency
of EI–R requires γ1 = −γ2 . If γ = (γ1 , γ2 ) is drawn from a continuous distribution on <2 ,
then the probability that γ1 = −γ2 is zero since a line has zero probability measure on a
two–dimensional plane. Hence, we say that the restriction γ1 = −γ2 is very tight and, when
logical consistency of EI–R depends on it, we conclude that EI–R will in all likelihood be
logically inconsistent.
In general, when the k–vector γ is randomly drawn from a continuous probability distribution on <k , then the probability that m elements of the vector, with 1 ≤ m ≤ k, lie in a
given hyperplane of dimension m − 1 is zero. Thus,
Definition 1 A k–vector γ from equation (1) is said to be non–generic if, for 1 ≤ m ≤ k,
m elements of γ must lie in an m − 1 dimensional hyperplane.
For example, suppose that γ is a 3–vector (k = 3) and that a restriction from equation
(3) implies that two (m = 2) of its elements must lie in a one (m − 1 = 2 − 1 = 1) dimensional
hyperplane. This is a complicated way of saying that these two elements must lie on a line.
The probability that two continuously distributed random variables fall exactly on a given
line is zero. Hence, if all k = 3 elements of γ are drawn from a continuous distribution on
<3 , the probability that the two key elements satisfy the restriction of being on a given line
is zero.
Of course, in actual applications of EI–R it is not the case that γ is drawn from a continuous distribution prior to EI–R’s being employed. Rather, according to standard frequentist
9
logic, estimating a model like equation (2) is equivalent to supposing that there exists a true γ
and that this value is fixed albeit unknown. Nonetheless, the idea behind a restriction which
leads to a non–generic γ is that the restriction is so tight that we should expect it never to
hold. Hence, if we conclude that a given application of EI–R is logically inconsistent unless
γ is non–generic, then we have effectively concluded that the application is in all likelihood
logically inconsistent.
Definition 2 An application of EI–R to an ecological dataset is generically logically inconsistent if the restriction in Proposition 1 requires γ to be non–generic.
It is straightforward to see that, if any of the Cov (Zi,j , Xi ) terms in equation (3) are non–
zero, then the set of γ vectors that satisfy the resulting restriction is non–generic. If, say, only
one covariance is non–zero, then the associated element of γ, that which multiplies the non–
zero covariance, will itself have to be exactly zero (which is a zero probability even). If the γ
term is not zero, the restriction in equation (3) will not be satisfied unless Cov (ν i , Xi ) exactly
offsets the product of the non–zero covariance and the non–zero element of γ. This sort of
offsetting will occur only by complete coincidence. Hence, when even a single Cov (Z i,j , Xi )
term is zero, then γ is non–generic.
Proposition 2 Let γ be the true parameter vector from equation (1). If there exists a j such
that 1 ≤ j ≤ k where Cov (Zi,j , Xi ) 6= 0, then γ is non–generic.
Hence, whenever an ecological dataset features at least a single, non–zero Cov (Z i,j , Xi ) term,
EI–R as applied to the dataset is generically logically inconsistent.
The result in Proposition 2 is, importantly, independent of whether Cov (νi , Xi ) is zero.
In fact, this latter covariance only makes matters worse from the perspective of satisfying the
restriction in equation (3). Namely, all Cov (Zi,j , Xi ) terms can be zero but the restriction can
still be violated if Cov (νi , Xi ) 6= 0. Hence, by ignoring this latter covariance, i.e., assuming
that it is zero, our test for logical consistency of EI–R is conservative. 8
8
Our discussion of restrictions and logical consistency of EI–R began with the premise that an ecological
dataset is given, and then we considered restrictions on γ such that equation (3) holds. We could, alternatively,
10
The matter of whether a proposed application of EI–R is generically logically inconsistent,
then, turns on whether there exists a Cov (Zi,j , Xi ) term that is non–zero. Such covariance
terms can be estimated by regressing Xi on the Zi vector:
Xi = λ 0 + λ 0 Zi + ² i .
(4)
If the F statistic for testing whether all slope coefficients in equation (4) are zero rejects at a
conventional confidence level, it follows that EI–R is generically logically inconsistent. Thus,
the hypothesis test for logical consistency of EI–R requires nothing more than estimating a
single regression with data that EI–R itself requires in the first place.
The intuition behind our regression test for logical consistency in EI–R is as follows.
Recall that King’s ecological inference technique requires that there is no aggregation bias,
¡
¢
i.e., that Cov βib , Xi = 0. But, if a second stage regression assumes that βib is a function of
covariates collected in a vector Zi and, if some of the elements in Zi are correlated with Xi ,
¢
¡
then in general it is not possible for Cov βib , Xi to be zero. Thus, when the test for logical
consistency of EI–R rejects, it follows that there is aggregation bias in the first stage and
EI–R in this situation breaks down.
The matter of non–rejection of logical consistency of EI–R is an important one. The
test we propose for logical consistency is one of necessity as opposed to one of sufficiency.
That is, an application of EI–R must pass our test; if not, it follows that the application
is logically inconsistent. However, if an application of EI–R does pass the test, it does not
therefore follow that the application is logically consistent. Rather, as we discuss later, it
is still possible that the application is logically inconsistent in some manner not covered by
our test. Or, the failure to reject could simply reflect that fact that our test does not have a
power of one. Similarly, if an application of the test does not reject, then it does not follow
that the ecological data used in the application are free from aggregation bias. 9
have started from the premise that the k elements of γ were given and asked about the resulting ecological
dataset which would satisfy the equation. From this latter perspective, when any elements of the vector γ are
non–zero, the set of covariances which satisfy equation (3) will be non–generic. Hence, it follows that EI–R
as applied to the dataset will be generically logically inconsistent.
9
For instance, the second stage could be misspecified due to omitted variables that are correlated with X i .
11
For the remainder of this paper, we use the term “logically inconsistent” in place of
“generically logically consistent” as the latter is tedious. Similarly, we refer to our test as a
test for logical consistency even though, technically, it is a test for generic logical consistency.
4
What to do when an Ecological Dataset Fails the Specification Test
When a researcher’s ecological dataset fails the test for logical consistency of EI–R, she
has two options. One, she can abandon EI–R entirely and choose another method of data
analysis. Or, two, she can model the aggregation bias that caused test failure and in so doing
avoid a second stage regression. We now discuss the latter option.
Modeling aggregation bias in the first stage of an application of EI–R requires King’s
extended model. Recall that in the standard implementation of his ecological inference tech¡
¢
¢ ¡
nique the mean of βib , βiw is Bb , Bw , and this mean vector is fixed across observations.
However, in the extended model the two elements of this vector are not restricted in this way.
¢
¡
where subscripts denote
Rather, in the extended model the means are denoted Bbi , Bw
i
dependence on observation i.
Readers interested in the details of the extended model should consult King (1997, pp.
168–74). In brief, this model posits that Bbi and Bw
i are linear functions of covariates. That is,
the extended model assumes something akin to Bbi = θb0 Wi where θb is a vector of parameters
that must be estimated and Wi is a vector of covariates. A product of the extended model,
then, is an estimate θ̂b of the parameter vector θb (and, similarly, an estimate θ̂w of θw based
0
on Bw
i = θw Wi ). For numerical reasons (equation 9.2, p. 170), King’s extended model does
not assume the precise form noted above for the two means Bbi and Bw
i . However, the linear
expressions for these means captures the essence of the model.
The extended model allows dependence of βib and βiw on covariates that are collected in
the Wi vector. If, for instance, Wi contains only a single covariate and is thus a scalar, then
θb describes how this covariate influences βib and θw describes how it influences βiw . Similarly,
when Wi contains multiple covariates, then θb and θw are vectors with elements corresponding
to covariates in Wi .
12
Thus, suppose a researcher carries out the test for logical consistency of EI–R on a given
dataset, and suppose that the test rejects. This implies that there is aggregation bias in the
dataset insofar as the researcher wants to assume that both his first and second stages apply.
To model this aggregation bias explicitly, the researcher would include as extended model
covariates – that is, as elements of the vector Wi – all elements of Zi and subsequently would
not estimate a second stage regression at all. In other words, when the specification test
rejects logical consistency of EI–R, then the two stage statistical procedure should not be
used. Rather, an abbreviated version – consisting solely of an application of King’s extended
model – should replace it; in this instance, dependence of βib and βiw on covariates would be
analyzed via first stage ecological inference.
When King’s extended model is used, assessing the impact of covariates on βib (and βiw , if
this is an object of study) requires considering first stage standard errors produced by King’s
technique. Namely, for each covariate (akin to an element of Wi ) used to parameterize the
w
means Bbi of βib and Bw
i of βi in the extended model, King’s method produces two coefficient
estimates (akin to elements of θ̂b and θ̂w ) and two associated t–statistics. Significance tests
based on these t–statistics can be used to determine whether covariates are related to β ib and
βiw .
A researcher may be tempted to include all or some elements of the second stage covariate
vector Zi in both an application of King’s extended model as well as in a second stage
regression. This practice should be avoided as it can lead to internal incoherence. Specifically,
it is possible for a first stage extended ecological inference model to indicate that a given
covariate has a significant impact on βib and/or βiw whereas the second stage finds no evidence
of such an effect, or vice versa. This would leave a researcher with disparate estimates of the
same quantity. In other words, to carry out EI–R either King’s standard model should be
used followed by a second stage regression or King’s extended model should be employed and
there should be no second stage regression. Obviously, once a full extended model is used,
there is no reason to estimate a second stage regression, since any relevant hypothesis about
the effect of covariates on βib and βiw can be tested directly using first stage estimates and
13
standard errors.
Suppose, to illustrate this point, that a researcher constructs an ecological dataset and
on it carries out the test for logical consistency of EI–R. Suppose as well that the test rejects
and that only one slope estimate from the test’s regression, i.e., only one element of the λ
vector from equation (4), is significant at, say, the 0.05 level. The researcher might want to
include the covariate associated with the statistically significant λ estimate in an extended
model and then estimate a second stage regression using the full Zi vector.
In such a case, the one significant covariate from the specification test would appear twice
in the EI–R procedure insofar as it influences βib in the first stage of EI–R and then influences
βib again in a second stage regression. Yet, both estimates of the impact of the variable would
be based on a single ecological dataset. Thus, the variable’s appearing twice means that there
will be two separate and dependent estimates of the impact of the variable on β ib . Simply put,
it is impossible to say how the two associated t–statistics should be combined and interpreted.
It is conceivable, and in fact we provide an illustration of this shortly, that the first stage
estimate of the impact of the covariate on βib can be statistically insignificant while the second
stage estimate of the impact of the covariate is significant. When all covariates are used only
once, either in a second stage regression or in an application of King’s extended model, this
type of incoherence can be avoided.
A final reason to use King’s extended model as opposed to EI–R concerns other sources
of logical inconsistency that can plague EI–R. We have focused thus far on whether the two
stages of EI–R make contradictory assumptions about aggregation bias, i.e, about whether
¡
¢
Cov βib , Xi is zero. But, there are other ways that the EI–R can be logically inconsistent.
¡
¢
For instance, the first stage assumption that βib , βiw have a truncated, bivariate normal
distribution has implications for the disturbance term νi in equation (1). In fact, it is unknown
if second stage assumptions about νi are even compatible with first stage bivariate normality.
The use of King’s extended model obviates the possibility that constraints on ν i cause logical
inconsistency of EI–R.
14
5
An Application of the Test for Logical Consistency in EI–R
We now apply the test for logically consistency in EI–R to the ecological dataset used by
Burden and Kimball (1998) in their published study of ticket–splitting in the United States
Congressional and Presidential elections of 1992.10 For Burden and Kimball’s House election
data, Xi is the proportion of House voters in Congressional district i who voted for Michael
Dukakis in the 1992 presidential race, and Ti is the fraction who voted for the Democratic
House candidate.11 Furthermore, βib is the fraction who voted for a Democratic Congressional
candidate as well as for Dukakis (straight ticket Democrats), and β iw is fraction who voted
for voted for Republican presidential candidate George H. W. Bush and a Democratic Congressional candidate (Burden and Kimball call these RD ticket splitters). In their application
of EI–R to the study of ticket splitting, Burden and Kimball use estimates β̂iw as dependent
variables in a second stage regression. Burden and Kimball also use values of 1 − β̂ib as dependent variables in a regression that explains DR ticket splitting, i.e., rates at which individuals
voted Democratic in the presidential race and Republican in a Congressional contest. 12
There are four elements of the second stage covariate vector Z i that Burden and Kimball
use to study RD ticket splitting. These consist of an indicator as to whether district i had
a Democratic incumbent (Democratic Incumbent), the Democratic candidate’s proportion of
House campaign spending (Spending Ratio), whether district i allowed straight–party votes
(Ballot Format), and whether district i was located in the South (South). In accordance with
the test for logical consistency of EI–R when studying RD ticket splitting, we regressed X i
on these four covariates and a constant. Resulting estimates – see equation (4) – are in Table
10
Cho and Gaines (2000) identify several errors in Burden and Kimball’s dataset, but we use the original
dataset to maintain compatibility with the published study. Our critique of Burden and Kimball is quite
different from that offered by Cho and Gaines, who argue, among other things, that King’s diagnostics
indicate the presence of aggregation bias in the data set. Burden and Kimball’s data is available from the
ICPSR (ftp://ftp.icpsr.umich.edu/pub/PRA/outgoing/s1140).
11
As Burden and Kimball note, because of rolloff Xi is not actually known and must be estimated. Like
Burden and Kimball, we estimated Xi using King’s ecological inference technique. As pointed out in Herron
and Shotts (2001), these estimated turnout rates are contaminated by errors. However, for purposes of
comparability we follow Burden and Kimball and treat these estimates as if they were true values of X i .
12
We comment here only on Burden and Kimball’s House data because their Senate election dataset is
extremely small (n = 32). Furthermore, Burden and Kimball augment their Senate data with House data in
a way that is very hard to justify.
15
Table 1: Testing for Logical Consistency in EI–R as Applied to Burden and Kimball (1998)
Variable
Constant
Democratic Incumbent
Republican Incumbent
RD Estimates
0.373
(40.7)
-0.186 —
(0.946)
—
Spending Ratio
0.203
(7.06)
Ballot Format
0.00355
(0.333)
South
−0.0882
(−7.85)
n
361
F
54.8
R2
0.381
Note: t–statistics in parentheses
DR Estimates
0.352
(17.5)
0.0232
(1.35)
0.208
7.86
0.00188
(0.177)
-0.087
(-7.75)
361
54.9
0.392
1. To study DR ticket splitting, Burden and Kimball use the same covariates as in their
RD analysis but with a Republican incumbent indicator in place of Democratic Incumbent.
Specification test results for this configuration of the second stage covariate vector Z i are also
in Table 1.13
Both F statistics in Table 1 are very large and have p–values that are approximately
zero. This means that, for studying both RD and DR ticket splitting, EI–R when using the
covariates in Table 1 is logically inconsistent.
Burden and Kimball, it is worth pointing out, include South in their first stage ecological
inference model (thus including a form of the extended model in EI–R) and then estimate
RD and DR second stage regressions using all the covariates in Table 1. At the very least,
the t–statistics in this table show that Burden and Kimball should have included Spending
Ratio in the first stage as this covariate is significantly related to Xi ; without such inclusion,
second stage results are based on a logically inconsistent application of EI–R. However, as
13
Some House districts in 1992 had no incumbent, and hence Democratic Incumbent and Republican Incumbent do not sum to one.
16
noted already, covariates should never be included in both stages of EI–R. More comments
on this issue, and why it is important for Burden and Kimball’s study, appear below.
Given the specification test result in Table 1, we conclude that EI–R should not be used in
an analysis of ticket splitting based on Burden and Kimball’s dataset of 1992 House elections.
As discussed in the previous section, an option available to a researcher who nonetheless wants
to study ticket splitting with this dataset is King’s extended model, a model that can be used
to estimate directly how covariates affect ticket splitting rates.
Table 2 presents results of applying King’s extended model to Burden and Kimball’s
dataset.14 The table also reports verbatim Burden and Kimball’s second stage regression
estimates which, as is now known, are based on a logically inconsistent application of EI–
R.15
It is important to note that EI–R and extended model estimates for RD ticket splitting
(and DR ticket splitting as well) are not directly comparable. Extended model coefficients
are not regression coefficients but rather represent shifts in the mean of the distribution from
which βib and βiw are drawn. However, to determine whether Burden and Kimball’s overall
EI–R conclusions differ from those produced by an analysis of ticket splitting that uses King’s
extended model, one can compare the signs and the t–statistics of the RD and DR estimates
in Table 2.
In their analysis of RD ticket splitting, Burden and Kimball found that Democratic Incumbent, Spending Ratio, and South all have a positive and significant effect on Republican
presidential voters’ propensities to split their tickets. Moreover, Burden and Kimball found
that Ballot Format had a negative and significant effect. In the extended model estimates
for RD ticket splitting, however, only one of these results holds: Democratic Incumbent has
a positive and significant effect on RD ticket splitting. The other three coefficient estimates,
14
Our implementation of the extended model, which is based on standard maximum likelihood theory, does
not incorporate priors. We tried a large variety of different starting values for the extended model, found
two local maxima, and present the results for the maximum with the higher loglikelihood value. We are
confident that this is the true function maximum since there was a huge difference between the two maxima
we found and because the other local maximum was clearly substantively unreasonable; its estimated fraction
of Democratic presidential voters who voted for Democratic House candidates was only slightly higher than
its estimate of the fraction of Republican presidential voters who voted for Democratic House candidates.
15
The Burden and Kimball (1998) EI–R results are from Tables 6 and 7 on pp. 539 and 540, respectively.
17
Table 2: King’s Extended Model as Applied to Burden and Kimball’s
Analysis of House Elections
Variable
Democratic Incumbent
RD Estimates
Extended
EI–R
0.116∗∗∗
0.107∗∗∗
(0.0273)
(0.015)
—
—
DR Estimates
Extended
EI–R
—
—
0.151∗∗∗
(0.0360)
∗∗∗
Spending Ratio
0.0845
0.350
-0.588
(0.324)
(0.021)
(0.375)
Ballot Format
0.201
-0.032∗∗∗
0.277
(0.199)
(0.008)
(0.225)
South
0.2196
0.052∗∗∗
0.219
(0.183)
(0.009)
(0.212)
Note: standard errors in parentheses; ∗ p < 0.05, ∗∗ p < 0.01,
Republican Incumbent
0.085∗∗∗
(0.011)
-0.252∗∗∗
(0.015)
0.003
(0.006)
0.007
(0.006)
∗∗∗ p < 0.001
which in the logically inconsistent EI–R analysis are highly significant, are in fact statistically
insignificant once this problem is corrected for.
As a side note, Burden and Kimball’s RD analysis suffers from internal incoherence as
well as logically inconsistency. As noted earlier, Burden and Kimball include South in their
first stage ecological inference model; they allow, to be precise, this covariate to influence β iw ,
the probability that an individual votes RD. For reasons that are unclear, they do not allow
South to influence 1 − βib , the probability that the individual votes DR. Needless to say, the
first stage estimate of the impact of South on RD ticket splitting is 0.0622 with a standard
error of 0.0410 (t = 1.52).16 In other words, according to Burden and Kimball’s first stage,
there is no compelling evidence that South affects RD ticket splitting. But, according to their
second stage, South does affect RD ticket splitting. This situation is obviously troubling, and
it is caused by the inclusion of South at both stages of EI–R. Furthermore, a contradiction
between the two RD estimates of South is hardly surprising given that EI–R is logically
inconsistent when used to study RD ticket splitting.
16
This estimate does not appear in Burden and Kimball’s published paper. Rather, we produced it by
working with their dataset.
18
We now turn to Burden and Kimball’s analysis of DR ticket splitting. In the rightmost
two columns of Table 2, the logically inconsistent EI–R results indicate that Republican
Incumbent and Spending Ratio have highly significant effects on DR splitting. In the extended
model analysis of such voting behavior, though, Republican Incumbent has a significant effect
whereas there is no evidence to support the claim that Spending Ratio is similarly significant.
There are, notably, no results in Table 2 that are significant in an extended model analysis
yet insignificant in EI–R. This means that by employing EI–R in a logically inconsistent way,
Burden and Kimball generated evidence of “findings” that, in fact, are not findings once
logical consistency is adjusted for.17
Thus, we conclude that Burden and Kimball’s main results on the causes of ticket splitting
are generated by an inappropriate statistical method. Using a method that is logically consistent we have shown that, with one exception (the effect of incumbent partisan affiliation),
all of Burden and Kimball’s substantive findings are statistically unfounded. The exception
is a notable one as it is quite intuitive. Furthermore the fact the extended model analysis of
Burden and Kimball’s House data returns some non–null results means that its failure to do
so in other cases does not reflect an overall inability to estimate anything.
For instance, Burden and Kimball claim that, “The most robust finding from [various
applications of EI–R] is that congressional campaign spending has a dramatic influence on
the percentages of voters who split their ballot[s] (p. 542).” In fact, we believe that there
is no evidence in Burden and Kimball’s dataset to support this assertion. Similarly, Burden
and Kimball claim that statistical significance of South in the RD analysis leads to a rejection
of theories of intentional ticket splitting (p. 540). In fact, South is insignificant (t = 1.2) in
the logically consistent RD analysis and, therefore, Burden and Kimball’s dataset does not
cast aspersions on theories of intentional ticket splitting insofar as South is concerned. These
comments do not mean, of course, that all of Burden and Kimball’s conclusions are incorrect.
But, it does mean that their data do not support their claims about the causes of split ticket
17
To confirm that the differences identified in Table 2 were due to the distinction between using King’s
extended model and EI–R, we replicated Burden and Kimball’s second stage analysis and found the same
regression results that they reported in their paper.
19
voting.
Our results on Burden and Kimball’s House dataset and the inaccuracies we have uncovered raise an important question: are problems with EI–R common or is the example in
this paper anomalous? Without an exhaustive study of numerous research programs that
use second stage regressions, we will never know precisely how many published applications
of EI–R are based on logically inconsistent assumptions.
It is, of course, possible that many applications of EI–R are logically consistent. However,
we are aware of many instances of EI–R in which it is hard to imagine that elements of the
second stage covariate vector Zi are related to βib (as assumed in a second stage regression)
yet unrelated to Xi (as assumed in first stage ecological inference). For example, in Cohen,
Kousser and Sides’s (1999) study of crossover voting in state legislative elections in Washington and California, Xi is the fraction of individuals in district i who are Democrats.18
Moreover, one element of Cohen, Kousser and Sides’s second stage covariate vector Z i is an
indicator for whether there is a Democratic incumbent in district i. It is conceivable that
Democratic incumbency in district i is unrelated to whether district i has a large number of
Democrats in it. However, this seems unlikely. This reasoning, then, suggests that logical
consistency in Cohen, Kousser and Sides’s application of EI–R should be questioned.
6
Discussion
We have proposed a test for logical consistency of the statistical procedure EI–R and
illustrated its usefulness by applying the test to an ecological dataset from Burden and
Kimball (1998). The test easily rejects logical consistency for Burden and Kimball’s analysis
of the causes of ticket splitting in House elections. Moreover, when we correct for this, then
the vast majority of Burden and Kimball’s findings on the causes of split ticket voting vanish.
We suspect that this result is not anomalous and that many applications of EI–R are, like
Burden and Kimball’s study of House races, logically inconsistent.
If our critique of EI–R is taken seriously, we should expect to see numerous researchers
18
Cohen, Kousser and Sides’s notation uses Pi instead of Xi and Vi instead of Ti .
20
who wish to estimate second stage regressions instead relying on King’s extended model, a
model with does not suffer from logical consistency problems, and eschewing second stage
regressions entirely. This, however, raises an important issue about EI–R and the enterprise
of using covariates in ecological inference.
Rivers (1998) notes that King’s extended model is fragile in the sense of being barely
identified. The importance of this point is as follows: if it is difficult to identify using
King’s extended model the impact of covariates on unknown, disaggregated quantities, then
it should be commensurately difficult to use a second stage regression to do this. However,
second stage regressions are easily identified and they make it appear as if identifying the
impact of covariates on disaggregated quantities is a simple matter. However, it should not be
forgotten that second stage regressions are identified precisely because they treat ecological
inference point estimates (β̂ib , in our notation) as fixed. But, such estimates are not fixed.
Instead, they are functions of estimated bivariate normal parameters ( ψ̂).
This point might explain why researchers often turn to EI–R in the first place. Indeed,
one might logically wonder why anyone would ever estimate a second stage regression when
covariates to be included in such a regression can be incorporated into a first stage extended
ecological inference model.
We suspect that researchers may wish to continue using EI–R because applications of
King’s extended model are likely to generate numerous null results from which few substantive
conclusions about political behavior can be drawn. This is, in fact, what happened when
Burden and Kimball’s (1998) study of ticket splitting was subjected to a logically consistent
statistical design. A prevalence of null results does not reflect flaws in King’s ecological
inference technique; rather, null results reflect the difficultly of incorporating covariates in
ecological inference.
It is the nature of contemporary political science research that statistically significant
“findings” are valued more than null results. But, findings should not be taken seriously if
they exist solely because of a statistical quirk, i.e., solely because they appear in a logically
inconsistent application of EI–R yet not in a logically consistent ecological inference model.
21
Some problems in political science, and the study of ticket splitting may be one of these,
may simply be intractable with aggregate data, and it is better to recognize and accept this
than to attempt to extract unsupportable findings using a potentially misleading statistical
technique.
References
Achen, Christopher H. and W. Phillips Shively. 1995. Cross–Level Inference. Chicago: University of Chicago Press.
Burden, Barry C. and David C. Kimball. 1998. “A New Approach to the Study of Ticket
Splitting.” American Political Science Review 92(3):533–44.
Cho, Wendy K. Tam and Brain J. Gaines. 2000. “Reassessing the Study of Split–Ticket
Voting.” Working Paper, University of Illinois.
Cohen, Jonathan, Thad Kousser and John Sides. 1999. “Sincere Voting, Hedging, and Raiding: Testing a Formal Model of Crossover Voting in Blanket Primaries.” Paper presented
at the annual meeting of the American Political Science Association, Atlanta, GA.
Freedman, D.A., S.P. Klein, M. Ostland and M.R. Roberts. 1998.
“On ‘Solutions’ to
the Ecological Inference Problem.” Journal of the American Statistical Association
93(444):1518–22.
Gay, Claudine. 2001. “The Effect of Black Congressional Representation on Political Participation.” American Political Science Review 95(3):589–602.
Herron, Michael C. and Kenneth W. Shotts. 2001. “Using Ecological Inference Point Estimates as Dependent Variables in Second Stage Linear Regressions.” Forthcoming, Political Analysis.
22
Kimball, David C. and Barry C. Burden. 1998. “Seeing the Forest and the Trees: Explaining
Split–Ticket Voting within Districts and States.” Paper presented at the annual meeting
of the American Political Science Association, Boston, MA.
King, Gary. 1997. A Solution to the Ecological Inference Problem. Princeton, NJ: Princeton
University Press.
Lublin, David and D. Stephen Voss. 2000. “Context and Francophone Support for the
Sovereignty of Quebec: An Ecological Analysis.” Forthcoming, Canadian Journal of
Political Science.
McCue, Kenneth F. 2001. “The Statistical Foundations of the EI Method.” The American
Statistician 55(2):106–110.
Rivers, Douglas. 1998. “Review of ‘A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data’.” American Political Science Review
92(2):442.
Tam, Wendy K. 1998. “Iff the Assumption Fits...: A Comment on the King Ecological
Inference Solution.” Political Analysis 7:143–63.
Voss, D. Stephen and David Lublin. 1998.
“Ecological Inference and the Comparative
Method.” APSA-CP: Newsletter of the APSA Organized Section in Comparative Politics
9(1):25–31.
Voss, D. Stephen and Penny Miller. 2000. “The Phantom Segregationist: Kentucky’s 1996
Desegregation Amendment and the Limits of Direct Democracy.” Working Paper, University of Kentucky.
Wolbrecht, Christina and J. Kevin Corder. 1999. “Gender and the Vote, 1920–1932.” Paper
presented at the annual meeting of the American Political Science Association, Atlanta,
GA.
23