Download A Specification Test for Linear Regressions that use

A Specification Test for Linear Regressions that use
King–Based Ecological Inference Point Estimates as
Dependent Variables1
Michael C. Herron2
Kenneth W. Shotts3
August 16, 2000
1
Paper prepared for the 2000 Annual Meeting of the American Political Science Association, Washington,
D.C.
2
Center for Basic Research in the Social Sciences, Harvard University, and Department of Political
Science, Northwestern University. Address: 34 Kirkland Street, Cambridge, MA 02138. Email: [email protected].
3
Department of Political Science, Northwestern University. Address: Scott Hall, 601 University Place,
Evanston, IL 60208–1006. Email: k–[email protected].
Abstract
Many researchers use point estimates produced by the King (1997) ecological inference technique
as dependent variables in second stage linear regressions. We show, however, that this two stage
procedure is at risk of logical inconsistency. Namely, the assumptions necessary to support the procedure’s first stage (ecological inference via King’s method) can be incompatible with the assumptions supporting the second (linear regression). We derive a specification test for logical consistency
of the two stage procedure and describe options available to a researcher whose ecological dataset
fails the test.
1 Introduction
Many researchers use point estimates produced by the King (1997) ecological inference technique as dependent variables in second stage linear regressions. King advocates this two stage
statistical procedure, which we call EI–R, saying that “[it] has enormous potential for uncovering
new information” (p. 279). Burden and Kimball (1998), for example, use King–based estimates
of district–level ticket splitting rates as dependent variables in second stage regressions which examine why individuals cast split ticket ballots. Similarly, Gay (1997) employs estimates of Black
turnout rates, produced by King’s ecological inference technique, in regressions that seek to determine whether black citizens vote at higher rates in elections that include black Congressional
candidates. Other examples of EI–R can be found in Kimball and Burden (1998), Voss and Lublin
(1998), Cohen, Kousser and Sides (1999), Wolbrecht and Corder (1999), Lublin (2000), and Voss
and Miller (2000).
As evidenced by the multiple citations above, EI–R has achieved widespread usage despite the
fact that it was only recently proposed. We show, however, that applications of this two stage procedure can be logically inconsistent in the sense that the assumptions necessary to support the procedure’s first stage (ecological inference via King’s method) can be incompatible with the assumptions
supporting the second (linear regression). In contrast, we say that an application of EI–R is logically
consistent when its first stage and second stage assumptions are not mutually contradictory.
The matter of logical consistency, as distinct from statistical consistency, of the two stage procedure EI–R merits consideration for the following reason: when combining two statistical techniques,
a researcher must simultaneously adopt the assumptions that support the technique’s first and second
stages. Previous work on EI–R has not sought to analyze the conditions under which its two sets of
underlying assumptions – those associated with first stage ecological inference and those with second stage linear regression – can be simultaneously adopted. In fact, to the best of our knowledge
all published and working–paper implementations of EI–R simply assume that the procedure’s first
and second stages are inherently compatible, i.e. that EI–R is logically consistent.
We argue here that this assumption is not always warranted. In particular, our analysis of EI–
R and its potential for logical inconsistency is based on the fact that standard implementations of
1
King’s technique to an ecological dataset (the first stage of EI–R) assume that there is no aggregation
bias. Nonetheless, we show that a second stage linear regression which uses King–based point
estimates as dependent variables (the second stage of EI–R) can imply the existence of aggregation
bias in the first stage.
As an illustration of this clash of assumptions, consider a classic case of ecological inference:
a researcher wants to estimate and explain disaggregated turnout rates for African–American and
white voters. Suppose, however, that she has only precinct–level racial composition and turnout
data. To apply King’s ecological inference technique as a component of EI–R to this turnout problem, the researcher must assume that there is no aggregation bias in her turnout dataset, i.e., that
black turnout is uncorrelated with the fraction of a precinct’s population that is African–American.
Now, suppose that the researcher estimates a second stage regression in which King–based
estimates of black turnout rates are assumed to be a linear function of precinct–level income and
education levels and other politically–relevant independent variables. If any of these right hand side
variables are correlated with the fraction in a precinct’s population that is African-American, then
the researcher’s second stage regression model virtually guarantees (in a sense that we make precise
later) that black turnout is correlated with the fraction of a precinct’s population that is AfricanAmerican, i.e., that there is aggregation bias. Thus, in this example the assumptions behind the
second stage of EI–R contradict those in its first stage. In particular, second stage linear regression
implies the existence of first stage aggregation bias, this despite the fact that the first stage assumed
that no aggregation bias was present.
This is problematic at two levels. At a fundamental level, any standard philosophy of science requires researchers to adopt internally consistent sets of assumptions. To do otherwise, as is possible
with EI–R, is to state simultaneously “A is true” and “A is false.” Once two mutually contradictory premises are adopted, any implication can be logically derived, i.e., all subsequent analysis is
meaningless.
At a more pragmatic level, logical inconsistency of EI–R is problematic because, in the presence of aggregation bias, King’s ecological inference technique produces statistically inconsistent
2
estimates of all the first stage population parameters that it tries to estimate.1 And, when all first
stage point estimates are inconsistent, second stage regression results – that is, EI–R’s final results –
will also be statistically inconsistent in ways that, notably, cannot be readily rectified. An estimator
that is statistically inconsistent is generally useless because it does not converge in probability to its
true value; rather, such an estimator tends to an incorrect value as the sample on which it is based
gets large.
Given the problems which result if EI–R is logically inconsistent, it is important for researchers
to determine whether a particular application of the technique is problematic. Fortunately, it turns
out that not all implementations of EI–R are logically inconsistent. Rather, we show that logical
consistency of EI–R is application–dependent, i.e., EI–R is logically consistent in some empirical
applications yet logically inconsistent in others.
Thus, we propose a simple specification test for logical consistency of EI–R. The test is implemented via a single linear regression using data from both the first and second stages of a proposed
application of EI–R. Interpreting the results of the specification test requires consideration of only a
single F statistic, the well–known and popular statistic which tests the null hypothesis that all slope
coefficients in a linear regression are zero. If the F statistic produced by an application of the test
to a given dataset is statistically significant, then EI–R is logically inconsistent when applied to this
dataset and should not be used. If, on the other hand, the F value is not significant, then logical
consistency of the application of EI–R cannot be rejected.
It is worth noting that, even in an application where EI–R is logically consistent, second stage
regression results will almost certainly be statistically inconsistent. This is because, in general,
errors in King–based estimates of disaggregated quantities are inversely correlated with the true
parameter values that the estimates measure. Fortunately, it is possible to correct for this statistical
inconsistency so long as EI–R is logically consistent (Herron and Shotts 2000).
However, when the test for logical consistency of EI–R rejects in a given ecological dataset, thus
implying the presence of aggregation bias, a researcher who wants to study the dataset has only two
options. One, she can abandon EI–R entirely and choose a different method of analysis. Or, two,
1
See Tam (1998) for a thorough discussion of the effects of aggregation bias on King’s technique.
3
she can explicitly model the aggregation bias in the first stage of the EI–R procedure. We discuss
this latter option in some detail.
We emphasize that our critique of EI–R and King’s ecological inference technique is fundamentally different from the argument in Tam (1998) and Cho and Gaines (2000), both of which
claim that aggregation bias is a common phenomenon in aggregate data and that King’s diagnostics provide little help in identifying and rectifying it. Our analysis, in contrast, is not based on a
claim that the assumptions supporting King’s ecological inference technique are unwarranted, either in general or in a particular empirical application. Rather, our critique applies only to the use
of King’s technique within the context of the combined EI–R procedure, and what we argue is that
the assumptions which support EI–R must be compatible across EI–R’s two stages.
Furthermore, our test for logical consistency of EI–R should not be thought of as a generic test
for aggregation bias in ecological data. Rather, the test asks whether the second stage of EI–R
implies that there is aggregation bias in the first stage. The test does not explore whether there are
other sources of aggregation bias that could cause first stage problems.
In Section 2 we provide notation and details on EI–R. Then, in Section 3 we present our specification test for logical consistency of EI–R and explain how to interpret it. Section 4 considers the
options faced by a researcher whose ecological dataset fails the test, and in Section 5 we apply the
test to the ecological dataset used in Burden and Kimball (1998). Section 6 offers general comments
on EI–R and concludes.
2 Overview of the EI–R Procedure
The first stage of EI–R consists of an application of King’s ecological inference technique, and
the second stage requires estimating a linear regression based on first stage output. Thus, we now
provide some details on ecological inference and King’s technique in particular and then explain
how the output of the technique is used in second stage regressions.
Suppose we have precinct–level data for an election of interest: let
denote precinct ’s turnout rate in the election and The goal of ecological inference is to use
and 4
, ,
the fraction of Blacks in precinct .
to estimate , the fraction of Blacks in precinct
who voted in the election, and , the corresponding fraction of Whites who voted. Achen and
Shively (1995) discuss the history of the so–called ecological inference problem and review various
solution attempts.
King distinguishes between two different implementations, standard and extended, of his ecological inference technique. The precise details of these implementations can be found in Chapter
6–8 of King’s book. Critiques and discussions of King’s approach to ecological inference can be
found in Freedman, Klein, Ostland and Roberts (1998), Tam (1998), and Cho and Gaines (2000).
Until further notice our comments on King’s technique apply to its standard implementation or what
we also call the standard model.
have a truncated bivariate normal distribu
tion on the unit square with means "! ! , standard deviations #$ $
&% and correlation ' . In
2
King’s notation, these parameters are collected in a 5–vector ()+* !, ! $ $
'- . Notably,
The standard implementation assumes that the two means ! and ! are fixed across , and this is the key feature of the standard model.
Furthermore, the standard model assumes .0/21 where .0/213#54 4 denotes covariance.
%
This is equivalent to assuming there exists no aggregation bias, i.e. there is no systematic relation-
ship between , the fraction of Blacks in precinct who voted, and , the fraction of Blacks in
precinct .3
6 , ! 6 , $ 6 , $ 6 , and ' 6 denote the point estimates of !, , ! , $ , $ , and ' , respectively,
Let !,
based on applying King’s standard model to an ecological dataset.4 Corresponding to the vector ( ,
the five point estimates are collected in a 5–vector ( 6 . Under standard model assumptions – recall,
one of which is that there is no aggregation bias – ! 6 8;=7:< 9
! as
tends to infinity where ;=7>< 9
denotes convergence in probability. In other words, ! 6 is a consistent estimate of ! , and a similar
6 ;@7:< 9 ( .
statement applies to the other four point estimates in ( 6 . In general, then, (?
As described in King’s book, the elements of A correspond to elements in a related 5–vector BA insofar as A is a
function of BA , i.e., knowledge of BA implies knowledge of A . In this paper we ignore BA and this is without loss of
generality. King’s motivation for parameterizing the elements of A with the elements of BA is numerical rather than
substantive.
3
HJKM
I L In. addition, King’s technique assumes that there is no spatial autocorrelation, that CED and CGF are independent for
4
King’s ecological inference technique can be estimated using standard maximum likelihood theory. Or, optional
priors can be used in which case King’s model yields a Bayesian framework. Both of these implementations of King’s
technique are compatible with the results described here.
2
5
Let 6 be the point estimate of produced by applying King’s technique to an ecological
dataset and
, N
O
(6 is a function of ( 6 , , and
). In EI–R, the 6 point esti-
mates are used as dependent variables in a second stage regression. Namely, King suggests that a
researcher who in theory might want to estimate
5
)PRQTSUWV QYX
(1)
5
6 )PRQTS U V QYX
(2)
instead estimate
where the difference between equation (1) and (2) is that 6 (observed) substitutes for (unobserved) in the latter. Here, P is a constant and S a Z –vector of parameters to be estimated, V
is a
Z –vector of exogenous variables, and X is a disturbance term assumed to be uncorrelated with and 6 . Let P 6 and S 6 denote the least squares estimates of P and S , respectively.5
Thus, the EI–R procedure consists of, first, estimating 6 using King’s ecological inference
technique and, second, estimating P 6 and S 6 with least squares applied to equation (2).6 The final
product of EI–R, therefore, consists of the estimates P 6 and S 6 , and, ultimately, whether EI–R is
; >7 < 9
statistically consistent in an overall sense depends on whether P 6 P and S 6 @; :7 < 9 S . On the other
hand, what can be thought of as first stage statistical consistency depends on whether ( 6 ;=7>< 9 (
where the [ –vector ( 6 , as noted previously, is the first stage estimate produced by applying King’s
standard model to an ecological dataset.
Suppose that EI–R is logically consistent so that its first and second stage assumptions do not
contradict one another. Then, presuming all assumptions behind King’s technique are satisfied,
( 6 =; :7 < 9 ( , meaning that EI–R’s first stage is statistically consistent. In this case of logical consis\
\
\
tency, except in very unusual circumstances P 6 ;@7:< 9 P and S 6 ;=7>< 9 S where ;=7:< 9 denotes a lack of
5
Equations (1) and (2) could just as easily have had ]:D ^ and :_] D ^ on their left hand sides. All of our results apply in
a symmetric fashion to such a case. Similarly, the results apply to regressions with `aN]:D b and `Ja _] D b as left hand side
variables as well.
6
King suggests a weighted least squares approach when estimating equation (2). Weighting, however, is not germane
to our discussion of logical inconsistency.
6
convergence in probability (Herron and Shotts 2000). In other words, EI–R is in general statistically
; \ 7>< 9
inconsistent when it is logically consistent; this is because 6 even when the first stage of
EI–R is statistically consistent. However, Herron and Shotts describe manipulations of P 6 and S 6
that produce consistent estimates of the population parameters P and S in equation (1) based on
inconsistent estimates produced by equation (2). Overall, then, when EI–R is logically consistent
; \ 7:< 9 P and S 6 ;=\ >7 < 9 S , it is possible
its first stage is statistically consistent and, despite the fact that P 6 =
to manipulate P 6 and S 6 so as to generate consistent second stage estimates of P and S .
; :7 < 9
These manipulations are possible only when ( 6 =
( , i.e. only when first stage ecological in-
ference estimates of the vector ( are statistically consistent. This is a key point for the following
reason: when EI–R is logically inconsistent then its second stage invalidates an assumption that
supports its first stage. Hence, logical inconsistency of EI–R implies first stage statistical incon-
; \ 7:< 9
sistency (( 6 =
( ) and EI–R completely falls apart.7 In particular, as far as are aware, no useful
manipulations of P 6 and S 6 are possible when EI–R is logically inconsistent. In this latter situation,
these point estimates are hopelessly statistically inconsistent and this cannot be rectified.
3 The Logical Consistency Test for EI–R
Recall that EI–R’s first stage assumption of no aggregation bias is mathematically equivalent to
.c/G1 . However, second stage regression implies dPeQfS U V QX from equation
(1). Hence, for EI–R to be logically consistent across its first and second stages, it must be true that
h
.c/210g " .c/21ijPRQTS U V Y
Q X " .c/21 S U V QYX l
q p 5 k S m c
. /213#jV m % Qr.c/213#X %
mnJo
7
King (1997) argues in Ch. 11 that his technique sometimes produces accurate estimates of A even in the presence
of aggregation bias. However, Tam (1998) argues that, in such situations, the technique produces inaccurate estimates.
In our opinion, the fact that an inconsistent estimator happens to be on target in some applications does not constitute
adequate justification for adopting it. Namely, inconsistent estimators are guilty until proven innocent, and, at the time of
this writing, the value of King’s technique in the presence of aggregation bias remains uncertain.
7
t p
where S m is the s th element of the parameter vector S and where V m is the s th element of the vector
V , su
Z . This derivation forms the basis of the following result:
Proposition 1 Suppose that an ecological dataset, consisting of ,
, and V , , is
given. Let S be the true second stage parameter vector from equation (1). When it is not the case
that
l
t p " . /213#vV m % Qr.c/213#vX % k S m c
mnJo
(3)
then EI–R as applied to the given dataset is logically inconsistent.
Proposition 1 places a restriction on the parameter vector S that must be satisfied for EI–R to be
logically consistent when applied to a given ecological dataset. When the restriction is violated, then
EI–R is logically inconsistent. Moreover, as described earlier, logical inconsistency of EI–R implies
; \ 7:< 9 ( ) and correcting resulting second stage inconsistencies
first stage statistical inconsistency (( 6 =
in P 6 and S 6 is not possible.
Suppose that, for a given ecological dataset, the ZwQ
covariances which appear in equation (3)
all happen to be exactly zero. Then, any Z –vector S , regardless of its elements, is consistent with
the restriction in Proposition 1. This is because the elements S m , when substituted into equation (3),
will be multiplied by zero and hence have no impact on whether the left hand side of the equation
is truly zero. In other words, when all covariances in equation (3) are zero it is not possible for the
elements of S to violate the restriction in Proposition 1 and in so doing cause rejection of logical
consistency of EI–R.
% terms are not zero, then the set of S vectors
which satisfy the restriction in Proposition 1 is not equivalent to the set of all possible S vectors.
" tp m % ,
For example, suppose that ZM)x so that S is a 2–vector, .c/213#vX % , and .c/G13#jV
or,
s
x . Then, for EI–R to be logically consistent it must be the case that S o QS:yY
equivalently, S o ; S:y . This example illustrates in a simple manner the fact that, when some of
tp the .c/G13#vV m % terms are non–zero, elements of the vector S will have to satisfy a non–trivial
On the other hand, if some of the .c/G13#vV
tp m restriction for EI–R to be logically consistent.
8
Thus, we can think of Proposition 1 as having two cases. In one case – when all covariances
in equation (3) are zero – there is no restriction on S . In the other case – when at least one of
the .c/G13#jV
tp m % terms is non–zero – the restriction in equation (3) is meaningful and logical
consistency of EI–R depends on whether S satisfies it.
When Proposition 1 imposes a meaningful restriction on S , we can enquire as to the nature of
the restriction. Is it meaningful yet weak – for example, requiring that S m{z
tight – for example, requiring, say, that S m |
for all s – or is it
for all s ? If the former, then we would say that the
restriction is relatively easy to satisfy and hence there are no compelling reasons to reject logical
consistency of EI–R based solely on the fact that S must satisfy the restriction.
On the other hand, if the restriction on S is very tight, then it is certainly possible that most
S vectors will fail to satisfy it. From this it would follow that EI–R is unlikely to be logically
consistent. Similarly, if given an ecological dataset the associated restriction on S is so tight that
is seems unimaginable that it can be satisfied, then we would commensurately argue that, in all
likelihood, EI–R is logically inconsistent when applied to the said dataset.
We now develop this idea of a very tight restriction on S , a restriction that seems virtually
impossible to satisfy. In particular, our conceptualization of such a restriction is one that will be
satisfied with probability zero if S is drawn randomly from a continuous distribution. To motivate
this conceptualization, consider the following example. Let ZM)x and suppose as before that logical
; S:y . If SY}#qS o S:y is drawn from a continuous distribution
%
y
o
;
Sy is zero since a line has zero probability measure on
on ~ , then the probability that S a two–dimensional plane. Hence, we say that the restriction S o ; Sy is very tight and, when
consistency of EI–R requires S o logical consistency of EI–R depends on it, we conclude that EI–R will in all likelihood be logically
inconsistent.
In general, when the Z –vector S is randomly drawn from a continuous probability distribution
on ~
k
, then the probability that 
of dimension 
elements of the vector, with
; is zero. Thus,
u€

€
Z , lie a given hyperplane
Definition 1 A Z –vector S from equation (1) is said to be non–generic if, for
elements of S must lie in an  ;
dimensional hyperplane.
9
f€

€
Z ,
Definition 1 codifies the idea of a very tight restriction in the sense that S is said to be non–generic
when it must satisfy such a restriction.
For example, suppose that S is a  –vector ( Z‚ ) and that a restriction from equation (3)
implies that two (ƒ„x ) of its elements must lie in a one (
; }x ; ) dimensional
hyperplane. (This is a complicated way of saying that these two elements must lie on a line.) In
general, the probability that two continuously distributed random variables fall exactly on a given
line is zero. Hence, if all Z…† elements of S are drawn from a continuous distribution on ~ ‡ , the
probability that the two key elements satisfy the restriction of being on a given line is zero.
Of course, in actual applications of EI–R it is not the case that S is drawn from a continuous distribution prior to EI–R’s being employed. Rather, according to standard frequentist logic, estimating
a model like equation (2) is equivalent to supposing that there exists a true S and that this value is
fixed albeit unknown. Nonetheless, the idea behind a restriction which leads to a non–generic S is
that the restriction is so tight that we should expect it never to hold. Hence, if we conclude that a
given application of EI–R is logically inconsistent unless S is non–generic, then we have effectively
concluded that the application is in all likelihood logically inconsistent. In particular, we would say
in this instance that EI–R is generically logically inconsistent.
Definition 2 An application of EI–R to an ecological dataset is said to be generically logically
inconsistent if the restriction in Proposition 1 requires S to be non–generic.
It is straightforward to see that, if any of the .c/213#jV
qp m % terms in equation (3) are non–zero,
then the S vector which satisfies the resulting restriction is non–generic. If, say, only one covariance
is non–zero, then the associated element of S , that which multiplies the non–zero covariance, will
itself have to be exactly zero. If not, the restriction in equation (3) will not be satisfied unless
.c/G13#X
"
% exactly offsets the product of the non–zero covariance and the non–zero element of S .
This sort of offsetting will occur only by complete coincidence. Furthermore, the probability that
a continuously distributed random variable on ~
Hence, when only a single .c/G13#jV
tp m is exactly zero is itself a probability zero event.
% term is zero, S is non–generic.
We have now argued the following.
10
Proposition 2 Let S be the true parameter vector from equation (1). If there exists a s such that
ˆ€
s
tp \ Z where .c/G13#jV m % , then S is non–generic.
€
Hence, whenever an ecological dataset features at least a single, non–zero .c/G13#jV
tp m % term,
EI–R as applied to the dataset is generically logically inconsistent.
The result in Proposition 2 is, importantly, independent of whether .c/213#jX
5
% is zero. In fact,
this latter covariance only makes matters worse from the perspective of satisfying the restriction in
tp m equation (3). Namely, all .c/G13#jV
.c/G13#X
"
% \ % terms can be zero but the restriction can still be violated if
. Hence, by ignoring this latter covariance, i.e., assuming that it is zero, our test
for logical consistency of EI–R is conservative.8
The matter of whether a proposed application of EI–R is generically logically inconsistent, then,
turns on whether there exists a .0/213#jV
tp m estimated by regressing vector. Then, if the standard F test for all slope coefficients
on the V
% term that is non–zero. Such covariance terms can be
being zero rejects at a conventional confidence level, it follows that EI–R is generically logically
inconsistent. Thus, the hypothesis test for logical consistency of EI–R requires nothing more than
estimating a single regression with data that EI–R itself requires in the first place. This regression
can, and should, be run before any application of EI–R.
The intuition behind our regression–based test for logical consistency in EI–R is as follows.
Recall that King’s ecological inference technique requires that there is no aggregation bias, i.e. that
.c/G1 . But, if a second stage regression assumes that J is a function of covariates
collected in a vector V
and, if some of the elements in V
it is not possible for .c/G1i are correlated with , then in general
to be zero. Thus, when the test for logical consistency of EI–R
rejects, it follows that there is aggregation bias in the first stage and EI–R in this situation completely
breaks down.
The matter of non–rejection of logical consistency of EI–R is an important one. The test we
propose for logical consistency is one of necessity as opposed to one of sufficiency. That is, an
8
Our discussion of restrictions and logical consistency of EI–R began with the premise that an ecological dataset is
given, and then we considered restrictions on ‰ such that equation (3) holds. We could, alternatively, have started from
the premise that the Š elements of ‰ were given and asked about the resulting ecological dataset which would satisfy the
equation. From this latter perspective, when any elements of ‰ are non–zero, the set of covariances which satisfy equation
(3) will be non–generic. Hence, it follows that EI–R as applied to the dataset will be generically logically inconsistent.
11
application of EI–R must pass our test; if not, it follows that the application is logically inconsistent.
However, if an application of EI–R does pass the test, it does not therefore follow that the application
is logically consistent. Rather, it is still possible that the application is logically inconsistent in some
manner not covered by our test.9 Or, the failure to reject could simply reflect that fact that our test
does not have a power of one. Similarly, if an application of the test does not reject, then it does not
follow that the ecological data used in the application are free from aggregation bias.10
For the remainder of this paper, we use the term “logically inconsistent” in place of “generically
logically consistent” as the latter is tedious. Similarly, we refer to our test as a test for logical
consistency even though, technically, it is a test for generic logical consistency.
4 What to do when an Ecological Dataset Fails the Specification Test
When a researcher’s ecological dataset fails the test for logical consistency of EI–R, she has two
options. First, she can abandon EI–R entirely and choose another method of data analysis. Second,
she can model the aggregation bias that caused test failure and in so doing avoid a second stage
regression. We now discuss the latter option and later comment on the former.
Modeling aggregation bias in the first stage of an application of EI–R requires what King calls
his extended model. Recall that in the standard implementation of his ecological inference technique
c is j! ! , and this mean vector is assumed to be fixed across observations.
the mean of J However, in the extended model the two elements of this vector are not restricted in this way. Rather,
O where subscripts denote dependence on
in the extended model the means are denoted ! ! observation .
Readers interested in the details of the extended model should consult King (1997, pp. 168–74).
In brief, this model posits that !, is a linear function of exogenous covariates. That is, the extended
model assumes something akin to ! Œ‹ UŽ
estimated and 
where ‹ is a vector of parameters that must be
a vector of covariates that includes a constant. A product of the extended model,
then, is an estimate 6‹ of the parameter vector ‹ . For numerical reasons (see p. 170) the extended
For example, logical consistency of EI–R will impose certain restrictions on the distribution of  D . Although we do
not explore the nature of these restrictions, it seems quite plausible that these restrictions are extremely tight.
10
For instance, the second stage could be misspecified due to omitted variables that are correlated with  D .
9
12
model does not assume the precise form noted above for !, . However, the above expression for !,
captures the essence of the model.
The idea behind the extended model is to capture dependence of on exogenous covariates
that are collected in what we have called the 
vector. For instance, if 
contains only a single
non–constant covariate, the corresponding element of ‹ describes how this covariate influences .
When, say, the single element of ‹ is positive, then it follows that large values of 
are associated
with large .
Thus, suppose that a researcher has an ecological dataset and on it carries out the test for logical
consistency of EI–R. Suppose as well that the test rejects. This implies that there is aggregation bias
in the dataset insofar as a researcher wants to assume that both his first and second stages apply. To
model this aggregation bias explicitly, the hypothetical researcher would include as extended model
covariates – that is, as elements of the vector 
noted above – all elements of V
and subsequently
would not estimate a second stage regression at all. In other words, when the specification test
rejects logical consistency of EI–R, then the two stage statistical procedure should not be used.
Rather, an abbreviated version – consisting solely of an application of King’s extended model –
should replace it; in this instance, dependence of on exogenous covariates should be analyzed
via first stage ecological inference and what we have labeled ‹ as opposed to the second stage
parameter vector S .
To summarize, the procedure we propose for EI–R is as follows.
1. Collect an ecological dataset consisting of 2. Regress on the elements of the vector V
,
, and V , ‘
J
.
and consider the F test that all slope coefficients
are zero. Then,
(a) If the F statistic does not reject, implement EI–R using the method described in Herron
and Shotts (2000).
(b) If the F statistic rejects, apply King’s extended model using all elements of V
as covari-
ates for parameterizing the means ! and ! of and , respectively.
The ultimate objective of EI–R is determining the extent to which elements of the exogenous
13
covariate vector V
are related to . When standard EI–R is employed because the test for logically
consistency did not reject, inference on the impact of V
can be based on second stage regression re-
sults, specifically, on confidence intervals for V –related slope coefficients produced by the method
described by Herron and Shotts.
On the other hand, when King’s extended model is employed as the first (and only) component
of an abbreviated EI–R, assessing the impact of V
on requires considering first stage standard
errors produced by King’s technique. Namely, for each covariate (akin to an element of 
, above)
used to parameterize the mean ! of J in the extended model, King’s method produces a coefficient estimate (akin to an element of 6‹ ) and an associated t–statistic. Standard significance tests
using these t–statistics (one for each element of V ) can form the basis of determining whether the
elements of V
are related to .
Suppose, for example, that V
contains only a single non–constant covariate and that, based
on failure of the logical consistency test, this covariate is included in an application of King’s extended model. Associated with the covariate will be two coefficient estimates, one which describes
dependence of on the covariate and which describes dependence of . Depending on what a re-
searcher is interested in studying – in other words, depending on the substantive meanings of and
in her problem – she would consult one or perhaps both of these estimates and, using estimated
standard errors produced by King’s extended model, form t–statistics and carry out hypothesis tests
in the usual fashion.
A researcher may be tempted to include all or some elements of V
in both an application of
King’s extended model as well as in a second stage regression. This practice – a concrete example
can be found in the next section – should be avoided as it can lead to internal incoherence. In
other words, we contend that either King’s standard model should be used followed by a second
stage regression or King’s extended model should be employed and there should be no second stage
regression. Mixing the extended model with a second stage regression can lead to serious problems.
Suppose, to illustrate this point, that a researcher gathers an ecological dataset and on it carries
out the test for logical consistency of EI–R. Suppose as well that the test rejects and that only one
coefficient estimate from the test’s regression is individually significant at, say, the
14
’
[ level. The
researcher might want to include the variable associated with the coefficient (that is, one element of
the V
V
vector) in an extended King model and then estimate a second stage regression using the full
vector.
In such a case, the one variable which is significantly related to in the specification test would
appear twice in the EI–R procedure insofar as it influences in the first stage of EI–R and then
influences again in the second stage. Yet, both estimates of the impact of the variable are based
on a single ecological dataset. Thus, the variable’s appearing twice means that there will be two
separate and dependent estimates of the impact of the variable on . Simply put, it is impossible to
say how the two associated t–statistics should be combined and interpreted. It is conceivable, and
we provide an example of this shortly, that the first stage estimate of the impact of the covariate on
can be statistically insignificant while the second stage estimate of the impact of the covariate is
significant. When all the V covariates are used only once, either in a second stage regression or in
an application of King’s extended model, this type of incoherence can be avoided.
Finally, a practice to which special attention is warranted is the inclusion of of the vector of exogenous covariates V
as an element
(e.g. Gay 1997, Lublin 2000). This practice – which
assumes explicitly that is a function of or incoherent insofar as – guarantees that EI–R is either logically inconsistent
appears twice in the EI–R analysis. Thus, if a researcher believes that
is dependent on , then the latter should be included as a parameter in the first stage ecological
inference step along with all other exogenous variables of interest. In this situation, no second stage
regression should be estimated.
In light of this possibility – including in a second stage regression – we now comment on
what seems like an attractive approach to EI–R but one that is flawed. Suppose that a researcher
wants to employ EI–R with a second stage regression that includes as an element of the exoge-
nous covariate vector V . The research wants to carry out our specification test by regressing the various elements of V . But, since stead regress on all the elements of V
on
itself is an element of V , she reasons that she should inexcept for . Suppose that she carries out this regression
and the resulting F test does not reject. That is, the test for all slope coefficients being zero does not
reject.
15
Following up on the non–rejection of the abbreviated specification test – so–called since regressed on an abbreviated V
model in which was
vector – the researcher then estimates a version of King’s extended
is included as the only exogenous covariate. Suppose that the resulting two
coefficient estimates – one describing dependence of on and one describing dependence
of – are statistically insignificant. Based on this result in conjunction with non–rejection of
the abbreviated specification test, the researcher might then want to conclude that she can safely
estimate a second stage regression, using the complete V
vector which includes itself, and not
worry about logical consistency.
However, the researcher should not conclude this. To do so would ignore the fact that King’s
extended model does not have a power of one when estimating dependence of (and , for that
matter) on . If the extended model coefficient estimates which describe the relationship between
and are not significant, it does not follow that they are zero.
That is, estimating a standard King model assumes that .c/21“J \
a second stage regression immediately implies that .c/G1 . Including in
, regardless of first stage
coefficient estimates. Therefore, the particular EI–R practice described above, while seemingly
attractive, should never be employed. It guarantees logical inconsistency.
5 An Application of the Test for Logical Consistency in EI–R
We now apply the test for logically consistency in EI–R to the ecological dataset used by Burden
and Kimball (1998) in their published study of ticket–splitting in the United States Congressional
and Presidential elections of 1992.11 Our application also illustrates in a concrete way the problems
that can arise if elements of an exogenous covariate vector are included in both the first and second
stages of an implementation of EI–R.
We obtained the Burden and Kimball data and divided it into House and Senate sections.12
For Burden and Kimball’s House data, the variable represents the proportion of individuals in
11
Cho and Gaines (2000) identify several errors in Burden and Kimball’s dataset, but we use the original dataset to
maintain compatibility with the published study. Our critique of Burden and Kimball is quite different from that offered
by Cho and Gaines, who argue, among other things, that King’s diagnostics indicate the presence of aggregation bias in
the data set.
12
The data is available from the ICPSR and in particular from ftp://ftp.icpsr.umich.edu/pub/PRA/outgoing/s1140.
16
Congressional district who voted for Michael Dukakis in the 1992 presidential race. And,
is the fraction who voted for the Democratic House candidate, J is the fraction who voted for
a Democratic Congressional candidate as well as for Dukakis, and is fraction who voted for
a Democratic Congressional candidate yet also voted for Bush. Thus, in district the fraction of
ticket–splitters is Q
; .
Furthermore, for the House analysis there are a total of four elements of the vector V . These
consist of an indicator as to whether district had a Democratic incumbent (“Democratic Incumbent”), the Democratic candidate’s proportion of House campaign spending (“Spending Ratio”),
whether district allowed straight–party votes (“Ballot Format”), and whether district was located
in the South (“South”). In accordance with the test for logical consistency of EI–R, we regressed
on the elements of V
and a constant. Resulting coefficient estimates and the overall F statistic
are in Table 1.13
In Burden and Kimball’s Senate data the variables noted above refer to states rather than Congressional districts, but the Senate V
vector contains one additional covariate (“Ideological Dis-
tance”) compared to the House version of V . This latter covariate measures the ideological distance
between George Bush and the Democratic Senate candidate in state . Senate results are in Table 1,
and all t–statistics in the table were calculated using heteroskedastic–consistent standard errors.
Before examining the results in Table 1 we consider at an intuitive level whether we expect be correlated with the elements of the V
voters in district (or state) and V
vector. Recall that to
represents the fraction of Dukakis–
includes, among other things, a South indicator variable. Given
all that is known about Democratic politics in the South, it would be hard to imagine that is not
correlated with whether district is in the south.
With respect to the House regression model, the F statistic for the hypothesis test that all slope
coefficients are zero is [•”
—–
. The associated p–value is approximately zero. This means that EI–R,
when its second stage regression uses the House variables listed in Table 1, is logically inconsistent.
In their analyses of ticket splitting in House and U.S. presidential elections, Burden and Kimball
include the south indicator in a first ecological inference model and subsequently include all the
13
Since Burden and Kimball try to explain ticket–splitting rates, their second stage dependent variables are
as opposed to :_] D ^ and _] D b . See fn. 5.
`˜a _] D b
17
_] D ^
and
Table 1: The Test for Logical Consistency in EI–R Applied to Burden and Kimball (1998)
Variable
Constant
House Estimate
Ideological Distance
–
|™
#t”Ex  %
; x ”
Ž
# ; ž %
š>x
#’|x| ™ % žš
;
x
# ; ’ x|[•” %
;
™|x|™
;# š> [•”
%
Democratic Incumbent
Spending Ratio
Ballot Format
South
Ÿ
F
#
361
54.9
, Note: t–statistics in parentheses
%
Senate Estimate
|›š ™|[
#™ ’œ
”–% —–|žš
# %
; x|x•”
—– š
# ; Ž
|
x %
x
#x ’œx•
”–% ’œ

# ’ž ™ – %
;
™
;#  ”E
%
32
2.42
’ž
#  %
Table 1 House variables, including “South,” in a variety of second stage regressions. This means
that, for each such second stage regression, there are two, dependent estimates of the impact of
“South” on ticket splitting. As we noted earlier, including a covariate in both stages of EI–R can
lead to seemingly incoherent results that are difficult to interpret. Indeed, in the case of Burden and
Kimball’s analysis of House elections, this is precisely what happens.
In particular, Burden and Kimball’s first stage (King’s extended model) produces two estimates
of “South,” one which describes dependence of on “South” and one which describes dependence
of . These point estimates are
and
š
x|[ , respectively, and associated estimated standard
ž–
x•” and x .14 These estimates and corresponding standard errors lead to t–statistics
—–  and x| , and both of these are well below conventional significance levels.
of approximately
errors are
Ž
–
Ž
2š–
Hence, from their first stage, Burden and Kimball are obligated to conclude that there is no evidence
that the level of ticket splitting in a given Congressional district in 1992 was a function of whether
14
In their published paper Burden and Kimball do not report these two point estimates; rather, we generated them
by applying Gary King’s software program EzI to the publicly–available Burden and Kimball dataset. In addition, our
estimation produced _A Kr¡£¢¤ ¥£¦§¨5¢¤ © `«ª ¨"¢2¤ ` §­¬G¨"¢2¤ ª ¦®¨"¢2¤ ®­¦¯©¯° . Based on private communication with David Kimball,
this vector estimate is very close to that which forms the basis of Burden and Kimball’s analysis of ticket splitting.
18
the district was located in the South.
Nonetheless, Burden and Kimball report in their paper a number of statistically significant
“South” coefficient estimates from House–based, second stage regressions. These latter regressions
imply that district location – in the South or not – was relevant to ticket splitting levels in House
elections. The fact that both first stage “South” estimates are statistically insignificant yet numerous
second stage estimates of the impact of “South” are significant is clearly a sign of incoherence. At
the least, this incoherence makes it very difficult, if not impossible, to understand whether “South”
was significantly important to ticket splitting rates.
In the case of Burden and Kimball’s Senate data, the relevant F statistic is x ”Ex and its p–value
is slightly greater than
’
[ . Hence, we cannot conclude that EI–R as applied to the Senate data
is logically inconsistent. However, the p–value is very close to the cutoff, and the Senate case is
problematic because its sample size is rather low (Ÿ
±|x ). These two facts mean that logical
consistency of EI–R in the Senate case is certainly not a foregone conclusion.
Our results on Burden and Kimball’s dataset raise an important question: are they anomalous?
On one hand, without an exhaustive study of numerous research programs that use second stage
regressions we will never know precisely how many published applications of EI–R are based on
logically inconsistent implementations.
On the other hand, there are many instances in which it is hard to imagine that elements of
the exogenous covariate vector V
are related to yet unrelated to . For example, in Cohen,
Kousser and Sides’s (1999) study of crossover voting in state legislative elections in Washington
and California, is the fraction of individuals in district who are Democrats.15 Moreover, one of
the elements in the associated V
vector is an indicator that measures whether there is a Democratic
incumbent in district . It is possible, of course, that Democratic incumbency in district is unrelated
to whether district has a large number of Democrats in it. However, this seems unlikely to be true.
This reasoning, then, suggests that logical consistency in Cohen, Kousser and Sides’s application of
EI–R should be questioned.
15
Cohen, Kousser and Sides’s notation uses ²D instead of 0D and ³D instead of CED .
19
6 Discussion
As we noted in the introduction, the number of research articles which use EI–R is not matched
by a parallel number of analyses of EI–R as a statistical technique. In fact, as far as we know, this
paper and Herron and Shotts (2000) are the only formal treatments of EI–R and, more broadly, the
only treatments of regression models that use ecological inference point estimates as dependent or
independent variables.
It seems fair to say that the conclusions of Herron and Shotts and the present paper are not
optimistic. Namely, Herron and Shotts show that EI–R’s final point estimates are statistically inconsistent even when EI–R is itself logically consistent. And, in the present paper we have proposed a
test for logical consistency of EI–R and applied it to the House and Senate datasets used in Burden
and Kimball (1998). The test rejects logical consistency in the House case and comes extremely
close to rejecting for the Senate. We suspect that these conclusions are not anomalous and that
many applications of EI–R are logically inconsistent. Moreover, we pointed out that, when the test
for logical consistency of EI–R rejects, a second stage regression should not be estimated. Rather,
covariates that would have been included in such a regression should instead find themselves in an
application of King’s extended model, that is, in an abbreviated version of EI–R.
Thus, if our critique of EI–R is taken seriously, we should expect to see numerous researchers
who wish to estimate second stage regressions instead using King’s extended model and eschewing
second stage regressions entirely. This, however, raises an important issue about EI–R and the
enterprise of using exogenous covariates in ecological inference.
Rivers (1998) notes that King’s extended model is fragile in the sense of being barely identified.
The importance of this point is as follows: if it is difficult to identify using King’s extended model
the impact of exogenous covariates on unknown disaggregated quantities, then it should be commensurately difficult to use a second stage regression to do this. However, second stage regressions
are easily identified and they make it appear as if identifying the impact of exogenous covariates on
disaggregated quantities is a simple matter. However, it should not be forgotten that second stage
regressions are identified precisely because they treat ecological inference point estimates (6 , in
our notation) as fixed. But, such estimates are not fixed. Instead, they are functions of estimated
20
bivariate normal parameters (( 6 ).
What this means is that researchers must tread extremely carefully when working with exogenous covariates and ecological inference. When a researcher estimates an extended King model
with numerous covariates, she must ensure that her resulting point estimates as to the impact of her
covariates are stable. If they are not, then this indicates potential identification problems. Models
that are poorly identified must be flagged as such, as their results are potentially meaningless.
References
Achen, Christopher H. and W. Phillips Shively. 1995. Cross–Level Inference. Chicago: University
of Chicago Press.
Burden, Barry C. and David C. Kimball. 1998. “A New Approach to the Study of Ticket Splitting.”
American Political Science Review 92(3):533–44.
Cho, Wendy K. Tam and Brain J. Gaines. 2000. “Reassessing the Study of Split–Ticket Voting.”
Working Paper, University of Illinois–Champaign.
Cohen, Jonathan, Thad Kousser and John Sides. 1999. “Sincere Voting, Hedging, and Raiding:
Testing a Formal Model of Crossover Voting in Blanket Primaries.” Paper presented at the
annual meeting of the American Political Ccience Association, Atlanta, GA.
Freedman, D.A., S.P. Klein, M. Ostland and M.R. Roberts. 1998. “On ‘Solutions’ to the Ecological
Inference Problem.” Journal of the American Statistical Association 93(444):1518–22.
Gay, Claudine. 1997. Taking Charge: Black Electoral Success and The Redefinition of American
Politics PhD thesis Harvard University.
Herron, Michael C. and Kenneth W. Shotts. 2000. “Using Ecological Inference Point Estimates in
Second Stage Linear Regressions.” Paper presented at the annual meeting of the Society for
Political Methodology, Los Angeles, CA.
21
Kimball, David C. and Barry C. Burden. 1998. “Seeing the Forest and the Trees: Explaining
Split–Ticket Voting within Districts and States.” Paper presented at the annual meeting of the
American Political Ccience Association, Boston, MA.
King, Gary. 1997. A Solution to the Ecological Inference Problem. Princeton, NJ: Princeton University Press.
Lublin, David I. 2000. “Context and Francophone Support for Quebec Sovereignty.” Working Paper,
American University.
Rivers, Douglas. 1998. “Review of ’A Solution to the Ecological Inference Problem: Reconstructing
Individual Behavior from Aggregate Data’.” American Political Science Review 92(2):442.
Tam, Wendy K. 1998. “Iff the Assumption Fits...: A Comment on the King Ecological Inference
Solution.” Political Analysis 7:143–63.
Voss, D. Stephen and David Lublin. 1998. “Ecological Inference and the Comparative Method.”
APSA-CP: Newsletter of the APSA Organized Section in Comparative Politics 9(1):25–31.
Voss, D. Stephen and Penny Miller. 2000. “The Phantom Segregationist: Kentucky’s 1996 Desegregation Amendment and the Limits of Direct Democracy.” Working Paper, University of
Kentucky.
Wolbrecht, Christina and J. Kevin Corder. 1999. “Gender and the Vote, 1920–1932.” Paper presented at the annual meeting of the American Political Ccience Association, Atlanta, GA.
22