Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A Specification Test for Linear Regressions that use King–Based Ecological Inference Point Estimates as Dependent Variables1 Michael C. Herron2 Department of Political Science Northwestern University Kenneth W. Shotts3 Department of Political Science Northwestern University March 8, 2002 1 The authors thank Wendy Tam Cho for comments. An earlier draft of this paper was presented at the 2000 Annual Meeting of the American Political Science Association. 2 601 University Place, Evanston, IL 60208–1006 (m–[email protected]). 3 601 University Place, Evanston, IL 60208–1006 (k–[email protected]). Abstract Many researchers use point estimates produced by the King (1997) ecological inference technique as dependent variables in second stage linear regressions. We show, however, that this two stage statistical procedure is at risk of logical inconsistency. Namely, the assumptions necessary to support the procedure’s first stage (ecological inference via King’s method) can be incompatible with the assumptions supporting the second stage (linear regression). We derive a specification test for logical consistency of the two stage procedure, describe options available to a researcher whose ecological dataset fails the test, and illustrate the problems that can arise if logical inconsistency is ignored. 1 Introduction Many researchers use point estimates produced by the King (1997) ecological inference technique as dependent variables in second stage linear regressions. King advocates this two stage statistical procedure, which Herron and Shotts (2001) call EI–R, saying that “[it] has enormous potential for uncovering new information” (p. 279). Burden and Kimball (1998), for example, use King–based estimates of district–level ticket splitting rates as dependent variables in second stage regressions which examine why individuals cast split ticket ballots. Similarly, Gay (2001) employs estimates of black turnout rates, produced by King’s ecological inference technique, in regressions that seek to determine whether black citizens vote at higher rates in elections that include black Congressional candidates. Other examples of EI–R can be found in Kimball and Burden (1998), Voss and Lublin (1998), Cohen, Kousser and Sides (1999), Wolbrecht and Corder (1999), Lublin and Voss (2000), and Voss and Miller (2000). Clearly, EI–R has achieved widespread usage despite the fact that it was only recently proposed. We show, however, that applications of this two stage procedure can be logically inconsistent in the sense that the assumptions necessary to support the procedure’s first stage (ecological inference via King’s method) can be incompatible with the assumptions supporting the second stage (linear regression). In contrast, we say that an application of EI–R is logically consistent when its first stage and second stage assumptions are not mutually contradictory. The matter of logical consistency, as distinct from statistical consistency, of EI–R is an important issue because this statistical procedure consists of two distinct stages. When a researcher employs the procedure, she must therefore simultaneously adopt assumptions that support both of EI–R’s stages. Previous work on EI–R has not sought to determine whether this is even possible. To the best of our knowledge, all published and working–paper implementations of EI–R simply assume that the procedure’s first and second stages are inherently compatible, i.e., that EI–R is logically consistent. We show that this may not be warranted and that, in fact, the assumptions necessary to support EI–R can be self–contradictory. In particular, our analysis of EI–R is based on 1 the fact that standard implementations of King’s technique to an ecological dataset (the first stage of EI–R) assume that there is no aggregation bias in the dataset. Nonetheless, we show that a linear regression which uses King–based point estimates as dependent variables (the second stage of EI–R) can imply the existence of aggregation bias. To illustrate this clash of assumptions, suppose that a researcher wants to estimate and explain disaggregated turnout rates of African–American and white voters but for this task has access only to precinct–level racial composition and turnout data. To apply King’s ecological inference technique to her turnout data and estimate African–American and white turnout rates by precinct (first stage of EI–R), the researcher must assume that there is no aggregation bias in her dataset, i.e., that black turnout is uncorrelated with the fraction of a precinct’s population that is African–American. Now, suppose that the researcher estimates a regression (second stage of EI–R) in which King–based estimates of black turnout rates are assumed to be a linear function of precinct– level income and education levels. If any of these right hand side variables are correlated with the fraction in a precinct’s population that is African-American, then the researcher’s second stage regression model virtually guarantees (in a sense that we make precise later) that black turnout is correlated with the fraction of a precinct’s population that is African-American, i.e., there is aggregation bias. Thus, in this hypothetical example the assumptions behind the second stage of EI–R contradict those in its first stage. This is problematic at two levels. At a fundamental level, any standard philosophy of science requires researchers to adopt internally consistent sets of assumptions. To do otherwise, as is possible with EI–R, is to state simultaneously “A is true” and “A is false.” Once two mutually contradictory premises are adopted, any implication can be logically derived, i.e., all subsequent analysis is meaningless. At a more pragmatic level, logical inconsistency of EI–R is problematic because, in the presence of aggregation bias, King’s ecological inference technique produces statistically inconsistent estimates (Tam 1998). And when this occurs, EI–R’s final results, i.e., second stage regression results that depend on first stage ecological inference estimates, will be statistically 2 inconsistent in ways that cannot be readily rectified. Given the problems that result if EI–R is logically inconsistent, it is important for researchers to determine whether a particular application of the technique is problematic in this way. Fortunately, it turns out that not all implementations of EI–R are logically inconsistent. Rather, we show that logical consistency of EI–R is application–dependent insofar as EI–R is logically consistent in some empirical applications yet logically inconsistent in others. We therefore propose a simple specification test for logical consistency of EI–R which can be implemented with a single linear regression. Interpreting the results of the specification test requires consideration of only a single F statistic, the well–known statistic which tests the null hypothesis that all slope coefficients in a linear regression are zero. If the F statistic is significant when applied to an ecological dataset, then an application of EI–R to the said dataset would be logically inconsistent. If, on the other hand, the F value is not significant, then logical consistency cannot be rejected. It is worth noting that, even in an application where EI–R is logically consistent, second stage regression results are statistically inconsistent. Fortunately, in some cases it is possible to correct for this statistical inconsistency so long as EI–R is logically consistent (Herron and Shotts 2001). In contrast, when the test for logical consistency of EI–R rejects in a given ecological dataset, a researcher who wants to study the dataset has only two options. One, she can abandon EI–R entirely and choose a different method of analysis. Or, two, she can explicitly incorporate covariates into the first stage of her EI–R procedure and use what King calls the extended ecological inference model. Later in the paper we discuss, and indeed recommend, this option. We emphasize that our critique of EI–R and King’s ecological inference technique is fundamentally different from the arguments in Tam (1998) and Cho and Gaines (2000), both of which claim that aggregation bias is a common phenomenon in aggregate data and that diagnostics advocated by King (1997, ch. 9) provide little help in identifying and rectifying it. Our analysis is not based on a claim that the assumptions supporting King’s ecological 3 inference technique are unwarranted, either in general or in a particular empirical application. Rather, our critique applies only to the use of King’s technique within the context of the combined EI–R procedure, and what we argue is that the assumptions which support EI–R must be compatible across EI–R’s two stages. Furthermore, our test for logical consistency of EI–R should not be thought of as a general test for aggregation bias in ecological data. Rather, the test asks whether the second stage of EI–R implies that there is aggregation bias in the first stage. The test does not explore whether there are other sources of aggregation bias that could cause first stage problems. In Section 2 we provide notation and details on EI–R. Then, in Section 3 we present our specification test for logical consistency of EI–R and explain how to interpret it. Section 4 considers the options faced by a researcher whose ecological dataset fails the test, and in Section 5 we apply the test to a House election dataset used in Burden and Kimball’s (1998) study of ticket splitting. This dataset fails the specification test, and thus we apply King’s extended model to it. The results of our extended model analysis of Burden and Kimball’s data are dramatically different from Burden and Kimball’s published findings on the causes of ticket splitting in House elections, and this illustrates the costs of ignoring logical inconsistency in EI–R. Section 6 offers general comments on EI–R and concludes. 2 Overview of EI–R This section provides some details on ecological inference and King’s technique in particu- lar and explains how the output of the technique is used in second stage regressions. Suppose we have precinct–level data for an election: let Ti ∈ [0, 1], i = 1, . . . , p, denote precinct i’s turnout rate in the election and Xi ∈ [0, 1] the fraction of Blacks in precinct i. The goal of ecological inference is to use Ti and Xi to estimate βib , the fraction of Blacks in precinct i who voted in the election, and βiw , the corresponding fraction of Whites who voted. Achen and Shively (1995) discuss the history of the ecological inference problem and review various solution attempts. King develops two different implementations of his ecological inference technique standard 4 and extended. The precise details of these implementations are found in King (1997, ch. 6–8). Critiques and discussions of King’s approach to ecological inference can be found in Freedman, Klein, Ostland and Roberts (1998), Tam (1998), Cho and Gaines (2000), and McCue (2001). Until further notice our comments on King’s technique apply to the standard model. ¡ ¢ This model assumes that βib , βiw have a truncated bivariate normal distribution on the ¡ ¢ unit square with means Bb , Bw , standard deviations (σb , σw ), and correlation ρ. These © ª parameters are collected in a 5–vector ψ = Bb , Bw , σb , σw , ρ , and, notably, the two means Bb and Bw are fixed across i.1 The importance of this assumption will be clear later when ¢ ¡ we discuss the extended model. Furthermore, the standard model assumes Cov βib , Xi = 0. This is equivalent to assuming there exists no aggregation bias, i.e., there is no correlation between βib , the fraction of blacks in precinct i who voted, and Xi , the fraction of blacks in precinct i.2 Let B̂b , B̂w , σ̂b , σ̂w , and ρ̂ denote the point estimates of Bb , Bw , σb , σw , and ρ, respectively, based on applying King’s standard model to an ecological dataset.3 Corresponding to the vector ψ, the five point estimates are collected in a 5–vector ψ̂. Under the standard P model’s assumptions (recall, one of which is that there is no aggregation bias) B̂b − → Bb P as p tends to infinity where − → denotes convergence in probability. In other words, B̂b is a consistent estimate of Bb , and a similar statement applies to the other four point estimates P in ψ̂. In general, then, ψ̂ − → ψ. Let β̂ib be the point estimate of βib produced by applying King’s technique to an ecological dataset Xi and Ti , i = 1, . . . , p; see McCue (2001) for a discussion of this estimator, which is a function of ψ̂, Xi , and Ti . In EI–R, the β̂ib point estimates are used as dependent variables 1 As described in King (1997), the elements of ψ correspond to elements in a related 5–vector ψ̆ where ψ is a function of ψ̆. We ignore ψ̆ and this is without loss of generality because King’s motivation for parameterizing the elements of ψ with the elements of ψ̆ is numerical rather than substantive. 2 In addition, King’s technique assumes that there is no spatial autocorrelation, i.e., that T i and Tj are independent for i 6= j. 3 King’s ecological inference technique can be estimated using standard maximum likelihood theory. Or, optional priors can be used in which case King’s model is based on a Bayesian framework. Both of these implementations of King’s technique are compatible our analysis. 5 in a second stage regression. Namely, King suggests that a researcher who wants to estimate βib = α + γ 0 Zi + νi , (1) β̂ib = α + γ 0 Zi + νi , (2) instead estimate where the difference between equation (1) and (2) is that β̂ib (observed) substitutes for βib (unobserved). Here, α is a constant, γ is a k–vector of parameters to be estimated, Z i is a k–vector of exogenous variables, and νi is a disturbance term assumed to be uncorrelated with βib and β̂ib . Let α̂ and γ̂ denote the least squares estimates of α and γ, respectively. 4 Thus, EI–R consists of, first, estimating β̂ib using King’s ecological inference technique and, second, estimating α̂ and γ̂ with least squares applied to equation (2).5 The final products of EI–R, therefore, are the estimates α̂ and γ̂, and, ultimately, whether EI–R is statistically P P consistent depends on whether α̂ − → α and γ̂ − → γ (henceforth we ignore the intercept α̂ and focus only on the slope estimates in γ̂). On the other hand, first stage statistical consistency P depends on whether ψ̂ − → ψ where the 5–vector ψ̂, as noted previously, is the first stage estimate produced by applying King’s standard model to an ecological dataset. Suppose that EI–R is logically consistent so that its first and second stage assumptions P do not contradict one another. This implies that ψ̂ − → ψ, meaning that EI–R’s first stage P P is statistically consistent. Nonetheless, even in this sanguine situation, γ̂ − 6 → γ, where − 6→ denotes a lack of convergence in probability, except in highly unusual circumstances (Herron and Shotts 2001). In other words, EI–R is in general statistically inconsistent when it is logically consistent; this is because β̂ib is not a perfect measure of βib even when the first stage of EI–R is statistically consistent. However, Herron and Shotts describe manipulations of γ̂ that, in the best of cases, can be used to produce consistent estimates of γ based on 4 Equations (1) and (2) could just as easily have had βiw and β̂iw on their left hand sides. All of our results apply in a symmetric fashion to such a case. 5 King suggests a weighted least squares approach when estimating equation (2). Weighting, however, is not germane to our discussion of logical inconsistency. 6 inconsistent estimates produced by equation (2).6 P These manipulations are feasible only when ψ̂ − → ψ, i.e., only when first stage ecological inference estimates of the vector ψ are statistically consistent. This is a key point for the following reason: when EI–R is logically inconsistent, its second stage invalidates an assumption that supports its first stage. Hence, logical inconsistency of EI–R implies first stage P statistical inconsistency (ψ̂ − 6 → ψ) and EI–R completely falls apart.7 In particular, as far as we are aware, no useful manipulations of γ̂ are possible when EI–R is logically inconsistent. In this latter situation, these point estimates are hopelessly statistically inconsistent. 3 The Logical Consistency Test for EI–R Recall that EI–R’s first stage assumption of no aggregation bias is mathematically equiv¡ ¢ alent to Cov βib , Xi = 0. However, second stage regression implies βib = α + γ 0 Zi + νi from equation (1). Hence, for EI–R to be logically consistent across its first and second stages, it must be true that ³ ´ 0 = Cov βib , Xi ¢ ¡ = Cov α + γ 0 Zi + νi , Xi ¡ ¢ = Cov γ 0 Zi + νi , Xi = k X γj Cov (Zi,j , Xi ) + Cov (νi , Xi ) j=1 where γj is the jth element of γ and where Zi,j is the jth element of the vector Zi , j = 1, . . . , k. This derivation forms the basis of the following result: Proposition 1 Suppose that an ecological dataset, consisting of Xi , Ti , and Zi , i = 1, . . . , p, 6 At the time of this paper’s writing, it is not known how EI–R estimates can be interpreted when Herron and Shotts’s manipulations are not applicable. 7 King (1997) argues in ch. 11 that his technique sometimes produces accurate estimates of ψ even in the presence of aggregation bias. However, Tam (1998) argues that, in such situations, the technique produces inaccurate estimates. In our opinion, the fact that an inconsistent estimator happens to be on target in some applications does not constitute adequate justification for adopting it. Namely, inconsistent estimators are guilty until proven innocent, and, at the time of this writing, the value of King’s technique in the presence of aggregation bias remains uncertain. 7 is given. Let γ be the true second stage parameter vector from equation (1). If k X γj Cov (Zi,j , Xi ) + Cov (νi , Xi ) 6= 0, (3) j=1 then EI–R as applied to this dataset is logically inconsistent. Proposition 1 places a restriction on the second stage parameter vector γ that must be satisfied for EI–R to be logically consistent. When the restriction is violated, EI–R is logically inconsistent. Suppose that, for a given ecological dataset, the k + 1 covariances in equation (3) all happen to be exactly zero. Then, any k–vector γ, regardless of its elements, is consistent with the restriction in Proposition 1. This is because the elements γj , when substituted into equation (3), will be multiplied by zero and hence have no impact on whether the left hand side of the equation is zero. In other words, when all covariances in equation (3) are zero it is not possible for the elements of the vector γ to violate the restriction in Proposition 1 and thereby cause rejection of logical consistency of EI–R. On the other hand, if some of the Cov (Zi,j , Xi ) terms are not zero, then only certain γ vectors satisfy the restriction in Proposition 1. For example, suppose that k = 2 so that γ is a 2-vector. Also suppose that Cov (νi , Xi ) = 0, and Cov (Zi,j , Xi ) = 1, j = 1, 2. Then, for EI–R to be logically consistent it must be the case that γ1 + γ2 = 0. In sum, Proposition 1 has two cases. In one case – when all covariances in equation (3) are zero – there is no restriction on γ. In the other case – when at least one of the Cov (Z i,j , Xi ) terms is non–zero – the restriction in equation (3) is meaningful and logical consistency of EI–R depends on whether γ satisfies it. When Proposition 1 imposes a meaningful restriction on γ, we can enquire as to the nature of the restriction. Is it meaningful yet weak – for example, requiring γ j < 100 for all j – or is it tight – for example, requiring γj = 1 for all j? If the former, then we would say that the restriction is relatively easy to satisfy and hence there are no compelling reasons to reject logical consistency of EI–R based solely on the fact that γ must satisfy the restriction. 8 On the other hand, if the restriction on the second stage parameter vector γ is very tight, then most γ vectors will fail to satisfy it. From this it would follow that EI–R is unlikely to be logically consistent. Similarly, if given an ecological dataset the associated restriction on γ is so tight that is seems unimaginable that it can be satisfied, then in all likelihood EI–R is logically inconsistent when applied to that dataset. We now develop this idea of a very tight restriction on γ, a restriction that seems virtually impossible to satisfy. In particular, our conceptualization of such a restriction is one that will be satisfied with probability zero if γ is drawn randomly from a continuous distribution. Consider the following example. Let k = 2 and suppose as before that logical consistency of EI–R requires γ1 = −γ2 . If γ = (γ1 , γ2 ) is drawn from a continuous distribution on <2 , then the probability that γ1 = −γ2 is zero since a line has zero probability measure on a two–dimensional plane. Hence, we say that the restriction γ1 = −γ2 is very tight and, when logical consistency of EI–R depends on it, we conclude that EI–R will in all likelihood be logically inconsistent. In general, when the k–vector γ is randomly drawn from a continuous probability distribution on <k , then the probability that m elements of the vector, with 1 ≤ m ≤ k, lie in a given hyperplane of dimension m − 1 is zero. Thus, Definition 1 A k–vector γ from equation (1) is said to be non–generic if, for 1 ≤ m ≤ k, m elements of γ must lie in an m − 1 dimensional hyperplane. For example, suppose that γ is a 3–vector (k = 3) and that a restriction from equation (3) implies that two (m = 2) of its elements must lie in a one (m − 1 = 2 − 1 = 1) dimensional hyperplane. This is a complicated way of saying that these two elements must lie on a line. The probability that two continuously distributed random variables fall exactly on a given line is zero. Hence, if all k = 3 elements of γ are drawn from a continuous distribution on <3 , the probability that the two key elements satisfy the restriction of being on a given line is zero. Of course, in actual applications of EI–R it is not the case that γ is drawn from a continuous distribution prior to EI–R’s being employed. Rather, according to standard frequentist 9 logic, estimating a model like equation (2) is equivalent to supposing that there exists a true γ and that this value is fixed albeit unknown. Nonetheless, the idea behind a restriction which leads to a non–generic γ is that the restriction is so tight that we should expect it never to hold. Hence, if we conclude that a given application of EI–R is logically inconsistent unless γ is non–generic, then we have effectively concluded that the application is in all likelihood logically inconsistent. Definition 2 An application of EI–R to an ecological dataset is generically logically inconsistent if the restriction in Proposition 1 requires γ to be non–generic. It is straightforward to see that, if any of the Cov (Zi,j , Xi ) terms in equation (3) are non– zero, then the set of γ vectors that satisfy the resulting restriction is non–generic. If, say, only one covariance is non–zero, then the associated element of γ, that which multiplies the non– zero covariance, will itself have to be exactly zero (which is a zero probability even). If the γ term is not zero, the restriction in equation (3) will not be satisfied unless Cov (ν i , Xi ) exactly offsets the product of the non–zero covariance and the non–zero element of γ. This sort of offsetting will occur only by complete coincidence. Hence, when even a single Cov (Z i,j , Xi ) term is zero, then γ is non–generic. Proposition 2 Let γ be the true parameter vector from equation (1). If there exists a j such that 1 ≤ j ≤ k where Cov (Zi,j , Xi ) 6= 0, then γ is non–generic. Hence, whenever an ecological dataset features at least a single, non–zero Cov (Z i,j , Xi ) term, EI–R as applied to the dataset is generically logically inconsistent. The result in Proposition 2 is, importantly, independent of whether Cov (νi , Xi ) is zero. In fact, this latter covariance only makes matters worse from the perspective of satisfying the restriction in equation (3). Namely, all Cov (Zi,j , Xi ) terms can be zero but the restriction can still be violated if Cov (νi , Xi ) 6= 0. Hence, by ignoring this latter covariance, i.e., assuming that it is zero, our test for logical consistency of EI–R is conservative. 8 8 Our discussion of restrictions and logical consistency of EI–R began with the premise that an ecological dataset is given, and then we considered restrictions on γ such that equation (3) holds. We could, alternatively, 10 The matter of whether a proposed application of EI–R is generically logically inconsistent, then, turns on whether there exists a Cov (Zi,j , Xi ) term that is non–zero. Such covariance terms can be estimated by regressing Xi on the Zi vector: Xi = λ 0 + λ 0 Zi + ² i . (4) If the F statistic for testing whether all slope coefficients in equation (4) are zero rejects at a conventional confidence level, it follows that EI–R is generically logically inconsistent. Thus, the hypothesis test for logical consistency of EI–R requires nothing more than estimating a single regression with data that EI–R itself requires in the first place. The intuition behind our regression test for logical consistency in EI–R is as follows. Recall that King’s ecological inference technique requires that there is no aggregation bias, ¡ ¢ i.e., that Cov βib , Xi = 0. But, if a second stage regression assumes that βib is a function of covariates collected in a vector Zi and, if some of the elements in Zi are correlated with Xi , ¢ ¡ then in general it is not possible for Cov βib , Xi to be zero. Thus, when the test for logical consistency of EI–R rejects, it follows that there is aggregation bias in the first stage and EI–R in this situation breaks down. The matter of non–rejection of logical consistency of EI–R is an important one. The test we propose for logical consistency is one of necessity as opposed to one of sufficiency. That is, an application of EI–R must pass our test; if not, it follows that the application is logically inconsistent. However, if an application of EI–R does pass the test, it does not therefore follow that the application is logically consistent. Rather, as we discuss later, it is still possible that the application is logically inconsistent in some manner not covered by our test. Or, the failure to reject could simply reflect that fact that our test does not have a power of one. Similarly, if an application of the test does not reject, then it does not follow that the ecological data used in the application are free from aggregation bias. 9 have started from the premise that the k elements of γ were given and asked about the resulting ecological dataset which would satisfy the equation. From this latter perspective, when any elements of the vector γ are non–zero, the set of covariances which satisfy equation (3) will be non–generic. Hence, it follows that EI–R as applied to the dataset will be generically logically inconsistent. 9 For instance, the second stage could be misspecified due to omitted variables that are correlated with X i . 11 For the remainder of this paper, we use the term “logically inconsistent” in place of “generically logically consistent” as the latter is tedious. Similarly, we refer to our test as a test for logical consistency even though, technically, it is a test for generic logical consistency. 4 What to do when an Ecological Dataset Fails the Specification Test When a researcher’s ecological dataset fails the test for logical consistency of EI–R, she has two options. One, she can abandon EI–R entirely and choose another method of data analysis. Or, two, she can model the aggregation bias that caused test failure and in so doing avoid a second stage regression. We now discuss the latter option. Modeling aggregation bias in the first stage of an application of EI–R requires King’s extended model. Recall that in the standard implementation of his ecological inference tech¡ ¢ ¢ ¡ nique the mean of βib , βiw is Bb , Bw , and this mean vector is fixed across observations. However, in the extended model the two elements of this vector are not restricted in this way. ¢ ¡ where subscripts denote Rather, in the extended model the means are denoted Bbi , Bw i dependence on observation i. Readers interested in the details of the extended model should consult King (1997, pp. 168–74). In brief, this model posits that Bbi and Bw i are linear functions of covariates. That is, the extended model assumes something akin to Bbi = θb0 Wi where θb is a vector of parameters that must be estimated and Wi is a vector of covariates. A product of the extended model, then, is an estimate θ̂b of the parameter vector θb (and, similarly, an estimate θ̂w of θw based 0 on Bw i = θw Wi ). For numerical reasons (equation 9.2, p. 170), King’s extended model does not assume the precise form noted above for the two means Bbi and Bw i . However, the linear expressions for these means captures the essence of the model. The extended model allows dependence of βib and βiw on covariates that are collected in the Wi vector. If, for instance, Wi contains only a single covariate and is thus a scalar, then θb describes how this covariate influences βib and θw describes how it influences βiw . Similarly, when Wi contains multiple covariates, then θb and θw are vectors with elements corresponding to covariates in Wi . 12 Thus, suppose a researcher carries out the test for logical consistency of EI–R on a given dataset, and suppose that the test rejects. This implies that there is aggregation bias in the dataset insofar as the researcher wants to assume that both his first and second stages apply. To model this aggregation bias explicitly, the researcher would include as extended model covariates – that is, as elements of the vector Wi – all elements of Zi and subsequently would not estimate a second stage regression at all. In other words, when the specification test rejects logical consistency of EI–R, then the two stage statistical procedure should not be used. Rather, an abbreviated version – consisting solely of an application of King’s extended model – should replace it; in this instance, dependence of βib and βiw on covariates would be analyzed via first stage ecological inference. When King’s extended model is used, assessing the impact of covariates on βib (and βiw , if this is an object of study) requires considering first stage standard errors produced by King’s technique. Namely, for each covariate (akin to an element of Wi ) used to parameterize the w means Bbi of βib and Bw i of βi in the extended model, King’s method produces two coefficient estimates (akin to elements of θ̂b and θ̂w ) and two associated t–statistics. Significance tests based on these t–statistics can be used to determine whether covariates are related to β ib and βiw . A researcher may be tempted to include all or some elements of the second stage covariate vector Zi in both an application of King’s extended model as well as in a second stage regression. This practice should be avoided as it can lead to internal incoherence. Specifically, it is possible for a first stage extended ecological inference model to indicate that a given covariate has a significant impact on βib and/or βiw whereas the second stage finds no evidence of such an effect, or vice versa. This would leave a researcher with disparate estimates of the same quantity. In other words, to carry out EI–R either King’s standard model should be used followed by a second stage regression or King’s extended model should be employed and there should be no second stage regression. Obviously, once a full extended model is used, there is no reason to estimate a second stage regression, since any relevant hypothesis about the effect of covariates on βib and βiw can be tested directly using first stage estimates and 13 standard errors. Suppose, to illustrate this point, that a researcher constructs an ecological dataset and on it carries out the test for logical consistency of EI–R. Suppose as well that the test rejects and that only one slope estimate from the test’s regression, i.e., only one element of the λ vector from equation (4), is significant at, say, the 0.05 level. The researcher might want to include the covariate associated with the statistically significant λ estimate in an extended model and then estimate a second stage regression using the full Zi vector. In such a case, the one significant covariate from the specification test would appear twice in the EI–R procedure insofar as it influences βib in the first stage of EI–R and then influences βib again in a second stage regression. Yet, both estimates of the impact of the variable would be based on a single ecological dataset. Thus, the variable’s appearing twice means that there will be two separate and dependent estimates of the impact of the variable on β ib . Simply put, it is impossible to say how the two associated t–statistics should be combined and interpreted. It is conceivable, and in fact we provide an illustration of this shortly, that the first stage estimate of the impact of the covariate on βib can be statistically insignificant while the second stage estimate of the impact of the covariate is significant. When all covariates are used only once, either in a second stage regression or in an application of King’s extended model, this type of incoherence can be avoided. A final reason to use King’s extended model as opposed to EI–R concerns other sources of logical inconsistency that can plague EI–R. We have focused thus far on whether the two stages of EI–R make contradictory assumptions about aggregation bias, i.e, about whether ¡ ¢ Cov βib , Xi is zero. But, there are other ways that the EI–R can be logically inconsistent. ¡ ¢ For instance, the first stage assumption that βib , βiw have a truncated, bivariate normal distribution has implications for the disturbance term νi in equation (1). In fact, it is unknown if second stage assumptions about νi are even compatible with first stage bivariate normality. The use of King’s extended model obviates the possibility that constraints on ν i cause logical inconsistency of EI–R. 14 5 An Application of the Test for Logical Consistency in EI–R We now apply the test for logically consistency in EI–R to the ecological dataset used by Burden and Kimball (1998) in their published study of ticket–splitting in the United States Congressional and Presidential elections of 1992.10 For Burden and Kimball’s House election data, Xi is the proportion of House voters in Congressional district i who voted for Michael Dukakis in the 1992 presidential race, and Ti is the fraction who voted for the Democratic House candidate.11 Furthermore, βib is the fraction who voted for a Democratic Congressional candidate as well as for Dukakis (straight ticket Democrats), and β iw is fraction who voted for voted for Republican presidential candidate George H. W. Bush and a Democratic Congressional candidate (Burden and Kimball call these RD ticket splitters). In their application of EI–R to the study of ticket splitting, Burden and Kimball use estimates β̂iw as dependent variables in a second stage regression. Burden and Kimball also use values of 1 − β̂ib as dependent variables in a regression that explains DR ticket splitting, i.e., rates at which individuals voted Democratic in the presidential race and Republican in a Congressional contest. 12 There are four elements of the second stage covariate vector Z i that Burden and Kimball use to study RD ticket splitting. These consist of an indicator as to whether district i had a Democratic incumbent (Democratic Incumbent), the Democratic candidate’s proportion of House campaign spending (Spending Ratio), whether district i allowed straight–party votes (Ballot Format), and whether district i was located in the South (South). In accordance with the test for logical consistency of EI–R when studying RD ticket splitting, we regressed X i on these four covariates and a constant. Resulting estimates – see equation (4) – are in Table 10 Cho and Gaines (2000) identify several errors in Burden and Kimball’s dataset, but we use the original dataset to maintain compatibility with the published study. Our critique of Burden and Kimball is quite different from that offered by Cho and Gaines, who argue, among other things, that King’s diagnostics indicate the presence of aggregation bias in the data set. Burden and Kimball’s data is available from the ICPSR (ftp://ftp.icpsr.umich.edu/pub/PRA/outgoing/s1140). 11 As Burden and Kimball note, because of rolloff Xi is not actually known and must be estimated. Like Burden and Kimball, we estimated Xi using King’s ecological inference technique. As pointed out in Herron and Shotts (2001), these estimated turnout rates are contaminated by errors. However, for purposes of comparability we follow Burden and Kimball and treat these estimates as if they were true values of X i . 12 We comment here only on Burden and Kimball’s House data because their Senate election dataset is extremely small (n = 32). Furthermore, Burden and Kimball augment their Senate data with House data in a way that is very hard to justify. 15 Table 1: Testing for Logical Consistency in EI–R as Applied to Burden and Kimball (1998) Variable Constant Democratic Incumbent Republican Incumbent RD Estimates 0.373 (40.7) -0.186 — (0.946) — Spending Ratio 0.203 (7.06) Ballot Format 0.00355 (0.333) South −0.0882 (−7.85) n 361 F 54.8 R2 0.381 Note: t–statistics in parentheses DR Estimates 0.352 (17.5) 0.0232 (1.35) 0.208 7.86 0.00188 (0.177) -0.087 (-7.75) 361 54.9 0.392 1. To study DR ticket splitting, Burden and Kimball use the same covariates as in their RD analysis but with a Republican incumbent indicator in place of Democratic Incumbent. Specification test results for this configuration of the second stage covariate vector Z i are also in Table 1.13 Both F statistics in Table 1 are very large and have p–values that are approximately zero. This means that, for studying both RD and DR ticket splitting, EI–R when using the covariates in Table 1 is logically inconsistent. Burden and Kimball, it is worth pointing out, include South in their first stage ecological inference model (thus including a form of the extended model in EI–R) and then estimate RD and DR second stage regressions using all the covariates in Table 1. At the very least, the t–statistics in this table show that Burden and Kimball should have included Spending Ratio in the first stage as this covariate is significantly related to Xi ; without such inclusion, second stage results are based on a logically inconsistent application of EI–R. However, as 13 Some House districts in 1992 had no incumbent, and hence Democratic Incumbent and Republican Incumbent do not sum to one. 16 noted already, covariates should never be included in both stages of EI–R. More comments on this issue, and why it is important for Burden and Kimball’s study, appear below. Given the specification test result in Table 1, we conclude that EI–R should not be used in an analysis of ticket splitting based on Burden and Kimball’s dataset of 1992 House elections. As discussed in the previous section, an option available to a researcher who nonetheless wants to study ticket splitting with this dataset is King’s extended model, a model that can be used to estimate directly how covariates affect ticket splitting rates. Table 2 presents results of applying King’s extended model to Burden and Kimball’s dataset.14 The table also reports verbatim Burden and Kimball’s second stage regression estimates which, as is now known, are based on a logically inconsistent application of EI– R.15 It is important to note that EI–R and extended model estimates for RD ticket splitting (and DR ticket splitting as well) are not directly comparable. Extended model coefficients are not regression coefficients but rather represent shifts in the mean of the distribution from which βib and βiw are drawn. However, to determine whether Burden and Kimball’s overall EI–R conclusions differ from those produced by an analysis of ticket splitting that uses King’s extended model, one can compare the signs and the t–statistics of the RD and DR estimates in Table 2. In their analysis of RD ticket splitting, Burden and Kimball found that Democratic Incumbent, Spending Ratio, and South all have a positive and significant effect on Republican presidential voters’ propensities to split their tickets. Moreover, Burden and Kimball found that Ballot Format had a negative and significant effect. In the extended model estimates for RD ticket splitting, however, only one of these results holds: Democratic Incumbent has a positive and significant effect on RD ticket splitting. The other three coefficient estimates, 14 Our implementation of the extended model, which is based on standard maximum likelihood theory, does not incorporate priors. We tried a large variety of different starting values for the extended model, found two local maxima, and present the results for the maximum with the higher loglikelihood value. We are confident that this is the true function maximum since there was a huge difference between the two maxima we found and because the other local maximum was clearly substantively unreasonable; its estimated fraction of Democratic presidential voters who voted for Democratic House candidates was only slightly higher than its estimate of the fraction of Republican presidential voters who voted for Democratic House candidates. 15 The Burden and Kimball (1998) EI–R results are from Tables 6 and 7 on pp. 539 and 540, respectively. 17 Table 2: King’s Extended Model as Applied to Burden and Kimball’s Analysis of House Elections Variable Democratic Incumbent RD Estimates Extended EI–R 0.116∗∗∗ 0.107∗∗∗ (0.0273) (0.015) — — DR Estimates Extended EI–R — — 0.151∗∗∗ (0.0360) ∗∗∗ Spending Ratio 0.0845 0.350 -0.588 (0.324) (0.021) (0.375) Ballot Format 0.201 -0.032∗∗∗ 0.277 (0.199) (0.008) (0.225) South 0.2196 0.052∗∗∗ 0.219 (0.183) (0.009) (0.212) Note: standard errors in parentheses; ∗ p < 0.05, ∗∗ p < 0.01, Republican Incumbent 0.085∗∗∗ (0.011) -0.252∗∗∗ (0.015) 0.003 (0.006) 0.007 (0.006) ∗∗∗ p < 0.001 which in the logically inconsistent EI–R analysis are highly significant, are in fact statistically insignificant once this problem is corrected for. As a side note, Burden and Kimball’s RD analysis suffers from internal incoherence as well as logically inconsistency. As noted earlier, Burden and Kimball include South in their first stage ecological inference model; they allow, to be precise, this covariate to influence β iw , the probability that an individual votes RD. For reasons that are unclear, they do not allow South to influence 1 − βib , the probability that the individual votes DR. Needless to say, the first stage estimate of the impact of South on RD ticket splitting is 0.0622 with a standard error of 0.0410 (t = 1.52).16 In other words, according to Burden and Kimball’s first stage, there is no compelling evidence that South affects RD ticket splitting. But, according to their second stage, South does affect RD ticket splitting. This situation is obviously troubling, and it is caused by the inclusion of South at both stages of EI–R. Furthermore, a contradiction between the two RD estimates of South is hardly surprising given that EI–R is logically inconsistent when used to study RD ticket splitting. 16 This estimate does not appear in Burden and Kimball’s published paper. Rather, we produced it by working with their dataset. 18 We now turn to Burden and Kimball’s analysis of DR ticket splitting. In the rightmost two columns of Table 2, the logically inconsistent EI–R results indicate that Republican Incumbent and Spending Ratio have highly significant effects on DR splitting. In the extended model analysis of such voting behavior, though, Republican Incumbent has a significant effect whereas there is no evidence to support the claim that Spending Ratio is similarly significant. There are, notably, no results in Table 2 that are significant in an extended model analysis yet insignificant in EI–R. This means that by employing EI–R in a logically inconsistent way, Burden and Kimball generated evidence of “findings” that, in fact, are not findings once logical consistency is adjusted for.17 Thus, we conclude that Burden and Kimball’s main results on the causes of ticket splitting are generated by an inappropriate statistical method. Using a method that is logically consistent we have shown that, with one exception (the effect of incumbent partisan affiliation), all of Burden and Kimball’s substantive findings are statistically unfounded. The exception is a notable one as it is quite intuitive. Furthermore the fact the extended model analysis of Burden and Kimball’s House data returns some non–null results means that its failure to do so in other cases does not reflect an overall inability to estimate anything. For instance, Burden and Kimball claim that, “The most robust finding from [various applications of EI–R] is that congressional campaign spending has a dramatic influence on the percentages of voters who split their ballot[s] (p. 542).” In fact, we believe that there is no evidence in Burden and Kimball’s dataset to support this assertion. Similarly, Burden and Kimball claim that statistical significance of South in the RD analysis leads to a rejection of theories of intentional ticket splitting (p. 540). In fact, South is insignificant (t = 1.2) in the logically consistent RD analysis and, therefore, Burden and Kimball’s dataset does not cast aspersions on theories of intentional ticket splitting insofar as South is concerned. These comments do not mean, of course, that all of Burden and Kimball’s conclusions are incorrect. But, it does mean that their data do not support their claims about the causes of split ticket 17 To confirm that the differences identified in Table 2 were due to the distinction between using King’s extended model and EI–R, we replicated Burden and Kimball’s second stage analysis and found the same regression results that they reported in their paper. 19 voting. Our results on Burden and Kimball’s House dataset and the inaccuracies we have uncovered raise an important question: are problems with EI–R common or is the example in this paper anomalous? Without an exhaustive study of numerous research programs that use second stage regressions, we will never know precisely how many published applications of EI–R are based on logically inconsistent assumptions. It is, of course, possible that many applications of EI–R are logically consistent. However, we are aware of many instances of EI–R in which it is hard to imagine that elements of the second stage covariate vector Zi are related to βib (as assumed in a second stage regression) yet unrelated to Xi (as assumed in first stage ecological inference). For example, in Cohen, Kousser and Sides’s (1999) study of crossover voting in state legislative elections in Washington and California, Xi is the fraction of individuals in district i who are Democrats.18 Moreover, one element of Cohen, Kousser and Sides’s second stage covariate vector Z i is an indicator for whether there is a Democratic incumbent in district i. It is conceivable that Democratic incumbency in district i is unrelated to whether district i has a large number of Democrats in it. However, this seems unlikely. This reasoning, then, suggests that logical consistency in Cohen, Kousser and Sides’s application of EI–R should be questioned. 6 Discussion We have proposed a test for logical consistency of the statistical procedure EI–R and illustrated its usefulness by applying the test to an ecological dataset from Burden and Kimball (1998). The test easily rejects logical consistency for Burden and Kimball’s analysis of the causes of ticket splitting in House elections. Moreover, when we correct for this, then the vast majority of Burden and Kimball’s findings on the causes of split ticket voting vanish. We suspect that this result is not anomalous and that many applications of EI–R are, like Burden and Kimball’s study of House races, logically inconsistent. If our critique of EI–R is taken seriously, we should expect to see numerous researchers 18 Cohen, Kousser and Sides’s notation uses Pi instead of Xi and Vi instead of Ti . 20 who wish to estimate second stage regressions instead relying on King’s extended model, a model with does not suffer from logical consistency problems, and eschewing second stage regressions entirely. This, however, raises an important issue about EI–R and the enterprise of using covariates in ecological inference. Rivers (1998) notes that King’s extended model is fragile in the sense of being barely identified. The importance of this point is as follows: if it is difficult to identify using King’s extended model the impact of covariates on unknown, disaggregated quantities, then it should be commensurately difficult to use a second stage regression to do this. However, second stage regressions are easily identified and they make it appear as if identifying the impact of covariates on disaggregated quantities is a simple matter. However, it should not be forgotten that second stage regressions are identified precisely because they treat ecological inference point estimates (β̂ib , in our notation) as fixed. But, such estimates are not fixed. Instead, they are functions of estimated bivariate normal parameters ( ψ̂). This point might explain why researchers often turn to EI–R in the first place. Indeed, one might logically wonder why anyone would ever estimate a second stage regression when covariates to be included in such a regression can be incorporated into a first stage extended ecological inference model. We suspect that researchers may wish to continue using EI–R because applications of King’s extended model are likely to generate numerous null results from which few substantive conclusions about political behavior can be drawn. This is, in fact, what happened when Burden and Kimball’s (1998) study of ticket splitting was subjected to a logically consistent statistical design. A prevalence of null results does not reflect flaws in King’s ecological inference technique; rather, null results reflect the difficultly of incorporating covariates in ecological inference. It is the nature of contemporary political science research that statistically significant “findings” are valued more than null results. But, findings should not be taken seriously if they exist solely because of a statistical quirk, i.e., solely because they appear in a logically inconsistent application of EI–R yet not in a logically consistent ecological inference model. 21 Some problems in political science, and the study of ticket splitting may be one of these, may simply be intractable with aggregate data, and it is better to recognize and accept this than to attempt to extract unsupportable findings using a potentially misleading statistical technique. References Achen, Christopher H. and W. Phillips Shively. 1995. Cross–Level Inference. Chicago: University of Chicago Press. Burden, Barry C. and David C. Kimball. 1998. “A New Approach to the Study of Ticket Splitting.” American Political Science Review 92(3):533–44. Cho, Wendy K. Tam and Brain J. Gaines. 2000. “Reassessing the Study of Split–Ticket Voting.” Working Paper, University of Illinois. Cohen, Jonathan, Thad Kousser and John Sides. 1999. “Sincere Voting, Hedging, and Raiding: Testing a Formal Model of Crossover Voting in Blanket Primaries.” Paper presented at the annual meeting of the American Political Science Association, Atlanta, GA. Freedman, D.A., S.P. Klein, M. Ostland and M.R. Roberts. 1998. “On ‘Solutions’ to the Ecological Inference Problem.” Journal of the American Statistical Association 93(444):1518–22. Gay, Claudine. 2001. “The Effect of Black Congressional Representation on Political Participation.” American Political Science Review 95(3):589–602. Herron, Michael C. and Kenneth W. Shotts. 2001. “Using Ecological Inference Point Estimates as Dependent Variables in Second Stage Linear Regressions.” Forthcoming, Political Analysis. 22 Kimball, David C. and Barry C. Burden. 1998. “Seeing the Forest and the Trees: Explaining Split–Ticket Voting within Districts and States.” Paper presented at the annual meeting of the American Political Science Association, Boston, MA. King, Gary. 1997. A Solution to the Ecological Inference Problem. Princeton, NJ: Princeton University Press. Lublin, David and D. Stephen Voss. 2000. “Context and Francophone Support for the Sovereignty of Quebec: An Ecological Analysis.” Forthcoming, Canadian Journal of Political Science. McCue, Kenneth F. 2001. “The Statistical Foundations of the EI Method.” The American Statistician 55(2):106–110. Rivers, Douglas. 1998. “Review of ‘A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data’.” American Political Science Review 92(2):442. Tam, Wendy K. 1998. “Iff the Assumption Fits...: A Comment on the King Ecological Inference Solution.” Political Analysis 7:143–63. Voss, D. Stephen and David Lublin. 1998. “Ecological Inference and the Comparative Method.” APSA-CP: Newsletter of the APSA Organized Section in Comparative Politics 9(1):25–31. Voss, D. Stephen and Penny Miller. 2000. “The Phantom Segregationist: Kentucky’s 1996 Desegregation Amendment and the Limits of Direct Democracy.” Working Paper, University of Kentucky. Wolbrecht, Christina and J. Kevin Corder. 1999. “Gender and the Vote, 1920–1932.” Paper presented at the annual meeting of the American Political Science Association, Atlanta, GA. 23