Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A Specification Test for Linear Regressions that use King–Based Ecological Inference Point Estimates as Dependent Variables Michael C. Herron1 Kenneth W. Shotts2 EXTREMELY ROUGH DRAFT – COMMENTS WELCOME July 13, 2000 1 Department of Political Science, Northwestern University. Address: Scott Hall, 601 University Place, Evanston, IL 60208–1006. Email: [email protected]. Fax 847–491–8985. 2 Department of Political Science, Northwestern University. Address: Scott Hall, 601 University Place, Evanston, IL 60208–1006. Email: k–[email protected]. Fax: 847–491–8985 Abstract Many researchers use point estimates produced by the King (1997) ecological inference technique as dependent variables in second stage linear regressions. We show, however, that this two stage procedure is at risk of logical inconsistency. Namely, the assumptions necessary to support the procedure’s first stage (ecological inference via King’s method) can be incompatible with the assumptions supporting the second (linear regression). We derive a specification test for logical consistency of the two stage procedure and describe options available to a researcher whose ecological dataset fails the test. 1 Introduction Many researchers use point estimates produced by the King (1997) ecological inference technique as dependent variables in second stage linear regressions. King advocates this two stage statistical procedure, which we call EI–R, saying that “[it] has enormous potential for uncovering new information” (p. 279). Burden and Kimball (1998), for example, use King–based estimates of district–level ticket splitting rates as dependent variables in second stage regressions which examine why individuals cast split ticket ballots. Similarly, Gay (1997) employs estimates of Black turnout rates, produced by King’s ecological inference technique, in regressions that seek to determine whether black citizens vote at higher rates in elections that include black Congressional candidates. Other examples of EI–R can be found in Kimball and Burden (1998), Voss and Lublin (1998), Cohen, Kousser and Sides (1999), Wolbrecht and Corder (1999), Lublin (2000), and Voss and Miller (2000). As evidenced by the multiple citations above, EI–R has achieved widespread usage despite the fact that it was only recently proposed. We show, however, that applications of this two stage procedure can be logically inconsistent in the sense that the assumptions necessary to support the procedure’s first stage (ecological inference via King’s method) can be incompatible with the assumptions supporting the second (linear regression). In contrast, we say that an application of EI–R is logically consistent when its first stage and second stage assumptions are not mutually contradictory. The possibility that the assumptions which support an application of EI–R contradict one another merits consideration because productive use of a statistical technique requires adopting the technique’s underlying assumptions. That is, to use any statistical technique a researcher must ultimately rely on its assumptions if he wishes to argue that the estimators produced by the technique have desirable properties such as consistency, asymptotic normality, and so forth. Moreover, and this is the key point which drives the results presented in this paper, when two statistical techniques are combined as in the case of EI–R a researcher must simultaneously adopt the assumptions that support the technique’s first and second stages. Previous work on EI–R has not sought to analyze whether its two sets of underlying assumptions – those associated with first stage ecological inference and those with second stage linear regression 1 – can be simultaneously adopted. In fact, to the best of our knowledge all published and working– paper implementations of EI–R simply assume without consideration that the procedure’s first and second stages are compatible, i.e. that EI–R is logically consistent. Nonetheless, we provide an analysis of the compatibility of EI–R’s two sets of assumptions and in so doing address the issue of aggregation bias in ecological data. In particular, our analysis is based on the fact that standard implementations of King’s technique (the first stage of EI–R) assume that there is no aggregation bias. Nonetheless, we show that it is often the case that linear regressions which use King–based point estimates as dependent variables (the second stage of EI–R) imply the existence of aggregation bias. As an illustration of this clash of assumptions, consider a classic case of ecological inference: a researcher wants to estimate and explain disaggregated turnout rates for African–American and white voters. Yet, suppose that she has only precinct–level racial composition and turnout data. To apply King’s technique as a component of EI–R to this turnout problem, the researcher must assume that there is no aggregation bias in the turnout dataset, i.e., that black turnout is uncorrelated with the fraction of a precinct’s population that is African–American. Now, suppose that such a researcher estimates a second stage regression in which King–based estimates of black turnout rates are assumed to be a linear function of precinct–level income and education levels and other politically–relevant independent variables. If any of these right hand side variables are correlated with the fraction in a precinct’s population that is African-American, then the researcher’s second stage regression model virtually guarantees (in a sense that we make precise later) that black turnout is correlated with the fraction of a precinct’s population that is AfricanAmerican, i.e., that there is aggregation bias. Thus, in this example the assumptions behind the second stage of EI–R contradict those in its first stage. This is problematic at two levels. At a fundamental level, any standard philosophy of science requires researchers to adopt internally consistent sets of assumptions. To do otherwise, as is often implicitly done in EI–R, is to simultaneously state “A is true” and “A is false.” Once two mutually contradictory premises are adopted, any implication can be logically derived, i.e., all subsequent analysis is meaningless. 2 At a more pragmatic level, logical inconsistency of EI–R is problematic because, in the presence of aggregation bias, King’s ecological inference technique produces statistically inconsistent estimates of all relevant population parameters.1 And, when all first stage point estimates are inconsistent, second stage regression results – that is, EI–R’s final results – will also be statistically inconsistent in ways that cannot be readily rectified. An estimator that is statistically inconsistent is generally useless because it does not converge in probability to its true value; rather, it tends to an incorrect value as the sample on which it is based gets large. Given the problems which result if EI–R is logically inconsistent, it is important for researchers to determine whether a particular application of the technique is problematic. Fortunately, it turns out that not all implementations of EI–R are logically inconsistent. Rather, we show that logical consistency of EI–R is application–dependent, i.e., EI–R is logically consistent in some empirical applications yet logically inconsistent in others. Thus, we propose a simple specification test for logical consistency of EI–R. The test is implemented via a single linear regression using data from both the first and second stages of a proposed application of EI–R. Interpreting the results of the specification test then requires consideration of only a single F statistic, the well–known and popular statistic which tests the null hypothesis that all slope coefficients in a linear regression are zero. If the F statistic produced by an application of the test to a given dataset is statistically significant, then EI–R is logically inconsistent when applied to this dataset and should not be used. If, on the other hand, the F value is not significant, then logical consistency of the application of EI–R cannot be rejected. It is worth noting that, even in an application where EI–R is logically consistent, second stage regression results will almost certainly be statistically inconsistent. This is because errors in King– based estimates of disaggregated quantities are inversely correlated with the true parameter values that the estimates measure. Fortunately, it is possible to correct for this statistical inconsistency so long as EI–R is logically consistent (Herron and Shotts 2000). However, when the test for logical consistency of EI–R rejects in a given ecological dataset, thus implying the presence of aggregation bias, a researcher who wants to study the dataset has only two 1 See Tam (1998) for a thorough discussion of the effects of aggregation bias on King’s technique. 3 options. One, she can abandon EI–R entirely and choose a different method of analysis. Or, two, she can explicitly model the aggregation bias in the first stage of the EI–R procedure. We discuss this latter option in some detail. We emphasize that our critique of EI–R and King’s ecological inference technique is fundamentally different from the argument in Tam (1998) and Cho and Gaines (2000), both of which claim that aggregation bias is a common phenomenon and that King’s diagnostics provide little help in identifying and rectifying it. Our analysis, in contrast, is not based on a claim that the assumptions supporting King’s ecological inference technique are unwarranted, either in general or in a particular empirical application. Rather, our critique applies only to the use of King’s technique within the context of the combined EI–R procedure, and what we argue is that the assumptions which support EI–R must be compatible across EI–R’s two stages. In Section 2 we provide notation and details on EI–R. Then, in Section 3 we present our specification test for logical consistency of EI–R and explain how to interpret it. Section 4 considers the options faced by a researcher whose ecological dataset fails the test, and in Section 5 we apply the test to the ecological dataset used in Burden and Kimball (1998). Section 6 offers general comments on EI–R and concludes. 2 Overview of the EI–R Procedure The first stage of EI–R consists of an application of King’s ecological inference technique, and the second stage requires estimating a linear regression based on first stage output. Thus, we now provide some details on ecological inference and King’s technique in particular and then explain how the output of the technique is used in second stage regressions. Suppose we have precinct–level data for an election of interest: let denote precinct ’s turnout rate in the election and The goal of ecological inference is to use and , , the fraction of Blacks in precinct . to estimate , the fraction of Blacks in precinct who voted in the election, and , the corresponding fraction of Whites who voted. Achen and Shively (1995) discuss the history of the so–called ecological inference problem and review various solution attempts. 4 King distinguishes between two different implementations, standard and extended, of his ecological inference technique. The precise details of these implementations can be found in Chapter 6–8 of King’s book. Critiques and discussions of King’s approach to ecological inference can be found in Freedman, Klein, Ostland and Roberts (1998), Tam (1998), and Cho and Gaines (2000). Until further notice our comments on King’s technique apply to its standard implementation or what we also call the standard model. have a truncated bivariate normal distribu , standard deviations and correlation . In tion on the unit square with means King’s notation, these parameters are collected in a 5–vector . Notably, The standard implementation assumes that and are fixed across , and this is the key feature of the standard model. where denotes covariance. Furthermore, the standard model assumes the two means This is equivalent to assuming there exists no aggregation bias, i.e. there is no systematic relationship between , the fraction of Blacks in precinct who voted, and , the fraction of Blacks in precinct .2 Let , , , , and denote the point estimates of , , , , and , respectively, based on applying King’s standard model to an ecological dataset.3 Corresponding to the vector , the five point estimates are collected in a 5–vector one of which is that there is no aggregation bias – . Under standard model assumptions – recall, as tends to infinity where ! is a consistent estimate of , and a similar statement applies to the other four point estimates in . In general, then, " . Let be the point estimate of produced by applying King’s technique to an ecological dataset and , ( is a function of , , and ). In EI–R, the point estidenotes convergence in probability. In other words, mates are used as dependent variables in a second stage regression. Namely, King suggests that a &()+ ' * In. addition, King’s technique assumes that there is no spatial autocorrelation, that #$ 2 3 and #% are independent for King’s ecological inference technique can be estimated using standard maximum likelihood theory. Or, optional priors can be used in which case King’s model yields a Bayesian framework. Both of these implementations of King’s technique are compatible with the results described here. 5 researcher who in theory might want to estimate (1) (2) instead estimate where the difference between equation (1) and (2) is that served) in the latter. Here, is a constant and (observed) substitutes for (unob a –vector of parameters to be estimated, is a –vector of exogenous variables, and is a disturbance term assumed to be uncorrelated with and . Let and denote the least squares estimates of and , respectively.4 Thus, the EI–R procedure consists of, first, estimating technique and, second, estimating and using King’s ecological inference with least squares applied to equation (2). 5 The final product of EI–R, therefore, consists of the estimates and , and, ultimately, whether EI–R is statistically consistent in an overall sense depends on whether ! and " . On the other hand, what can be thought of as first stage statistical consistency depends on whether where the –vector ! , as noted previously, is the first stage estimate produced by applying King’s standard model to an ecological dataset. When EI–R is logically consistent then its first and second stage assumptions do not contradict one another. Hence, when EI–R is logically consistent then there is no aggregation bias and EI–R’s first stage is statistically consistent ( stances " and " where ). In this case, except in very unusual circum- denotes a lack of convergence in probability (Herron and Shotts 2000). In other words, EI–R is in general statistically inconsistent even when it is logically even when the first stage of EI–R is statistically consistent. consistent, and this is because 4 $ $ $ $ Equations (1) and (2) could just as easily have had and on their left hand sides. All of our results apply in a symmetric fashion to such a case. Similarly, the results apply to regressions with and as left hand side variables as well. 5 King suggests a weighted least squares approach when estimating equation (2). Weighting, however, is not germane to our discussion of logical inconsistency. Furthermore, Herron and Shotts point out that there is no theory which says that weighted least squares estimation of equation (2) is superior to ordinary least squares estimation. 6 However, Herron and Shotts describe manipulations of of the population parameters and and that produce consistent estimates in equation (1) based on inconsistent estimates produced by equation (2). Overall, then, when EI–R is logically consistent its first stage is statistically consistent and, despite the fact that ! and consistent second stage estimates of " , it is possible to manipulate and . ! These manipulations are possible only when and so as to generate , i.e. only when first stage ecological inference estimates are statistically consistent. This is a key point for the following reason: when EI–R is logically inconsistent, then its first stage is statistically inconsistent ( " ! ) and EI– R completely falls apart.6 In particular, as far as are aware, no useful manipulations of and are possible when EI–R is logically inconsistent. In this latter situation, these point estimates are hopelessly statistically inconsistent and this cannot be rectified. 3 The Logical Consistency Test for EI–R Recall that EI–R’s first stage assumption of no aggregation bias is mathematically equivalent to . However, second stage regression implies from equation (1). Hence, for EI–R to be logically consistent across its first and second stages, it must be true that where , is the th element of the parameter vector and where is the th element of the vector . This derivation forms the basis of the following result: 6 King (1997) argues in Ch. 11 that his technique sometimes produces accurate estimates of even in the presence of aggregation bias. However, Tam (1998) argues that, in such situations, the technique produces inaccurate estimates. In our opinion, the fact that an inconsistent estimator happens to be on target in some applications does not constitute adequate justification for adopting it. Namely, inconsistent estimators are guilty until proven innocent, and, at the time of this writing, the value of King’s technique in the presence of aggregation bias remains uncertain. 7 , , and , , is Proposition 1 Suppose that an ecological dataset, consisting of given. Let be the true second stage parameter vector from equation (1). When it is not the case that (3) then EI–R as applied to the given dataset is logically inconsistent. Proposition 1 places a restriction on the parameter vector that must be satisfied for EI–R to be logically consistent when applied to a given ecological dataset. When the restriction is violated, then EI–R is logically inconsistent. Moreover, as described earlier, logical inconsistency of EI–R implies first stage statistical inconsistency ( in and ) and correcting resulting second stage inconsistencies is not possible. Suppose that, for a given ecological dataset, the covariances which appear in equation (3) all happen to be exactly zero. Then, any –vector , regardless of its elements, is consistent with the restriction in Proposition 1. This is because the elements , when substituted into equation (3), will be multiplied by zero and hence have no impact on whether the left hand side of the equation is truly zero. In other words, when all covariances in equation (3) are zero it is not possible for the elements of to violate the restriction in Proposition 1 and in so doing cause rejection of logical consistency of EI–R. On the other hand, if some of the terms are not zero, then the set of which satisfy the restriction in Proposition 1 is not equivalent to the set of all possible vectors vectors. , For example, suppose that so that is a 2–vector, , and . Then, for EI–R to be logically consistent it must be the case that or, equivalently, the . This example illustrates in a simple manner the fact that, when some of terms are non–zero, elements of the vector will have to satisfy a non–trivial restriction for EI–R to be logically consistent. Thus, we can think of Proposition 1 as having two cases. In one case – when all covariances in equation (3) are zero – there is no restriction on . In the other case – when at least one of 8 the terms is non–zero – the restriction in equation (3) is meaningful and logical consistency of EI–R depends on whether satisfies it. When Proposition 1 imposes a meaningful restriction on , we can enquire as to the nature of the restriction. Is it meaningful yet weak – for example, requiring that tight – for example, requiring, say, that for all – or is it for all ? If the former, then we would say that the restriction is relatively easy to satisfy and hence there are no compelling reasons to reject logical consistency of EI–R based solely on the fact that On the other hand, if the restriction on must satisfy the restriction. is very tight, then it is certainly possible that most vectors will fail to satisfy it. From this it would follow that EI–R is unlikely to be logically consistent. Similarly, if given an ecological dataset the associated restriction on is so tight that is seems unimaginable that it can be satisfied, then we would commensurately argue that, in all likelihood, EI–R is logically inconsistent when applied to the said dataset. We now develop this idea of a very tight restriction on , a restriction that seems virtually impossible to satisfy. In particular, our conceptualization of such a restriction is one that will be satisfied with probability zero if is drawn randomly from a continuous distribution. To motivate this conceptualization, consider the following example. Let and suppose as before that logical . If is drawn from a continuous distribution on , then the probability that is zero since a line has zero probability measure on a two–dimensional plane. Hence, we say that the restriction is very tight and, when consistency of EI–R requires logical consistency of EI–R depends on it, we conclude that EI–R will in all likelihood be logically inconsistent. In general, when the –vector on , then the probability that of dimension is zero. Thus, Definition 1 A –vector elements of must lie in an is randomly drawn from a continuous probability distribution elements of the vector, with , lie a given hyperplane from equation (1) is said to be non–generic if, for , dimensional hyperplane. Definition 1 codifies the idea of a very tight restriction in the sense that when it must satisfy such a restriction. 9 is said to be non–generic For example, suppose that implies that two ( is a –vector ( ) and that a restriction from equation (3) ) of its elements must lie in a one ( ) dimensional hyperplane. (This is a complicated way of saying that these two elements must lie on a line.) In general, the probability that two continuously distributed random variables fall exactly on a given line is zero. Hence, if all elements of are drawn from a continuous distribution on , the probability that the two key elements satisfy the restriction of being on a given line is zero. Of course, in actual applications of EI–R it is not the case that is drawn from a continuous dis- tribution prior to EI–R’s being employed. Rather, according to standard frequentist logic, estimating a model like equation (2) is equivalent to supposing that there exists a true and that this value is fixed albeit unknown. Nonetheless, the idea behind a restriction which leads to a non–generic is that the restriction is so tight that we should expect it never to hold. Hence, if we conclude that a given application of EI–R is logically inconsistent unless is non–generic, then we have effectively concluded that the application is in all likelihood logically inconsistent. In particular, we would say in this instance that EI–R is generically logically inconsistent. Definition 2 An application of EI–R to an ecological dataset is said to be generically logically inconsistent if the restriction in Proposition 1 requires It is straightforward to see that, if any of the then the to be non–generic. terms in equation (3) are non–zero, vector which satisfies the resulting restriction is non–generic. If, say, only one covariance is non–zero, then the associated element of , that which multiplies the non–zero covariance, will itself have to be exactly zero. If not, the restriction in equation (3) will not be satisfied unless exactly offsets the product of the non–zero covariance and the non–zero element of . This sort of offsetting will occur only by complete coincidence. Furthermore, the probability that a continuously distributed random variable on Hence, when only a single is exactly zero is itself a probability zero event. term is zero, is non–generic. We have now argued the following. Proposition 2 Let where be the true parameter vector from equation (1). If there exists a such that , then is non–generic. 10 Hence, whenever an ecological dataset features at least a single, non–zero EI–R as applied to the dataset is generically logically inconsistent. The result in Proposition 2 is, importantly, independent of whether term, is zero. In fact, this latter covariance only makes matters worse from the perspective of satisfying the restriction in equation (3). Namely, all terms can be zero but the restriction can still be violated if . Hence, by ignoring this latter covariance our test for logical consistency of EI–R is conservative.7 The matter of whether a proposed application of EI–R is generically logically inconsistent, then, turns on whether there exists a estimated by regressing on the term that is non–zero. Such covariance terms can be vector. Then, if the standard F test for all slope coefficients being zero rejects at a conventional confidence level, it follows that EI–R is generically logically inconsistent. Thus, the hypothesis test for logical consistency of EI–R requires nothing more than estimating a single regression with data that EI–R itself requires in the first place. This regression can, and should, be run before any application of EI–R. The intuition behind our regression–based test for logical consistency in EI–R is as follows. Recall that King’s ecological inference technique requires that there is no aggregation bias, i.e. that is a function of covariates collected in a vector and, if some of the elements in are correlated with , then in general to be zero. Thus, when the test for logical consistency of EI–R it is not possible for . But, if a second stage regression assumes that rejects, it follows that there is aggregation bias in the first stage and EI–R in this situation completely breaks down. The matter of non–rejection of logical consistency of EI–R is an important one. The test we propose for logical consistency is one of necessity as opposed to one of sufficiency. That is, an application of EI–R must pass our test; if not, it follows that the application is logically inconsistent. However, if an application of EI–R does pass the test, it does not therefore follow that the application 7 Our discussion of restrictions and logical consistency of EI–R began with the premise that an ecological dataset is given, and then we considered restrictions on such that equation (3) holds. We could, alternatively, have started from the premise that the elements of were given and asked about the resulting ecological dataset which would satisfy the equation. From this latter perspective, when any elements of are non–zero, the set of covariances which satisfy equation (3) will be non–generic. Hence, it follows that EI–R as applied to the dataset will be generically logically inconsistent. 11 is logically consistent. Rather, it is still possible that the application is logically inconsistent in some manner not covered by our test.8 Or, the failure to reject could simply reflect that fact that our test does not have a power of one. For the remainder of this paper, we use the term “logically inconsistent” in place of “generically logically consistent” as the latter is tedious. Similarly, we refer to our test as a test for logical consistency even though, technically, it is a test for generic logical consistency. 4 What to do when an Ecological Dataset Fails the Specification Test When a researcher’s ecological dataset fails the test for logical consistency of EI–R, she has two options. First, she can abandon EI–R entirely and choose another method of data analysis. Second, she can model the aggregation bias that caused test failure in the first place and in so doing avoid a second stage regression. Modeling aggregation bias in the first stage requires what King calls his extended model. Recall that in the standard model the mean of is , and this mean vector is assumed to be fixed across observations. However, in the extended model the two elements of this vector are not restricted in this way. Rather, in the extended model the means are denoted where the means are dependent on . Readers interested in the details of the extended model should consult King (1997, pp. 168–74). In brief, this model posits that is a linear function of exogenous covariates. That is, the extended model assumes something akin to estimated and where is a vector of parameters that must be a vector of covariates that includes a constant. A product of the extended model, then, is an estimate of the parameter vector . For numerical reasons (see p. 170) the extended model does not assume the precise form noted above for . However, the above expression for captures the essence of the extended model. Thus, suppose that a researcher has an ecological dataset and on it carries out the test for logical consistency of EI–R. Suppose as well that the test rejects. Since it consequently follows that there $ 8 For example, logical consistency of EI–R will impose certain restrictions on the distribution of . Although we do not explore the nature of these restrictions, it seems quite plausible that these restrictions are extremely tight. 12 is aggregation bias in the ecological dataset, the hypothetical researcher would then include as extended model covariates all elements of and subsequently would not estimate a second stage regression at all. In other words, when the specification test rejects logical consistency of EI–R, then the two stage statistical procedure should not be used. Rather, an abbreviated version – consisting solely of an application of King’s extended model – should replace it. To summarize, the procedure we propose for EI–R is as follows. 1. Collect an ecological dataset consisting of 2. Regress on the elements of the vector , , and , . and consider the F test that all slope coefficients are zero. Then, (a) If the F statistic does not reject, implement EI–R using the method described in Herron and Shotts (2000). (b) If the F statistic rejects, apply King’s extended model using all elements of ates for parameterizing the means and of and as covari- , respectively. The ultimate objective of EI–R is determining the extent to which elements of the exogenous covariate vector are related to . When standard EI–R is employed because the test for logically consistency did not reject, inference on the impact of sion results, specifically, on confidence intervals for can be based on second stage regres- –related slope coefficients produced by the method described by Herron and Shotts. That is, the Herron and Shotts method produces consistent estimates and valid confidence intervals for and , and these intervals can be used to conduct inference in the standard way. On the other hand, when King’s extended model is employed as the first (and only) component of an abbreviated EI–R, assessing the impact of on requires considering first stage standard errors produced by King’s technique. Namely, for each covariate used to parameterize the mean of in the extended model, King’s method produces a coefficient estimate and an associated t–statistic. Standard significance tests using these t–statistics (one for each element of the basis of determining whether the elements of 13 ) can form are related to through its mean . Suppose, for example, that contains only a single non–constant covariate and that, based on failure of the logical consistency test, this covariate is included in an application of King’s extended model. Associated with the covariate will be two coefficient estimates, one which relates the covariate to the mean and one to the mean . Depending on what a researcher is interested in and in her problem – studying – in other words, depending on the substantive meanings of she would consult one or perhaps both of these estimates and, using estimated standard errors, form t–statistics and carry out hypothesis tests in the usual fashion. A researcher may be tempted to include all or some elements of in both an application of King’s extended model as well as in a second stage regression. This practice – an example follows shortly – should be avoided as it can lead to internal incoherence. In other words, either King’s standard model should be used followed by a second stage regression or King’s extended model should be employed and there should be no second stage regression. Mixing the extended model with a second stage regression can lead to serious problems. Suppose, to illustrate this point, that a researcher gathers an ecological dataset and on it carries out the test for logical consistency of EI–R. Suppose as well that the test rejects and that only one coefficient estimate from the test’s regression is individually significant at, say, the level. The researcher might want to include the variable associated with the coefficient (that is, one element of the vector) in an extended King model and then estimate a second stage regression using the full vector. In such a case, the one variable which is significantly related to appear twice in the EI–R procedure insofar as it influences the mean EI–R and then influences in the specification test would of in the first stage of again in the second stage. Yet, both estimates of the impact of the variable are based on a single ecological dataset. Thus, the variable’s appearing twice means that there will be two separate and dependent estimates of the impact of the variable on . Simply put, it is impossible to say how the two associated t–statistics should be combined and interpreted. It is conceivable, for instance, that the first stage estimate of the impact of the covariate on the mean of can be statistically insignificant while the second stage estimate of the impact of the covariate is significant. When all the covariates are used only once, either in a second stage regression or in 14 an application of King’s extended model, this type of incoherence can be avoid. Finally, a practice to which special attention is warranted is the inclusion of the vector of exogenous covariates as an element of (e.g. Gay 1997, Lublin 2000). This practice – which assumes is a function of – guarantees that EI–R is logically inconsistent. Thus, if a researcher believes that is dependent on , then the latter should be included as a parameter in explicitly that the first stage ecological inference step along with all other exogenous variables of interest. In this situation, no second stage regression will be estimated. 5 An Application of the Test for Logical Consistency in EI–R We now apply the test for logically consistency in EI–R to the ecological dataset used by Burden and Kimball (1998) in their published study of ticket–splitting in the United States Congressional and Presidential elections of 1992.9 We obtained the Burden and Kimball data and divided it into House and Senate sections.10 For Burden and Kimball’s House data, the variable represents the proportion of individuals in Congressional district who voted for Michael Dukakis in the 1992 presidential race. And, is the fraction who voted for a Democratic Congressional candidate as well as for Dukakis, and is fraction who voted for is the fraction who voted for the Democratic House candidate, a Democratic Congressional candidate yet also voted for Bush. Thus, in district ticket–splitters is . the fraction of Furthermore, for the House analysis there are a total of four elements of the vector . These elements consist of an indicator as to whether district had a Democratic incumbent (“Democratic Incumbent”), the Democratic candidate’s proportion of House campaign spending (“Spending Ratio”), whether district allowed straight–party votes (“Ballot Format”), and whether district was located in the South (“South”). In accordance with the test for logical consistency of EI–R, we regressed on the elements of and a constant. Resulting coefficient estimates and the overall F 9 Cho and Gaines (2000) identify several errors in Burden and Kimball’s dataset, but we use the original dataset to maintain compatibility with the published study. Our critique of Burden and Kimball is quite different from that offered by Cho and Gaines, who argue, among other things, that King’s diagnostics indicate the presence of aggregation bias in the data set. 10 The data is available from the ICPSR and in particular from ftp://ftp.icpsr.umich.edu/pub/PRA/outgoing/s1140. 15 Table 1: The Test for Logical Consistency in EI–R Applied to Burden and Kimball (1998) Variable Constant House Estimate Ideological Distance – Senate Estimate Democratic Incumbent Spending Ratio Ballot Format South 361 F 54.9 Note: t–statistics in parentheses 32 2.42 statistic are in Table 1.11 In Burden and Kimball’s Senate data the variables noted above refer to states rather than Congressional districts, but the Senate vector contains one additional covariate (“Ideological Dis- tance”) compared to the House version of . This latter covariate measures the ideological distance between George Bush and the Democratic Senate candidate in state . Senate results are in Table 1, and all t–statistics in the table were calculated using heteroskedastic–consistent standard errors. Before examining the results in Table 1 we consider at an intuitive level whether we expect be correlated with the elements of the voters in district (or state) and vector. Recall that to represents the fraction of Dukakis– includes, among other things, a South indicator variable. Given all that is known about Democratic politics in the South, it would be hard to imagine that is not correlated with whether district is in the south. With respect to the House regression model, the F statistic for the hypothesis test that all slope coefficients are zero is . The associated p–value is approximately zero. This means that EI–R, 11 $ $ $ Since Burden and Kimball try to explain ticket–splitting rates, their second stage dependent variables are as opposed to and . See fn. 4. 16 $ and consistency of EI–R in the Senate case is certainly not a foregone conclusion. Our results on Burden and Kimball’s dataset raise an important question: are our results on Burden and Kimball’s work anomalous? On one hand, without an exhaustive study of numerous research programs that use second stage regressions we will never know precisely how many published applications of EI–R are problematic. On the other hand, there are many instances in which it is hard to imagine that elements of the exogenous covariate vector yet unrelated to . For example, in Cohen, are related to Kousser and Sides’s (1999) study of crossover voting in state legislative elections in Washington and California, is the fraction of individuals in district who are Democrats.13 Moreover, one of the elements in the associated vector is an indicator that measures whether there is a Democratic incumbent in district . It is possible, of course, that Democratic incumbency in district is unrelated to whether district has a large number of Democrats in it. However, this seems unlikely to be true. This reasoning, then, suggests that logical consistency in Cohen, Kousser and Sides’s application of EI–R should be questioned. 6 Discussion As we noted in the introduction, the number of research articles which use EI–R is not matched by a parallel number of analyses of EI–R as a statistical technique. In fact, as far as we know, this paper and Herron and Shotts (2000) are the only formal treatments of EI–R and, more broadly, the only treatments of regression models that use ecological inference point estimates as dependent or independent variables. It seems fair to say that the conclusions of Herron and Shotts and the present paper are not optimistic. Namely, Herron and Shotts show that EI–R’s final point estimates are statistically inconsistent even when EI–R is itself logically consistent. And, in the present paper we have proposed a test for logical consistency of EI–R and applied it to the House and Senate datasets used in Burden and Kimball (1998). The test rejects logical consistency in the House case and comes extremely close to rejecting for the Senate. We suspect that these conclusions are not anomalous and that 13 Cohen, Kousser and Sides’s notation uses $ instead of 18 $ and $ instead of # $. many applications of EI–R are logically inconsistent. Moreover, we pointed out that, when the test for logical consistency of EI–R rejects, a second stage regression should not be estimated. Rather, covariates that would have been included in such a regression should instead find themselves in an application of King’s extended model, that is, in an abbreviated version of EI–R. Thus, if our critique of EI–R is taken seriously, we should expect to see numerous researchers who wish to estimate second stage regressions instead using King’s extended model and eschewing second stage regressions entirely. This, however, raises an important issue about EI–R and the enterprise of using exogenous covariates in ecological inference. Rivers (1998) notes that King’s extended model is fragile in the sense of being barely identified. The importance of this point is as follows: if it is difficult to identify using King’s extended model the impact of exogenous covariates on unknown disaggregated quantities, then it should be commensurately difficult to use a second stage regression to do this. However, second stage regressions are easily identified and they make it appear as if identifying the impact of exogenous covariates on disaggregated quantities is a simple matter. However, it should not be forgotten that second stage regressions are identified precisely because they treat ecological inference point estimates ( , in our notation) as fixed. But, such estimates are not fixed. Instead, they are functions of estimated bivariate normal parameters ( ). What this means is that researchers must tread extremely carefully when working with exogenous covariates and ecological inference. When a researcher estimates an extended King model with numerous covariates, she must ensure that her resulting point estimates as to the impact of her covariates are stable. If they are not, then this indicates potential identification problems. Models that are poorly identified must be flagged as such, as their results are potentially meaningless. References Achen, Christopher H. and W. Phillips Shively. 1995. Cross–Level Inference. Chicago: University of Chicago Press. Burden, Barry C. and David C. Kimball. 1998. “A New Approach to the Study of Ticket Splitting.” 19 American Political Science Review 92(3):533–44. Cho, Wendy K. Tam and Brain J. Gaines. 2000. “Reassessing the Study of Split–Ticket Voting.” Working Paper, University of Illinois–Champaign. Cohen, Jonathan, Thad Kousser and John Sides. 1999. “Sincere Voting, Hedging, and Raiding: Testing a Formal Model of Crossover Voting in Blanket Primaries.” Paper presented at the annual meeting of the American Political Ccience Association, Atlanta, GA. Freedman, D.A., S.P. Klein, M. Ostland and M.R. Roberts. 1998. “On ‘Solutions’ to the Ecological Inference Problem.” Journal of the American Statistical Association 93(444):1518–22. Gay, Claudine. 1997. Taking Charge: Black Electoral Success and The Redefinition of American Politics PhD thesis Harvard University. Herron, Michael C. and Kenneth W. Shotts. 2000. “Using Ecological Inference Point Estimates in Second Stage Linear Regressions.” Working Ppaer, Northwestern University. Kimball, David C. and Barry C. Burden. 1998. “Seeing the Forest and the Trees: Explaining Split–Ticket Voting within Districts and States.” Paper presented at the annual meeting of the American Political Ccience Association, Boston, MA. King, Gary. 1997. A Solution to the Ecological Inference Problem. Princeton, NJ: Princeton University Press. Lublin, David I. 2000. “Context and Francophone Support for Quebec Sovereignty.” Working Paper, American University. Rivers, Douglas. 1998. “Review of ’A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data’.” American Political Science Review 92(2):442. Tam, Wendy K. 1998. “Iff the Assumption Fits...: A Comment on the King Ecological Inference Solution.” Political Analysis 7:143–63. 20 Voss, D. Stephen and David Lublin. 1998. “Ecological Inference and the Comparative Method.” APSA-CP: Newsletter of the APSA Organized Section in Comparative Politics 9(1):25–31. Voss, D. Stephen and Penny Miller. 2000. “The Phantom Segregationist: Kentucky’s 1996 Desegregation Amendment and the Limits of Direct Democracy.” Working Paper, University of Kentucky. Wolbrecht, Christina and J. Kevin Corder. 1999. “Gender and the Vote, 1920–1932.” Paper presented at the annual meeting of the American Political Ccience Association, Atlanta, GA. 21