Download Regression Analysis and the Philosophy of Social Sciences --

Regression Analysis and the Philosophy of Social Sciences -a Critical Realist View
Amit Ron*
University of Minnesota
December 20, 1999
* [email protected]. I would like to thank Professor James Farr and
Professor John Freeman for their helpful advice, comments and criticism.
An earlier version of this paper was presented at the Third International
Conference of the Center for Critical Realism, Örebro University, August
20-22 1999.
2
Abstract:
This paper challenges the connection conventionally made between regression analysis
and the empiricist philosophy of science and offers an alternative explication for the way
regression analysis is being practiced. The alternative explication is based on critical realism, a
competing approach to empiricism in the field of philosophy of science. The paper argues that
critical realism can better explicate the way in which scientists ‘play’ with the data as part of the
process of inquiry. The practice of regression analysis is understood by the critical realist
explication as a post hoc attempt to identify a restricted closed system. The gist of successful
regression analysis is not being able to offer a law-like statement but to bring forth evidence of
an otherwise hidden mechanism. Through the study methodological debates regarding regression
analysis, it is argued that critical realism can offer conceptual tools for better understanding the
core issues that are at stake in these debates.
3
1. Introduction
The procedure of regression analysis is conventionally considered as an exemplar of the
positivist empiricist approach to research in political science. Those that use the procedure are
forced to defend empiricism with the entire philosophical burden that it carries; and those that
attack the empiricist philosophy of science condemn the prevalence of this statistical procedure
in the field. Very few, however, challenge the connection that was established between the
procedure and the empiricist philosophy.
By the term regression analysis, I refer to various mathematical methods that aggregate
observations into a form in which a dependent variable is a mathematical function of independent
variables (y = f(x1,x2,…xn)), often in a way that allows a statistical inference regarding the
parameters of the function outside the specific sample. Thus, my argument covers not only the
methods that are based on least square estimations, but also maximum likelihood estimations and
Bayesian inference. The question, then, is how this mathematical function is part of the overall
project of increase in scientific knowledge. The answer to such a question requires not only a
technical understanding of the mathematical procedures, but also an explication of the meaning
of scientific laws.
I will use the term “empiricism” to describe the interpretation of the law-like relation as
an Hempelian general law.1 According to this interpretation, regression analysis is a useful tool
for establishing such laws in the social sciences. However, the conventional wisdom continues,
social life is complex and for various reason it is never possible to satisfy the demanding
1
I use the term empiricism following Bhaskar (1975). It is used here in a different sense from
the way it is used in the rationalist-empiricist debate on the possible sources of knowledge. I
prefer the term empiricism to the more commonly used term positivism, since the definition of
empiricism used here is broader than what is conventionally included under the term positivism.
4
conditions of the mathematical theorems, and hence our ability to identify laws is always
problematic and questionable. This view appears in textbooks as the only possible interpretation
of the procedure. Thus, the practitioner of regression analysis who follows the textbook might
unnecessarily subscribe herself to the problems of empiricism: the demand of operational
definitions, the assumption of a world ruled by laws, insufficient place for agency, lack of
sensitivity to the problems of interpretation, and a limited place for emancipatory practice.
In this paper, I argue for an alternative philosophical framework to interpret more
adequately the way regression analysis is actually used by political scientists. This alternative is
based on Critical Realism, a school in philosophy of science, which is increasingly becoming
influential, particularly in Anglo-American social theory (Isaac, 1990: 2). In particular, I develop
a version of Critical Realism first articulated by Roy Bhaskar (1975, 1979). According to this
view, laws should not be understood as descriptions of constant conjunctions of events but as
tendencies of "powerful particulars" (Harré & Madden, 1975: 5). Thus, when a scientist cites a
law, she is describing a property of a thing, and not trying to predict in a specific circumstance if
the thing will behave in a specific way.
The empiricist view holds, among other things, that it is very difficult to conduct
experiments in the social sciences. Therefore, procedures like regression analysis can be used to
identify laws based on statistical analysis of passive observations. Through regression analysis,
the scientist can control for all the effects that govern real-life phenomena, and to identify the
best way to describe the relation between the observations. I claim here to the contrary that in
using the procedure of regression analysis, scientists are trying to identify situations in which it is
possible to observe the activity of a mechanism. When a scientist offers a regression equation,
she does not necessarily mean that the whole model or part of it approximates a universal general
law. Instead, she argues, at least implicitly, that she was able to demonstrate the activity of a
mechanism that could not be observed from the data alone. The gist of successful regression
5
analysis is not being able to offer a law-like statement but to bring forth evidence of an otherwise
hidden mechanism.
The account of regression analysis that I offer here is a matter of "explication."
Explication entails the analysis of a given notion, used intuitively by scientists, in order to
provide it with coherent philosophical foundations. The debate of the meaning of the notion of
scientific explanation is itself a famous example of explication (Salmon, 1990). In this essay,
therefore, I am offering an alternative explication of the notion of regression analysis. My
strategy is not to proceed by immanent critique, that is, to point out internal contradictions in the
empiricist interpretation. Instead, I offer an alternative explication, which better represents
regression analysis as actually practiced by social and political scientists. Thus, my explication
is directed to the scientists who practice regression and not only the philosophers who examine
the adequacy of arguments.
Explication is never only a descriptive discussion but also always a normative one too.
From the moment that the philosopher establishes a coherent explication, that is, it can become
an evaluative standard for evaluating scientific works. Hempel, for example, uses his explication
of the notion 'scientific explanation' to determine that some notions of explanation (such as
functional explanations) are not truly explanatory. In this paper, I focus only on establishing an
alternative, convincing interpretation other than the empiricist one. Thus, instead of criticizing
works that use regression for being empiricist, I offer a realist reconstruction of these works.2
The structure of my argument is as follows. In the next section, I offer a brief
description of critical realism as an alternative to the dominant tradition of empiricism in the
philosophy of science. It is far beyond the scope of this essay to offer a comprehensive
2
For an argument on the reconstructive potential of realism see Isaac, 1990. Collier (1994,
chapter 7) offers examples of such reconstruction.
6
discussion of critical realism in general or Bhaskar's version in particular.3 Instead, I focus
briefly only on those aspects of critical realism which are relevant for the argument presented in
this paper. The main point to be taken from this section is that we can and should think about
scientific laws in a way that is not related to relations between observations. Put differently,
goodness of fit is not a definitive character of a scientific law. It is important to emphasize that
critical realism does not call for a new type of scientific laws. Instead, it argues that the notion,
as used by scientists and properly explicated, refers to tendencies, not constant conjunctions of
events.
This point is elaborated in the second section, in which I present and contrast an
empiricist and a realist explication of regression analysis. It is important to understand the nature
of the argument I present. My starting point is regression analysis as practiced by political
scientists. I do not criticize the practice itself or its appropriateness for social sciences. Instead, I
assume that this practice is appropriate, and then ask what philosophical foundations can best
explicate it. Next, I look at the explication offered by some important textbooks and
methodological works on regression analysis. I argue that they implicitly rely on an empiricist
philosophy of science.4
My argument must be further clarified at the outset. Even though some econometrics
texts, by adopting the terms "true models" and "true betas," present regression analysis as a
method to reveal the general laws that govern social interactions, I do not think that most
political scientists understand regression analysis in this way. Nevertheless, they do accept the
empiricist definition of scientific laws as a deductive claim that allows the knowledge of the
3
For such a discussion see Collier, 1994, Outhwaite, 1987, Isaac, 1990 and Archer et al., 1998.
4
Here I discuss these works in general, I give particular examples and references in the second
section.
7
dependent variable from the independent variables. Thus, they think that compared to the natural
sciences we can make only more circumscribed and qualitative formulation of laws, but
nonetheless in the same formal representation, as a function with the form y=f(xn).
To see how this is achieved it is necessary to understand the roots of the empiricist
philosophy of science. Empiricism belongs to the analytic tradition in the philosophy of science
(Gunnell, 1969). This tradition tries to establish a set of logical criteria that both describes
scientific activity and justifies its logical validity. It does so, in short, by describing scientists in
terms that are borrowed from logic. As a footnote, it is necessary to mention that so far
empiricism was not able to establish such a criterion. It is only claims that its goal is to establish
one, which will be identical to all sciences. The empiricist explication of regression analysis
rests on the claim that goodness of fit together with additional criteria, which I discuss in detail
later, can offer a formal criteria for deciding whether a specific regression equation qualifies as a
law.5
The immediate response to the above claim is that without goodness of fit as a formal
criterion we lose the scientific nature of the inquiry. If we do not look at the relations between
our models and the data, why collect data in the first place? My argument in the second part of
the section is that a realist perspective can defend the scientific character of regression analysis
without relying on goodness of fit as a formal criterion for establishing laws. In this view,
goodness of fit is required only to ensure that the specification being used indeed makes the
activity of the mechanism in question observed.
5
There is a debate between methodologists regarding the proper measure of goodness of fit, but
underlying this debate is an agreement that a formal criterion of goodness of fit is necessary to
establish explanation (see King, 1986, 1991; Luskin, 1991).
8
In the third section, I focus on one theoretical debate and show how the realist
interpretation can be used to reformulate the issues that are at stake in the debate. I focus on the
'Norwegian Exceptionalism' debate in the field of comparative political economy not only
because of its theoretical importance but also because of the methodological issues that it
consciously raises (Lange & Garrett, 1985, 1986, 1987; Jackman, 1987, 1989). I argue that the
debate is not properly about which model produces the best goodness of fit or which model better
explains the dependent variable (in this case economic growth). Rather, the question is whether
the specification used by Lange and Garrett is sufficient to demonstrate the mechanism for which
they argue. I conclude with the implication of my argument to the use of regression analysis and
for broader questions in the philosophy of science.
Philosophy of science cannot and does not pretend to replace the practice of scientists.
Realism claims that in their practice scientists attempt to discover the real mechanisms of social
structures. This, however, does not mean that every social structure offered by scientists are real,
or even that we can have secure grounds in believing that any particular structure is real. The
work of scientists is best understood as an attempt to demonstrate the reality of otherwise
hypothetical structures. Realism does not make the work of scientists easier but offers coherent
philosophical grounds for their activity.
2. Two Philosophies of Science:
This section presents critical realism as an alternative to the dominant empiricist
philosophy of science. Empiricism is the philosophical tradition that understands scientific laws
as a description of relations between sense observations expressed as constant conjunctions of
events. This understanding of science is shared not only by those advocating a naturalist
approach to social sciences but also by those that oppose it. Many scholars who see themselves
as anti-naturalists and emphasize the difference between the study of nature and the study of
9
society in fact share the empiricist view of laws in natural science but reject the possibility of
applying scientific laws to the realm of society (Bhaskar, 1979: 2-3). Thus, despite the
disagreements between the naturalistic and the anti-naturalist tradition, they both rest upon the
same definition of scientific laws. Critical realism challenges the empiricist view of science and
laws, and offers a different understanding of scientific activity.
My interest in this paper is the applicability of realism and not any possible
inconsistencies of empiricism. For this reason, I will not go into details about possible
philosophical inconstancies in the empiricist or realist philosophy of science. Instead, I argue
that even if empiricism can offer a systematic model of scientific activity, lacking any logical
contradiction, it fails in explicating the way political scientists use regression analysis. Thus, in
the discussion I linger less on the logical structure of both philosophies of science and more on
the way that they explicate the activities of scientists.
According to Critical Realism, “the world consists of things, not events” and thus,
science is "concerned essentially with what kinds of things they are and with what they tend to
do; it is only derivatively concerned with predicting what is actually going to happen" (Bhaskar,
1975: 51). Scientific laws describe tendencies that things have in virtue of their internal
structure. The predicate of scientific laws is not observations but real tendencies of things.
"Thus," according to Bhaskar, "in citing a law one is referring to the transfactual activity of
mechanisms, that is to their activity as such, not making a claim about the actual outcome (which
in general will be co-determined by the activity of other mechanisms too)" (Bhaskar, 1979: 12).
Does it make any difference which definition of scientific laws we choose? To answer
this question it is necessary to discuss the definition of open and closed systems. A closed
system is a system where a constant conjunction of events obtains. In a close system, as a matter
10
of definition, the formula 'whenever this, then that' applies (Bhaskar, 1975: 69). 6 Empiricism
assumes that a "universal closure" obtains. This means that all events in the world can be
described under the formula 'whenever this, then that,' or at least that science can be applied only
to that part of the world which is a closed system. This is a metaphysical assumption that cannot
be corroborated empirically. One of the main challenges of the empiricist philosophy of science
is to offer an analytical distinction between law-like statements and real scientific laws, because
the two are similar in form.
The task of distinguishing between law-like statements and laws is not an easy task for
philosophers. Not surprisingly, it is not an easy one for scientists either. Law-like statements,
even those that are considered as corroborated by science, are often found inadequate in real life
situations. In such situations empiricists usually adopt one of the following strategies. First, it
can be claimed that the law-like statement is only an accidental generalization.7 For example, if
third parties would begin to appear in simple-majority simple-ballot systems, it would be
possible to argue that "Duverger's law" does not hold anymore.8 Second, the empiricist can use
the ceteris paribus clause and claim that the laws hold only if certain conditions apply or that it is
only a probabilistic law. Third, it can be claimed that the law-like statement is not complete.
Either a relevant variable was omitted from the formulation, or a complex unit has to be
separated into a more basic one. Duverger's law, according to such strategy, holds in all cases
6
More precisely, "For every event y there is an event x or set of events x1...xn such that x or
x1...xn and y are regularly conjoined under some set of descriptions" (Bhaskar, 1975: 69).
7
It can also be argued that the law-like statement was once a law but is no longer so (as in
historically bounded laws, see Ball, 1972 for a critical discussion of the concept)
8
According to Duverger's law plurality election rules bring about and maintain two-party
competition (see Riker, 1976).
11
"except in countries where (1) third parties nationally are continually one of two parties locally,
and (2) one party among several is almost always the Condorcet winner in elections" (Riker,
1982: 761, which provides an example of an empiricist understanding of Duverger's law). Thus,
empiricism can maintain the assumption of universal closure in two ways, it can either limit the
applicability of the law (in time, space or for special conditions) or refine the law by adding
more and more variables and making the law more complex.
Realism, to the contrary, claims that law statements, as used by scientists and in contrast
with the way understood in empiricist philosophy, apply in the same way in open as well as in
closed systems. "The citation of law," according to Bhaskar, "presupposes a claim about the
activity of some mechanism but not about the conditions under which the mechanism operates
and hence not about the results of its activity, i.e. the actual outcome on any particular occasion"
(Bhaskar, 1975: 95). The predicate of scientific laws, be it Duverger's law or Hooke's law, is a
mechanism that, once set in action, generates events open to empirical observations. The
mechanism acts in a certain way because the things in question (simple-majority simple-ballot
systems or springs) have internal structures that make them behave in certain ways and not
others.
This revised definition of laws allows a different perspective on the question of agency
(the ability to initiate change spontaneously). Within the empiricist tradition science and agency
cannot reside together. If we ascribe agency to a particular, it follows that we cannot describe its
behavior in law-like form. In the realist tradition, things can have tendencies to act in a certain
way and yet at the same time to have agency, i.e. the ability to initiate action. Some people have
a tendency to lose patience in certain situations but still have the ability to keep calm. For the
empiricist, a law in the form of 'whenever irritated, then lose patience' cannot be fully complete if
the person can decide whether she wants to keep calm. From a realist view, there is no
philosophical problem in saying that the person has a tendency even if she rarely does so, based
12
for example on an analysis of her personality, although it might be difficult from the standpoint
of the scientists to support such a claim.
What does it mean to ascribe powers or tendencies to a particular? 9 It is to claim that
this particular has an inner structure that generates a mechanism, which then put the particular
in a tendency to act in certain way. The analysis of the structure can teach us about the internal
conditions necessary for this particular to act. Nevertheless, it is possible that the thing will not
act although the internal conditions are satisfied because of another mechanism or because that
thing has agency (Harré, 1970: 260ff). It is important to note that tendencies and agency are
terms used by philosophy of science and not by the scientist themselves. A scientist, therefore,
"is never just content to ascribe power but moves immediately to the construction of possible
explanations for it with the paradigms and other instruments of thought at his disposal. That is
his job" (Bhaskar, 1975: 176). 10 The ascription of a tendency leads the scientists to study the
structure of the thing in question. This includes both theoretical elaboration of the supposed
structure of the entity and an empirical experimental work to show that the description of the
mechanism is correct. In experiments, the scientist artificially creates a closure that allows the
formula 'whenever this, then that,' to apply. The construction of an experimental design requires
not only knowledge of the structure of the thing studied but also creativity in creating the closed
9
We can ascribe powers not only to material things (that occupy place). We can also speak
about tendencies of relations such as spin in physics or marriage. These relations endure through
time but have no physical space (Bhaskar, 1975: 181; see also Harré & Madden, 1975, especially
103-105 and Harré, 1993: 44-5).
10
In the language of the social sciences, we encounter the terms of "power" and "tendency" much
more frequently, partly because the underdevelopment of these sciences, as the empiricist would
put it, and partly because the nature of its subject matter.
13
system. Finally, in the process of scientific change, the description of structure at one level leads
to an inquiry of a deeper more basic structure. 11
Critical Realism does not deny that science has something to do with the empirical
world. It denies that this relation can be put forward as the definitive characteristic of science.
This, however, does not deny that scientists are working with a criterion for judging between the
truth-value of alternative models. Such criteria exist in each discipline and are part of the
implicit and explicit knowledge that scientists acquire. While some interpret this as evidence for
ontological relativism -- the claim that science does not correspond with any real substance,
critical realism insists that this is a result of the limits of knowledge and not of the lack of reality.
Our inability to fully know or describe future events does not imply that mechanisms do not
exist. It is the openness of our world that prevents us from being able to use analytical criteria to
distinguish between explanations
Can we speak of social structures? It is far beyond the scope of this essay to study the
different positions regarding this issue. Yet, we should not necessarily assume a 'yes' answer for
this question. Instead, I want to suspend the question for the purpose of the discussion and to
adopt the language of social structures and mechanisms. Thus, I assume that social things have
an internal structure and that this structure generates mechanisms that create tendencies for these
things to behave in certain ways. Now, it is possible to claim that either all these mechanisms
can be or should be described in the language of empiricism (in 'whenever this then that' form) or
to claim that no such mechanism can ever be found. My argument here does not necessarily
refute these positions. I argue here only that realism can explicate better than empiricism what
scientists are doing when they are using regression. Nevertheless, even if my argument is
correct, one can still argue that in following realism the scientists are doing wrong and they
11
for a philosophical discussion of this process, see Harré, 1970 and Harré & Madden, 1975.
14
should follow more carefully the rules of empiricism or that they should avoid the search of laws
in social science.
3. Two interpretations of regression
In this section, I want to contrast an empiricist with a realist explication of the procedure
of regression analysis. I then argue that realism can better explicate the way political scientists
practice regression analysis than can empiricism. There is an inherent interpretative difficulty in
the argument I suggest. To the extent that scientists rely on the empiricist interpretation in their
use of the practice, this interpretation explicates their practice. Political scientists, especially
those who practice regression analysis, often use the terminology of empiricism to interpret their
work. However, in many cases this terminology does not correspond to the actual content of
their work or the nature of their practice (Farr, 1985; Isaac, 1987: 194-5). Other political
scientists feel less contented with the empiricist terminology and try to practice regression
analysis without discussing the philosophical foundations of their work. I believe that the reason
for this is the inability of empiricism, the dominant explication of regression analysis, to offer a
coherent explication of the way regression analysis is actually practiced.
One further clarification is required before proceeding. My aim in the first part of the
discussion is to reconstruct a systematic explication of regression analysis from an empiricist
perspective. For this purpose, I discuss some of the most prominent methodologists and
practitioners of regression analysis in the field of political science. I do not want to claim that
the way that they practice science is somehow wrong or flawed because they are empiricist.
Instead, I argue that their appeal to empiricism makes it difficult for them to explicate the
statistical procedures that they are using, and consequently to make sense of their results. Put
differently, I do not try to build a straw man of empiricism in order to burn it, but instead to
reconstruct an explication that is implicit in the way political scientists understand their own
work.
15
2.1. Regression Analysis as a Formal Method -- The empiricist interpretation
Knowing the intercept and the slope, we can predict Y for a given X value. For
instance, if we encounter a Riverside city employee with 10 years of schooling,
then we would predict his or her income would be $12,398... Of course,
predictive models are not completely distinct from explanatory models.
Commonly, a good explanatory model will predict fairly well. Similarly, an
accurate predictive model is usually based on causal variables, or their
surrogates. (Lewis-Beck, 1980: 19-20).
For example, suppose a researcher is interested in the average vote difference in
general elections with high Union Leader Bias. In this case, the variable Slant
equals 1, but the Primary and Slant X Primary variables are each 0. The
appropriate forecast from equation 2* is then -19%+14%(1)+23%(0)+5%(0), or 5%. That is, in high-slant general elections, the Manchester vote averaged 5%
less for the favored candidate than in the rest of the state (Achen, 1982: 21).
These citations, taken from textbooks discussing regression analysis, reveal the concept
of explanation that regression analysis is assumed to supply. It is essentially the Hempelian
model of scientific explanation. According to this model, "any explanation of a particular
occurrence is an argument to the effect that the event-to-be-explained was to be expected by
virtue of certain explanatory facts. The explanatory facts must include at least one general law.
The essence of scientific explanation can thus be described as nomic expectability -- that is,
expectability on the basis of lawful connections” (cited in Salmon, 1990: 57). The Hempelian
model is the most famous, even paradigmatic logical formulation of the empiricist assumptions.12
The regression's equation has the form of a general law. As shown in the examples,
given the equation and the values of the independent variables, it is possible to know, for any set
of independent values, the range of values that the dependent variable can take. Since the use of
the statistical model is justified in such a way, then the question of what is the philosophical
status of this general law arises. As we shall see, the textbooks give only few hints about the
12
I am intentionally ignoring the discussion of the different forms of inference, deductive or
inductive; Hempel's model covers both types of inference, and others beside (Hempel, 1965).
16
answer to this question. Based on these hints, I suggest here two approaches to answer this
question from an empiricist perspective.
The structure of the first approach is as follows:
1. Scientific laws are expressed in terms of relations between observations, in the form y=f(xn).
2. Regression analysis rests on a mathematical theorem that assumes the existence of a true
model, in the form y=f(xn).
3. Social relations cannot be modeled in such a way, either for epistemological or for
ontological reasons.
4. Hence: Regression analysis cannot be used for a causal inference but only for a descriptive
inference.
The first approach is to frame the answer in terms of a continuum between a 'mere
description' and a 'true model.' The term 'true model' appears in the statistical model upon which
regression analysis is based. The true model is a mathematical function that connects the
variables in a population. Statistical models can be used to estimate the parameters of the
function in a known level of precision based on data taken from a sample. For example, the most
basic and widely-used form of regression analysis -- ordinary least squares -- allows an
estimation of the parameters of a model, assuming that there are linear relations between the
variables in the population (and other assumptions). The first way to answer the question of the
metaphysical status of the regression equation is to claim that it is an attempt to estimate a true
model that exists in the real world. In this view, since the mathematical model has the same form
as a scientific law, there is no reason why the former cannot be used as an estimate of the latter.
To argue that the result of a regression analysis is an estimate of a true model, two
further assumptions must be sustained. The first is an ontological assumption that social
relations are modeled in a way that can be described as a mathematical function. The second is
an epistemological assumption that it is possible to know the parameters of this function. Both
17
assumptions are difficult to sustain. First, regarding the ontological assumption, there is no
procedure that can be used to support or refute it, so it is a matter of belief. One may believe that
social relations are in fact modeled by a true model that is yet unknown to us (see Elster, 1998:
62 for such a belief). The second assumption is even harder to sustain. Even if we believe that
social relations follow a law-like mathematical function, the current status of our knowledge in
the social sciences provides few clues regarding the form of this function.
Although the term 'true model' appears in the technical presentation of the statistical
model of regression analysis (and is often heard in discussions of regression analysis), I have not
found any political scientist who argues that he or she is trying to find such a true model.
However, one of the responses of political scientists to the problem of justifying a 'true model' is
to argue that regression analysis cannot be used for explanatory inference but only for a
descriptive one (for the distinction, see King, Keohane & Verba, 1994: ch. 2-3). For example,
Achen (1982: 16) argues that in the social sciences "neither the form of the theoretically
specified relationship nor the list of the controlling factors has law-like character." However,
holding the Hempelian concept of scientific explanation, the logical result is that the regression
equation is merely "one aspect of a good description of the data" (Achen, 1982:16).13 Luskin
(1991: 1038) seems to adopt the same answer when he claims that "uniquely true models exist
only in the assumptions of econometric proofs. A given y can always be explained in a number
13
Achen (1982: 16) also suggests that "even the investigation and testing of causal statement in
social science reduces to descriptive work -- description whose main purpose is the discovery
and testing of theory, but descriptive work all the same." I have difficulties in understanding this
formulation. However it seems to me that what Achen claims here may be salvaged by the
critical realist interpretation of regression analysis.
18
of equally valid ways ... at most, there may plausibly be a single true model of a given type -- at a
given level of conceptual aggregation" (Berry (1993: 7), approvingly cites Luskin on this issue).14
There is no doubt that some forms of statistical inference are sometimes useful for
descriptive purposes only, such as in Election Day sampling. But, it is more difficult to explain
the use of regression analysis for this purpose. To see why this is the case, we can look at the
example Achen (1982) uses to exemplify what he sees as a good application of regression
analysis. This is a study by Eric Veblen (1975) that tries to assess the impact of a newspaper, the
Union Leader, on election results. Regression analysis is used to show that the newspaper's
position had a considerable effect in the primary elections, and moderate effect in the general
elections. Veblen's estimates are not based on "unknown and unwanted" 'true' functional form
and there was no "pretense made that the regression coefficients being estimated represented true
effects constant across space and time" (Achen, 1982: 28). However, at the same time, Achen
concludes that "it has been constructed a prima facie case that the Union Leader makes a nonnegligible difference, and thus by implication, that other printed media also strongly color the
perceptions of their readers" (p. 28). This last generalization cannot be reconciled with Achen's
explication of regression analysis. If the analysis is only descriptive then the conclusion is
clearly a fallacy. I assume that for this reason Achen concludes the discussion with the assertion
14
Luskin (1991: 1038) too tries to avoid the radical relativism implied by his argument -- "the
truest model may not be the best. Parsimony also matters, If what makes model A truer than
model B is the inclusion of real but minor influences, we may prefer model B... But ceteris
paribus, at least, it seems reasonable to prefer greater to lesser truth in the model." It seems from
this citation that Luskin understand truth merely as empirical adequacy. However, he does not
elaborate further on this issue suggesting that "these are deep waters, into which we shall venture
no further."
19
that although his descriptive approach "has the advantage of characterizing what investigators
actually do, it lacks the firm foundation in statistical theory that the older, fictional version [the
'true model' approach] had." Until this "conceptual deficit" will be remedied, "researchers have
an obligation to the nature of their task, even when it is not fully understood theoretically" (p. 2930, see also Achen, 1983: 71). Achen's inability to remedy the “conceptual deficit,” I would
argue, is due to his empiricist view of scientific laws, and that realism can offer a theoretical
foundation for regression analysis that is faithful to the way it is actually done. However, before
moving to the realist explication, I want to discuss the more sophisticated version of the
empiricist explication of regression analysis.
The structure of this version is as follows:
1. Scientific laws are expressed in terms of relations between observations, in the form y=f(xn).
2. Regression analysis rests on a mathematical theorem that assumes the existence of a true
model, in the form y=f(xn).
3. For epistemological reasons, we cannot know the functional form of the 'true model,' and
therefore we can always construct competing models to explain the same process.
4. The only way that these competing models can be corroborated or falsified is by collecting
more and more empirical observations. Hypothetically, if we could collect all available data on
the population, the 'true model' will be the one that best fits the data. However, since we only
have a sample of observation, it is not possible to use "goodness of fit" as the only criterion to
distinguish between the models.
5. Hence: Additional criterion, or criteria, should be used to distinguish between the competing
models.
Both the 'descriptivist' and the 'true model' approaches are hard to attain. As if
acknowledging this, most political scientists usually call for some middle-of-the-road position.
20
However, philosophical considerations must push the empiricist to one of the extreme positions.
If laws are statements of constant conjunction of events, then "goodness of fit" must be a
necessary criterion for the adequacy of scientific laws. Holding such criterion for adequacy, one
can believe either that the empirically best model has no intrinsic value except its pragmatic
usefulness, or that the true model is the best possible fit for the data. Otherwise, one needs other
criterion to select between the possible specifications of the law. Thus, in the discussions about
regression analysis, it is possible to identify 'surplus-elements' to the criterion of empirical
adequacy.15 These attempts can be divided into three groups.
One group adds a different criterion for explanation, usually expressed in terms of
models, theories or causal mechanisms. From all the possible models that can fit the data, only
those that are supported by a theory are considered 'explanatory.' There is no controversy that
theories are important for any explication of scientific activity. The question is how the
empirical criterion and the theoretical criterion are brought together. If one remains faithful to
the view that perfect law should fit the data better than any other alternative then the place of
theory becomes redundant. In an imaginary world, where laws indeed explain the data in an
empiricist sense, the theory that stands behind these laws is no longer important.
Other criterion that can be added to empirical adequacy is the personal knowledge,
beliefs and needs of the scientist. One advocate of this position is Gary King (1991). At the
heart of King's explication is his understanding of models:
... a model is necessarily (and preferably) an abstraction and thus a drastic
simplification, one that if successful will enable one to study only the essential
elements of reality. Models may be good or bad for some purpose or another,
but labeling models as true or false is not fruitful. Can one distinguish between
true and false models of an airplane? Presumably either all models are false or
the only true (sufficiently realistic) model is the airplane itself (although even
15
The term is coined by Buchdahl (1969) and is used by Bhaskar in his analysis of theories and
models (1975: 148ff).
21
actual airplanes do differ from one another). In either case, the goal of finding a
"true model" is neither worthy nor useful (1991: 1048, see also King, Keohane &
Verba, 1994; 49).
Therefore, according to King, "[t]he usefulness of a particular model specification
depends entirely on what causal or forecasting goals one pursues” (1991: 1048). King takes a
model of objects as its paradigmatic case rather than looking at processes. However, when we
think about scientific models we usually think about models of processes. Scientific models are
offered to processes such as a motion of a spring or of a free fall of an object, but one cannot find
a scientific model of an airplane.16 The purpose of a model of an airplane that is used in a wind
tunnel is not to model the airplane itself, but to model its motion. When we look for a model of
the motion of a spring, we neither expect the model to replicate the motion of the original spring
nor to present only one aspect of the motion. Instead, we expect the model to help us better
understand the underlying forces that generate the apparent motion of the spring. Hooke's law
offers such a model.
By offering this objection, I do not want to argue that King is wrong in his understanding
of models. Instead, I suggest that King’s view of social sciences is implicit in his understanding
of models. If models in social science are like models of airplanes, then explaining social
relations is not different from describing them. A full explanation of a phenomenon or a
situation would be a description of all the components that are involved in the situation, in the
same way that a complete model of an airplane would be the airplane itself. When describing
complex social phenomena it is hard to specify all the relevant components. Therefore, a model
16
Scientists are engaged also in modeling objects such as atoms, genes or the solar system. But,
this is a different process from modeling an airplane. The purpose of modeling a gene is to gain
a better understanding of how this unobservable object works. It is not for abstracting a simple
model out of a complex phenomenon.
22
of these phenomena necessarily presents one aspect of the phenomenon being studied. This
approach views social relations as ontologically 'flat,' that is, as lacking any deeper dimension
that might explain the ongoing pattern of social relations.17 One implication of King's approach
is that there is a trade-off between parsimony and precision. A simple model would be
necessarily general but inaccurate. The more accurate one tries to make the model, the more
complex or less general it becomes.
It is important to emphasize that King does not simply recapitulate Achen's descriptivist
approach. While for Achen, the regression equation is merely "one aspect of a good description
of the date" (Achen, 1982:16), King insists that scientists can consult their knowledge for
choosing the correct model. He complains that "researchers are often too quick to claim
ignorance about many parts of their models," and suggests that "[t]he wealth of knowledge most
empirical researchers have about their subject matter should be mined as much as traditional data
sources to improve estimation" (1989: 34). If we are to remain faithful to the claim that there is
no such thing as a "true model" then the question is how knowledge can be used to assess the
model? King's implicit answer to the question is that the knowledge that experienced researchers
have is a practical one, knowledge regarding the necessary aspect of the situation for the required
use of the model. If we take the airplane analogy, a model of the exterior of the airplane cannot
be used to improve the comfort of the passengers. Similarly, a model that looks only at aggregate
indicators of the economy at the national level cannot be used for explaining the distribution
17
In his Unifying Political Methodology, King presents a different definition of models, one that
resembles the 'true model' approach. His definition of statistical model is "a formal
representation of the process by which social system produces output. The essential goal is to
learn about the underlying process that generates output and hence the observed data" (1989: 8).
I do not see how this definition can be reconciled with the ‘airplane’ definition of models.
23
wealth. Regression analysis, therefore, is a tool for the researchers to segment aspects of social
relations that are necessary for their purpose. An experienced and knowledgeable researcher
would use the tool efficiently, while an inexperienced one would make it in a way that gives her
information that she does not need, or leave her without information that she needs.
A second group models the subjective standpoint of the scientist into the regression
analysis is by using Bayesian inference. The problem that the practitioners of Bayesian method
address is of nonstochastic and "weak data" (Western & Jackman, 1994). The problem, which is
more acute in the field of comparative political economy, is that the data is not generated by
random sampling according to the protocols of inferential statistics. Bayesian statistics allows
the researcher to incorporate her insecurities regarding the trustworthiness of the data into the
process of analysis, and then to see whether the results are sensitive to her choices. The result of
this analysis is not a single "best" model, but "parameter estimates and standard errors that
honestly reflect the observed variation of results across a range of plausible models" (Bartels,
1997: 643).
Again, Bayesian statistics is a mathematical model that does not necessarily instantiate
empiricist or any other philosophy of science. One can think about Bayesian analysis resting on
different sorts of prior assumptions, like the beliefs of the scientist regarding the moral value of
the model. However, it seems that by focusing on problems with the data, Bayesian statistics is
implicitly understood in empiricist terms. This can be seen by the focus on weak data. Weak
data, according to Jackman, "provide little information about parameters of statistical models"
(1994: 414). Implicit in this view is the claim that strong data would provide more information
about the parameters of statistical models, and hence that such models exist. Thus, the problem
that is addressed by this application of Bayesian statistics is a specific methodological problem,
namely, the weakness of the data. The answer to this problem is a sophisticated additional
criterion that takes into account the intuition or the knowledge that the scientist has about the
24
subject matter. Nevertheless, implicit in the answer is the view that scientific laws should be
represented as formal mathematical functions.
Bayesian statistics is often understood as a subjectivist approach to the study of society.
However, if the interpretation I offer here is correct, then the subjectivism of this approach is
very minimal and answering only a methodological problem. Underlying this subjectivist
element is a commitment to the idea of true model that is stronger than other approaches.
The third group of answers attacks the problem from a different angle. Instead of adding
a theoretical criterion, one can add a different empirical criterion to select between models.
Different methods of robustness check and sensitivity analysis may be used in addition to the
criterion of goodness of fit (Leamer, 1983). I will discuss the question of robustness in detail
later. For now, however, it is important to see that by adding different empirical criteria one
does not avoid theoretical implications. Each criterion implies an underlying assumption about
what makes one model more adequate than other.
In the discussion of these alternative answers, I do not want to de-emphasize the
importance of theories, prior beliefs, or alternative methods of empirical adequacy for the work
of good scientists. It should be clear, again, that my aim is to examine critically the way that
these criteria are offered as part of an empiricist explication of regression analysis. The problem
with the empiricist explication, even in its sophisticated form, is not that it does not make sense
in itself, as a philosophical doctrine. On the contrary, it makes a lot of sense, especially since we
are predisposed to think about scientific laws in terms of relation between observable events.
The problem with this explication lies elsewhere, in the difficulties that it has in explaining the
way scientists actually practice regression analysis. In particular, it has a problem in accounting
for the interactive ways in which scientists go back-and-forth from the theoretical modeling to
the empirical patterns given by the data. Thus, the empiricist explication fails exactly where its
promise lies, in offering a description of the activity of scientists using the tools of logic.
25
Without an additional criterion for mediating between goodness of fit and the additional
criterion, we are left exactly at the point where we began. With two criteria it is possible, post
hoc, to reconstruct the decision making of scientists in choosing the model (such as "the model is
not the best in terms of goodness of fit, but is more robust than the others"). However, it does
not offer sufficient tools for the scientists in the process of choosing between the models.
Leamer describes the process of choosing between models in an instructive way: " [t]he
econometric art as it is practiced at the computer terminal involves fitting many, perhaps
thousands, of statistical models. One or several that the researcher finds pleasing are selected for
reporting purposes" (1983: 36)".18 Leamer is perhaps too harsh. It is not that scientists do not
have agreed-upon procedures to select between models; they do. The problem is that it is
difficult to explicate these procedures merely using the tools of logic. Empiricism fails to give a
proper explication of the way practitioners of regression analysis 'play' with the data as an
integral part of their activity. 19 This failure leads to an ambiguous attitude of textbook
discussions regarding this practice.
Bartels and Brady (1993: 141) warn against what they call "data dredging," which they
define as "an extensive exploration of alternative specifications in search of one that produces
'significant' parameter estimates, a high R squared value, or any other desired results." They
18
Leamer himself offers in this article a methodological solution to this problem, based on
fragility analysis. The methodological solution, as I argue in this paper, does not solve the
philosophical question of explicating the way the procedure is practiced.
19
Kritzer (1996) makes an analogy between quantitative analysis and performance art. Leamer
(1983: 36) also uses the term 'econometric art.' However, I will argue in the next part that to
understand the practice of regression analysis it is not necessary to make the problematic analogy
with art.
26
even cite Green's claim that "data-dredgers are made, not born," a phrase that is commonly used
in reference to criminals (Green, 1990). A different tone can be heard in Tufte's discussion
(1974: 146-7; see also Freeman, Williams & Lin, 1989: 842)
There is, then, an interplay between explanatory ideas and the examination of
data. Some variables were tried out on the basis of a vague idea and were then
discarded when they yielded no explanatory ideas and the examination of the
data. Some variables were tried out on the basis of a vague idea and were then
discarded when they yielded no explanatory return. ... but the results just did not
seem solid enough to warrant inclusion in the final model ... Now, looking at
several different multiple regressions and sorting around through different
variables may not fit some abstract models of scientific research procedure -- but
it is normally done in constructing explanatory models, and it is precisely this
sorting through of various notions that is the heart of data analysis.
There is no contradiction between the two arguments. ‘Data-dredging’ is unwanted and
yet 'playing' with the data is an integral part of statistical research. What I want to emphasize by
these citations is some unarticulated guilt feelings that hang over the practice of regression
analysis, which are more than a warning against fraud in science. I claim that the source of this
ambiguity can be traced to empiricist assumptions about science.
As we have seen, under empiricism the scientist becomes, metaphorically speaking, a
slave of two masters, empirical and other criteria of validity, where empirical adequacy is a
necessary but not sufficient criterion for an explanation. The assumption of a closed world leads
to a situation were theory is necessarily to be blamed when the empirical evidence does not
match theoretical assumptions. In real life, theory never fully matches the data and therefore it is
always up to the scientist whether to revise the theory or to declare that it is false or at least at
fault. This formulation creates numerous problems for the philosopher of science who tries to
defend the rationality of scientific practice from within the empiricist tradition. It makes even
more problems for the scientists, who have to ‘play’ with the data behind closed doors, and then
in order to present the result to translate it into empiricist terms.
27
The solution to this problematic situation is offered by critical realism. It is simple,
brave and relies on a radical change in our understanding of regression analysis.20 Critical
realism argues that since scientific laws are not just statements of constant conjunctions of
events, but rather are tendencies. Therefore, empirical adequacy is not only insufficient, but also
unnecessary for establishing an explanation (Bhaskar, 1975: 14). By doing so, critical realism
moves beyond being another middle-of-the-road approach between what I have called a
"descriptivist" and a "true-model" approach. However, it must be admitted that this solution
raises a different set of problems. The most difficult problem is this, if empirical adequacy is not
necessary for establishing an explanation then why bother engaging in empirical research. This
problem is especially difficult since I have claimed that critical realism can better explicate the
activity of regression analysis. In the following section, I will try to solve this problem.
2.2. Regression Analysis as an Activity -- The Realist Interpretation
According to critical realism, the scientific activity of explanation consists of three
analytically different stages. During the first stage, a regularity is observed; during the second
stage hypothetical causal mechanism is offered to explain the regularity; and during the third
stage scientific activity is directed at isolating the mechanism itself. Adopting this model of
explanation, we can see that the empiricist interpretation of regression analysis understands this
procedure only as part of either the first or the second stage. According to the descriptivist,
regression is used merely to identify regularities and cannot say much about the mechanisms that
cause them. According the to 'true model' or the 'theory' interpretations, an ideal form of
regression equation should represent the form of the causal process. The critical realist
20
There is a debate between critical realists about the interpretation of regression analysis. Sayer
(1992) views regression analysis as internally connected to empiricism. Porpora’s (1998)
argument is close to the argument presented here.
28
interpretation that I suggest here sees regression analysis as a part of the third stage. In
regression analysis, the scientist tries to demonstrate the existence of a causal mechanism by
controlling for other mechanisms that could have acted at that time. The scientist shows that by
using the technique of regression analysis, she was able to identify a statement of the form of
'whenever this, then that,' which could not be simply observed from the data. However, in doing
so the scientist does not simply put the theory into a simple empirical test. Rather, he or she is
experimenting with the data in order to reveal the underlying mechanism generating it.
Regression analysis, in this view, is more than a tool for corroborating or falsifying theories; it is
a means for the revelation of mechanisms.
To make the argument, I first discuss the parallels between regression analysis and
experiments as understood by realism. Then, I discuss some possible objections to this view.
The starting point of the discussion is the definition of a closed system as one in which a constant
conjunction of events occurs. A further distinction should be made between universal and
restricted closures, as well as between artificial and spontaneous closures (Bhaskar, 1975: 91).
Some empiricists assume that a universal closure exists and that the aim of science is to connect
all events in law-like statements. So far, however, we know only restricted closures.
Nonetheless, we are able to recognize many situations wherein restricted closures occur
spontaneously. These are situations in which we are able to say, with more or less confidence,
that whenever one thing happens, the other will happen too. The billiard table as well as the
planetary system are the prototypical examples. In these settings Newtonian laws can be applied
in a way that not only explains but allows predictions. Social sciences were also able to identify
closed systems, although in a looser sense. In the process of coalition formation or in some small
decision-making designs it seems that it is possible to identify a set of law-like generalizations.
These are situations that occur spontaneously in everyday life without the intervention of
scientists.
29
Scientists do not study only closed systems that occur spontaneously, but also artificially
create closed systems by using experiments. In experimental designs, the scientists set the
environment in such way that regularities occur. Experiments, however, are difficult to obtain in
social science, and therefore alternative forms of inference are used. Statistical procedures, such
as regression, allows 'statistical control' instead of 'experimental control' (Lewis-Beck, 1980: 49).
Traditionally, as I said before, regression analysis was understood as operating under the
assumption that universal closure is attainable. However, it is possible, and even more accurate,
to see regression analysis as an attempt to identify spontaneously occurring closures. Thus, the
same way that a phenomenon or regularity that is not observed directly in open systems is
revealed in the experimental design, a hidden regularity is revealed through the use of regression
analysis.
The conventional wisdom correctly recognizes that since scientists do not have a causal
role in assigning cases into categories, regression analysis offers weaker internal validity than
experiments. However, the conventional wisdom overlooks the role of the scientist in setting
both the experimental design and the regression model. Thus, it is assumed that the design of
both experiments and regression analysis follow directly from the questions set by the scientist.
Achen, for example, claims that "so long as the experimental design is faithfully executed, the
experimental results will give an honest estimate of the treatment effect in the population tested"
(Achen, 1986: 2).
This understanding of experiments is vividly challenged by the realist Ian Hacking
(1983):
That may sound as if we believe in the electrons because we predict how our
apparatus will behave. That too is misleading. We have a number of general
ideas about how to prepare polarized electrons, say. We spend a lot of time
building prototypes that don't work. We get rid of innumerable bugs. Often we
have to give up and try another approach. Debugging is not a matter of
theoretically explaining or predicting what is going wrong. It is partly a matter
of getting rid of 'noise' in the apparatus ... We are completely convinced of the
reality of electrons when we regularly set out to build -- and often enough
30
succeed in building -- new kinds of device that use various well-understood
causal properties of electrons to interfere in other more hypothetical parts of
nature (p 265).
... roughly speaking, no one ever repeats an experiment. Typically serious
repetitions of an experiment are attempts to do the same thing better -- to
produce more stable, less noisy version of the phenomenon. A repetition of an
experiment usually uses different kind of equipment. ... The point of those
classroom exercises is never to test or elaborate the theory. The point is to teach
people how to become experimenters -- and to winnow out those for whom
experimental science is not the right career. (p 231).
I believe that Hacking's point is relevant not only to experiments in the natural sciences,
but also to experiments in the social sciences. More importantly, it is relevant to regression
analysis.21 Regression analysis is an activity of trying to show some phenomena can be explained
by a specific mechanism. It rarely succeeds on the first try and various models are needed to
establish a good demonstration. Any explication of regression analysis must be able to account
for the active role of the scientist in the process. Besides, being able to do good regression
analysis is a function not only of knowledge but also of experience. The experience is not
necessarily with the subtleties of the method but also with the unpredictable characteristics of the
subject matter.
The realist interpretation of experiment is different from the empiricist interpretation in
two elements. First, according to realism, the research design aims to create conditions for the
powerful particular to operate and be observed, and not to assign cases for each category of the
independent variable. Second, the realist interpretation distinguishes between two functions in
the experimental design, experimental production and experimental control. The experimental
21
In regression analysis, we should distinguish between two types of repetitions. A repetition of
regression analysis with the same database is prevalent, but this ensures only that there was no
mistake or fraud in the first place. Repetitions with a different database are much less frequent,
exactly because the same equation cannot be applied, as is, for a different sample.
31
production triggers the mechanism that causes the thing to operate; the experimental control tries
to prevent any interference with the operation of the mechanism and thus make the action
observed (Bhaskar, 1975: 53).
In regression analysis, the scientist cannot take an active role in triggering the
mechanism to operate. However, the scientist still has an important role in designing the
experimental control. Theory does inform the scientist on the internal conditions that need to be
satisfied for the mechanism to operate. But, it can never fully inform the scientist about other
interference with the operation of the mechanism. Moreover, statistical manipulation may be
necessary for the action to be observed. Here, it is not only the theory that directs the research
but it is the scientist's imaginative technical knowledge that makes the regression analysis
meaningful.
An important question, however, is whether this attempt is a fruitful one. Are there
many situations in social life in which closed system occurs spontaneously and can be identified
using the tools of regression analysis? I would argue that the answer is yes, and that is exactly
the reason why empiricism is as convincing philosophy of science as it is. Society, as
interpretative sociologists observed long ago, is rule-governed. In many cases we have the
feeling that the agents' behavior can be described using the formula 'whenever this, then that.'
However, attempts to capture the regularity through a law-like statement reveals that it is much
more slippery than what we thought in the beginning. Under empiricism, we are forced into one
of two directions. Either, we can claim that society cannot be described in law-like form and
hence cannot be scientifically studied; or we can try to add more variables to the law to restrain
what seems to be a "rebellious" law statement. The understanding of laws as of 'tendencies' that
exist in an 'open world' help us to understand that a law-like regularity can be identified and
explained, although it cannot be articulated as a universal law. Sayer (1992: 214-5) makes a
similar point when he claims that "Although we often cannot predict when an event will happen,
32
e.g. when the fish will be hooked, or when the value of the £ will rise, we can explain how it
happens when it does, by closely examining the nature of the objects possessing the relevant
powers and liabilities and the mechanisms by which they work, when they work. In the case of
the value of the £, we could find out the reasons why currency speculator and others bought
sterling…." Regression can be used, and indeed is used retroactively, to identify these
mechanisms and not to predict the future.
Another possible objection to the argument I suggest is that it is not possible to identify
closure in large-scale phenomena. Thus, allegedly, social analysis must focus on small-scale
situations (Harré, 1993). I have two objections to this argument. First, I believe that these types
of questions should not and cannot be answered by philosophers but by scientists; the question is
empirical and not philosophical. Moreover, as I said before, large-scale regularities can be easily
observed in real-life situations. This is the reason that scientists believe that the can readily
identify law-like statements. If scientists would comply with the philosophical requirement not
to study large-scale phenomena such as inflation, unemployment, or war, this would not mean
that the apparent regularities in these phenomena would go away. The alternative would be a
crude explanation rather than no explanation.
However, the empiricist might object, regression equations can be used and are indeed
used for post-sample predictions, exactly because they are more than demonstrations. There is
no doubt that regression analysis can improve predictions in many cases, and are useful, for
example in psychometrics. The question is whether post-sample prediction can be genuinely
explanatory. I want to examine such a claim by looking at one study that uses this method.
Atesoglu and Congleton (1982) use Kramer's analysis of economic voting for post-sample
prediction of eight subsequent elections. Bartels and Brady (1992: 141-2) cite this work in their
'The State of Quantitative Political Methodology' as an example of the usefulness of post-sample
predictions as a model specification test. Realists agree that tendencies are not case-specific and
33
can be identified outside of the original sample. However, for realists, the equation itself is casespecific and cannot be applied as is to other cases. Atesoglu and Congleton take Kramer’s
regression equation and try to use it, as is, for a post-sample prediction. Apparently, one cannot
get more empiricist oriented than in claiming that the "acid test of time series model is its ability
[of the equation itself] to perform well outside the sample period" (Atesoglu & Congleton, 1982:
873). However, a closer look at the result obtained reveals that empiricism cannot tell the full
story.
Kramer (1971) tried to show in the original paper that voters choose a candidate using a
decision rule that is based on readily available information. Thus, the past performance of the
incumbent party gives the voter some indication of what it would do if returned to office. 22
Unfortunately for the empiricist, the Watergate event convinced many voters not to follow
Kramer's rule and hence Kramer's original equations do not predict better than a naive prediction
based on past vote. Nevertheless, "the results become more favorable for the Kramer equations
when the unexpected Watergate event and its apparently persistent repercussions are deleted
form the post sample-period" (Atesoglu & Congleton, 1982: 875). Thus, to show the usefulness
of the model, they use dummy variables and modify the original equation to account for the
effect of Watergate. If Atesoglu and Congleton try to demonstrate the post-sample predictive
power of Kramer's equations then the demonstration is a poor one.23 However, if they try to
22
As far as I understand from the paper, Kramer (1971: 140) does not suggest that the models he
specifies have predictive power. He summarizes his results in qualitative terms arguing that
"election outcomes are in substantial part responsive to the objective changes occurring under the
incumbent party."
23
In one of the models they specify they delete as much as seven years out of their 15 years'
post-sample period.
34
demonstrate that voters tend to follow Kramer's rule, then they offer a strong argument. I believe
that their argument tries to do the latter and therefore it is not possible to simply take Kramer’s
equation and try to apply it to other samples. According to the realist interpretation, it takes
different specifications to demonstrate the way a mechanism operates in different situations.
Equations cannot travel from one place to another but are always case specific. It might be
possible that the regression equation that is used in one sample can be a useful starting point for
the study of a different sample, but this is because the same mechanism is at work in both cases
and not because the equation itself is a model that describes the exact way the mechanism
operates by using a mathematical function. The lesson to be taken from this example is that even
when scientists explicitly understand their work in terms of one philosophy of science, we have
to be very careful in accepting this claim because another may be better instantiated in fact. We
must examine closer if indeed their actual works indeed corresponds to what they claim to be
doing.
4. The 'Norwegian exceptionalism' debate
As I have emphasized several times above, philosophy of science cannot replace
scientists in their work. However, a philosophical discussion can clarify the underlying issues
that are at stake. In this section, I argue that critical realism can offer conceptual tools for better
understanding the core issues that are at stake in the debate between Peter Lange and Geoffrey
Garrett on one side, and Robert Jackman on the other side regarding the issue of the 'Norwegian
exceptionalism' (Lange & Garrett, 1985, 1986, 1987; Jackman, 1987, 1989). Then, I want to use
issues that are raised in the debate to discuss topics in applied regression.
The importance of this debate exceeds the specific theoretical issues that are discussed.
The debate highlights a major problem in comparative politics, the weakness of the data
(Western & Jackman, 1994: 414). This debate revolves around fifteen cases of what is defined
as the advanced capitalist democracies. Jackman's response to the argument presented by Lange
35
and Garrett is based on the fact that if Norway is excluded from this population then Lange and
Garrett's results are changed substantively. Therefore, the debate is an important case for the
study of the role of data in statistical research and as such reappears in other discussions of
methodology, beyond its contribution to the study of European politics (Western & Jackman,
1994; Bartels, 1997).
Lange and Garrett (1985) claim that both organizational power and the political power of
the left have marginal influence on economic growth. Strong organization of labor makes
collective bargain possible, while political power ensures that future increase in societal product
will be distributed favorably to workers. Only when labor is strong in both dimensions advanced
industrial democracies will have the political conditions necessary for growth. When labor is
weak in both dimensions the play of market forces will ensure medium-term economic growth.
Lange and Garrett offer empirical evidence for their argument. Jackman (1985) challenges these
findings and claims that they were created by the influence of Norway's exceptional economic
growth that followed the supply of North Sea oil. Jackman goes on and claims that the
theoretical assumptions of Lange and Garrett are problematic and suggests that labor's political
power does not give leftist politicians significant influence on state's policy.
In comparing a critical realist with an empiricist interpretation of the debate over the
Norwegian exceptionalism, the central issue is the relation between the statistical analysis and
the theoretical explanations that are offered. To discuss this, it is useful to rephrase the argument
in the language of mechanisms, although the participants in the debate are not using these
phrases themselves.24 Both sides in the debate agree that a mechanism that relates domestic
24
It should be noted that Lange and Garrett do not use explicit empiricist phrases either. They
talk about their interaction thesis using terms such as hypothesis (1985: 793), association,
36
structure and economic growth exists, and that the mechanism acts in the way described by
Lange and Garrett (Jackman, 1987: 254). However, Jackman argues that although such
mechanism exists it is rarely set in action because the initial conditions are not satisfied. Leftist
parties, while in government, can pursue policies favorable for labor, but for various reasons they
not necessarily do so (p. 254-5; 1989: 656-7). This point is important, because Jackman does not
challenge the existence of the mechanism but only argues that Lange and Garrett were unable to
make this mechanism observed. The regression analysis that they use does not demonstrate the
mechanism in question but the effect of Norway. "The presence of North Sea oil," according to
Jackman, "has made Norway's economic growth since 1973 so unusual that Norway does not
belong to the analysis" (1985: 251).
Lange and Garrett claim that not only does Norway belong to the analysis, but that it is
actually Norway's corporatist structure that helped in transferring the North Sea oil into
economic growth (1987: 259). However, at the level of empirical observations Jackman's
analysis challenges their results. To put it in Bhaskar's terms, Jackman argues that the closed
system identified by Lange and Garrett does not teach us about the mechanism in question. From
an empiricist viewpoint, Jackman's analysis almost put a checkmate on Lange and Garrett's
argument. Every alternative analysis that they offer would be post- hoc evidence bound to a
grave suspicion. However, in their 1987 response to Jackman's argument, Lange and Garrett
extend their analysis and bring further evidence to support their claim.
What are Lange and Garrett trying to do in bringing more evidence? I think that it would
be hard to argue that they are trying to improve their generalization. A better description would
use the term "demonstration." Through the further evidence they bring they try to identify a
likelihood to be inclined (p. 798) and positive contribution of variables to economic growth (p.
799, 821).
37
closed system in which the activity of the domestic structure can be discerned, and thus to
demonstrate the marginal activity of domestic structure on economic growth. Anyway, by saying
this, I do not want to argue that either Lange and Garrett or Jackman offer a substantively better
analysis of the relation between domestic structure and economic growth nor do I want to claim
that the mechanisms they discuss necessarily exist. This, again, is a question for scientific
inquiry. In my analysis here, I only argue that the structure of the analysis can be best
understood in realist terms.
I now want to use the issues raised in the debate to reexamine some topics in applied
regression. Again, I want to argue that the procedures used in the practice of regression analysis
can be better understood in realist terms. The first topic that I want to study is the use of dummy
variables.25 In his analysis, Jackman adds Norway as a dummy variable to control for its
influence on the results. Bartels (1997: 653) comments that the choice to use a model in which
Norway is a dummy variable is a posterior decision and actually every other case could be used
as a dummy variable. For this reason, there is no any strong basis for preferring a model without
Norway to a model with Norway. Bartels, in this line of argument, presents an empiricist view,
which argues that the task of scientific activity is to choose between competing formal models.26
Critical realism sees scientific activity differently. Scientists are trying to identify closures either
by using experiments or by 'playing' with the data in regression analysis. For this reason, the
central question debated is not whose model is better, but whether Lange and Garrett were able
to demonstrate the effect of the domestic structure. Jackman's claim that "Norway does not
25
Dummy variables can be used to add a nominal variable to the regression equation or to
exclude some cases from the analysis. I discuss here only the second use.
26
Bartels argues in this article that one has to take into consideration his or her prior beliefs and
not test only the model's goodness of fit.
38
belong to the analysis" (1985: 251) can only be understood in context of an attempt to identify a
restricted closure in which the activity of the mechanism can be observed. Therefore, the dummy
variable is not added for the sake of improving the formal model but for correctly identifying the
mechanism.
My critique here is directed against Bartels' philosophical view of scientific activity. As
a scientist, Bartels has the right to challenge the prior assumptions of Lange and Garrett or of
Jackman. Indeed, Bartels does not accept the claim for Norwegian exceptionalism and adds a
variable that measures the dependence on important oil to his model. This exemplifies the point
I have made in the first part. Scientists try to make their explanations as comprehensive as
possible and thus would not accept dummy variables or exceptional cases but try to explain these
cases. This, however, does not imply that scientific activity can be best viewed as an attempt to
arrive at the most comprehensive model.
Another topic that is raised by the debate is that of robustness check. Robustness check
is the use of various methods to examine to what extent the key estimates would be changed
under plausible alternative specifications of the model (Bartels & Brady, 1993: 141). Jackman
(1985: 251) raises the importance of robustness as the 'broader methodological issue' that is
implied by his argument. The question that I want to examine here is why robustness can serve
as a virtue of a good model.
Little (1995: 261) makes a useful distinction between robustness and autonomy.
Robustness is “a measure of the degree to which the results of the model persist under small
perturbations in the settings of parameters, formulation of the equation and so on” while
autonomy is “the stability of the model's results in face of variations of contextual factors.”
Lange and Garrett and Jackman use the term robustness in both senses, either to examine the
effect of Norway or to examine the effect of change in the time lags (Lange & Garrett, 1987:
263).
39
Robustness, in the sense of autonomy can be viewed as a form of out of sample
predictions. As such, I have discussed the usefulness of this method earlier. Nevertheless, there
is more in robustness check than simple out of sample prediction. The question is what? From a
strict empiricist viewpoint there is no value in robustness check, since the criterion for the
inclusion of a variable in a model is its contribution to the goodness of fit. Robustness check,
therefore, implies that there is a real mechanism that produces the data and that the regression
equation aims at identifying this mechanism. But, it implies more. It implies that this
mechanism cannot be described by one regression equation in the sense of a 'true model.' This
means that the mechanism cannot be completely described as a set of relations between the
observations but must, in some sense, exceed the observations. It should be emphasized that this
does not necessarily imply the critical realist view of an open world. One can argue, based on
the assumption of a universal closure, that theoretically these mechanisms can be described as a
relation between observations, but the weakness of the data we have requires us to use other
criteria besides goodness of fit. Again, my strategy in this paper is not to refute the empiricist
interpretation but to articulate the realist as a more plausible one. I would argue that even though
more observations might make it possible to better study the relations between domestic structure
and economic growth, it would never be possible to describe these relations in a law-like form.
Moreover, the scientists that participate in this debate are not trying to do so.
I believe that critical realism can be useful in understanding other topics in applied
regression as well. Although I would not develop these specific issues here, it is not hard to see
how a realist view, as presented here, can address the issues of specification, functional form, the
relative importance of the different variables in the equation or multicolinearity.
40
5. Conclusions
In the conclusion, I want to discuss the importance of the argument I have presented
above for three different frontiers. The first frontier is that of critical realism itself. There is a
debate between critical realists whether their framework implies a specific method for social
science, or only a philosophical framework for understanding social science as practiced by
social scientists (Porpora, 1998). Those that hold the former position argue that an analysis of
the interpretative aspects of social mechanisms excludes various methods of social analysis,
including regression analysis. The argument I present here belongs to the second position. I
believe that the power of critical realism is not in inventing new methods for social science but in
making sense of the existing methods (although looking for new or improved methods is still an
important task). In the argument I present here, I do not intend to argue that regression analysis
can be used for every inquiry in social sciences nor do I want to defend every mindless throwing
of variables into a regression model. However, I argue that it would be shortsighted for social
scientists to simply ignore the conclusions that were gained by using methods such as regression
analysis. Critical realism can offer a powerful way for understanding the results obtained by
these works without subscribing to the empiricist philosophy of science.
The second frontier in concerns the aim of the social sciences as a whole. Following
Habermas (1971), there is a common argument that various forms of social inquiry can be
discerned according to the cognitive interest that motivates them. In this distinction, empiricalanalytic sciences are guided by the technical interest of gaining control while other forms of
scientific inquiry are motivated by the interests of communication or of emancipation. In
maintaining methodological rigor and in using methods such as regression analysis, social
scientists are often understood as following the logic of the technical interest. This view is based
on the identification of statistical procedures with the empiricist understanding of science, in
which to explain an event is to be able to predict it. However, if we follow critical realism in
breaking the identification of science with empiricism, as I have suggested in this paper, then it is
41
questionable whether scientific activity is aimed only at achieving control. Moreover, it can be
argued that it is the scientific activity itself that is an important tool for achieving effective
communication and emancipation (Bhaskar, 1991).
The third frontier is regression analysis. Here the question is what are the implications
of the argument presented here for the practitioners of regression analysis. To the extent that I
am correct in the claim that critical realism can better explicate the practice of regression
analysis then there are not many direct or practical implications to the practice of regression
analysis itself. However, the shift from an empiricist to a realist self-understanding of regression
analysis has important implications. First, critical realism can help to bridge the gap between
textbook prescriptions of how to run regression and the way it is actually practiced 'behind closed
doors.' In this sense, critical realism allows the practitioner of regression analysis to open the
door when 'playing' with the data. Then, critical realism helps to articulate the relations between
the theoretical analysis and the empirical inquiry, and the appropriate relation between the verbal
description of the mechanisms in question and the empirical evidence given to support the
argument. Finally, critical realism helps to chart the limits of regression analysis. Critical
realism would strongly disagree with King's view of statistical model as "the formal
representation of the process by which a social system produces output" (1989: 8). Social
mechanisms, according to critical realism, cannot be described in terms of the observations they
produce. Therefore, statistical analysis can never stand alone as a formal representation of a
mechanism. However, to the extent that critical realism fails to explicate the activity of political
scientists, and they indeed do try to find formal models that describe constant conjunctions of
events, critical realism offers a philosophical standpoint from which it is possible to criticize this
type of scientific activity.
At this point a skeptical reader might wonder whether it is possible to describe social
mechanisms at all. I think that the answer to this question depends on the way one understands
social mechanisms. In this sense, the term “mechanism” may be misleading. Social mechanisms
cannot be described in the way that a mechanism of a clock or a car is described, mainly because
42
they are not working in such a way. Therefore, in my view, social mechanisms can be described
only in a very loose sense. This, however, does not mean that their influence is only marginal. I
believe that social sciences are more successful in describing social mechanisms than are often
argued. I am, at least, convinced by Lange and Garrett that domestic structure influences
economic growth, by Riker that coalitions tend to be minimal, by Michels' iron law of oligarchy,
and by Marx's that capitalism has self-destructive mechanisms. This, however, does not mean
that any of these mechanisms is fully articulated. Moreover, it does not mean that some of these
mechanisms are equal in their importance nor that these mechanisms necessarily indeed act in
every situation. The answer to these questions should, in my view, be given by scientific inquiry.
I want to conclude with a more general statement regarding the task of the philosophy of
science. One can ask what are the practical implications of the argument I present here; what is
the lesson for the scientists? I do not think that there is any technical lesson that can be taken
from the argument. Scientists should not ask philosophers to choose their methods for them.
The choice of method is their job (and of course they can do it wrong or thoughtlessly).
However, I do believe that an adequate explication of scientific activity is important for the work
of the scientists themselves. An adequate explication is needed for connecting social sciences
with social activity. Realism can be used to warn scientists that although they study real aspects
of society, their knowledge does not give them, nor anyone else, control over it. But, it does give
them an advantageous position, from which it is possible to bring change to society.
6. Reference List:
Achen, C. A. (1982). Interpreting and Using Regression. California: Sage Publications.
Achen, C. A. (1983). “Toward Theories of Data: The State of Political Methodology.” In,
Finifter, A. W. (Ed)., Political Science: The State of the Discipline. Washington: APSA. 69-94.
Achen, C. A. (1986). The Statistical Analysis of Quasi-Experiments. Berkeley: University of
California Press.
Archer, M. Bhaskar, R. Collier, A. Lawson, T. Norrie, A. (Eds). (1998). Critical Realism:
Essential Readings. London: Routledge.
43
Atesoglu, H. S., Congleton, R. (1982). "Economic Conditions and National Elections, PostSample Forecasts and the Kramer Equations." The American Political Science Review, (76), 4.
873-5.
Ball, T. (1972). “On ‘Historical’ Explanation.” Philosophy of Social Sciences, (2). 181-192.
Bartels, L. M. (1997). “Specifying Uncertainty and Model Averaging.” American Journal of
Political Science, (41), (2). 641-674.
Bartels, L. & Brady, D. H. (1993). "The State of Quantitative Political Methodology." In,
Finifter, S. W. (Ed)., Political Science: The State of the Discipline. Washington: APSA. 121162.
Bhaskar, R. (1975). A Realist Theory of Science. NJ: Humanities Press.
Bhaskar, R. (1979). The Possibility of Naturalism. NJ: Humanities Press.
Bhaskar, R. (1991). Philosophy and the Idea of Freedom. Oxford: Blackwell.
Berry D. W. (1993). Understanding Regression Assumptions. California: Sage Publications.
Buchdahl, G. (1969). Metaphysics and the Philosophy of Science. Oxford: Basil Blackwell.
Collier, A. (1994). Critical Realism: An Introduction to Roy Bhaskar's Philosophy. London:
Verso.
Elster, J. (1998). "A Plea for Mechanism." In, Hedstrom P. and Swedberg R. (Eds). Social
Mechanisms. Cambridge: Cambridge University Press.
Farr, J. (1985). "Situational Analysis: Explanation in Political Science." The Journal of
Politics, (47), 4. 1085-1107.
Freeman, J. R., Williams, J. T. and Lin, T. (1989). "Vector Autoregression and the Study of
Politics." American Journal of Political Science, (33), 4. 842-77.
Green, D. P. (1990). “On the Value of Not Teaching Students to be Dangerous.” Political
Methodology (3). 7-9.
Gunnell, J. G. (1969). "Deduction, Explanation, and the Social Scientific Inquiry." The
American Political Science Review, (63), 4. 1233-46.
Habermas, J. (1971). Knowledge and Human Interests. Boston: Beacon Press.
Hacking, I. (1983). Representing and Intervening. Cambridge: Cambridge University Press.
Harré, R. (1970). The Principles of Scientific Thinking. London: Macmillan.
Harré, R., Madden, E. H. (1975). Causal Powers. New-Jersey: Rowman and Littlefield.
Harré, R. (1993). Social Being. Oxford: Blackwell.
Hempel, C. G. (1965). Aspects of Scientific Explanation, and other Essays in the Philosophy of
Science. New York: Free Press.
Hempel, C. G., Oppenheim P. (1965). "Studies in the Logic of Explanation." In, Hempel C. G.,
Aspects of Scientific Explanation. New York: The Free Press.
Isaac, J. C. (1987). Power and Marxist Theory: A Realist View. Ithaca: Cornell University
Press.
Isaac, J. C. (1990). "Realism and Reality: Some Realistic Reconsiderations." Journal for the
Theory of Social Behaviour. 1-31.
44
Jackman R. W. (1987). "The Politics of Economic Growth in the Industrial Democracies, 197480: Leftist Strength or North Sea Oil?" Journal of Politics (49), 1. 242-56.
Jackman, R. W. (1989). "The Politics of Economic Growth, Once Again." Journal of Politics
(51), 3. 646-61.
King, G. (1986). “How to Lie with Statistics: Avoiding Common Mistakes in Quantitative
Political Science.” American Journal of Political Science (30). 666-87.
King, G. (1989). Unifying Political Methodology. Cambridge: Cambridge University Press.
King, G. (1991). “’Truth’ is Stranger than Prediction, More Questionable than Causal
Inference.” American Journal of Political Science (35), 4. 1047-53.
King, G., Keohane, R. O. & Verba, S. (1994). Designing Social Inquiry. Princeton: Princeton
University Press.
Kritzer, H. M. (1996). "The Data Puzzle: The Nature of Interpretation in Quantitative
Research." American Journal of Politics (40), 1. 1-32.
Lange, P. Garrett, G. (1985). "The Politics of Growth: Strategic Interaction and Economic
Performance in Advanced Industrial Democracies, 1974-1980." Journal of Politics, (49), 1. 257274
Lange, P. Garrett, G. (1987). "The Politics of Growth Reconsidered." Journal of Politics, (47).
792-827.
Leamer, E. E. (1983). "Let's Take the Con Out of Econometrics." The American Economic
Review (73), 1. 31-43.
Lewis-Beck, M. (1980). Applied Regression. Beverly Hills: Sage Publications.
Little, D. (1995). "Economic Models in Development Economics." In, Little, D. (Ed).. On the
Reliability of Economic Models. Boston: Kluwer Academic Publishers.
Luskin, R. C. “Abusus Non Tollit Usum: Standardized Coefficients, Correlations, and R2s.”
American Journal of Political Science (35), 4. 1032-46.
Outhwaite, W. New Philosophies of Social Science: Realism, Hermeneutics and Critical Theory.
London: Macmillan.
Porpora, D. V. (1998). “Do Realists Run Regression?” Paper presented at the Second Annual
Conference on Critical Realism. University of Essex. September 1998.
Riker, W. H. (1982). “The Two - Party System and Duverger's Law: An Essay on the History of
Political Science.” American Political Science Review (76), 4. 753-766.
Salmon, W. C. (1990). Four Decades of Scientific Explanations. University of Minnesota
Press.
Sayer, A. (1992). Method in Social Science: A Realist Approach. London: Routledge.
Tufte, E. R. (1974). Data Analysis for Politics and Policy. Englewood Cliffs: Prentice-Hall.
Western, B. Jackman, S. (1994). “Bayesian Inference for Comparative Research.” American
Political Science Review (88), 2. 412-423.
Veblen, E. (1975). The Manchester Union Leader in New Hampshire Elections. New
Hampshire: The University Press of New-England.