Download THE RELIABILITY OF THE ANES FEELING THERMOMETERS: AN

THE RELIABILITY OF THE ANES
FEELING THERMOMETERS: AN
OPTIMISTIC ASSESSMENT
Robert N. Lupton
Michigan State University
[email protected]
William G. Jacoby
Michigan State University
[email protected]
Prepared for presentation at the 2016 Annual Meetings of the Southern
Political Science Association. San Juan, Puerto Rico, January 7, 2016.
ABSTRACT
The feeling thermometer is a rating-scale survey question intended to measure affective responses
toward a stimulus object, such as a political candidate or party. Items using this format have
been included on the American National Election Study’s interview schedule since 1964. Recently,
however, they have been criticized for being “notoriously unreliable.” To our knowledge, however,
the reliability of the feeling thermometer items has never been tested directly. We employ methods
for three-wave panel data to test the reliability of the thermometer items included in the 1980 NES
Major Panel Study and the ANES 2000-2002-2004 Panel Study. We show that most thermometer
items are extremely reliable, both in absolute terms and relative to other commonly-used items
from the ANES surveys. To the extent that there are thermometer ratings with lower reliability
levels, they are due more to the stimuli than to a problematic item format. For these reasons,
we believe the feeling thermometers can continue to be used as measures of affective responses to
candidates, parties, groups, and other political phenomena.
Feeling thermometers are ubiquitous in the study of American political behavior. But, despite
their wide usage, the feeling thermometer survey items have been criticized for possessing questionable measurement characteristics. While this criticism has taken several forms over the years, a
recent and particularly salient charge is that the feeling thermometers are “notoriously unreliable”
(Broockman, Kalla, Aranow 2015). Interestingly, however, the reliability of the individual feeling
thermometer items apparently has never been tested directly. This seems like a major omissions
for survey questions that are used so widely.
The current study attempts to fill this void. Using a methodology for measuring the reliability of
a single item with three-wave panel data, we find that the feeling thermometers are quite reliable.
And, variations in reliability seem to be explained by the nature of the stimulus objects being
evaluated, rather than the format of the thermometer items. In the interest of brevity, the analysis
reported below only uses data from the American National Election Studies’ (ANES) 1980 Major
Panel Study and the ANES 2000-2002-2004 Panel Study. But, it is important to emphasize that
the results generalize: Virtually identical findings are produced using the other three-wave panel
studies that the ANES deployed over the years. While there may be other measurement concerns
about the feeling thermometers, item reliability is not a hindrance to using these survey questions
in order to elicit affective responses to stimulus objects.
BACKGROUND
The term, “feeling thermometer” refers to a question format used to obtain rating scale responses
from survey respondents toward specific stimulus objects. These items were first used in the 1964
ANES Pre-Post Survey to obtain evaluations of various sociodemographic and political groups.
Starting in 1968, feeling thermometers have been used for presidential candidates, other political
figures (e.g., vice-presidential candidates), and governmental institutions (e.g., the U. S. Supreme
Court). While the exact wording has varied somewhat, depending upon the year and the interview
format (face-to-face or over the telephone), a typical version of the question is as follows:
Please look at page 2 of the booklet. I’d like to get your feelings toward some of our
political leaders and other people who are in the news these days. I’ll read the name
of a person and I’d like you to rate that person using something we call the feeling
thermometer. Ratings between 50 degrees and 100 degrees mean that you feel favorable
and warm toward the person. Ratings between 0 degrees and 50 degrees mean that you
don’t feel favorable toward the person and that you don’t care too much for that person.
You would rate the person at the 50 degree mark if you don’t feel particularly warm or
cold toward the person. If we come to a person whose name you don’t recognize, you
don’t need to rate that person. Just tell me and we’ll move on to the next one.
The booklet page mentioned in the question contains a stylized rendering of a thermometer,
with nine labelled points ranging from 100◦ (labelled “Very warm or favorable feeling”) through
50◦ (labelled “No feeling at all”) to 0◦ (labelled “Very cold or unfavorable feeling”). In principle,
respondents can answer a thermometer question with any number between zero and one hundred;
thus, the format could be regarded as a 101-point scale. In reality, most respondents give integer
responses that end in zero (i.e., 50, 60, etc.), leading many researchers to treat the thermometers
as nine- or eleven-point scales.
The feeling thermometers have proven to be extremely popular, and they (or close variants) have
been deployed by several different survey research organizations beyond the ANES (e.g., Alwin 1997;
Delavande and Manski 2010; Adida, Davenport, McClendon 2015). Feeling thermometers have
been used for many different purposes, including measuring attitudes toward single candidates and
political figures (e.g., Reynolds 1974; Feldman 1988; Adida et al. 2015); modeling vote intentions
for single candidates (Delavande and Manski 2010) or between a pair of candidates (Markus and
Converse 1979; Page and Jones 1979; Jacoby 1999); developing multidimensional representations
of ideological orientations (Conover and Feldman 1981) and party identification (Weisberg 1980),
operationalizing spatial theories of voting (e.g., Rabinowitz 1978; Rabinowitz and Macdonald 1989),
modeling public perceptions of presidential candidates (Weisberg and Rusk 1970; Jacoby 1986;
Jacoby and Armstrong 2014), and modeling perceptions of, and affect toward, sociodemographic
groups (Zinni, Mattei, Rhodebeck 1997; Kalkan, Layman, Uslaner 2009). Thus, analyses of feeling
thermometer responses have played a major role in studies of the mass public’s orientations toward
politics and the social world more generally.
Potential Measurement Problems
Given their widespread usage, it may be somewhat surprising to note that the measurement
characteristics of feeling thermometer ratings remain open to some serious questions. For example,
most analysts treat the thermometers as if they provide interval-level measurement— although few
say so explicitly. Nevertheless, several researchers have suggested that the function relating “true”
2
feelings about stimulus objects to thermometer ratings of those objects is monotonic rather than
linear in form (e.g., Rabinowitz 1976; Winter and Berinksy 1999; Jacoby 1999), thereby implying
that the proper level of measurement is ordinal. Another potential problem is that different people
may use the thermometer scale in different ways. For example some may use the full range of
possible values from zero to 100 to express their feelings toward a battery of stimuli, while others
confine their ratings to a narrower interval along the 101-point scale. Or individuals may make
finer distinctions about their feelings toward objects they like than they do toward objects they
dislike. In either case, this is the problem of interpersonal incomparability of scores, or differential
item functioning. Several studies have used indirect evidence to suggest that this may be occurring
among thermometer ratings (e.g., Brady 1985; Wilcox, Sigelman, Cook 1989; Winter and Berinsky
1999).
Still another potential problem involves the effects of random measurement error on the thermometer scores. To the extent that this occurs, it lowers the reliability of the measurement. And,
recent events suggest that this might be the case: Broockman, Kalla, and Aronow (2015) state
that “Feeling thermometers are notoriously unreliable survey items. That is, in a technical sense,
subject’s responses to feeling thermometers typically contain a fairly large amount of random measurement error” (page 14). Now, the paper in which this statement appears is unpublished and
currently only available online. But it still has had a major impact on the social science research
community: It was precisely this supposed unreliability that led Broockman and his colleagues
to question the integrity of the data analysis in the article, “When Contact Changes Minds: An
Experiment on Transmission of Support for Gay Equality,” by Michael J. LaCour and Donald P.
Green (2014).
Broockman et al. show that there are implausibly high test-retest correlations between feeling
thermometers for “gay men and lesbians” on different waves of a panel study used in the LaCour
and Green article. The average of the six pairwise coefficients (across the four waves of the panel
study) is 0.952. We agree completely that such correlations are far too large to be believable.
But, we are more hesitant about the motivation that led to this conclusion. Broockman et al.
provide no evidence about the poor measurement properties of the feeling thermometers beyond
the conventional wisdom or common knowledge that seems to be implied by the term “notoriously
3
unreliable” (emphasis added). We believe that more systematic investigation is necessary before
tarring a seemingly useful battery of survey items with such a broad brush.
We start with the general assumption that no measurement is perfect— that is, random errors
always occur, regardless of the exact context of the measurement. Hence, all measures are less than
perfectly reliable. In the context of survey research, feeling thermometers are intended to reduce
measurement error by giving respondents a “longer” continuum on which to place themselves
than is the case with three-, five-, or seven-point bipolar response scales (often called “Likert-type
items”). If respondents really do represent their true feelings about the stimulus object with greater
precision, then the feeling thermometer format should reduce random measurement error. But,
many researchers have noted that, despite the potential for 101 different scores, the vast majority
of respondents place themselves at one of the nine labelled points on the feeling thermometer card or
at one of the eleven values that end in zero (e.g., Rabinowitz 1976; Alwin 1992; 1997; Broockman et
al. 2015). The resultant “lumpy” distributions of scores may suggest that people are not capable of
making the fine distinctions implied by the relatively continuous thermometer format (e.g., Winter
and Berinsky 1990). If that is the case, then error is introduced, again reducing the reliability of
the resultant measure. Thus, there probably is reason for concern about the reliability of the feeling
thermometers.
Prior Research on Measurement Error and Reliability
Previous research on this topic is surprisingly limited. Green (1988) used a structural equation
model with feeling thermometers as manifest indicators of latent variables to test multidimensional
representations of party identification and ideology. He shows that conclusions of multiple partisan and ideological dimensions probably were incorrect and due to the combination of random
and nonrandom measurement error in the feeling thermometers. Nevertheless, Green’s results also
show that the feeling thermometers perform at least as well as other indicators of latent partisan and ideological affinities. Furthermore, they show robust correlations with relative candidate
preferences— which are measured by taking differences between feeling thermometer ratings of the
Democratic and Republican presidential candidates. So, while there certainly is measurement error
present, the variables work in a manner consistent with theoretical expectations. Green concludes
that “(t)he evidence . . . speaks well not only for the validity of the measures used in our model but
also for the substantive importance of group evaluations as a topic of study” (1988, page 776).
4
Krosnick (1991) examined the temporal stability of symbolic versus non-symbolic attitudes.
The former category included party identification and ideology. Krosnick included attitudes toward social groups in the latter category, and six of his seven measures were based on thermometer
ratings. He reported that “the party (and) ideology . . . reliabilities are higher than the . . . social
groups reliabilities . . . which in turn are significantly larger than the efficacy and trust reliability”
(page 561). But, any comparison for present purposes is inconclusive, because Krosnick also used
feeling thermometers to measure party identification and ideological orientations. Thus, the reliabilities of the thermometer-based measures range from high to moderate levels; it is difficult to
draw more precise conclusions from the results presented in Krosnick’s article.
Probably most relevant to the present study, Duane Alwin carried out several analyses comparing the reliabilities of feeling thermometer items to the reliabilities of survey items with fewer
response categories. One study used three-wave panels from the ANES (Alwin 1992) and the other
employed a multitrait-multimethod approach to examine items from the University of Michigan’s
1978 Quality of Life Survey (Alwin 1997). The results in both studies showed that feeling thermometers produce more reliable measures than do response formats with fewer categories. Nevertheless,
there are limitations to the utility of these findings for present purposes. Specifically, the NES
results are only presented as average reliabilities across a set of items; therefore, it is impossible
to discern from Alwin’s article the reliability of specific thermometer items. And, the Quality of
Life Survey had respondents use the thermometers to rate their satisfaction with various aspects
of their own lives. This is a very different substantive context than the evaluations of political
stimulus objects for which the feeling thermometers are used by most political scientists.
A recent major study of measurement error and reliability in survey questions focuses on the
methodological and substantive benefits of combining responses to individual items into multipleitem measures (Ansolabehere, Rodden, Snyder 2008). Several of the examples used in this analysis
combined feeling thermometers with other items to form relatively reliable measures of issue attitudes. We certainly agree that the benefits of multiple-item measures are enormous. But, the use
of such measures implies that the scores on the constituent items are not of intrinsic interest because they are merely error-laden replications of the attitude object being tapped by the multi-item
measure. Thus, Ansolabehere et al. really do not address the reliability of the feeling thermometer
items per se, other than asserting that each one is relatively unreliable when taken on its own.
5
To summarize, the overall number of relevant studies is very small. And several of those had
different objectives than evaluating the reliability of the feeling thermometers. Furthermore, the
results often are not presented in great detail, making it impossible to evaluate the reliability of
thermometers toward specific objects. In several of the studies, the feeling thermometers, themselves, are regarded as limited manifest indicators of more general concepts. We believe the latter
assumption may be problematic because each feeling thermometer is intended to measure a person’s attitude toward a specific stimulus object. The stimuli represent important elements of the
political, such as presidential candidates, prominent public figures, political parties, governmental
institutions, and sociodemographic groups. Therefore, we believe that more attention needs to be
given to the reliabilities of the feeling thermometer items.
METHODOLOGICAL STRATEGY AND DATA
Following standard principles of measurement theory, we assume that a feeling thermometer
score, shown as X, can be broken down into two components as follows:
Xi = Ti + Ei
(1)
In equation (1), Xi is respondent i’s thermometer rating of some stimulus, Ti is i’s “true” attitude
toward the stimulus, and Ei is the error or discrepancy between the observed thermometer score
and the actual attitude. Taking the variance of X produces:
2
2
σX
= σT2 + σE
+ 2σT,E
(2)
The last term on the right-hand side of equation (2) is the covariance of the true attitudes and the
errors. But, let us assume (following fairly standard practice) that the two are uncorrelated. In
2 ) breaks down
that case, the last term in equation (2) drops out, and the total observed variance (σX
neatly into two components, the variance in the true attitudes ((σT2 ) and the variance of the errors
2 ). Dividing both sides by σ 2 shows that the total variance in the observed thermometer scores
(σE
X
can be divided into the prortion that is due to variance in the true attitudes, and the proportion
6
that is due to error. The former is defined as the reliability of the feeling thermometer, X, when it
is used as a measure of the attitude toward the object:
RelX =
σT2
2
σX
(3)
Of course, equation (3) cannot be evaluated directly, because the true attitude, T , is unobserved.
Therefore, it must be estimated indirectly.
If X were a scale, defined as a linear combination of multiple indicators of T , then the reliability
of X can be estimated by evaluating the internal consistency of the indicators, defined as the intercorrelations among the constituent indicators. This is precisely the basis for Cronbach’s coefficient
α, by far the most widely-used estimate of scale reliability (e.g., Nunnally and Bernstein 1994).
Unfortunately, we cannot use this approach because we do not have multiple indicators. Again, we
are regarding each thermometer as a single measure of attitude toward the stimulus object.
With single measures, the usual approach is to use test-retest correlations to obtain estimates
of reliability. The logic is that, if a question is administered to respondents at two time points, and
the underlying attitude does not change during the interval between the two administrations, then
the only source of empirical change is error. Hence, one minus the squared correlation between the
thermometer scores at the two time points should give the proportion of the observed variance that
is due to error. And, since the variance breaks down into two additive components, the squared
correlation itself is the proportion due to “true” variance, or the reliability.
Again, the preceding logic is based on the assumption that the true attitude is perfectly stable.
In fact, it might change over time. And, with only two administrations of the feeling thermometers,
there is no way to differentiate substantive change in the true attitude from error fluctuations in
the scores. On the other hand, if the thermometers are administered at three time points, then
there are several ways to differentiate temporal stability in the true attitude from reliability in the
thermometer scores. Let j be an index for the time point, so that j = 1, 2, 3. Now, if we are willing
to assume that the Tj ’s and xj ’s are both standardized to unit variance at each time point, that
Tj is related to Tj−1 by a single-lag autoregressive process, and that the reliability is constant (i.e.,
it does not vary across the time points) then Heise (1969) showed that the reliability of X can be
estimated from the correlation matrix for the Xj ’s. Wiley and Wiley (1970) relax some of Heise’s
7
assumptions. They continue to assume the single-lag autoregressive process across the true scores
2 , is constant over time. But they argue that it is more realistic
and also that the error variance, σE
to assume that the variances of the true attitudes do change over time. This, implies that there
can be different reliabilities for Xj at each time point, rather than a single global estimate for X.
Wiley and Wiley (1970) show that maximum likelihood estimates of the three reliabilities can be
obtained from the covariance matrix for the Xj ’s. We will use the Wiley and Wiley approach to
estimate the reliabilities of feeling thermometer items and, for comparison, several other items from
some of the ANES panel studies.
Over the years, the ANES has conducted five three-wave panel studies that include feeling thermometers for the same stimulus objects over all three waves.1 Here, we will analyze data from two
of these panel studies. First, the 1980 ANES Major Panel Study interviewed survey respondents in
January, June, and September 1980. Thus it measures individual attitudes across the course of a
presidential campaign. The 1980 Major Panel Study includes eleven feeling thermometers toward
individual political figures (most of whom were presidential candidates), six more for the Democratic Party, the Republican Party, Liberals, Conservatives, Independents, and Political Parties
(generally). The Study also includes the standard seven-point party identification index and the
seven-point ideological self-placement variable, along with four additional issue attitudes measured
on seven-point bipolar scales.
The second data source is the 2000-2002-2004 ANES Panel Study which includes 29 feeling
thermometers that were administered on all three waves. Seven of these are for individual people,
two are for Liberals and Conservatives, four tap attitudes toward governmental institutions, and 16
measure evaluations of social groups. This Panel Study also includes the party identification index
on all three waves. And there are questions asking respondents whether federal spending should
be increased, decreased, or kept at current levels in each of ten policy areas.
Currently, we examine only the 1980 and 2000-2002-2004 panel studies in order to keep the
paper relatively brief. But, future iterations will incorporate data from all five ANES panel studies that contain suitable data. For now, we want to emphasize that the results we present below
1
Feeling thermometers were included in all three waves of the 1972-1974-196 ANES Panel Study, the 1980 Major
Panel component of the 1980 ANES, the 1990-1991-1992 ANES Panel Study, the 1992-1994-1996 ANES Panel Study,
and the 2000-2002-2004 ANES Panel Study.
8
definitely are generalizable. Reliabilities for the feeling thermometers from the other three panel
studies are very similar to those we analyze here.
EMPIRICAL RESULTS
The Wiley-Wiley estimates of the reliabilities for the feeling thermometer items in the 1980
Major Panel Study are given in Table 1, and they are depicted graphically in Figure 1. The
reliabilities range over a fairly wide interval, from a minimum of 0.395 (for Political Parties in
general in the June 1980 panel wave) to a maximum of 0.940 (for George H. W. Bush in the
January 1980 wave). But, the central tendency for the 1980 feeling thermometers leans toward
the upper range, with a mean reliability of 0.732. On average, nearly three-fourths of the variance
in the thermometer ratings is shared with the underlying attitudes toward the respective stimulus
objects. This certainly appears to be an acceptable level of reliability for survey data, where
previous research showed much lower average reliabilities for a variety of different kinds of survey
items (Krosnick 1991; Alwin 1992).
Admittedly, the feeling thermometers are not perfect measures. Here, the mean reliability still
leaves 26.8% of the observed variance in the feeling thermometers as error. Of course, we want the
error variance to be as small as possible. So, is this too much? Like judgments about the size of
the R2 in a regression equation, it is difficult to evaluate particular reliability values on their own.
Therefore, it is useful to present some comparative evidence. Table 2 and Figure 2 present the
reliabilities for the other items from the 1980 Major Panel Study.
As expected, the reliabilities for the party identification index are consistently high, ranging
from 0.862 to 0.870. Reliability is lower for ideological self-placement, with coefficients ranging
from 0.635 to 0.657. The reliabilities for the issue questions range from 0.498 (for programs to
help Russia in September 1980) to 0.724 (for defense spending in June 1980), with a mean value of
0.616. So, on survey questions about policy issues, just over 60% of the observed variance is shared
with the true attitude, on average. Of course, this is quite a bit lower than the mean for the feeling
thermometers.
Let us next consider the results for the 2000-2002-2004 Panel Study. The reliabilities for the
feeling thermometer items are presented in Table 3 and Figure 3. Here, the coefficients vary over a
wider range, from a minimum of 0.338 (for evaluations of Blacks in 2004) to a maximum of 0.888
9
(for evaluations of Dick Cheney in 2004). The mean reliability for these thermometers is 0.597. In
the twenty-first century data, the average feeling thermometer variable only shares about 60% of
its variance with the corresponding underlying attitude.
What accounts for the apparent difference between the results obtained for the two datasets?
One obvious possibility is the differing sets of stimulus objects to which the thermometers were
applied. In the 1980 data, 11 of the 15 stimuli were individual figures, and the remaining four
were related to the political parties. In the 2000-2002-2004 data, only seven out of 29 stimuli
were individual political figures; the rest were either related to the federal government or were
sociodemographic groups. Even a quick glance at Figure 3 reveals that the objects in the latter two
subsets tend to have lower reliabilities than do those in the first subset. Specifically, the reliabilities
for individual figures range from 0.706 (for Jesse Jackson in 2002) to 0.888 (again, for Cheney in
2004), with a mean value of 0.787. The coefficients for the remaining stimulus objects range from
0.338 (again, for Blacks in 2004) to 0.745 (for fundamentalists in 2004), with a mean of 0.537.
Thus, the critics may have a point when it comes to evaluations of institutions and groups. For
those stimuli, the feeling thermometers share just over half of their variance (i.e., 53.7%) with the
underlying attitudes. But, the much higher reliabilities for the thermometer ratings of individual
figures strongly suggests that the problem lies with the nature of the stimulus objects rather than
with the feeling thermometer format, itself.
Once again, some comparative evidence should prove useful. Table 4 and Figure 4 provide the
2002 through 2004 reliability estimates for the party identification index and for the 10 questions
about federal spending on specific policies. Party identification again shows extremely high reliability, with coefficients from 0.898 in the first panel wave to 0.912 in the third wave. In contrast,
the spending questions have much lower and more variable reliabilities. The coefficients for latter
items range from a very low 0.263 (for spending on programs to reduce crime in 2004) to a more
acceptable 0.699 (for spending on programs to help Blacks in 2000). The mean reliability for the
spending items is 0.509. We believe these results are quite telling. The reliabilities for another
battery of widely-used survey questions from the ANES (i.e., the spending items) tend to be quite
a bit lower than the reliabilities for the feeling thermometers— even taking into account the fact
that a number of the latter are also quite low. Overall, the feeling thermometers come off very well
in comparisons to other survey questions.
10
While the evidence from the two ANES datasets provides very few examples of unacceptably
low reliabilities among the feeling thermometers, the fact remains that the reliability does vary
markedly from one stimulus object to the next. And this begs the question of why the variability
exists. As we saw earlier, with the 2000-2002-2004 data, the reliabilities for institutions and groups
are almost uniformly lower than those for individual people. This suggests that the reliability for a
given thermometer is affected by the nature of the stimulus object being evaluated. But rather than
focusing on substantively-defined categories of the objects (i.e., groups versus individuals, etc.), we
will consider three characteristics that are common to all of the objects being evaluated with the
feeling thermometers.
First, there is the degree of polarization with respect to the object. That is, how divided
are the ANES respondents in their opinions about the “target” of the feeling thermometer. This
characteristic is measured very easily by simply taking the variance of the thermometer ratings for
each object. Polarization is hypothesized to have a positive effect on item reliability, since more
extreme opinions typically are more strongly held and resistant to change.
Second, there is the degree of opinionation with respect to each object. Stated differently, this
is the extent to which people possess nonneutral feelings about each stimulus object. Opinionation
also is expected to have a positive effect on reliability, since a neutral response implies that a person
really has no affective reaction one way or the other with respect to the object. Opinionation is
measured by taking the proportion of ANES respondents who give feeling thermometer responses
that are either greater or less than 50 (i.e., the neutral value on the thermometer scale).
Third, there is the degree of consensus in feelings about the stimulus object. Here, consensus is
defined as a situation in which most respondents report the same general evaluation of an object.
For example, there appears to be a consensus of very positive feelings toward the elderly. This
stimulus almost uniformly receives very high thermometer scores, with mean values of 80.815,
76.316, and 78.514 in 2000, 2002, and 2004, respectively. Of course, there is variability around
each of these central tendencies; the corresponding standard deviations are 17.500, 17.697, and
17.282, respectively. So even the lower evaluations of the elderly will still tend to fall above the
neutral point of 50. We hypothesize that consensus will be negatively related to reliability. Our
reasoning is that, if everyone feels the same way about a given stimulus object, then variability
in ratings will tend to stem from random fluctuations rather than meaningful differences in the
11
thermometer scores. For each stimulus, consensus is measured by summing the thermometer scores
greater than fifty, summing the scores less than fifty, and then taking the absolute value of the
difference between the two sums.2 To the extent that thermometer scores are either consistently
positive or consistently negative, the absolute difference between the two sums should be greater
than zero.
Polarization, opinionation, and consensus are used as independent variables in a regression
equation to predict the reliability of the thermometers. The OLS estimates of the model parameters
are obtained separately for the two ANES datasets. Note that the three “observations” for each
stimulus object— corresponding to the three waves of each panel study— are not independent of
each other. Therefore, robust standard errors are reported, clustered on the 15 stimuli in the 1980
data, and on the 29 stimuli in the 2000-2002-2004 data. Preliminary results for the 1980 data
revealed that the three observations for George H. W. Bush are outliers that are highly influential
on the estimates. Therefore, a dummy variable is included in the 1980 model coded one for these
three observations and zero otherwise. Finally, inspection of diagnostic plots revealed a strongly
nonlinear relationship between polarization and reliability. This pattern appeared in both datasets.
Accordingly, the regressor for that variable in each model is actually -1 times the inverse of the
polarization value for each stimulus object. This nonlinearity makes it a bit more difficult to
describe the relationship between polarization and reliability. But, it definitely provides a more
accurate depiction of the structure in the data than does a linear functional form.
The OLS estimates for the regression models are shown in Table 5. Note that the model fit is
very good in both datasets, with R2 = 0.739 in the 1980 data, and R2 = 0.834 in the 2000-2002-2004
data. The standard deviation of the residuals is 0.060 for the 1980 model and 0.059 for the 20002002-2004 estimates, showing that the average accuracy with which reliability can be predicted
from the regressors is basically identical for both equations. The signs on the coefficients for the
three main independent variables are all consistent with prior expectations. And, again, the results
are very similar for the two equations. Polarization and opinion both show significant positive
effects on reliability while consensus has a significant negative impact. The outlying observations
for Bush show up clearly with the highly significant positive coefficient for that variable.
2
This variable’s values are divided by 10,000 in order to make the coefficients in the regression model more
reasonable.
12
Given the transformations carried out on two of the independent variables, the coefficients are
somewhat difficult to interpret on their own. Therefore, effects plots for the two models are shown
in Figures 5 and 6. Each panel in the two figures shows the predicted reliability across the range of
one independent variable, with the other independent variables held at their mean values. The first
panel in each figure shows how polarization corresponds to increasing reliability, with the rate of
increase getting smaller across the range of the polarization variable. Although discernible in both
panels, the nonlinearity is especially pronounced in the 2000-2002-2004 data. Similarly, the overall
effect of polarization is greater in the latter dataset. In 1980, reliability increases approximately
0.15 units, from about 0.65 to approximately 0.8 across the range of polarization values. In the
later dataset, the total increase is about 0.4 units, from about 0.36 to about 0.76. The linear effects
of the other two independent variables are clearly evident in the remaining two panels within each
figure. Note also that the effects for each of these variables are quite similar across the datasets.
The results from these regressions show that the reliabilities of the feeling thermometer variables
definitely are affected by the nature of the stimulus object being evaluated. Divisions of opinion—
captured here by the polarization variable and inversely by the consensus measure— lead to greater
systematic structure in the thermometer scores and higher reliability. Similarly, the attitudinal
salience of the stimulus object, as manifested in the opinionation variable, promotes reliability.
Reasonably enough, sizable segments of the mass public must have non-neutral feelings about an
object in order to provide meaningful thermometer evaluations about the object. In summary, the
thermometer reliabilities vary in fairly obvious ways. There is nothing in the regression results to
suggest that they are particularly suspect representations of affective reactions toward the stimulus
objects.
CONCLUSION
The results from this study should provide strong reassurance to the many social scientists
who use feeling thermometers as empirical variables in their research. The charge that the feeling
thermometers are extremely unreliable is simply wrong. The reliabilities for most of the feeling
thermometer variables examined in this study are very high, both in absolute terms and in comparison to the reliabilities for other well-known and commonly-used variables from the ANES. And
in the relatively few cases where reliabilities are low, the problem clearly lies with the nature of
13
the stimulus object rather than the characteristics of the measurement instrument. When used
to evaluate a salient stimulus for which there are sizable differences of opinion within the public,
the feeling thermometers provide very reliable gauges of sentiment. But, when they are applied
to ambiguous objects toward which fewer crystallized opinions actually exist, it should be no surprise that the resultant thermometer scores contain a higher amount of “noise” variation. This is
precisely how any measurement procedure or tool should work.
While the results presented here are clearcut, they definitely are not the final word on the
measurement characteristics of the feeling thermometers. Other issues, such as the measurement
level of the scores and the potential for differential item functioning, still require attention. But
there would be little reason to worry about these latter measurement properties if the thermometer
scores proved to be extremely unreliable— essentially randomly fluctuating numbers rather than
meaningful reflections of individual attitudes. The evidence presented here indicates that definitely
is not the case.
In fact, the findings about reliability should provide even stronger motivation for pursuing the
other measurement characteristics. The large amount of prior research that has been carried out
using the feeling thermometers demonstrates clearly that the latter are related to a variety of other
interesting phenomena. This makes it all the more important to understand more precisely how the
specific thermometer scores correspond to individual attitudes (i.e., the question of measurement
level) and whether one person’s scores can be meaningfully compared to the scores provided by
another person (i.e., the issue of differential item functioning). Scientific progress in any field
depends heavily on the quality with which its central concepts are measured. And, the current
study provides important evidence on exactly that point. The general conclusion is that, if the
feeling thermometers are “notoriously unreliable” then that notoriety is badly misplaced. The
actual data show that the ANES feeling thermometers definitely comprise a reliable measurement
instrument.
14
REFERENCES
Adida, Claire L.; Lauren D. Davenport; Gwyneth McClendon. (2015) “Ethnic Cueing Across Minorities: A Survey Experiment on Candidate Evaluation in the U.S.” Public Opinion Quarterly. Forthcoming.
Alwin, Duane F. (1992) “Information Transmission in the Survey Interview: Number of Response
Categories and the Reliability of Attitude Measurement.” Sociological Methodology 22: 83118.
Alwin, Duane F. (1997) “Feeling Thermometers Versus 7-Point Scales: Which Are Better?” Sociological Methods and Research 25: 318-340.
Ansolabehere, Stephen; Jonathan Rodden; James M. Snyder, Jr. (2008) “The Strength of Issues:
Using Multiple Measures to Gauge Preference Stability, Ideological Constraint, and Issue
Voting.” American Political Science Review 102: 215-232.
Brady, Henry. (1985) “The Perils of Survey Research: Inter-Personally Incomparible Responses.”
Political Methodology 11: 269-290.
Broockman, David; Joshua Kalla; Peter Aronow. (2015) “Irregularities in LaCour (2014).” Unpublished manuscript. Available online at https://people.stanford.edu/dbroock/sites/
default/files/broockman_kalla_aronow_lg_irregularities.pdf
Conover, Pamela Johnston, and Stanley Feldman. (1981) “The Origins and Meaning of Liberal/Conservative Self-Identifications.” American Journal of Political Science 25: 617-45.
Delavande, Adeline and Charles F. Manski. (2010) “Probabilistic Polling and Voting in the 2008
Presidential Election: Evidence from the American Life Panel.” Public Opinion Quarterly74:
433-459.
Feldman, Stanley. (1988) “Structure and Consistency in Public Opinion: The Role of Core Beliefs
and Values.” American Journal of Political Science 32: 416-40.
Green, Donald Philip. (1988) “On the Dimensionality of Public Sentiment toward Partisan and
Ideological Groups.” American Journal of Political Science32: 758-780.
Heise, David R. (1969) “Separating Reliability and Stability in Test-Retest Correlation.” American Sociological Review 34: 93-101.
Jacoby, William G. (1986) “Levels of Conceptualization and Reliance on the Liberal- Conservative
Continuum.” Journal of Politics 48: 423-432.
Jacoby, William G. (1999) “Levels of Measurement and Political Research: An Optimistic View.”
American Journal of Political Science 43: 271-301.
Jacoby, William G. and David A. Armstrong II. (2014) “Bootstrap Confidence Regions for Multidimensional Scaling Solutions” American Journal of Political Science 58: 264-278.
Kalkan, Kerem Ozan; Geoffrey C. Layman; Eric M. Uslaner. (2009) “‘Bands of Others’ ? Attitudes
Toward Muslims in Contemporary American Society.” Journal of Politics71: 847-862.
Krosnick, Jon A. (1991) “The Stability of Political Preferences: Comparisons of Symbolic and
Nonsymbolic Attitudes.” American Journal of Political Science35: 547-576.
LaCour, Michael J. and Donald P. Green. (2014) “When Contact Changes Minds: An Experiment
On Transmission Of Support For Gay Equality.” Science 346: 1366-1369.
Markus, Gregory B. and Philip E. Converse. (1979) “A Dynamic Simultaneous Equation Model
of Electoral Choice.” American Political Science Review 73: 1055-1070.
Nunnally, Jum C. and Ira H. Bernstein. (1994) Psychometric Theory (Third Edition). New York,
NY: McGraw-Hill.
Page, Benjamin I. and Calvin Jones. (1979) “Reciprocal Effects of Policy Preferences, Party
Loyalties, and the Vote.” American Political Science Review 73: 1071-1089.
Rabinowitz, George B. (1976) “A Procedure for Ordering Object Pairs Consistent with the Multidimensional Unfolding Model.” Psychometrika 41: 349-373.
Rabinowitz, George B. (1978) “On the Nature of Political Issues: Insights from a Spatial Analysis.”
American Journal of Political Science22: 793-817.
Rabinowitz, George and Stuart E. Macdonald. (1989) “A Directional Theory of Issue Voting.”
American Political Science Review 83: 93-121.
Reynolds, H. T. (1974) “Rationality and Attitudes Toward Political Parties and Candidates.”
Journal of Politics 36: 983-1005.
Weisberg, Herbert F. (1980) “A Multidimensional Conceptualization of Party Identification.”
Political Behavior 2: 33-60.
Weisberg, Herbert F. and Jerrold G. Rusk. (1970) “Dimensions of Candidate Evaluation.” American Political Science Review 64: 1167-1185.
Wilcox, Clyde; Lee Sigelman; Elizabeth Cook. (1989) “Some Like It Hot: Individual Differences
in Responses to Group Feeling Thermometers.” Public Opinion Quarterly53: 246-257.
Wiley, David E. and James A. Wiley. (1970) “The Estimation of Measurement Error in Panel
Data.” American Sociological Review 35: 112-117.
Winter, Nicholas and Adam Berinsky. (1999) “What’s Your Temperature? Thermometer Ratings
and Political Analysis.” Paper presented at the 1999 Annual Meetings of the American
Political Science Association.
Zinni, Frank P., Jr.; Franco Mattei; Laurie A. Rhodebeck. (1997) “The Structure of Attitudes
toward Groups: A Comparison of Experts and Novices.” Political Research Quarterly 50:
595-626.
16
Table 1: Wiley-Wiley reliability estimates for feeling thermometers from the 1980 ANES Major
Panel Study.
Panel wave
January
June
September
Jimmy Carter
0.802
0.834
0.838
Edward Kennedy
0.856
0.829
0.822
George McGovern
0.781
0.702
0.702
Walter Mondale
0.709
0.632
0.690
Gerald Ford
0.717
0.697
0.674
George Bush
0.940
0.927
0.940
Jerry Brown
0.800
0.745
0.694
Howard Baker
0.693
0.643
0.645
John Connally
0.823
0.783
0.780
Ronald Reagan
0.806
0.790
0.812
George Wallace
0.779
0.740
0.736
Democratic party
0.769
0.776
0.729
Republican party
0.718
0.719
0.675
Independents
0.660
0.546
0.520
Parties in general
0.550
0.395
0.506
Table 2: Wiley-Wiley reliability estimates for party identification index, ideological self-placement
variable, seven-point issue scales from the 1980 ANES Major Panel Study.
Panel wave
January
June
September
Party identification
0.862
0.869
0.870
Ideology self-placement
0.635
0.654
0.657
Defense spending
0.701
0.724
0.689
Govt. spending vs. services
0.634
0.637
0.602
Aid to Russia
0.594
0.527
0.502
Inflation vs. Unemployment
0.619
0.612
0.498
Table 3: Wiley-Wiley reliability estimates for feeling thermometers from the 2000-2002-2004 ANES
Panel Survey.
Panel wave
2000
2002
2004
Bill Clinton
0.820
0.812
0.808
Hillary Clinton
0.849
0.832
0.827
Al Gore
0.789
0.726
0.756
Ralph Nader
0.770
0.736
0.736
Jesse Jackson
0.773
0.707
0.708
George W. Bush
0.739
0.767
0.836
Dick Cheney
0.798
0.850
0.889
Liberals
0.635
0.571
0.617
Conservatives
0.646
0.651
0.694
Federal Government
0.573
0.557
0.657
Supreme Court
0.466
0.493
0.489
Congress
0.511
0.463
0.563
Military
0.608
0.529
0.550
Whites
0.405
0.411
0.397
Blacks
0.431
0.376
0.338
Note: Table is continued on the next page.
Table 3: Wiley-Wiley reliability estimates for feeling thermometers from the 2000-2002-2004 ANES
Panel Survey (Continued ).
Panel wave
2000
2002
2004
Hispanics
0.418
0.376
0.368
Asian Americans
0.426
0.419
0.413
Catholics
0.463
0.491
0.545
Jews
0.536
0.511
0.512
Protestants
0.492
0.502
0.508
Labor Unions
0.692
0.627
0.678
Big Business
0.530
0.489
0.567
Poor People
0.486
0.473
0.497
People on Welfare
0.568
0.523
0.512
Fundamentalists
0.715
0.700
0.745
The Elderly
0.410
0.470
0.432
Environmentalists
0.629
0.632
0.630
Gays and Lesbians
0.742
0.705
0.740
Feminists
0.561
0.510
0.571
Table 4: Wiley-Wiley reliability estimates for party identification index and questions about increasing or decreasing federal spending from the 2000-2002-2004 ANES Panel Survey.
Panel wave
2000
2002
2004
Party identification
0.898
0.900
0.912
Fed spending, welfare
0.460
0.476
0.397
Fed spending, fighting AIDS
0.593
0.529
0.476
Fed spending, foreign aid
0.487
0.522
0.514
Fed spending, helping the poor
0.526
0.405
0.263
Fed spending, Social Security
0.647
0.597
0.587
Fed spending, protecting environment
0.583
0.558
0.446
Fed spending, public schools
0.542
0.469
0.500
Fed spending, reducing crime
0.342
0.400
0.305
Fed spending, helping children
0.624
0.546
0.503
Fed spending, assistance to Blacks
0.699
0.689
0.597
Table 5: Predicting the reliability of the feeling thermometer variables from the characteristics
of the objects being evaluated with each thermometer. Table entries are OLS regression
coefficients with standard errors in parentheses. Standard errors are clustered by stimulus
objects. All coefficients are statistically different from zero at the 0.05 level, with one-sided
tests.
Data source
1980 ANES Major
Panel Study
2000-2002-2004
ANES Panel Study
-(1/Polarization)
79.578
(28.326)
164.297
(14.224)
Opinionation
0.581
(0.204)
0.530
(0.117)
Consensus
-0.032
(0.016)
-0.016
(0.004)
George H. W. Bush
0.246
(0.024)
—
Intercept
0.511
(0.178)
0.638
(0.099)
R2
0.739
0.834
Se
0.060
0.059
N of observations
45
87
N of clusters
15
29
Figure 1: Wiley-Wiley reliability estimates for feeling thermometers from the 1980 ANES Major
Panel Study.
Reliability, Jan. 1980
●
Reliability, June 1980
Reliability, Sept. 1980
George Bush
●
Edward Kennedy
●
Jimmy Carter
●
John Connally
●
Ronald Reagan
●
Jerry Brown
●
George McGovern
●
George Wallace
●
Democratic Party
●
Republican Party
●
Gerald Ford
●
Walter Mondale
●
Howard Baker
●
Independents
●
Parties in General
●
0.0
0.2
0.4
0.6
0.8
Wiley−Wiley Reliability Estimates
1.0
Figure 2: Wiley-Wiley reliability estimates for party identification index, ideological self-placement
variable, seven-point issue scales from the 1980 ANES Major Panel Study.
Reliability, Jan 1980
●
Reliability, June 1980
Reliability, Sept 1980
Party ID
●
Def spending
●
lib−con ID
●
Gov spending
●
Inflation
●
Russia
●
0.0
0.2
0.4
0.6
0.8
Wiley−Wiley Reliability Estimates
1.0
Figure 3: Wiley-Wiley reliability estimates for feeling thermometers from the 2000-2002-2004 ANES
Panel Survey (Continued ).
Reliability, 2000
●
Reliability, 2002
Reliability, 2004
Dick Cheney
Hillary Clinton
George W. Bush
Bill Clinton
Al Gore
Jesse Jackson
Ralph Nader
Fundamentalists
Gays and Lesbians
Conservatives
Labor Unions
Federal Government
Liberals
Environmentalists
Military
Feminists
People on Welfare
Big Business
Congress
Catholics
Jews
Protestants
Poor People
Supreme Court
The Elderly
Blacks
Asian Americans
Hispanics
Whites
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0
0.2
0.4
0.6
0.8
Wiley−Wiley Reliability Estimates
1.0
Figure 4: Wiley-Wiley reliability estimates for party identification index and questions about increasing or decreasing federal spending from the 2000-2002-2004 ANES Panel Survey.
Reliability, 2000
●
Reliability, 2002
Reliability, 2004
Party ID
●
Fed spending, Blacks
●
Fed spending, Soc Secur
●
Fed spending, children
●
Fed spending, AIDS
●
Fed spending, environ
●
Fed spending, schools
●
Fed spending, help poor
●
Fed spending, for aid
●
Fed spending, welfare
●
Fed spending, crime
●
0.0
0.2
0.4
0.6
0.8
Wiley−Wiley Reliability Estimates
1.0
Figure 5: Effect plots for the independent variables predicting feeling thermometer reliabilities in
the 1980 ANES Major Panel Study.
●
●
●
0.9
Wiley−Wiley reliability for feeling thermometer
●
●
●
●
0.8
●
●
●
●
●
0.7
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
0.6
●●
●
●
0.5
0.4
●
400
600
800
Polarization
●
●
●
0.9
Wiley−Wiley reliability for feeling thermometer
●
●
0.8
●
●
●
●
●
●
0.7
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.6
●
●
●
●
0.5
0.4
●
0.6
0.7
0.8
0.9
Opinionation
●
●
●
0.9
Wiley−Wiley reliability for feeling thermometer
●
● ●
●
●
●
0.8
●
●
● ●●
●
●
●
●
●
● ●
0.7
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
0.6
●
●
●
●
0.5
0.4
●
0
1
2
3
Consensus
4
5
Figure 6: Effect plots for the independent variables predicting feeling thermometer reliabilities in
the 2000-2002-2004 ANES Panel Study.
0.9
●
Wiley−Wiley reliability for feeling thermometer
●
●
●
●●
0.8
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
0.7
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●●
●
●
0.6
●
● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
0.5
●
●
●
●●
●
●
0.4
●
●
●
●
●
●
●
400
600
800
1000
Polarization
0.9
●
Wiley−Wiley reliability for feeling thermometer
●
●
●
●
●
0.8
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
0.7
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
0.6
● ●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
0.5
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
0.4
●
●
●
●
●
●
●
●
●
0.6
0.7
0.8
0.9
Opinionation
0.9
●
Wiley−Wiley reliability for feeling thermometer
●
●
●●
0.8
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
0.7
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.6
●●●
●
●
●
●
●
●
●
●
●
0.5
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
0.4
●
●
●
●
●
●
0
2
●
●
●
●
●
●
●
4
6
Consensus
8
10