Download Using Questionnaire Design to Fight Nonresponse Bias in Web

Using Questionnaire Design
to Fight Nonresponse
Bias in Web Surveys
Social Science Computer Review
Volume 28 Number 2
May 2010 251-267
# 2010 SAGE Publications
10.1177/0894439309340751
http://ssc.sagepub.com
hosted at
http://online.sagepub.com
Paula Vicente
Elizabeth Reis
ISCTE-Lisbon University Institute
The technical potential of the Internet offers survey researchers a wide range of possibilities for
web surveys in terms of questionnaire design; however, the abuse of technical facilities can
detract respondents from cooperating rather than motivating them. Within the web survey
methodology literature, many contributions can be found on how to write a ‘‘good’’
questionnaire. The outcomes are however scattered and researchers and practitioners may
find it difficult to obtain an overall picture. The article reviews the latest empirical research
on how questionnaire characteristics affect response rates. The article is divided into three
main sections: an introduction where the various forms of nonresponse in web surveys are
described; a second section presenting questionnaire features affecting nonresponse—
general structure, length, disclosure of survey progress, visual presentation, interactivity,
and question/response format—and a final section that summarizes the options in terms of
questionnaire design and its implications for nonresponse rate.
Keywords:
rate
web surveys; questionnaires; nonresponse rate; dropout rate; item nonresponse
Introduction
With the growing number of Internet users and the increasing popularity of broader
bandwidth, it is clear that web survey research is becoming more and more prevalent especially in organizational environments, in access panels, or generally in contexts where the
target population can be well identified.
Like mail questionnaires, the web questionnaires are self-administered but the Internet
offers researchers a number of powerful advantages over mail surveys. In web surveys,
there is no printing or mailing of the questionnaires, data already come in an electronic format, and there can be permanent, 24-hr a day data collection. All this permits significant
cost and time savings. Additionally, the Internet allows interaction with the respondent that
can work in favor of better completed questionnaires, without missing or commission
errors.
Authors’ Note: Please address correspondence to Paula Vicente, ISCTE-Lisbon University Institute, Lisbon,
1649-029, Portugal; e-mail: [email protected].
251
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
252 Social Science Computer Review
But along with the positive aspects offered by Internet as a survey research tool, there are
important methodological concerns that cannot be ignored if survey quality is to be
guaranteed. One of those concerns relates to response rates. Web surveys are generally connected with low response rates (Couper, 2000; Lozar Manfreda, Berzelak, Vehovar, Bosnjak, & Haas, 2008; Lozar Manfreda, Bosnjak, Haas, & Vehovar, 2005; Umbach, 2004) and
this can be a cause for bias in survey estimates. Other problems include coverage and sampling error (Couper, 2000). Because not everybody has access or uses the Internet, webbased survey estimates may suffer from coverage bias because they exclude some segments
of the population; moreover Internet penetration is not evenly distributed across segments
of the population: Internet users tend to be younger, with higher educational and income
levels than the general population (European Commission, 2008; Hudson, Seah, Hite, &
Haab, 2004). Sampling error occurs during the process of selecting the sample of
respondents, for example, a nonprobability selection scheme is a cause of sampling error
as not every element of the sampling frame has a nonzero chance of being selected for the
survey.
The designing and implementation of a web survey therefore involves devoting
much attention to procedures aimed at controlling the risk of nonresponse, coverage, and
sampling error.
When developing and implementing a strategy aimed at reducing nonresponse rates in a
survey, several design features can be manipulated. Basically, the surveyor can act in two
ways: (a) proactively, either by offering the individuals some kind of incentive that indeed
seduces them into cooperating or by implementing a contact strategy with the respondents
that convinces them of the importance of cooperating (e.g., conducting multiple contacts) or
(b) alternatively (or additionally) defensively, by manipulating certain aspects of the survey
in order not to put the individuals off cooperating; designing a ‘‘good’’ questionnaire is one
way of doing this.
Differences in questionnaire design have long been acknowledged as affecting response
rates in mail surveys (Carroll, 1994; Dillman, Sinclair, & Clark, 1993; Heberlein and Baumgartner, 1975; Jansen, 1985). Although some of the design principles established for mail
surveys may be translated to web surveys because both methods are self-administered, others require specific treatment in the Internet context because the technical features of the
Internet allow a respondent-survey interaction that is distinct from that of paper questionnaires (Couper, Traugott, & Lamias, 2001).
Within the web survey methodology literature, many contributions provide tips and suggestions on how to write a ‘‘good’’ questionnaire in order not to detract respondents from
cooperating. The outcomes are however scattered and researchers and practitioners may
find it difficult to obtain an overall picture. The objective of this article is to use the latest
empirical research to propose a categorization of questionnaire characteristics which
influence response rates and to summarize a set of questionnaire design principles for web
surveys that help researchers considering conducting a web-based survey to (a) know the set
of possibilities of questionnaire design features, (b) be aware of the impact each design
choice may have on response rates, and (c) choose the ‘‘best’’ design for the questionnaire.
Nonresponse rate is defined in survey methodology as the percentage of people contacted
that did not respond to the survey (Groves et al., 2004). For web surveys, there are a number
of steps in this whole process of contacting and trying to get information from the
252
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
253
respondents and several types of (non)response rates can be distinguished during this process. However, because we are dealing with questionnaire design, we restrict our analysis to
the web survey process from the moment the respondent starts to answer the web questionnaire. We shall concentrate on questionnaire design features that can influence respondents’
participation during the filling out of the questionnaire but have limited impact on the initial
decision to participate in a web survey. Specifically we will refer to
1.
2.
3.
Dropout rate: the percentage of those who prematurely abandoned the questionnaire, that
is represents the percentage of respondents who started the questionnaire but did not end it.
A dropout can occur immediately after the start, in the middle, or near the end of the
questionnaire.
Item nonresponse rate: the unanswered questions (‘‘no opinions,’’ ‘‘don’t knows,’’ or ‘‘no
answer’’) as a percentage of the total number of questions in the questionnaire.
Overall completion rate: the percentage of respondents who completed the questionnaire
of all those who started the questionnaire.
Questionnaire Features Affecting Nonresponse
Web survey methodology offers much advice on how to write a questionnaire in order
not to detract individuals from cooperating. Although a lot of experimental studies are
available about the effects of questionnaire features on nonresponse, there are few categorizations structuring that knowledge.
The work by Couper, Tourangeau, and Kenyon (2004) distinguishes two types of web
questionnaire features that are likely to influence participants’ motivation to start and complete a survey till the end. The first is task elements, which includes factors that influence
the way the respondents deal with the questionnaire (e.g., question wording, response
options, instructions, navigational clues); the second is style elements, comprising a set
of elements related to the look of the questionnaire (e.g., typefaces, background color, logotypes, and other elements). Similarly, Jackob and Zerback (2006) arrange web questionnaires features into two groups: visual appearance and structural appearance. The first
category comprises the stylistic elements of the questionnaire and the second includes the
structural/task features of the questionnaire. However, these classifications are quite generalist, especially in relation to structural features, because most experimental research
reported in the literature allows several factors with distinct effects on response rates to
be distinguished within this group, that are worth separating into specific categories of web
questionnaires features. It is our conviction that a more refined classification can be
achieved.
Ganassali (2008) has proposed a framework to study the effects of questionnaire
characteristics on response quality in web surveys based on the response process of a web
questionnaire. According to the author, this process begins by the respondent giving a quick
look through the questionnaire to apprehend its length and make a preliminary assessment
of the level of effort required to answer it. Second, the respondents will get an impression of
the friendliness of the form according to the balance between text and illustrations. The
third step will be the reading of the questions and the assessment of the wording. Then the
respondent is supposed to apprehend the interactive components of the survey interview,
253
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
254 Social Science Computer Review
and finally, the respondent will look at the response format to know exactly what kind of
task and what kind of data is expected: ticks, numbers, texts, and so on. Based on these
successive stages, the author identified five groups of questionnaire factors with influence
on response quality.
Alongside the process that determines the quality with which the respondent answers the
questions, there is another process that determines whether the respondent gives an answer
or leaves a particular question unanswered, whether the respondent continues till the questionnaire is completed or gives up before finishing it. When at the outset, the respondent
gives a quick look through the questionnaire to apprehend its general layout and its length,
he or she will come to the decision whether to proceed with the questionnaire or to immediately give up. On deciding to advance, the respondent will read the questions and find out
how they are supposed to be answered; the questions and response format may determine
whether he or she leaves a particular question unanswered or even whether he or she drops
out or continues to the next question. As the respondent advances through the questionnaire,
he or she gets a clearer picture of its visual presentation, how to navigate across questions,
and how much still remains till the end of the questionnaire; all these factors can act as
motivators or detractors to continue.
Starting from the two-group categorization of questionnaire features with influence on
response rates already present in the literature and joining the successive stages of the contact process between the respondents and the questionnaire itself, we suggest a six-category
classification for web questionnaires features capable of affecting nonresponse: general
structure, length, disclosure of survey progress, visual presentation, interactivity, and
question/response format. When describing the effect of each feature-type category on
nonresponse, we will distinguish, whenever possible, the effect on dropout rates, item
nonresponse rates, and overall completion rates.
General Structure
One of the features that define the general structure of a questionnaire is how the questions are presented. Basically the options can balance between a scroll design and a screen
design. A scroll design is like the electronic version of a paper questionnaire. The questions
are displayed on a single web page, the respondent can scroll from question to question to
read and select the responses. A button is placed at the end of the page to submit the
responses to the web site of the survey and works much like the act of mailing a paper
questionnaire. At the other extreme, the screen design is very similar to a Computer
Assisted Telephone Interviewing (CATI) survey conducted in the Internet environment.
Questions appear on several screens/pages with a button to submit the responses on each
screen; each screen can contain just one or a number of questions. Responses on each screen
are submitted prior to receiving the next screen. Hybrid surveys fall between these two
extremes and may provide a mixture of screen and scroll, for example, multiple scrolling
sections.
Nonresponse due to the choice of one design or the other affects mainly the item nonresponse rate. The research by Lozar Manfreda, Batagelj, and Vehovar (2002) in which a
comparison between single, long scrolling page, and multiple page design was made taking
a national web survey revealed higher item nonresponse in the scroll design. The two
254
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
255
designs were also compared in terms of dropout rates but the difference between them was not
statistically significant. Because no forced response procedure (an aspect that will be discussed in Section Interactivity) was used in either design, respondents might have skipped
individual survey questions, either by mistake or on purpose, owing to the scrolling between
questions required in the one-page design, thus causing a higher item nonresponse rate.
A comparison of the scroll and screen versions of the same questionnaire was also made
by Tourangeau, Couper, Galesic, and Givens (2004) using a national survey on physicians.
Among other issues, the study evaluated completion rates and item nonresponse. Results
revealed no significant differences between the two designs in terms of completion rates.
However, item omission rates were higher in the scroll version and the most significant
differences were found in items involving skipping patterns.
Peytchev, Couper, McCabe, and Crawford (2006) compared the scroll version with the
screen version of the same questionnaire in a survey of university students. The two designs
were equivalent in terms of completion rates and dropout rates, and in terms of item
nonresponse, the scroll version produced only a slightly higher rate than the screen version.
The authors justified the similarities between the two designs by the fact that the sample
was composed of university students who are typically very experienced in using the
Internet and therefore naturally comfortable with any design.
A second issue in the structure of the questionnaire relates to how it is sent: either
embedded or in an attached format. An embedded questionnaire means the researcher
embeds the questions in the text of the e-mail. Alternatively, the questionnaire can be sent
as an attachment to the e-mail, either as a link or as a document. To respond to an attached
e-mail survey, the respondent must have the hardware and the software that enables him or
her to download, read, and upload the file and he or she must know how to execute the various response steps. While scroll designs can be either embedded or attached, screen
designs require the attached format. The embedded format was more common in the early
days of web surveys when computer hardware and software were not so sophisticated.
Nowadays, the most usual procedure is to attach the questionnaire namely by providing
links to the survey in the e-mail. We have found just one study dealing with this topic; the
research by Dommeyer and Moriarty (2000) compared the embedded version (i.e., the
questions embedded in the body of the e-mail) with the attached version (i.e., the survey
was accessed by means of a link included within the e-mail) of the same questionnaire in
a survey on students’ attitudes toward drinking. The authors found significant differences
in completion rates between the modes with the embedded version producing an 8% higher
response rate than the attached version. No differences were found for item nonresponse.
The authors concluded that the embedded version was more comfortable and secure
because it did not require the downloading and uploading of a foreign file and there was
no threat of virus.
Length
The participation in a survey always implies a burden for the respondent translated in the
time he or she has to spend answering the questions. Therefore, the decision to carry on till
the end of a survey is to a great extent influenced by the effort required of the respondent.
Common sense tells us that there is likely to be a lower response rate in longer
255
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
256 Social Science Computer Review
questionnaires than in shorter ones. The length is not always measured in the same way in
the studies: this can be done by using number of pages, number of questions, or the completion time. Some studies show that questionnaire length has a modest effect on response
rates in mail surveys (Dillman et al., 1993, Haraldsen, Stalnacke, & Blackwell, 1999). Some
meta-analyses of surveys also indicate that the impact of length on response rates is modest
(Yammarino, Skinner, & Childers, 1991; Yu & Cooper, 1983).
Several studies have examined the effect of questionnaire dimension on response rates in
web surveys. The meta-analysis conducted by Cook, Heath, and Thompson (2000) to investigate the determinants of response rates in web surveys found no significant correlation
between questionnaire length and response rates. However, questionnaire length was found
to affect response rates in subsequent studies. In their research on design features
influencing response rates, Lozar Manfreda and Vehovar (2002) questioned web survey
researchers on their experience with web surveys. The respondents selected questionnaire
length, from among the possible reasons, as the problem that most accounted for dropouts in
web surveys. The effect of questionnaire length on web surveys nonresponse has been studied by Deutskens, Ruyter, Wetzels, and Oosterveld (2004) in a multiclient attitude and
usage study where a shorter version of the questionnaire (15 to 30 min to complete) was
compared to the long version (30 to 45 min to complete). The short version yielded a higher
response rate (24.5% vs. 17.1%). The dropout rate was higher in the long version and the
respondents stopped sooner in the long version than in the short version. Item nonresponse
(measured by the percentage of ‘‘don’t know’’ answers) was also higher in the long version.
In a more recent survey among university students on young peoples’ consumption
patterns, Ganassali (2008) evaluated the effect of questionnaire size by comparing a short
questionnaire (20 questions) with a longer one (42 questions). The author proved that
questionnaire length had a significant effect on dropout rates and found higher dropout rates
in the longer version of the questionnaire.
Another issue related to questionnaire length is the announcement a priori of questionnaire length. Hogg and Miller (2003) used two survey experiments—a concept test survey
regarding interactive television and a dining chain customer loyalty survey—to investigate
how revealing questionnaire length in advance affected dropout rates. Respondents were
chosen from a panel of web users. In both surveys, the shorter version of the questionnaire
was assigned to individuals who previously indicated they had no more than 15 min to
complete the survey, while the longer version was assigned for individuals who previously
indicated they had up to 30 min to complete the survey. The authors found higher dropout
rates among those assigned to the longer version of the questionnaire in both surveys.
Crawford, Couper, and Lamias (2001) also conducted an experiment to evaluate whether
the previous announcement of questionnaire length would affect the percentage of people
who begin the survey and whether dropouts would be higher when the survey took longer
than the promised completion time. The author found that respondents who were informed
that the survey would take 8 to 10 min had a lower nonresponse rate than those who were
told it would take 20 min. However, the longer version group had a lower dropout rate once
they started the survey. The two groups were not significantly different in terms of the point
at which the respondents stopped filling in the questionnaire. Galesic (2005, 2006) replicated to some extent this experiment and investigated the effect of announced length on
response rates, specifically the effect of announcing a shorter (10 min) versus a longer
256
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
257
(30 min) completion time and the effect if the announced length did not correspond to the
actual length. A web survey on unemployment in Croatia was used for the experiment. The
author found that 76% of the respondents of the 10-min group started the questionnaire
compared with 64% of the 30-min group. Moreover, the dropout rate was higher in the long
version (50% vs. 28%). When the actual length was much longer than the announced length,
the dropout rates were higher than when the actual length matched the announced length but
dropouts occurred later in the questionnaire.
Disclosure of Survey Progress
Besides questionnaire length, there is the issue of the respondent’s perception of
progress. While it is easy for a paper-questionnaire respondent to run over the pages of the
questionnaire and see how much longer it will take to reach the end, in web surveys this may
not be so easy. In the case of scroll design questionnaires, the respondents can scroll up and
down to find out how many questions there are in much the same way as he or she would do
in a paper questionnaire. In the case of screen design questionnaires, it is not usually
possible to scroll between screens so a progress indicator can be included in the survey
when considered necessary to give respondents some feedback on their progress through
the questionnaire. The progress information can be displayed in a graphical format (e.g.,
as a bar chart or as a pie chart) or as a text message (e.g., x% of survey complete). The
progress indicator can be permanently visible, that is, appears in every screen or it can
appear only at the respondent’s request (e.g., display a show progress link on each
screen where the respondent can click whether he or she wants to know his or her progress).
Alternatively, or additionally, progress information can be displayed on some screens at key
(transition) points of the survey (e.g., display a screen with the message you are about one
third through the survey).
The objective of progress indicators in web surveys is to reduce dropouts and thus
increase the completion rate. Several studies have sought to investigate whether the inclusion of a progress indicator is beneficial to the response rate. Couper et al. (2001) designed
an experiment using a survey with university students where a random half of the sample
was given a questionnaire with a progress indicator while the other half was not. Contrary to
what the authors expected, the difference between groups in terms of completion rates was
not statistically significant; moreover, the group with the progress indicator took longer on
average to complete the survey which the authors explained by the longer download times
of the progress indicator version of the questionnaire. The conclusion was that longer download times associated with the progress indicator dampened the potential positive effect of
including a progress indicator.
In the study by Crawford et al. (2001), the problem of slow downloading was controlled
and the progress indicator was found to have a negative effect when the with-progress indicator and the no-progress indicator groups were compared, namely the completion rate was
lower for those receiving the progress indicator version. A possible explanation for this
rather unexpected outcome came from the fact that the progress indicator reported the progress through the questionnaire inaccurately, that is, the progress indicator (that detected the
number of questions completed out of the total number of questions in the survey) underestimated the amount of time (or effort) that had been completed. This difference was
257
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
258 Social Science Computer Review
caused mainly by the open-ended questions sections of the questionnaire where it is more
difficult than in the closed-ended questions section to predict the completion time.
Conrad, Couper, Tourangeau, and Galesic (2005) studied two additional issues regarding
the effectiveness of progress indicators in reducing dropouts in web surveys: the impact of
progress speed (constant, fast-to-slow, or slow-to-fast) of the progress indicator and the
impact of frequency with which the progress information was displayed (always-on, intermittent, or on-demand). The survey was on lifestyle attitudes and respondents were
recruited from two commercial opt-in panels. The higher dropout rates were found in the
slow-to-fast version (slower progress at the start increasing to fast near the end of the questionnaire) of the progress indicator and most of the dropouts occurred earlier in this version
than in the other two versions. As for the frequency of the progress information display, the
results in the sample that could request progress information on-demand revealed that only
37% of the respondents requested that information and progress information was displayed
on only 1.1% of the screens. The authors also found an interaction effect between speed of
progress and the display of feedback information: there are more dropouts where there is
slow early progress and more frequent feedback (it seems that frequent display is seen
by the respondents as something that delays their progress); the dropout rate is reduced
where there is faster early progress and more frequent feedback (either permanently on
or intermittent). In short, in long questionnaires, the inclusion of a permanently-on progress
indicator may not be very effective in reducing dropouts because it conveys that much of
the questionnaire remains unanswered; and respondents are very sensitive to early
information when presented on every page (whether this information is good news or bad),
therefore intermittent feedback may be a good solution for minimizing costs and
maximizing benefits.
Visual Presentation
The possibility of using the advanced graphics technology supported by current web
browsers is one of the most often cited advantages of web surveys. Graphics can be used
to illustrate survey questions, decreasing the burden on respondents if they are designed for
easy navigation through the questionnaire and/or improving respondents’ motivation and
satisfaction. The latter can be achieved by generating a valuable feeling of having a ‘‘good
time’’ or ‘‘fun’’ while answering the web questionnaire (Lozar Manfreda et al., 2002).
Web survey questionnaires are based largely on visual language. Redline and Dillman
(1999) distinguishes between three different types of visual language: graphic language
(fonts, font sizes, borders, and tables); symbolic language (e.g., arrows that guide the
respondents through the questionnaire); and numerical language (used to number the
questions and response items). Couper (2001) includes font size, font type, color, layout,
symbols, images, animation, and other graphics as components of visual language.
Ganassali (2008) refers that the visual information in a questionnaire can be displayed
on three different levels: questions in which images play a major part (e.g., logotype recognition), images as supplements to the text (e.g., illustrations), and incidental images (e.g.,
the screen background).
Dillman, Tortora, Conradt, and Bowker (1998) studied the effect of using a ‘‘fancy’’
design or a ‘‘plain’’ design in the same questionnaire on the response rates for a web survey.
258
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
259
Both questionnaires used individual screen formats, with scrolling reserved for individual
questions that required more than one screen for viewing. No progress indicator was used in
either format. Both questionnaires had the same wording and question order but they
differed in some design specific features. More specifically, the fancy version used bright
colors (purple and pink) with a constantly changing figure/ground format (vs. black and
white colors in the plain version), html tables for constructing tables (vs. no use of html
tables in the plain version), and placement of answer categories on the extreme right of each
screen (vs. placing answers on the left side of the screen in the plain version). The plain
version proved to be better in terms of completion rate (41% in the plain vs. 36% in the
fancy); additionally, dropouts occurred on a later page in the plain version than in the fancy
version which meant that the plain version was more likely to be fully completed. The
authors explained their findings by the fact that the plain version required less transmission
time and less time to be completed than the fancy version.
The influence of advanced graphics for illustrating survey questions, that is, logotypes,
on nonresponse was evaluated in the experiment by Lozar Manfreda et al. (2002). In a
questionnaire on web site recognition, a random group of respondents was administered
a version of the questionnaire with a page presenting site logotypes and their names; the
other group was administered the version where only the names of the sites were written.
All the other features of the questionnaire were identical. The use of logotypes proved to
have a positive effect on item nonresponse because a lower item nonresponse rate was
found for the group with the logotype version. However, the use of logotypes caused a
higher dropout rate: the percentage of respondents abandoning the survey at the page
containing the question with the logotypes was higher than in the version of the questionnaire without logotypes; this was more marked among respondents using the slowest
Internet connection indicating that the use of advanced graphics can cause some downloading problems for those with the slowest types of Internet connection to the point that they
quit the survey.
Deutskens et al. (2004) also evaluate the impact in a multiclient attitude and usage study
on response rate of visual versus text version of the same questionnaire. The visual version
meant that respondents could see the brand name and the picture or logo in a list of products
while in the text version they only saw the name of the products. The visual presentation of
the questionnaire had a lower response rate than the text version although item nonresponse
was higher in the text version than in the visual version; respondents in the text group
indicated ‘‘don’t know’’ more often than respondents in the visual group. As in the studies
previously cited, the authors justify the lower overall response rate in questionnaires with
visual enhancements by the longer downloading times of visual presentations.
Interactivity
In every self-administered survey, the respondents tend to seek information from the
questionnaire that allows them to dissipate any doubt about the meaning of the questions,
how the answers should be given, or how to navigate through the questionnaire. One of the
advantages of web questionnaires over paper questionnaires is that it can ‘‘interact’’ with
the respondents, thus helping them to know what to do during questionnaire completion.
259
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
260 Social Science Computer Review
Screen designs have several interactive options that can be implemented in web questionnaires: the automatic jumps between questions which allow the omission of inapplicable
questions based on previous responses, the display of missing data messages alerting the
respondent to unanswered questions, the presentation of instructions or explanations based
on answers given by the respondent to previous questions, or the adaptation of question text
according to answers given to previous questions.
Most studies on interactivity focus on its effect on data quality (e.g., Conrad, Couper,
Tourangeau, & Peytchev, 2006; Conrad et al., 2005); little research has been conducted
on the impact of automatic interactive questionnaires on response rates. We have found
three studies on the effect of interactivity on response rates, namely the using of forcedresponse procedures. These procedures consist of building the questionnaire in such a way
that respondents are forced to provide an answer to each question before being allowed to
answer any subsequent ones. In these cases, a ‘‘next’’ button is placed on each screen that
prevents the respondent proceeding if a valid answer has not been given to the questions
presented on the screen. A warning message is displayed to alert the respondents to
unanswered questions.
DeRouvray and Couper (2002) examined the effect of alerting respondents to unanswered questions in a comparison of various strategies aimed at reducing ‘‘no-opinion’’
responses. They found that item nonresponse went down when the alerting message was
displayed and was even lower if the options ‘‘don’t know’’/’’no opinion’’ were not provided. More recently, Stieger, Reips, and Voracek (2007) conducted an experiment with
university students to assess the effect of forced response on dropout rates. The same questionnaire was compared between two design mode groups—the forced-response group, that
is, those who tried to skip a question without answering it and therefore saw an error page
and the nonforced-response group, that is, those who never saw the error page because they
answered all the items without trying to skip any item. The authors found a dropout rate of
64% in the forced-response group as opposed to only 19% in the nonforced-response group;
moreover, when analyzing the moment/screen of the questionnaire where dropouts
occurred, it was clear that the dropout rate was higher for the forced-response group in all
the screens of the survey.
The research by Ganassali (2008) compared the interactive with the noninteractive
version of the same questionnaire on a survey about young people’s consumption patterns
and proved that the interactive version (in which an alerting message was displayed for the
open-ended question in case of no answer) produced a lower item nonresponse rate than the
noninteractive version.
Question/Response Format
Questionnaires can contain closed-ended questions or open-ended questions.
Closed-ended questions require respondents to select the answers that best fit their ideas
or characteristics from a set of response options. Open-ended questions allow respondents
to formulate the answers in their own words rather than selecting them from a list of answer
categories. Closed-ended questions are usually easier to respond and are likely to reduce the
completion time of the questionnaire. Open-ended questions require more thinking and
260
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
261
writing than closed-ended questions and are therefore more burdensome for the respondent
(Groves et al., 2004).
In web questionnaires, open-ended questions require that a text field or a text area be supplied on the screen/page for respondents to insert his or her answer, while a decision has to
be made with closed-ended questions on how to present the answer categories, for example,
radio buttons (all answering options are shown; the respondent selects his or her options by
a mouse click on the corresponding ‘‘button’’) or drop-down boxes (the list of answer
options is hidden and it is only shown after a mouse click; the respondent selects his or her
options by pointing and clicking the mouse on the intended option).
The meta-analysis by Lozar Manfreda and Vehovar (2002) on determinants of nonresponse in web surveys proved question type has a significant effect on dropout rates; higher
dropout rates were likely to be found in surveys that included open-ended and/or difficultto-answer questions. Jackob and Zerback (2006) report the results of a survey on media trust
conducted with students where a first sample of students was presented with an open-ended
question format and a second sample in which the same question was presented as closedended with a list of response options. The results showed an item nonresponse rate of 11%
in the open-ended sample against 16% in the closed-ended sample. Although this evidence
is not striking, it indicates that open-ended questions are not necessarily worse.
Turning to the impact of response format on response rates, Couper et al. (2001) studied
the impact of radio buttons versus entry boxes on response rates in a survey of university
students. In the radio buttons version, a ‘‘don’t know’’ and ‘‘no answer’’ response had to
be explicitly selected, whereas respondents had the option of simply leaving an entry box
blank in the entry box version. The outcomes revealed higher item nonresponse in the entry
box version.
Couper, Tourangeau, Conrad, and Singer (2006) conducted an experiment to explore the
utility of a particular input device in web surveys—visual analog scales—comparing it with
radio button input and numeric entry in a text box on a series of bipolar questions eliciting
views on genetic versus environmental causes of various behaviors. A visual analog scale is
a kind of graphic rating scale on which the respondent directly marks his or her position on a
‘‘continuous’’ line; this line represents the scale.
The input type was found to have a significant effect on the overall completion rate, with
the radio button version showing the higher completion rate. Regarding missing data, none
of the versions had an explicit ‘‘don’t know’’ option and respondents could skip items without answering them; the radio button version had the lowest item nonresponse rate followed
by the text box version and then by the visual scale analog version. The item nonresponse
rate in the latter version was more than double that of the other two versions. Although this
experiment was designed to evaluate the effect of the visual scale analog, it also allowed
comparisons to be made of the other two formats, leading to the conclusion that the performance of the radio buttons version was better in terms of response rates.
The impact of response format was also evaluated in the experiment by Healey (2007).
Using a survey on the general population of New Zealand, the author compared two
response formats, namely radio buttons versus drop-downs boxes, by randomly assigning
one half of the sample to one treatment and the second half of the sample to the other. Three
aspects of response rates were evaluated: completion rates, item nonresponse at survey
level, and item nonresponse at question level. Although no significant difference was found
261
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
262 Social Science Computer Review
between the two treatments regarding completion rates and item nonresponse at survey
level, at question level there was evidence that drop-down boxes led to slightly higher item
nonresponse than radio buttons.
Table 1 outlines the major findings from the literature revision. It presents the design features that can be considered when designing the questionnaire, where the major impact of
each design feature is likely to be felt—dropout rate, item nonresponse rate, or overall completion rate—and the literature sources cited in this review on each design feature.
Web Questionnaires: Which Design Is Best?
The purpose of this article was to classify web questionnaire features and discuss the
impact of questionnaire design on nonresponse by gathering the most relevant knowledge
available about web survey methodology and web survey nonresponse. The results of the
empirical research are not always consensual about the effect of a particular design practice
on response rates. Several factors can account for this. First, the target population is not the
same across experiments and the level of expertise of the target population dealing with the
Internet influences the contact process between the respondent and the questionnaire and
thus the outcomes in terms of response rates. Second, the lack of a standard definition for
some concepts makes it difficult to compare the various experiments, for example, there is
no consensus on how the length of a questionnaire should be measured and therefore it is
hard to establish the length of the questionnaire which detracts respondents from cooperating. Finally, there are factors that are difficult for the researcher to control but that are capable of influencing the contact process between respondent and questionnaires, for example,
the speed of the Internet connection of the respondents’ computer.
This review of the literature on web questionnaire design and survey nonresponse presents a picture of what is known about the effects of questionnaire design on web surveys
response rates. In short:
Screen designs tend to yield a lower item nonresponse rate than scroll designs. Moreover,
screen designs allow for a better control of the response process, for example, in the case of
dropouts, the question where the respondent abandoned the questionnaire can be located
thus enabling the analysis of causes behind this.
Surveys with lengthy questionnaires tend to have lower overall completion rates mainly
because the increased burden of long questionnaires tends to increase dropout rates.
The respondents’ perception of the burden is more important than the burden itself. The
inclusion of a progress indicator can give the respondent some feedback as to the point
of the questionnaire where he or she stands. If the respondent does not see his or her progress moving significantly forward, the likelihood of dropout increases. This is more likely
to happen in longer questionnaires. Therefore, an intermittent progress indicator instead of
one that is permanently on can work better to keep respondents’ motivation higher and thus
decrease dropouts.
Illustration enhancements of the questionnaires can make the questionnaire more attractive
and pleasant thus contributing to higher completion rates. But visual enhancements can
also cause premature abandon of the survey if the questionnaire becomes difficult to navigate or download due to software or hardware compatibility problems. Although these
262
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
263
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
f. Include versus not include forced
response procedures
g. Closed-ended versus open-ended
questions
h. Radio buttons versus other response
format
d. Include versus not include progress
information
e. ‘‘Fancy’’ versus ‘‘plain’’ design
b. Embedded versus attached
questionnaire
c. Long versus short questionnaires
a. Scroll versus screens designs
Design Feature
Overall
Item
Nonresponse Completion
Rate
Rate
Dropout
Rate
Couper et al. (2001); Couper, Tourangeau, and Peytchev (2006); Healey
(2007)
Lozar Manfreda and Vehovar (2002); Deutskens, Ruyter, Wetzels, and
Oosterveld (2004); Ganassali (2008)
Couper, Traugott, and Lamias (2001); Crawford, Couper, and Lamias
(2001); Conrad, Couper, Tourangeau, and Galesic (2005)
Dillman, Tortora, Conradt, and Bowker (1998); Lozar Manfreda et al.
(2002); Deutskens et al. (2004)
DeRouvray and Couper (2002); Stieger, Reips, and Voracek (2007);
Ganassali (2008)
Lozar Manfreda and Vehovar (2002); Jackob and Zerback (2006)
Lozar Manfreda, Batagelj, and Vehovar (2002); Tourangeau, Couper,
Galesic, and Givens (2004); Peytchev, Couper, McCabe, and Crawford
(2006)
Dommeyer and Moriarty (2000)
Literature Sources
Table 1
Design Features for Web Survey Questionnaires, (Non)Response Rates, and Literature Sources
264 Social Science Computer Review
technical problems can be expected to decline in the future as computer and Internet connection speed increases, for now the empirical evidence on the use of graphics in web questionnaires points to a moderate use of graphically complex designs in order not to detract
respondents from completing the questionnaire.
The radio button format instead of drop-down boxes or text entry for closed-ended questions seems to work in favor of lower item nonresponse.
When designing a web questionnaire, decisions must be taken on the six features described
here—general structure, length, disclosure of survey progress, visual presentation, interactivity, and question/response format. Ideally, these features are combined to simultaneously
guarantee low dropouts, low item nonresponse, and high overall completion rate. The literature review clearly demonstrates that it is not easy to build a set of methodological rules
that grants good results in all types of (non)response rates—dropouts, item nonresponse,
and overall completion rate—in all web surveys. To fulfill the objective on one aspect, it
is not uncommon for other aspects to be put at risk. In such cases, researchers must try
to find the ‘‘best’’ mix of design features to counterbalance the different aspects.
For example, the combination ‘‘screen design þ forced response procedure þ radio
buttons’’ may have a favorable effect on reducing item nonresponse because according
to the literature all these features affect item response. Respondents are less likely to unintentionally skip a question in screen designs than in scroll designs because the respondent
sees few questions in each screen in screen designs; in contrast, the whole questionnaire can
be seen by using the scroll bar in scroll designs. As a result, item nonresponse rate is likely
to be reduced in screen designs. However, if a scroll design is necessary for some reason
(e.g., cost or software constraints), it is prudent to implement a procedure to alert for unanswered questions to control the risk of high item nonresponse. The inclusion of a procedure
to force response to all questions may also be pertinent in surveys where a complete data set
is crucial (e.g., when required by the data analysis procedures that are to be used) to
guarantee a zero item nonresponse rate, whether a screen or a scroll design is chosen.
A combination of the type ‘‘moderate length þ progress information þ moderately fancy
design’’ may act favorably to control for dropouts because the literature review presents the
length, the disclosure of survey progress, and visual presentation as factors with a
significant impact on dropout rate. A ‘‘moderately long’’ questionnaire is more likely to
be completed till the end than a longer one. However, if a long questionnaire is unavoidable
(e.g., due to the objectives of the survey), something must be done to counterbalance the
negative effect that size is expected to have on the completion rate—the insertion of a
‘‘motivating’’ progress indicator, namely one that shows real progress every time the
respondent looks at it, is likely to help keep the respondent till the end of the questionnaire;
more recent research suggests that an intermittent display of progress information seems to
be the best option.
The literature review does not provide a direct answer for the best combination of
‘‘general structure, length, disclosure of survey progress, visual presentation, interactivity,
and question/response format’’ to enhance response rates. On one hand, the effect of some
design features on response rates is not yet conclusive. On the other, further research is
required for a full understanding of the effect of some specific combinations of design
features on response rates as most experimental research designs on questionnaire features
264
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
265
focus on the effect of one of the key factors—general structure, length, disclosure of survey
progress, visual presentation, interactivity, or question/response format—and not on the
entire sequence of the six factors. Nevertheless, taking into consideration the specific
context in which web surveys are made—an electronic environment, with some degree
of technical complexity, and where there is no interviewer to help the respondent to fill
in the questionnaire—it can be emphasized that the web surveyor should note that it is more
important for a questionnaire to be easy and motivating for respondents to understand and
complete than to have one that it is easy to administer.
Web surveys research is a rapidly changing field so future updates of best practices to
lower nonresponse will undoubtedly accommodate ongoing developments. Moreover, the
shortcomings found in the literature and referred above certainly make web surveys
research a rich area for future investigation.
References
Carroll, S. (1994). Questionnaire design affects response rate. Marketing News, 28, 14.
Conrad, F., Couper, M., Tourangeau, T., & Galesic, M. (2005). Interactive feedback can improve quality of
responses in web surveys. Paper presented at the 60th American Association for Public Opinion Research
Annual Conference, Phoenix, AZ.
Conrad, F., Couper, M., Tourangeau, T., & Peytchev, A. (2006). Use and non-use of clarification features in web
surveys. Journal of Official Statistics, 22, 245-269.
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or Internet-based
surveys. Educational and Psychological Measurement, 60, 821-836.
Couper, M. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494.
Couper, M. (2001). Web surveys: The questionnaire design challenge. Proceedings of the 53rd Session of the
International Statistical Institute, Seoul, Korea.
Couper, M., Tourangeau, R., Conrad, F., & Singer, E. (2006). Evaluating the effectiveness of visual analog
scales: A web experiment. Social Science Computer Review, 24, 227-245.
Couper, M., Tourangeau, R., & Kenyon, K. (2004). Picture this! Exploring visual effects in web surveys. Public
Opinion Quarterly, 68, 255-266.
Couper, M., Traugott, M., & Lamias, M. (2001). Web survey design and administration. Public Opinion
Quarterly, 65, 230-253.
Crawford, S., Couper, M., & Lamias, M. (2001). Web surveys: Perceptions of burden. Social Science Computer
Review, 19, 146-162.
DeRouvray, C., & Couper, M. (2002). Designing a strategy for reducing ‘‘no opinion’’ responses in web-based
surveys. Social Science Computer Review, 20, 3-9.
Deutskens, E., Ruyter, K., Wetzels, M., & Oosterveld, P. (2004). Response rate and response quality of Internetbased surveys: An experimental study. Marketing Letters, 15, 21-36.
Dillman, D., Sinclair, M., & Clark, J. (1993). Effects of questionnaire length, respondent-friendly design, and a
difficult question on response rates for occupant-addressed census mail surveys. Public Opinion Quarterly,
57, 289-304.
Dillman, D., Tortora, R., Conradt, J., & Bowker, D. (1998). Influence of the plain vs fancy design on response
rates for web surveys. Proceedings of the Survey Research Methods Section, American Statistical
Association.
Dommeyer, C., & Moriarty, E. (2000). Comparing two forms of an e-mail survey: Embedded vs. attached.
Journal of the Market Research Society, 42, 39-53.
European Commission. (2008). Eurobarometer 293, European Commission, Brussels.
Galesic, M. (2005). Effects of questionnaire length on quality of responses in web surveys. Paper presented at
the ESF Workshop, Dubrovnik, Croatia.
265
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
266 Social Science Computer Review
Galesic, M. (2006). Dropouts on the web: Effects of interest and burden experienced during an online survey.
Journal of Official Statistics, 22, 313-328.
Ganassali, S. (2008). The influence of the design of web survey questionnaires on the quality of responses.
Survey Research Methods, 2, 21-32.
Groves, R., Fowler, F., Jr., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2004). Survey methodology. New York: John Wiley and Sons.
Haraldsen, G., Stalnacke, M., & Blackwell, K. (1999). Empirical contributions to a general survey response
model. Paper presented at the International Conference on Survey Nonresponse, Portland, OR.
Healey, B. (2007). Drop down and scroll mice: The effect of response option format and input mechanism
employed on data quality in web surveys. Social Science Computer Review, 25, 111-128.
Heberlein, T., & Baumgartner, R. (1975). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43, 447-462.
Hogg, A., & Miller, J. (2003). Watch out for dropouts. Quirk’s Marketing Research Review, 1137. Retrieved from
http://www.websm.org/uploadi/editor/Watch%20out%20for%20dropouts.doc.
Retrieved date 19th March 2010.
Hudson, D., Seah, L. -H., Hite, D., & Haab, T. (2004). Telephone presurveys, self-selection and non-response
bias to mail and Internet surveys in economic research. Applied Economic Letters, 11, 237-240.
Jackob, N., & Zerback, T. (2006). Improving quality by lowering non-response: A guideline for online surveys.
Paper presented at the WAPOR Seminar ‘‘Quality Criteria in Survey Research VI,’’ Cadenabbia, Italy.
Jansen, J. (1985). Effect of questionnaire layout and size and issue-involvement on response rates in mail
surveys. Perceptual and Motor Skills, 61, 139-142.
Lozar Manfreda, K., Batagelj, Z., & Vehovar, V. (2002). Design of web survey questionnaires: Three basic
experiments. Journal of Computer-Mediated Communication, 7, 3. Retrieved from http://www.
websm.org/uploadi/editor/Lozar_2002_Design.doc. Retrieved date 19th March 2010.
Lozar Manfreda, K., Berzelak, J., Vehovar, V., Bosnjak, M., & Haas, I. (2008). Web surveys versus other survey
modes: A meta-analysis comparing response rates. International Journal of Market Research, 50, 79-104.
Lozar Manfreda, K., Bosnjak, M., Haas, I., & Vehovar, V. (2005). Web survey response rates compared to other
modes—A meta analysis. Paper presented at the 60th American Association for Public Opinion Research
Annual Conference, Miami Beach, FL.
Lozar Manfreda, K., & Vehovar, V. (2002). Survey design features influencing response rates in web surveys.
Paper presented at the International Conference on Improving Surveys, Copenhagen, Denmark.
Peytchev, A., Couper, M., McCabe, S., & Crawford, S. (2006). Web survey design: Paging versus scrolling.
Public Opinion Quarterly, 70, 596-607.
Redline, C., & Dillman, D. (1999). The influence of auxiliary, symbolic, numeric and verbal language on
navigational compliance in self-administered questionnaires. U.S. Census Bureau. Retrieved from
http://survey.sesrc.wsu.edu/dillman/papers/Auxiliary,Symbolic,Numeric%
20paper--with%20Cleo.pdf. Retrieved date 19th March 2010.
Stieger, S., Reips, U.-D., & Voracek, M. (2007). Forced-response in online surveys: Bias from reactance and an
increase in sex-specific dropout. Journal of the American Society for Information Science and Technology,
58, 1653-1660.
Tourangeau, R., Couper, M., Galesic, M., & Givens, J. (2004). A comparison of two web-based surveys:
Static versus dynamic versions of the NAMCS questionnaire. RC33 6th International Conference on
Social Science Methodology: Recent Developments and Applications in Social Research Methodology,
Amsterdam, Holland.
Umbach, P. (2004). Web surveys: Best practices. New Directions for Institutional Research, 121, 23-38.
Yammarino, F., Skinner, S., & Childers, T. (1991). Understanding mail survey response behavior: A meta-analysis.
Public Opinion Quarterly, 55, 613-639.
Yu, J., & Cooper, H. (1983). A Quantitative review of research design effects on response rates to questionnaires. Journal of Marketing Research, 20, 36-44.
266
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016
Using Questionnaire Design to Fight Nonresponse Bias / Vicente, Reis
267
Paula Vicente is an assistant professor in the quantitative methods department, ISCTE-Lisbon University Institute, Portugal. Her research interests are focused on survey methodology, Internet surveys, and mixed-mode
effects. She may be reached at [email protected].
Elizabeth Reis is a full professor in the quantitative methods department, ISCTE-Lisbon University Institute,
Portugal. She is currently vice president of GIESTA, a multidisciplinary centre at ISCTE that combines
methodological and substantive research in survey methodology and statistics. She may be reached at
[email protected].
267
Downloaded from ssc.sagepub.com at Aalto University on March 29, 2016