Download Design Effects

Design Effects
Ethan Noel
WEC
Survey Process
 Operational Design
 Two primary issues affect the
choice of the method of data
collection:
 What is the most appropriate
method for a particular research
question?
 What is the impact of a particular
method of data collection on
survey errors and cost?
Define research
objectives
Choose mode of
collection
Choose sampling
frame
Construct and
pretest
questionnaire
Design and
select sample
Design and
implement data
collection
Code and edit
data
Make postsurvey
adjustments
Perform analysis
History and Future of Survey Research
 Groves (2011) – Three major eras
 1930-1960: The Era of Invention
 1960-1990: The Era of Innovation
 1990 – present: “Designed Data” Supplemented by “Organic Data”
 Smith (2013) – New techniques for data collection and how they compare to
survey research
 Data mining
 Internet
 Social Media
 Administrative Data
 Issues?
 Solutions?
Data Collection
 Term can be slightly misleading
 Survey data is usually produced/created at the time of the interview or the
completion of a questionnaire.
 Historically there were 3 basic data collection methods
 Mailing paper questionnaires
 Telephone interviews
 Face-to-face interviews
 Computers altered traditional methods and added new ones
 The combination of modes allows researchers to minimize cost and error
The Evolution of Survey Technology
Paper
Computer
OCR/ICR
FAX
Disk by
mail
E-mail
Web
CATI
TDE
IVR/TACASI
AudioCASI
VideoCASI
Mail
Telephone
CAPI
Face-toface
SAQ
Experimental Integration
 Sniderman and Grob (1996) –Changing emphasis for experimental design
 Split-ballot -> vignette -> CATI
 Non-directive designs – randomized assignment of respondents to question form
without an intent to sway, influence, or control the direction of their response
 Directive designs – experimentation interventions are active and deliberate
 Postdecisional
 Predecisional
 Issue framing
 Context of choice
 Characteristics of chooser
 Mode
Mode Selection Considerations
 Interviewer involvement
 Face-to-face vs. mail survey
 SAQ vs ACASI
 Interaction between interviewer and respondent
 More data with increased interaction -> more control over measurement process
 Privacy
 Different levels of privacy can influence answers
 Interviewer presence
 Modes to improve reporting?
 Channels of communication
 Technology use
Choosing a Mode
 No one ideal mode for all survey applications
 Alternative approaches
 Face-to-face and telephone surveys
 Mail surveys and web surveys
 Proximate modes as reasonable alternatives
 Considerations:
 Sources of error
 Logic/Feasibility
 Coverage and sampling frame
 Appropriateness of topic to the method
 Cost constraints
 Value of timely results
Mode Application
 Bassili (1993) – Comparison of two measures of attitude strength:
accessability and certainty, for the purpose of predicting discrepancies
between voting intentions and voting behavior.
 Accessibility – response latency to a question of intent
 Certainty – question about the finality of the respondent’s intention
 CATI – Computer-assisted telephone interviewing
 27% initial response rate -> 19% final response rate
 What were the benefits to using a CATI mode? Disadvantages?
 What other modes might have sufficed?
Mode Effects
 Aquilino (1994) – The effect of research mode on respondents’ willingness to
reveal illicit or undesirable behavior.
 3 randomly assigned modes:
 SAQ – more likely to admit illicit drug use
 Face-to-face – slightly less likely than with SAQ
 Telephone – Least likelihood of admission
 Privacy concerns are much greater when dealing with sensitive
information.
 Social distance
 Response anonymity increases willingness to reveal sensitive behavior.
Designing a Questionnaire
 Dillman – Social Exchange Theory - questionnaire recipients are most likely
to respond if they expect that the perceived benefits of doing so will
outweigh the perceived costs of responding.
 Respondent burden
 Channels of communication
 Mail questionnaires
 Telephone questionnaires
Considerations
 Sampling
 Available sampling frames
 Mail, web, telephone surveys
 Method choice often has indirect sample design implications
 Cost/efficiency
 Coverage
 Respondent access to phones/Internet
 Researcher access to mailing list
 Considerations for mode choice: speed, importance of precision, existing
inferences (ex: telephone ownership correlated with voting participation)
Considerations
 Response Rates
 Face-to-face > telephone > mail
 No evidence that technology affects response rates in interviewer-administered
surveys
 Self-administered surveys see higher response rates with paper-based methods
than electronic equivalents
 Measurement Quality
 Completeness of the data, social desirability bias, and response effects (wording,
ordering)
 Open-ended questions vs closed questions
 Sources of Error