Download European Social Survey

Methodological Rigour
in Survey Research
A case study of the
European Social Survey
Michael Breen
Issues in Survey Research
•Questionnaire design
•Sampling strategy
•Fieldwork oversight
•Response rates
•Cross national research issues
European Social Survey (ESS)
• The European Social Survey was established in 2001 as an
academically-driven social survey designed to chart and explain
the interaction between Europe's changing institutions and the
attitudes, beliefs and behaviour patterns of its diverse
populations.
• Currently in the midst of its sixth round, this biennial crosssectional survey covers more than thirty nations and employs the
most rigorous methodologies.
• All data, questionnaires, an interactive analysis tool (NESSTAR)
and an educational resources for use in HEIs are all available
online at http://www.europeansocialsurvey.org/.
Questionnaire design
• The core questionnaire items cover both socio-demographic and
substantive themes. The content of the core remains largely
constant at each round. It includes both independent and
dependent variables; the latter designed to measure shifts over
time in what are considered to be key components of Europe’s
social fabric.
• These core questions have been designed in collaboration with a
group of experts in different fields. This is overseen by the
Scientific Advisory Board of the ESS.
• Translations are required for each language used as a first
language by 5% or more of the population.
Sampling strategy
• Persons aged 15 years and over who are resident in private
households in the Republic of Ireland
• Must be a randomly derived sample
• Must conform to the ESS sampling specification
• Strategy must be agreed with the field agency and signed off by
the Core Scientific Team of ESS before fieldwork begins.
Margin of Error
(95% Confidence Interval)
Dist./Size 50
5 95 6.04
10 90 8.32
15 85 9.90
20 80 11.09
25 75 12.00
30 70 12.70
35 65 13.22
40 60 13.58
45 55 13.79
50 50 13.86
100
4.27
5.88
7.00
7.84
8.49
8.98
9.35
9.60
9.75
9.80
200
3.02
4.16
4.95
5.54
6.00
6.35
6.61
6.79
6.89
6.93
300
2.47
3.39
4.04
4.53
4.90
5.19
5.40
5.54
5.63
5.66
400
2.14
2.94
3.50
3.92
4.24
4.49
4.67
4.80
4.88
4.90
600
1.74
2.40
2.86
3.20
3.46
3.67
3.82
3.92
3.98
4.00
800
1.51
2.08
2.47
2.77
3.00
3.18
3.31
3.39
3.45
3.46
1000
1.35
1.86
2.21
2.48
2.68
2.84
2.96
3.04
3.08
3.10
1200
1.23
1.70
2.02
2.26
2.45
2.59
2.70
2.77
2.81
2.83
1600
1.07
1.47
1.75
1.96
2.12
2.25
2.34
2.40
2.44
2.45
2000
0.96
1.31
1.56
1.75
1.90
2.01
2.09
2.15
2.18
2.19
3000
0.78
1.07
1.28
1.43
1.55
1.64
1.71
1.75
1.78
1.79
4000
0.68
0.93
1.11
1.24
1.34
1.42
1.48
1.52
1.54
1.55
Sample Design in the ESS (3 stage)
• Stage 1: Selection of 225 PSUs (i.e. clusters of addresses) from a
national total of 3,123 clusters formed from the GeoDirectory.
Clusters are sorted geographically within eight strata (high/low
elderly population; urban/rural; high/low education). Clusters
are selected using systematic sampling with a random start. The
probability of selection for the clusters is proportional to the
number of addresses in the cluster. The minimum cluster size is
500 residential addresses.
• Stage 2: Systematic sample of 20 addresses in each cluster.
Within each cluster, addresses are sorted geographically.
• Stage 3: Within dwellings, the next birthday method will be used
for equal-probability selection of one person who is 15 years old
or over, with no upper age limit within each household. No
substitution of respondents will be permitted
Effective sample size
• Most statistical tests assume data is obtained by simple random
sampling.
• If random sampling is not used, the traditional statistical tests
become incorrect to an unknown degree.
• The sample needs to be corrected to include design effects
• There is a difference between the effective sample size and the
actual sample size.
Design effects
• Design effect due to clustering:
• DEFFc = 1.31= 1+ (10.68-1)* 0.0321.
• Design effect due to different selection probabilities within
households (from population statistics, updated to Q4 2009):
DEFFp = 1.212
• Total design effects
• DEFF = 1.59= 1.31 * 1.21.
Sample Size for ESS Round 5
• Target response rate is 60% for Ireland from previous experience
• The gross sample size must be 4500. With 11% of ineligibles and
a response rate of 60% the net sample size must be 2,403.
• N_net = (4500 * 0.89 * 0.6) = 2,400
• For each cluster 20 addresses have to be drawn. That results in
an average of 10.68 interviews per cluster. With an overall DEFF
of 1.59, that results in an effective sample size of 1515.
• N_eff = (2405 * 1.59) = 1515
Fieldwork Agency & Oversight
•
•
•
•
•
•
•
•
•
•
Contract based on tender document
Tender handled at European level
90+ Expressions of Interest
4 tenders received
Agency liaison (via postdoctoral fellows)
Clearly defined deliverables
Weekly reporting
Quality control back checks
Coding issues/data format
Billing control
Deliverables: Data files
• 1. Main and Supplementary questionnaire, with additional ESS
variables
• 2. Interviewer questionnaire
• 3. Contact form
• 4. Parents' occupation
• 5. Raw data
• 6. Sample design data file (SDDF)
Deliverables: Documents
1.
National Technical Summary
2.
Population statistics
3.
Main questionnaire
4.
Supplementary questionnaire
5.
Interviewer questionnaire
6.
Contact form
7.
Show cards
8.
Interviewer and fieldwork instructions
9.
Advance letters, brochures and other written information to the
respondents
10. CAPI programmes, if applicable
Response rates enhancement
• Four personal visits by interviewers to each sampling unit on different
days of the week and times of day, of which at least 1 must be at the
weekend and 1 in the evening. These visits should be spread over at
least two different weeks
• In order to allow difficult-to-contact people to be located, the
fieldwork period can not be less than 30 days
• Use of incentives
• Provision of telephone numbers for potential respondents to contact
• Extra training of interviewers in response-maximisation techniques and
doorstep interactions,
• Implementing refusal avoidance and conversion techniques
• Re-issuing of refusals and non-contacts
• Refusal conversion, i.e. obtaining the cooperation of initially reluctant
sample persons
Response rate calculation
• The target for the ‘ESS’ response rate is 70%.
• The Irish rate was set at 60% from previous experience
• It is calculated as
Response rate =
number of achieved interviews
number of individuals – ineligibles
• Achieved response rates in ESS Ireland Round 5 was 65.4%, the
highest rate ever achieved in the ESS in Ireland.
• (Highest in Round 5 was Bulgaria at 81%, lowest Lithuania at 40%
Cross national research issues
• For validity, multinational surveys have to be based pm
probability sample designs of comparable national populations.
• All measurements should be well controlled for comparability.
• Translations must be accurate and comparable
• Significance of “media events” must be noted
• Multilevel modelling is required using hierarchical linear
regression
• Structural equation modeling is also used to build causal
pathways on a country by country basis.
• While the software is becoming more plentiful and more
powerful (AMOS, MLWiN, R, Rasch) it requires clearly defined
theoretical foundations before analysis.