Download probability sampling - Fraser Health Research and Evaluation

Designing Surveys for Research and
Evaluation
© Fraser Health Authority, 2007
The Fraser Health Authority (“FH”) authorizes the use, reproduction and/or
modification of this publication for purposes other than commercial redistribution. In
consideration for this authorization, the user agrees that any unmodified
reproduction of this publication shall retain all copyright and proprietary notices. If
the user modifies the content of this publication, all FH copyright notices shall be
removed, however FH shall be acknowledged as the author of the source
publication.
Reproduction or storage of this publication in any form by any means for the purpose
of commercial redistribution is strictly prohibited.
This publication is intended to provide general information only, and should not be
relied on as providing specific healthcare, legal or other professional advice. The
Fraser Health Authority, and every person involved in the creation of this publication,
disclaims any warranty, express or implied, as to its accuracy, completeness or
currency, and disclaims all liability in respect of any actions, including the results of
any actions, taken or not taken in reliance on the information contained herein.
1
Your fortune…
Workshop Outline
What is a survey?

Survey



A method or process of gathering information from a sample of
individuals.
The physical format or tool employed to gather information:
Opinion poll, questionnaire, customer satisfaction, voting, census
etc.
Why are surveys used?

1. To answer a question.
• Program Evaluation- Did we deliver the services that we planned to deliver?

2. To gain information on a population or topic.
• Patient Satisfaction.

3. To gather objective data to make informed decisions.
• Priority setting.

4. To provide a benchmark.
• "snapshot" in time, establish a baseline to measure change.
When to use a Survey?
 When
questions and information needs
are best answered by the people
themselves.
 When information is not available from
existing sources.
So You Would Like to Design a
Survey…..
 Will
need to complete
some basic survey
planning steps.
Identifying Information Needs

1) State the problem.
 2) Identify parameters related to problem.

Who, where, when, what?

3) Identify concepts to be measured.
 4) How much detail for each item.
 5) How will the results be presented.

Statistics Canada (2005)
Survey Sampling and Sample
Size
 Frame
• The physical means that you will use to access the
information for the population.

Phone book, map, telephone numbers, address,
department, hospital ward, employee listing etc.

Example: Program registration forms.
Survey Population

The survey population is the population that is
covered by the frame.

Example, all 10,000 educational program
recipients.
Now that you have determined who will
comprise your survey population, it is time to
think about sampling.
Probability Sampling Methods

A probability sampling method is any method
of sampling that utilizes some form of random
selection.
 Must set up some process or procedure that
assures that the different units in the population
have equal probabilities of being chosen.
Sampling: Things to Keep in Mind

Probabilistic or random sampling methods
generally preferred over non-probabilistic ones,
because they are more accurate and rigorous.

If you want to make accurate generalizations to
the population regarding your sample, you will
need to use probability sampling methods.
Sampling: Things to Keep in
Mind

If you select your sample by convenience, the
conclusions drawn from the survey results apply
only to the SURVEY SAMPLE.
E.g. Non-probability sampling methods.
Non-probability methods are acceptable
providing the method reflects the purpose of why
you are surveying, and will give you the
information you are looking for.
Probability Sampling Methods:
Random

There are several methods to choose from:
 Simple random
sampling.
Probability Sampling Methods:
Stratified

Stratified sampling (divide the
population into non-overlapping
strata and sample from within each stratum
independently).
 Guarantees representation of all important
groups.
Probability Sampling Methods:
Systematic

Selection of the sample using an interval “k” so
that every “k” unit in the frame is selected, is
called systematic
random sampling.
Probability Sampling Methods:
Systematic

Steps to achieve a systematic random sample:
1. Number the units in the population from 1 to N.
2. Decide on the n (sample size) that you want or need.

k = N/n = the interval size.
3. Randomly select an integer between 1 and k.
4. Then take every kth unit.
Example:
1. N=200
2. n=40, take N/n, 200/40=5 (interval size).
3. Randomly select a number between 1 and 5 (let’s pick 4).
4. Begin with 4, and take every 5th unit.
Probability Sampling Methods:
Cluster
 Cluster
sampling.
 Divide population into clusters and
randomly sample clusters.
 Measure all units within sampled clusters.
 Example: See blue areas on map.
 Not just geographic areas, could
select hospitals, schools
etc.
Probability Sampling
 In
social research, most people use a
combination of these sampling methods.
 Called multi-stage sampling.
 Tailor your sampling method to the
purpose of your survey.
Non-Probability Sampling Methods
 There
are different types of non-probability
sampling methods as well:




Convenience (not representative of population).
Purposive (certain group in mind).
Expert sampling (seek out specific expertise).
Snowball sampling (ask people to participate, they
ask more people).
If you select non-probability sampling
methods, the conclusions drawn from the
survey results apply only to the SURVEY
population.
Sample Size: How many?

Considerations:
 Level of precision desired (when generalizing
to population, the larger the sample, the
greater the precision).
 Size of population.
 Budget.
 Time.
 Sampling method.
 Response rate.
 Type of survey.
Good
general guideline: Sample size table.
Remember…
 If
you sample the entire population, you
do not need to worry about statistical
tests, or generalizing your findings.
 Example: Census
 All cases are represented, can speak
with 100% confidence about entire
group.
Data Collection Methods

Now that we have
decided on sampling
methods, we need to
think about the way in
which we collect the
information.

What type of data
collection method will
you choose?
Methodological Considerations for
Surveys

It is possible to incorporate experimental and
quasi-experimental designs in survey research.

Link back to purpose of survey.

Do you need to have a control group?
 Do you need to use probability sampling?
 Will you have multiple data collections?


Will the same individuals participate at both times?
How will you identify these individuals?
Data Collection Methods

Face to face interview.

Exit survey (ex. Leaving mall).

Telephone interview.

Self-completed questionnaire.

Web-based survey.
Data Collection Methods
 Data

collection methods.
What type of method will you select?
 Considerations.




Cost.
Time.
Accessibility and type of population (are mail
or telephone an option?).
Amount, type, and quality of data required.
Data Collection Methods
Method of
data
collection
Cost
Time
Response Length
rate
Complex
FACE TO
FACE
HIGH
MEDIUM
HIGH
LONG
HIGH
EXIT
MEDIUM
FASTSLOW
MEDIUM
SHORT
LOW
TELEPHONE MEDIUM
FAST
MEDIUMHIGH
MEDIUM
MEDIUM
MAIL
SLOW
VERY
LOW
MEDIUM
LOW
FAST
LOWMEDIUM
SHORTMEDIUM
MEDIUM
LOW
WEB-BASED LOW

Once you have decided
on the data collection
method, it is time to
think about the content
of your survey.

What are the questions
that will comprise your
survey?
Question Development: Types

Open-ended questions: Respondents are free to
express answers in own words.

Closed-ended questions: Respondents must
choose responses from a list.

Partially-open questions: Respondents can
choose from categories provided, or compose
their own answers.
Open-ended Questions
 Used
often in qualitative research (i.e.
focus groups, case studies).
 Advantages:


Provide opportunity for self-expression.
Can obtain exact numerical data if required
• Example: How many times have you been to the
hospital in the last year? 16

Can obtain natural wording.
Open-ended Question Example
 Hello
Sir, can you tell us about the service
you received at the grocery store today?
“My experience at your
store was horrible! The
employees were rude,
you did not have what I
wanted, and I got a
ticket because my
parking meter ran out of
time!”
Open-ended Questions
Limitations for
researcher:

• Can yield irrelevant
answers.
• Difficult/time
consuming to code and
analyze.
• Expensive.
•
Limitations for
respondent:
Questions are
demanding.
• Time consuming.
Closed-ended Questions
 Very
common in surveys.
 Advantages:






Easy to answer.
Fast to answer.
Easy to code.
Easier and faster to analyze.
Less expensive (survey processing and
analysis).
Consistent response categories.
Closed-ended Question
Example

Binary:
 Were you satisfied with the service you
received?


Yes
No

Multiple Choice:
 How long have you worked for the clinic?




Less than 1 year.
1 to 5 years.
6-10 years.
More than 10 years.
Closed-ended Questions


Limitations:
More effort in design stage.
 May elicit answer where no option/knowledge
exists.
 Response categories must be exhaustive and
non-overlapping.
Partially-Open Questions

Allows respondent to choose an option, BUT also gives
them the opportunity to choose “OTHER, PLEASE
SPECIFY”.

Example:
Were you happy with the service you received at the
diabetes clinic?

Yes
No
Not Sure
If you selected No, why was this the case?
_____________________________________________________
Rating Scales
Using the scale below, please rate our staff on
the following items:
1=Poor
2=Fair
3=Good
4=Excellent
Knowledge
Friendliness
Promptness
1
1
1
2
2
2
3
3
3
4
4
4
Commonly Used Rating Scales

Satisfaction scales


Agree/Disagree scales



(strongly disagree to strongly agree).
Universal scale: Can be applied to all ratings.
Performance scales



(very dissatisfied to very satisfied).
(poor to excellent).
Can be used widely, very good for overall ratings.
Frequency scales


(never to very often).
Not very common, but can sometimes be used.
Common Satisfaction Scales

2 point satisfaction



4 point satisfaction





1-satisfied
2-unsatisfied
1-very satisfied
2-satisfied
3-dissatisfied
4-very dissatisfied
5 point satisfaction





1-very satisfied
2-satisfied
3-neither dissatisfied or satisfied (neutral)
4-dissatisfied
5-very dissatisfied
Common Satisfaction Scales

Also 7 point scales (have a somewhat satisfied
and somewhat dissatisfied categories)…use if
applicable for your particular purpose.







1-very satisfied
2-satisfied
3-somewhat satisfied
4-neither dissatisfied or satisfied (neutral)
5-somewhat dissatisfied
6-dissatisfied
7-very dissatisfied
Can
mean
similar
things...
Common Performance Scales

4 point excellence
scale.





1-excellent
2-good
3-fair
4- poor
5 point excellence
scale.





1-excellent
2-good
3-fair
4-poor
5-very poor
Common Performance Scales

Can also have expectation scales:





4 point expectation.
1-exceeded
2-met
3-nearly met
4-missed

Can also have importance scales (very to not
very important).
Frequency Scales

Use when you have questions related to how
often.






Never
Rarely
Sometimes
Often
Always
E.g. How often are you kept waiting at your doctor’s
office?
Rating Scales: Factors to Consider

Factors to consider:
1) Select the appropriate scale!



Match scale to question. Find the most natural scale
through informal pre-testing.
E.g. I can easily get the information I need to do my
job well.
Would you use a frequency scale (how often), or an
agreement scale (strength of agreement)?
Rating Scales
2) Direction
 Doesn’t matter as long as it’s clear to
respondent.
 Do not change scale direction, keep one
direction for all questions.
3) Number of choices: there is no specific
number, pick the one that works best for you.
• Large scales (10 pt +), harder to answer and label.
• Shorter scales not as sensitive.
Rating Scales
4) Label Categories instead of using only numbers
 Will make distinction between categories clear.
5) Midpoint
 Will yield more information than forcing a
pro/con response.

E.g. Neither dissatisfied or satisfied (neutral).
Rating Scales
6) Don’t know, not applicable, and neutral are
different.

Important to consider these options when
appropriate.
Don’t Know: respondent lacks knowledge to
make judgment.
 Not Applicable: respondent cannot relate to
statement.
 Neutral: respondent has come to middle of two
extremes.

Rating Scales
7) Response set.
 Respondents tend to repeat previous answers in
rating questions.
 Long series of rating questions should be broken
up.
 Insert other question types between rating
scales.
Visual Analogue Scales

Visual Analogue Scale (VAS) is a measurement
instrument that tries to measure a characteristic
or attitude that is believed to range across a
continuum of values and cannot easily be
directly measured (Gould, 2001).

Example:
Considerations for Wording
•
•
•
•
•
Keep your audience in mind.
Are the questions easy to understand?
Do the questions have the same meaning to
all?
Define important terms.
Limit bias
Bias – Question Design

Wording

Ambiguous/inappropriate question
• Were you happy with the service you received at the diabetes clinic?

Complex question
• Are you in favour, or not in favour of a law that would not allow store
to be open on Sundays nor stat holidays?”

Double-barrelled question
• If you watch TV regularly, what kind of shows do you watch?

Technical jargon
• Do you agree that IH should have access to SPSS or SAS to
support quantitative survey analysis?”

Uncommon word
• Assist, reside, sufficient, deleterious etc.

Vague word
• Regularly, generally, (can also apply to emotional or value based
words – e.g. feel, respect, positive)
Choi & Pak. Prev Chronic Dis. 2005: 2:1-13
Bias – Question Design

Missing or inadequate data for intended purpose

Belief vs. behaviour
• Do you believe smoking is harmful? vs. Do you smoke?

Starting time
• In the past year……(changing time reference)

Data degradation
• Date of birth vs. age in years vs. age category

Insensitive measure
• (worse 1 - 2 - 3 better) limited categories, floor and ceiling
effects
Choi & Pak. Prev Chronic Dis. 2005: 2:1-13
Bias – Question Design

Faulty scale

Forced choice
• Were you happy with the service you received at the diabetes
clinic? Yes No

Missing interval
• Complete range of response interval options is not present

Overlapping Interval
• Interval anchors or parts of ranges overlap

Scale Format
• Odd numbers tend to result in neutral options (choose middle
category)
• Even numbers tend to force choice to one side or another
• No consensus to best approach
Choi & Pak. Prev Chronic Dis. 2005: 2:1-13
Bias – Question Design

Leading questions

Framing
• Questions framed so that respondent may choose incorrect answer
• What surgery would you prefer?



Outcome with 5% mortality.
Outcome with 90% survival.
Leading question
• Do you do physical exercise, such as cycling?
• Don’t you agree that….
• Please rate our excellent service

Mind-set
• Try to maintain response option consistency (categories, anchors,
order)
Choi & Pak. Prev Chronic Dis. 2005: 2:1-13
Bias – Question Design

Intrusiveness


Reporting
Sensitive information
• Both involve selective suppression of personal or confidential
information

Inconsistency

Case definition
• ICD classifications (first event vs. recurrent)
• Coke, soda, pop

Change of scale and/or wording
• Scale consistency is especially important if comparisons are made
over time or with the results of other surveyors

Diagnostic vogue
• Same illness may have different labels (region, time, type of
respondent)
Choi & Pak. Prev Chronic Dis. 2005: 2:1-13
Survey Design Process
1) Define objectives and requirements.
Keep “need to know” questions, be cautious
about “like to know”.
2) Consult with experts familiar with, or are part
of interest group.
3) Draft questions while thinking about data
collection method and burden on respondent.
4) Review/revise the questionnaire.
5) Pre-test or ‘pilot’ the questionnaire
Keys points to remember





Write in everyday terms.
Follow basic writing principles (direct/to the
point, no spelling errors, grammar etc).
Use consistent scales.
Use consistent wording.
Be clear about directions (what you would like
the respondent to do).
Reliability and Validity

Reliability - degree to which an instrument
measures the same way each time it is used
under the same condition with the same subjects.
 Validity - degree to which a survey accurately
reflects or assesses the specific concept that the
researcher is attempting to measure.
 There are many reliable and valid surveys that
might be suitable for your research.




Internet
Electronic databases – CINAHL, Medline
Provincial or National Surveys
Validated surveys are only valid if use entire tool.
Choosing a Ready-Made Survey

Reliability



Test-retest
Internal Consistency
Validity




Predictive - is it correlated with a known outcome?
Concurrent - is it correlated with a known and
accepted measure/survey?
Content - does it contain all appropriate information
(based on theory and expert opinion)?
Construct – does the survey measure what it is
supposed to measure.
Survey Length
How long is too long?
Avoid long surveys when they
are unnecessary!
Sequencing and Layout
Intro Paragraph
 Always begin a survey with an introductory
statement.

Often includes:





Asking the participant to participate.
Ends with thanking the participant for participating.
Discussing confidentiality.
Purpose of survey.
Discusses sharing of findings with participants.
Sequencing and Layout
 Begin
with easy questions
(demographics).
 Group questions by topic.
 Respect chronological order when
appropriate.
 Always include comments section at end.
 No name on survey.
 Reduce number of “skip to” questions
(easy for web-based surveys).
Sequencing and Layout
 Use
at least 12 pt font (larger for older
audience).
 Pastel colors work well.
 Instructions in different style (i.e. Italics).
Maximizing Survey Response
Burns, K. E.A. et al. CMAJ 2008;179:245-252
Copyright ©2008 Canadian Medical Association or its licensors
Data Coding and Analyses
 Once
you have decided on survey
questions, it is time to think about coding
the data.
 Important to think about before the
collection of data!
 Coded data can be entered into a
spreadsheet, which will help when
analyzing data.
 Data should be coded numerically for ease
of analysis.
Codebook
 What
is a codebook?
 A codebook is a log of your variables
(survey items) and how you will code
them.
 A codebook will help everyone
understand the coding schemes to
ensure that they are on the same page!
Data Processing and Analyses:
Codebook Example
Variable
Name
Variable
Label
Values
Coding
Missing
Variable
Type
age
age
1,2,3,4,5
1=10-20 years
2=21-30 years
3=31-40 years
4=41-50 years
5=51+ years
97=Incorrect
response
98=No response
99=Not
Applicable
Ordinal
sex
sex
1,2
1=male, 2=female
97=Incorrect
response
98=No response
99=Not
Applicable
Nominal
happiness
happiness
at
work
1,2,3
1=not happy
2=somewhat happy
3=very happy
97=Incorrect
response
98=No response
99=Not
Applicable
Ordinal
Spreadsheet Example
ID#
Age
Sex Happiness
1
1
1
2
2
2
2
2
3
3
1
2
4
57
2
2
5
45
2
3
6
66
2
3
7
2
2
3
8
88
2
3
Open-ended Data coding
It’s easy to code closed response or rating
questions, but how do you code open-ended
data?
 Objective: to create codes and classify
responses into categories respondents would
have chosen, had they been offered categories.



Two phases: 1) Scan responses, 2) Scan responses
and then code them.
Themes will emerge.
Data Cleaning

There are several ways you can clean survey
data.

Editing checks:

1) Structure checks-identify non-response.
 2) Range edits- make sure there are valid
ranges (E.g. No 7’s on a scale of 1-5).
 3) Make sure ‘not stated’ codes are put into
unanswered responses.
Data Processing and Analysis:
SURVEY SAYS!
Response Rate

The actual number of completed surveys, NOT
the number of surveys distributed.
 Low response rates (of less than 60%) may put
you at risk for non-response error.


Non-Response error: People who do not respond may
be different from those that did, in ways that are
important to your study .
Try to get the highest response rate possible.
• Reminders
• Oversample
• Incentives
Descriptive or Inferential Statistics?

Descriptive Statistics.
 Descriptive statistics are used to describe the
basic features of data.
 Provide simple summaries about the sample
and the measures.

Example:


Measures of central tendency (means, medians,
modes).
Frequencies.
Central Tendency

The level of data will dictate which measure of central
tendency you should use.
 Categorical = Data that is classified into categories and
cannot be arranged in any particular order (e.g. Apples
and pears, gender, eye colour, ethnicity).
 Ordinal = Data ordered, but distance between intervals
not always equal. (e.g. Low, middle and high income).
 Continuous = equal distance between each interval
(e.g. 1,2,3., age).



If data is categorical – MODE.
If data is ordinal – MEDIAN.
If data is continuous – MEAN.
Central Tendency
Mean
What is a mean?
 The sum of all the
scores divided by the
number of scores.
 Often referred to as
the average.
 Good measure of
central tendency.
 Central tendency is
simply the location of
the middle in a
distribution of scores.
Central Tendency
A median is the middle of a distribution.
 Half the scores are above the median and half are below
the median.
 How do I compute the median?
• If there is an odd number of numbers, the median
is the middle number. For example, the median of
5, 8, and 11 is 8.
• If there is an even number of numbers, the median
is the mean of the two middle numbers. The
median of the numbers 4, 8, 9, 13 is (8+9)/2 =8.5.
Central Tendency
What is a mode?
 Most frequently occurring score in a distribution.
 Distributions can have more than one mode, called
"multimodal“.
Descriptive or Inferential Statistics?
 Inferential
Statistics
 Use inferential statistics when trying to
reach conclusions that extend beyond the
immediate data alone.
 Examine relationships between variables,
comparisons etc.
 Make conclusions about the population
from the sample.

Requires probability sampling.
Descriptive or Inferential Statistics?
 Your
sampling method and question, or
reason you are conducting the survey will
dictate the analysis.
 Descriptive
statistics are a common way to
analyze survey data.
Post-analysis

Once you have analyzed your data, it is time
to interpret the findings.

What do your findings mean?

Depends on the purpose of the survey.
• Program evaluation
• Research
• Customer Satisfaction (gap analysis)



Are you surprised by the information?
Can you use the information to improve, support or
develop a program?
Has the information collected raised other
questions?