Download PowerPoint

Part Three:
Observation,
Sampling
Pierre-Auguste Renoir, 1841-1919
Experiments
Pierre-Auguste Renoir: Barges on the Seine, 1869
Topics Appropriate to Experiments
1. Experiments Allow for Control of Variables
• How much is learned about a topic.
• How much time is allowed for tasks.
• The composition of groups.
• Who speaks and how long they speak in
groups.
• Manipulation of opinions by the use of
confederates in group settings.
2. Laboratory and Field Experiments
• Field experiments provide a natural setting, but
allow for less control over variables.
The Classical Experiment
1. Independent and Dependent Variables
• Typically, the operational definitions of
independent and dependent variables are
defined prior to the experiment.
• The way in which independent variables will be
introduced into the experiment is
predetermined.
• The purpose of an experiment is to manipulate
the score on the dependent variable by
introducing different independent variables or
manipulating existing ones.
The Classical Experiment
1. Independent and Dependent Variables (Cont.)
• Included among the independent variables are
one or more variables specifically intended to
manipulate the score on the dependent
variable.
• These are called by various names, each
meaning the same thing:
• Stimulus
• Treatment
• Experimental
The Classical Experiment
2. Pretesting and Posttesting
• The pretest is the measurement of variables
prior to introducing the treatment variable.
• The posttest is the measurement of variables
after introducing the treatment variable.
3. Experimental and Control Groups
• The Experimental group is exposed to the
treatment variable(s).
• The Control group is not exposed to the
treatment variable(s).
The Classical Experiment
Experimental
Group
Control
Group
Time 1
Measure Dependent
Variable (pretest)
Measure Dependent
Variable (pretest)
Time 2
Administer Stimulus
Time 3
Measure Dependent
Variable (posttest)
Schedule
Measure Dependent
Variable (posttest)
Quasi-Experimental Designs
1. Rationale
• Sometimes, the researcher examines events in
the field that cannot be easily anticipated.
• Responses to disasters.
• Responses to rapid social change.
• Sometimes, the added expense of a classical
experiment is not necessary.
• The one-shot case study is common to
market testing of low-involvement products.
Quasi-Experimental Designs
2. One-Shot Case Study
• Posttest only of the experimental group.
Schedule
Experimental Group
Time 1
Time 2
Administer Stimulus
Time 3
Measure Dependent
Variable (posttest)
Control Group
Quasi-Experimental Designs
3. One-Group Pretest-Posttest Design
• Pretest and posttest of one group.
Schedule
Experimental Group
Time 1
Measure Dependent
Variable (pretest)
Time 2
Administer Stimulus
Time 3
Measure Dependent
Variable (posttest)
Control Group
Quasi-Experimental Designs
4. Static Group Comparison
• Control at Time 3.
Schedule
Experimental Group
Control Group
Time 1
Time 2
Administer Stimulus
Time 3
Measure Dependent
Variable (posttest)
Measure Dependent
Variable (posttest)
Experimental Designs
This chart summarizes the experimental designs:
Control Group
Yes
No
Yes
Classical
One-group,
Pretest-Posttest
No
Static-Group
Comparison
One-Shot
Case Study
Pretest
Elements of Experiments
1. The Blind Experiment
• In some cases, the experimenter might
influence the scores on the variables.
• Example: In evaluating the efficacy of a new
medicine, if the subjects know they are taking
the medicine, they might respond because of
this knowledge rather than because of the
medicine. Therefore, all subjects are given
“medicine,” but the control group is given a
placebo: a false medicine (e.g., a pill filled with
sugar rather than medicine.)
Elements of Experiments
2. The Double Blind Experiment
• In some circumstances, the experimenter
might influence scores on the variables.
• Example: If the experimenter knows which
subjects are taking the real medicine and this
person wants the medicine to be effective, then
the experimenter might evaluate the subject’s
outcomes more favorably.
• In the double-blind experiment, neither the
subject nor the experimenter know which
subjects are in the experimental group.
Elements of Experiments
3. Representation
• Typically, experiments focus on building or
testing theory rather than attempting to predict
population characteristics.
• Therefore, as long as the subjects have key
characteristics of interest, then it is not often
necessary that the sample be representative.
• It is critical, however, for subjects to be evenly
matched in characteristics across the
experimental and control groups.
Elements of Experiments
4. Probability Sampling
• Probability sampling is used to achieve
representativeness with large samples.
Therefore, it is not often used for experiments.
5. Randomization
• It is essential for subjects to be randomly
assigned to the experimental and control
groups.
6. Matching
• To assure even distribution of key
characteristics between groups, experimenters
might assign subjects to groups.
Validity Issues in Experiments
1. Sources of Internal Invalidity
• Do the results reflect the effect of the stimulus
variable?
• Factors affecting internal validity:
1. History: Unplanned events that occur
during the experiment.
2. Maturation: Change in people from Time 1
to Time 3.
3. Testing (cueing): The process of the
experiment itself creates changes in
responses to the dependent variable.
Validity Issues in Experiments
1. Sources of Internal Invalidity (Continued)
• Factors affecting internal validity:
4. Instrumentation: Do the pretest and
posttest measures exactly match each
other?
5. Regression Toward the Mean: Changes
might occur because subjects begin at the
extreme.
6. Selection Bias: Subjects are not matched
across groups.
Validity Issues in Experiments
1. Sources of Internal Invalidity (Continued)
• Factors affecting internal validity:
7. Experimental Mortality: Subjects drop out
of the study before it is completed.
8. Causal Time Order: In some cases, it is
difficult to time the stimulus after the
pretest.
9. Diffusion: Subjects across groups share
information with one another.
10. Compensation: Experimenters might treat
the control group differently.
Validity Issues in Experiments
1. Sources of Internal Invalidity (Continued)
• Factors affecting internal validity:
11. Compensatory Rivalry: Subjects who know
they are in the control group might behave
with more interest.
12. Demoralization: Subjects in the control
group might behave with less interest.
Validity Issues in Experiments
2. Sources of External Invalidity
• Can the results be generalized to the
population?
• Interaction: Subjects who know they are being
studied might be more receptive to the
stimulus.
• Cueing: The administration of the pretest might
sensitize subjects to the content of the
stimulus.
Alternative Experimental Settings
1. Web-Based Experiments
• Subjects answer questions or perform tasks
online.
• Subjects might be asked questions prior to
being assigned to a group, or they might be
assigned at random at the outset of their
session.
See: Online Social Psychology Studies
See: Small World Phenomenon
Alternative Experimental Settings
2. Natural Experiments
• Behavior occurring during or after natural
events can be investigated.
• “Control” groups can be persons in similar
settings that did not experience the natural
event.
Survey Research
Pierre-Auguste Renoir: Luncheon of the Boating Party, 1881 [D]
Topics Appropriate for Survey Research
1. Data Collection from Large Numbers
• Survey research is a relatively inexpensive
procedure for collecting information from a
large number of elements.
• They are efficient means of learning about
attitudes, intentions to act.
• Unfortunately, they are overused by marketing
and political organizations, to the point where
many persons refuse to respond to any type of
survey.
Issues Related to Survey Research
1. Survey Response Rate
• When survey response rates are low then
questions arise about how well the results can
be generalized to the population.
• What were the opinions of those elements
who did not respond?
• Various types of extrapolation procedures can
be used to estimate the responses of those
who did not complete the questionnaire, but no
procedure is widely accepted by the community
of scholars.
Issues Related to Survey Research
1. Survey Response Rate (Continued)
• A large volume of research addresses the issue
of how to improve survey response rates.
• Key features that affect response rates:
• The topic of the survey, sponsorship,
incentives, survey mode, time/day of
contact, question wording, question ordering,
formatting of the questionnaire, follow-ups,
interview style, interviewer style, envelope
style, cover letter, time to complete.
Issues Related to Survey Research
1. Survey Response Rate (Continued)
• The survey response rate is calculated as:
Number of completed questionnaires
(Number of attempted contacts
- Unavailable for contact)
• For university-sponsored research, survey
response rates of 50% are acceptable, 60% is
preferred, 70% is very good.
Issues Related to Survey Research
2. Item Response Rate
• The item response rate is the percentage of
responses to an item among all returned and
completed questionnaires.
• Questionnaires might be returned and
otherwise complete except for missing
responses to certain questions.
• Example: Responses to questions about
income can receive low response rates.
• Key features that affect item response rates:
• Topic, question wording, question ordering,
questionnaire formatting, interviewer style.
Guidelines for Asking Questions
1. Choose Appropriate Question Forms
• Open Ended Questions: Respondents are
asked to give information in their own words.
• Closed-Ended Questions: Respondents are
asked to state which response most closely
represents their answers to questions.
2. Respondents Must be Competent to Answer
• Typically, surveys are administered to adults.
• Sometimes, screening questions are needed to
insure that the respondent is qualified to
answer (i.e., “Do you live in the city limits of
Sappville?”).
Guidelines for Asking Questions
3. Respondents Must Be Willing to Answer
• Respondents must consent to complete the
survey and must be competent and legally able
to give their consent.
4. Questions Should be Relevant
• Questions should apply to most respondents.
• Questions should address the main topic of the
survey.
5. Short Items are Best
• Persons are more likely to understand and
respond to simple, short questions.
Guidelines for Asking Questions
6. The First Question
• Should directly address the main topic of the
survey, even if the survey covers sensitive
topics (e.g., sex, religiosity, employment).
• Attempts to “warm up” the subject with
background questions or ones not directly
related to the topic will give the impression of
being evasive, coy, misleading in purpose.
• Should apply to all potential respondents.
• Should be short and easy to answer.
Guidelines for Asking Questions
7. Question Wording
• See the: Question Wording website for detailed
directions and examples of how to avoid
common mistakes in writing survey questions.
Question Wording
1. Avoid the loaded question
• The loaded question provides only one
reasonable response for the subject.
• The Surgeon General states that cigarette
smoking is harmful to one’s health. Do you
encourage your children to smoke
cigarettes?
Note: Sometimes one might deliberately want to
bias wording to help balance a controversial topic:
• Do you support cigarette advertising in foreign
countries to promote job creation in the U.S.?
Question Wording
2. Avoid using inflammatory words
• Inflammatory words bias the response.
• Do you think rude people should be able to
smoke their cigarettes while attending a
baseball game?
3. Avoid being too folksy
• Informal language assumes knowledge and
familiarity.
• Ok, let’s look at some questions on smoking
cigarettes.
Question Wording
4. Avoid using slang terms
• Slang assumes knowledge and familiarity.
• Would you hang with a cigarette smoker?
5. Avoid using technical terms
• Most persons do not know the meaning of
technical terms.
• Approximately how many PCP’s are inhaled
from smoking one cigarette?
Question Wording
6. Use precise wording
• Imprecision can create misunderstanding.
• Should tobacco be banned?
7. Be precise regarding time
• Imprecision can create misunderstanding.
• Have you ever smoked cigarettes?
[meaning “as a habit” rather than “ever tried
one”]
Question Wording
8. Use accurate facts
• Inaccuracy distorts the meaning of the
question.
• How concerned are you about the possibility
of contracting HIV from smoking cigarettes?
9. Do not assume knowledge or behavior
• The assumed knowledge or behavior should be
asked as a prior question.
• Do you agree with the Surgeon General’s
latest report on cigarette smoking?
Question Wording
10. Use correct grammar
• Inaccuracy distorts the meaning of the
question.
• Should cigarette smoking be gotten done
with?
11. Avoid double negatives
• Double negatives create confusion about
meaning.
• Do you disagree that cigarette smoking is
disagreeable?
Question Wording
12. Avoid the double-barreled question
• The word “and” can create two questions in
one.
• Do you think that cigarette smoking is bad
for your health and well-being?
This error is very common in questionnaire
wording. Be very skeptical of the use of “and”
in question wording
Question Wording
13. Response categories should match the
question
• Using a common set of response categories
can create misunderstandings.
• Should the national health care bill include
a $1.00 tax increase on a pack of
cigarettes?
1. never
2. sometimes
3. often
4. always
Question Wording
14. Response categories should be mutually
exclusive
• Inclusive response categories create
confusion about how to mark the item.
• How much do you spend on cigarettes each
week?
1. Do not smoke
2. less than $10
3. $10 to $15
4. $15 or more
Question Wording
15. Use a time frame to measure future behavior
• An open time frame allows for too many
possibilities.
• Wrong: Will you ever smoke a cigarette?
• Right: Do you intend to smoke a cigarette
with the next six weeks?
Question Wording
16. Avoid determinism
• Deterministic questions do not leave open the
possibility for changes or exceptions.
• Is cigarette smoking in public places ever
acceptable?
17. Provide clear instructions on responses
• Ambiguity will create confusion about how to
respond.
• Please rate your opinion about smoking
cigarettes on a scale of 1 to 10.
Question Wording
18. Avoid specifying too many response
alternatives in the question
• Long, complex questions create confusion.
• Do you strongly agree, agree, neither agree
nor disagree, disagree, or strongly disagree
that cigarette is harmful to one’s health?
Question Wording
19. Split complex questions into two parts
• Questions should be easy to answer.
• Wrong: What percentage of your weekly
income do you spend on cigarettes?
• Right:
• What was your approximate total
income before taxes in 2016?
• Approximately how much money do
you spend on cigarettes each week?
Question Wording
20. Include “Don’t Know” only when appropriate
• Too much use of this response option can
create problems when interpreting the data.
• Dr. Sapp advises to use a “don’t know”
response category when requesting factual
information (e.g., Do your children smoke
cigarettes?), but not when requesting opinions
(e.g., Should billboard advertisements for
cigarettes be banned?).
Question Wording
21. Avoid lists longer than five items
• Questions should be easy to answer.
• Please rank in order of importance the
following 15 reasons for avoiding cigarette
smoking?
22. Avoid too much abstraction
• Too much abstraction can create confusion.
• Does cigarette smoking erode the moral
integrity of the American citizenry?
Question Wording
23. Be simple without being condescending
• Questions should respect the intelligence of
the respondent.
• Should the Surgeon General (i.e., the head
person in charge of health promotion) ban
cigarette smoking?
Other Notes
• Avoid lengthy questions.
• Special instructions to interviewers should be
clear and easy to follow.
General Questionnaire Format
1. Formats for Respondents
• The questionnaire should present an easy to
understand format for respondents.
• It should be clear about how to respond.
• The questionnaire should have a professional
appearance. Money spent on professional
formatting and graphics is money well spent.
• The response options should tend to flow down
the page to give the sense of completing the
questionnaire quickly.
General Questionnaire Format
2. Contingency Questions
• These questions screen respondents for
characteristics and then direct them to the next
appropriate question on the survey.
3. Matrix Questions
• A lead in question is followed by a series of
statements that complete the question.
• Example: “How much do you trust...
1. Federal legislators.
2. State legislators.
3. Local legislators.
General Questionnaire Format
4. Ordering Items in a Questionnaire
• The ordering should flow logically from one
topic to the next.
• Question ordering affects responses. It is
difficult to avoid this bias altogether.
5. Questionnaire Instructions
• A self-administered questionnaire will need to
be especially easy to understand.
• Short introductions help the respondent grasp
the intent of the questions that follow.
• Providing opportunities to give additional
information helps gain validity and responses.
General Questionnaire Format
6. Pretesting Questionnaire
• Pretesting helps uncover many flaws types of
flaws that can occur when writing a
questionnaire.
• Pretesting will reveal the approximate time it
takes to complete the questionnaire.
Data Collection
1. The Total Design Method
• The Total Design Method, developed by Don
Dillman, has been very effective in improving
survey and item response rates.
• The TDM includes a wide range of guidelines
for questionnaire development and survey
administration.
• One key feature of the TDM is the suggested
guideline for administering follow-up
questionnaires in a mailed survey.
Data Collection
1. The Total Design Method (Continued)
• The TDM approach to follow-ups:
Day 1: Mail questionnaire and cover letter.
Day 7: Mail a reminder postcard to all elements.
Day 21: Mail a replacement questionnaire and cover
letter to those elements who have not yet
replied.
Day 49: Mail a replacement questionnaire by
certified mail and cover letter to those
elements who have not yet replied.
Data Collection
1. The Total Design Method (Continued)
• The TDM approach to follow-ups increases the
cost of conducting a survey.
• Be prepared to hear complaints from elements,
especially after the fourth mailing (See: Angry
Letter).
• Instructor’s Note: I feel justified in using the
TDM approach to follow-ups when I am
conducting research with taxpayer’s money
because I think that elements have a
responsibility to reply on behalf of other
members of the society.
Data Collection
1. The Total Design Method (Continued)
• The TDM approach to follow-ups increases the
cost of conducting a survey.
• Be prepared to hear complaints from elements,
especially after the fourth mailing (See: Angry
Letter).
• Instructor’s Note: I feel justified in using the
TDM approach to follow-ups when I am
conducting research with taxpayer’s money
because I think that elements have a
responsibility to reply on behalf of other
members of the society.
Data Collection
2. The Cover Letter
The cover, or introductory, letter conveys
important information to potential respondents:
• Legitimacy of the survey.
• Contact information.
• “Social contract” for the study. “What’s in it for
the respondent.”
• Requirements of the Institutional Review Board.
• Appeals to respond.
The Mailed Survey
1. Introduction
• Questionnaires are mailed to potential
respondents and returned by mail.
• Business reply envelopes can be used for the
return of the questionnaire.
• The questionnaire is accompanied by a cover
letter.
• Sometimes, incentives are used to increase the
response rate.
The Mailed Survey
2. Comparison with Other Modes
 Best mode for collecting sensitive, extensive, or
complex information.
 Must have strong content validity of items.
 Most time to gather data.
 Requires entry of data after collection.
 Does not allow for clarification or probing.
 Does not allow for editing questions.
 Must have very clear directions.
 No control over who completes the
questionnaire.
The Telephone Survey
1. Introduction
• The preferred method for marketing research
and political polling.
• The direct contact and rapid data acquisition of
a telephone survey are well suited to collection
of information on a few, easy to understand
items.
• Telephone surveying is not restricted by the “Do
Not Call” registry.
The Telephone Survey
2. Comparison with Other Modes
 Best mode for collecting a few, easy to answer
questions in a short time frame.
 Does not require entry of data after collection
(Computer Assisted Telephone Interviewing).
 Allows for clarification or probing.
 Allows for editing questions.
 More expensive than a mailed survey.
 Questions cannot be overly complicated and
must be answered with knowledge at hand.
 Requires trained interviewers.
The Personal Interview
1. Introduction
• There is no substitute for a personal interview
when the research requires much in-depth
information from elements.
• Personal interviewing is an excellent mode for
exploratory research.
• Personal interviewing allows the researcher to
collect additional information not included in the
interview itself: body language, setting,
condition of the home, nuances of meaning,
personal contact with hard-to-reach elements.
The Personal Interview
2. Comparison with Other Modes
 Best mode for collecting in-depth responses.
 Can collect information other than what is
asked on the questionnaire.
 Allows for clarification or probing.
 Allows for editing questions.
 Allows for unstandardized interviewing.
 The only reasonable procedure for collecting
data from some elements.
 Most expensive type of survey.
 Requires trained interviewers.
 Data entry after collection might be required.
The Internet Survey
1. Introduction
• This method is gaining in popularity as more
persons have home computers and access to
the internet.
• Internet surveys can combine some of the
advantages of mailed and telephone surveys.
• The quality and affordability of software
packages is improving rapidly.
The Internet Survey
2. Comparison with Other Modes
 Least expensive approach to surveys.
 Can ask many questions.
 Can ask detailed information.
 Respondents can complete the survey at their
own pace and at a time convenient to them.
 Allows for randomization of questions by
respondent.
 Immediate data entry.
 Still unfamiliar to some respondents.
 Not a good way to contact older or less affluent
adults.
The Recommended Approach
Mail push to web.




Less expensive than a mailed only.
Can ask many questions.
Can ask detailed information.
Respondents can complete the survey at their
own pace and at a time convenient to them.
 Web version allows for randomization of
questions by respondent.
 Web version has immediate data entry.
 Still unfamiliar to some respondents.
Secondary Analysis
1. Introduction
• Analysis of data from existing surveys.
• Typically, this term refers to analysis of largescale, government sponsored surveys.
• Various censuses of the population.
• National Longitudinal Survey
• National Survey of Families and Households
• General Social Survey
• National Health and Nutrition Examination
Survey
• Many others....
Secondary Analysis
2. Comparison with Other Modes
 Large number of subjects.
 Longitudinal studies.
 Data available to many researchers.
 Expert data collection procedures.
 Take questions as they come.
 Questions over time do not always match.
 Can be slow to adopt new theories and
concepts.
The Logic of Sampling
Pierre-Auguste Renoir: Yvonne & Christine Lerolle Playing the Piano, 1897.
A Brief History of Sampling
Some Failures
• President Alf Landon: In 1936, the Literary
Digest predicted that Landon would defeat
President Franklin Roosevelt. It based this
opinion on a poll it conducted, wherein it
selected its sample from a list of telephone
numbers and automobile registrations.
• The flaw in this sampling procedure was that
this sample frame was biased toward the
educated and affluent, who tended to vote for
Landon (Roosevelt won in a landslide!).
A Brief History of Sampling
Some Failures (Continued)
• President Thomas Dewey: In 1948, the George
Gallop agency predicted that Dewey would
defeat Harry S. Truman. It based this opinion
on a poll it conducted, wherein the sample was
selected by quota using the 1940 Census
figures.
• The sample was biased because the 1940
Census did not reflect the rapid move to urban
areas following WWII. The many new
unaccounted for urban dwellers tended to vote
for Truman.
Probability Sampling
Definition
• A sample that selects subjects with a known
probability.
• Probability samples are important when one
wishes to generalize to the larger population
because one knows how to weight the
responses to fit the characteristics of the
population.
Nonprobability Sampling
Definition
• A sample that relies upon available subjects.
• Researchers sometimes rely upon available
subjects rather than draw samples using
probability sampling.
• Available subjects are selected because:
1. Lack of access all members of the
population,
2. Reduction in costs and time,
3. Lack of need for a probability sample.
Nonprobability Sampling
Purposive or Judgmental Sampling
• Selection of individuals with specific
characteristics.
• One might:
1. Request certain individuals within a
population (e.g., ask for an adult male in the
household in a telephone survey),
2. Restrict the sampling to certain audiences
(e.g., select a sample from readers of
Popular Mechanics),
3. Specify a need for individuals with certain
characteristics (e.g., solicit with ads).
Nonprobability Sampling
Purposive or Judgmental Sampling
• This type of sampling has the advantage of
collecting information from a targeted element.
For example, one might place an advertisement
in the newspaper to solicit “all current or former
members of the armed forces who have served
in Iraq” to join your sample.
• Thus, one can request that elements with
specific characteristics join the sample.
Nonprobability Sampling
Key Informants
• Selection of individuals who know information
about other individuals or events.
• Assurances of confidentiality can become
important in this type of sampling.
Nonprobability Sampling
Snowball Sampling
• Selection of individuals who are recommended
by others already selected.
• This procedure is appropriate for difficult to
locate populations or persons with specific
characteristics:
• Vietnam veterans who fought in a specific
area of the country.
• Influential leaders in a community.
• Persons who wish to remain anonymous,
but who will respond to introductions from
their associates.
Nonprobability Sampling
Snowball Sampling
• Snowball sampling allows the researcher to
screen potential members of the sample,
thereby building a sample of only those whom
the researcher wants to study.
• Snowball sampling takes more time and money
to implement than purposive sampling.
Nonprobability Sampling
Quota Sampling
• Selection of individuals to fill a quota for a
certain characteristic.
• This procedure is appropriate for building a
representative sample.
• One must have an accurate depiction of the
total sample.
• Filling out some cells in the quota might require
an unreasonable amount of resources.
Nonprobability Sampling
Quota Sampling
• Quota sampling has the advantage of collecting
information from elements of interest. For
example, the researcher might want to survey
100 males and 100 females. So, the
researcher continues to contact individuals until
the sample has 100 males and 100 females.
Nonprobability Sampling
Quota Sampling
• If the characteristics of interest become too
complicated, however, it can be difficult to fill all
the cells of a quota sample.
• For example, the researcher might have to
contact many persons before finding 100 white
females, aged 65+, with a college education to
join the sample.
Nonprobability Sampling
Summary
• Nonprobability sampling sometimes is the only
reasonable procedure for building a sample.
• Probability samples are unnecessary for
studies aimed at theory building or testing,
wherein the researcher is not attempting to
generalize the findings to a population.
Nonprobability Sampling
Summary
• Nonprobability sampling typically is less
expensive than probability sampling, but not in
all cases.
• The theoretical assumptions necessary for
inferential statistics requires a probability
sample. Therefore, non-probability samples
should not be used to make inferences to a
population.
Probability Sampling
Representativeness
• Representativeness: The extent to which a
sample has the same characteristics as the
population.
• Representativeness is judged by comparing
selected characteristics.
• Representativeness is not needed for
accurate generalization to a population:
• some characteristics are not important.
• weighting can adjust differences between
sample and population.
Probability Sampling
Conscious and Unconscious Sample Bias
• A biased sample is one whose characteristics
do not match those of the population.
• If the sample is biased, and the responses are
not weighted to reflect this bias, then
generalizations to the population will be flawed.
• A randomly selected sample is not necessarily
an unbiased one. A minority subpopulation, for
example, might be missed or underrepresented
when selecting a sample at random.
Types of Sampling Designs
Simple Random Sampling
1. Number the elements of the sample frame.
2. Generate n unique random numbers within the
range of numbers assigned to the sample
frame, where n = the size of the initial sample.
 This is the simplest procedure for drawing
a probability sample.
 Might not capture minority elements of a
sample frame.
 Cannot draw independent samples for
specific sub-populations.
Types of Sampling Designs
Stratified Sampling
• In its simplest form, a stratified sample is a set
of simple random samples selected from subsegments of the sample frame.
• Example: One might select a simple random
sample of males and a simple random sample
of females.
Types of Sampling Designs
Stratified Sampling (Continued)
 Allows one to control the number of elements
selected from each sub-segment of the sample
frame.
 Homogeneous sub-samples will have smaller
standard errors on parameter estimates than
will more heterogeneous samples of the entire
sample frame.
 More expensive and time consuming.
 Must know the size of each segment in the
sample frame.
Types of Sampling Designs
Cluster Sampling
• Elements are divided into groups of equal
number of elements (i.e., clusters).
• The clusters are selected at random.
• All elements within a cluster are included in the
initial sample.
 Saves time and money for personal interviews.
 Do not have to know the exact size of the
sample frame.
 Increased sampling error because of the
clustering procedure.
Types of Sampling Designs
Weighting
• Suppose we want to survey a city with a
population of 4,500 whites and 500 blacks.
• We want a sample of 1/10 = 500.
• If we selected at random, we would obtain only
50 blacks in our initial sample.
• We might want to over sample blacks to
improve the validity and reliability of our
estimates of their opinions.
• If we do so, we need to adjust the weights of
their opinions when we generalize to the total
population.
Types of Sampling Designs
Weighting (Continued)
Whites
Number in population…………………… 4,500
Percentage of population……………….
90
Sampling fraction (oversample blacks)..
1/10
Number in initial sample…………………
450
Unweighted percentage of sample……..
81.8
Weight (to adjust for oversampling)…….
1
Weighted number in initial sample………
450
Weighted percentage of initial sample….
90
Blacks
500
10
1/5
100
18.2
1/2
50
10
Probability Sampling
Sampling Distributions
• Probability Theory: A branch of mathematics
that provides the tools for estimating the
representativeness of a sample.
• A key aspect of probability theory is the Central
Limit Theorem: If the sum of variables has a
finite variance (i.e., set end points), then it will
be approximately normally distributed (i.e.,
have a bell-shaped curve).
• A normal distribution sometimes is called a
Gaussian distribution.
Probability Sampling
Sampling Distributions (Continued)
• The normal distribution is very useful because it
defines boundaries by which to judge the
representativeness of a sample.
• The key boundary of interest is the standard
deviation, which is a range of values from the
mean that includes a certain percentage of area
beneath the bell-shaped curve.
• For example, one standard deviation accounts
for all values from the mean included within
≈34.1% of the bell-shaped curve.
The Normal Distribution
Each standard deviation (σ) represents a defined
area from the mean (μ) beneath the curve.
Probability Sampling
Sampling Distributions (Continued)
• Variance is a statistic that represents how
spread out the observations are from the mean.
• The standard deviation is the square root of the
variance.
Probability Sampling
Sampling Distributions (Continued)
• Therefore, if one knows the mean and variance
of the population and the mean and variance of
the sample, one can estimate how closely the
sample characteristics match the population
characteristics using a standardized criterion of
judgment: the bell-shaped, normal distribution.
Probability Sampling
Sampling Distributions (Continued)
• Parameter: A summary description of a variable
in the population (e.g., the mean and standard
deviation are parameters).
• Statistic: A summary description of a variable in
the sample.
• Confidence Level: The amount of error the
researcher is willing to tolerate (e.g., 5%)
• Confidence Interval: The range of values about
a statistic where the parameter might be
located for a given confidence level.
Probability Sampling
EPSEM
• Equal Probability of Selection Method: All
members of a population have an equal
probability of selection in the sample.
• This is the basic principle of probability
sampling.
• Perfect representation still might not be
achieved.
• EPS is not always desirable. Sometimes,
one wants to oversample some segments of
a population.
Probability Sampling
EPSEM (Continued)
• Element: The unit from which information is
collected.
• An ISU student.
• Population: The aggregate of elements.
• All ISU students.
• Study Population: The part of the population
that is known or available to be sampled.
• All ISU students properly registered.
Probability Sampling
EPSEM (Continued)
• Sample Frame: The list from which the sample
is drawn.
• ISU students listed in the telephone
directory.
• Sampling Unit: The element or set of elements
considered for selection into the sample.
• ISU students taking 12 or more hours this
semester.
Probability Sampling
EPSEM (Continued)
• Initial Sample: Sampling units selected from the
sample frame.
• ISU students taking 12 or more hours this
semester who were selected at random
from the telephone directory.
• Final Sample: The elements who complete the
survey.
• ISU students in the initial sample who
completed the survey.
Types of Sampling Designs
Multistage Cluster Sampling
• Clusters are selected at random.
• Elements are selected at random within each
cluster.
 Saves time and money for personal interviews.
 Do not have to know the exact size of the
sample frame.
 Increased sampling error because of the
clustering procedure.
 Increased sampling error because of the
selection of elements within a cluster.
Types of Sampling Designs
Multistage Cluster Sampling: Example
•
•
•
•
•
•
•
The city has 10,000 households.
The city has 1,000 blocks of 10 hh each.
We want an initial sample of 500.
We want to select 1/20 households.
In Stage 1, we select 100 blocks.
In Stage 2, we select 5 hh per block.
Probability of selection for each hh:
• 1/10 (block) x ½ (hh in block) = 1/20.
Types of Sampling Designs
Probability Proportionate to Size (PPS)
• What if a few city blocks contain many more
households than others, and we anticipate that
density of housing is an important
characteristic that will affect our study?
• Then, we want to select clusters and
households proportionate to the number of
households in each city block to insure that we
select households from the large city blocks.
Types of Sampling Designs
PPS Sampling: Example
•
•
•
•
•
•
The city has 10,000 households.
The city has 110 blocks.
10 blocks contain 500 hh each.
100 blocks contain 50 hh each.
We want an initial sample of 500 hh.
If we selected blocks at random, we might miss
all 10 of the very large blocks.
• So, we use PPS sampling.
Types of Sampling Designs
PPS Sampling: Example
• The 10 large blocks contain 1/2 of the hh.
• So, we want to select blocks and hh so that we
obtain 1/2 of our sample from the large blocks
and 1/2 from the small blocks.
• We decide to select 50 hh from each block.
• Therefore, we need to select 5 of the 10 large
blocks and 5 of the small blocks.
• Probabilities:
• Large blocks: 1/2 (block) x 1/10 (hh) = 1/20.
• Small blocks: 1/20 (block) x 1/1 (hh) = 1/20.
Evaluation Research
Pierre-Auguste Renoir: Le Grenouillere, 1869
Evaluation Research
Introduction
• Evaluation research refers to a research purpose
rather than to a specific method.
• Evaluation research can include many different
types of methods aimed at understanding the
effectiveness of a social program that is intended
to bring about desired change.
• This form of research helps sociologists complete
the tasks of identifying social problems and
assessing the efficacy and consequences of social
change programs.
Evaluation Research
Evaluation research includes:
1. Needs assessment studies.
• Determine the existence, extent, and
awareness of social problems.
2. Cost-benefit studies.
• Assess the extent to which the outcomes of
social change programs justify their costs.
3. Monitoring studies.
• Provide information about ongoing social
problems.
Evaluation Research
Evaluation research includes:
4. Program evaluation (outcome assessment).
• Determine the extent to which social programs
are reducing social problems.
Formulating the Problem
1. Issues of Measurement
• One cannot measure efficacy and desired
outcomes unless one knows specifically the
outcomes of a social program or policy
expected within a certain time frame.
• Sometimes, goals are not initially welldefined, change over time, or broaden in
scope over time.
• Sometimes, intended outcomes require a
long time to materialize, but funding
guidelines require early evaluation of
programs or policies.
Formulating the Problem
2. Specifying Outcomes
• The response variable, or outcome, must be
clearly defined.
• Sometimes, outcomes are defined by the
guidelines of the funding agency.
• Ideally, definitions of outcomes are specified
prior to the implementation of the program
or policy being evaluated.
• But, things change….
Formulating the Problem
3. Measuring Experimental Contexts
• Obviously, to assess the efficacy of a program
or policy, one needs to know and be able to
measure its characteristics.
• Sometimes, characteristics are easy to
identify (e.g., hours of contact, labor hours,
funding, time, guidelines for behavior).
• In some cases, characteristics are more
difficult to identify (e.g., quality of contact,
expertise of labor, timing of funding,
flexibility in guidelines).
Formulating the Problem
4. Specifying Interventions
• Evaluation research often does not enjoy the
level of control available in a laboratory
experiment.
• Thus, specifying the independent variables, the
“interventions,” is not necessarily a
straightforward task.
• People participate differentially in programs.
• People come and go within programs.
• Program delivery varies over time and
space.
Formulating the Problem
5. Specifying the Population
• Specifying the participants in a program is not
always straightforward.
• People vary in the characteristics they bring
into a social change program.
• People vary in the extent to which they have
adopted and adapted to the desired
changes of the program.
Formulating the Problem
6. New versus Existing Measures
• Specifying new or existing measures affects
the validity and reliability of the evaluation.
• The use of new or existing measures also can
affect the extent of acceptance of an evaluation
by funding agencies, the public, and the
community of scholars.
• Standardized measures, often specified by
funding agencies, can have advantages and
disadvantages for evaluation of programs
and policies.
Formulating the Problem
7. Operationalizing Success and Failure
• Specifying what constitutes success or failure
can be challenging.
• How much change is success?
• What types of change are success?
• Are unanticipated changes success?
• When should success happen, immediately
or over a long period of time?
• Which measures indicate success?
• What happens when some measures
indicate success and others indicate failure?
The Social Context
1. Logistical Problems
• Evaluation research implies an assessment of
employee performance.
• Employees of organizations and agencies
being evaluated, therefore, often are reluctant
to reveal problems with a program or policy.
• Motivating personnel to participate fully in an
evaluation can be a challenge.
• Administrators, in particular, might feel
threatened by evaluation research.
• Administrators might hinder the quality of the
evaluation research.
The Social Context
2. Ethical Issues
• Evaluation research implies becoming involved
in the programs being conducted. Hence, the
evaluator might disturb the normal functioning
of the program.
• The results of an evaluation sometimes reveal
a need for immediate change to protect human
subjects. But the aims of the evaluation argue
for nonintervention to best complete the
evaluation.
The Social Context
3. Use of Research Results
• Evaluation research sometimes is funded with
the goal of applauding or discrediting a
program or policy.
• When the purposes are biased, then the quality
of the research is more likely to become
biased.
• When the results of evaluation research do not
support biased goals, then they might be
critiqued or squashed.
Social Indicators Research
1. Social Indicators
• Social indicators are aggregated statistics that
reflect various forms of societal well-being.
• Consumer price index
• Poverty levels
• Levels of illiteracy
• Infant mortality statistics
• Divorce rates
• Although such indicators provide only rough
approximations of societal health, they are part
of common practice.
Social Indicators Research
2. Computer Simulation
• High speed, large capacity computers allow for
complex simulations using many indicators of
societal conditions to forecast trends or predict
the outcome of suggested programs or
policies.
• Simulations are restricted by knowledge of
current technologies and conditions, which
might change dramatically over the course of
the simulation period.
Types of Evaluation Research Designs
1. Experimental Designs
• Typically, evaluation research involves
assessments of programs and policies in field
(i.e., natural) experiments.
• One does not have the level of control
available within the laboratory.
• Unless evaluation is planned within the context
of social change programs and policies, one
might not be able to conduct a classical
experiment.
Types of Evaluation Research Designs
2. Quasi-Experimental Designs
• Subjects are not randomly assigned to
experimental and control conditions.
• Assessments do not occur both at Time 1 and
Time 3 (i.e., pretest and posttest for all
subjects).
Types of Evaluation Research Designs
2. Quasi-Experimental Designs (Continued)
• Time-Series Designs: If time-series evaluations
do not involve classical experiments, it can be
challenging to infer an effect of the treatment.
• Consider this situation:
• An instructor introduces the use of
“controversial discussion topics” midway
through the semester, and then observes
the level of classroom participation.
• Which of the following patterns of classroom
participation support a treatment effect?
Types of Evaluation Research Designs
2. Quasi-Experimental Designs (Continued)
• Pattern One:
• Classroom participation is low at the
beginning of the semester, but steadily
increases at a constant rate throughout the
semester.
• Pattern Two:
• Classroom participation has a random
pattern of low and high levels of interaction
throughout the semester.
Types of Evaluation Research Designs
2. Quasi-Experimental Designs (Continued)
• Pattern Three:
• Classroom participation is low at the
beginning of the semester, but steadily
increases at a constant rate throughout the
semester.
Types of Evaluation Research Designs
2. Quasi-Experimental Designs (Continued)
• Time-Series Designs
• In observing Pattern 1, the researcher might
conclude that participation increases
throughout the semester, regardless of the
introduction of a treatment.
• In observing Pattern 2, the researcher might
conclude that participation is erratic and not
related to the introduction of a treatment.
• Pattern 3 indicates a treatment effect.
Types of Evaluation Research Designs
2. Quasi-Experimental Designs (Continued)
• Nonequivalent Control Groups
• Researchers seek naturally-occurring
control groups with similar characteristics to
the experimental group.
• Multiple Time-Series Designs
• Comparison of trends across naturallyoccurring groups, wherein one group
experiences some type of treatment effect.
Types of Evaluation Research Designs
3. Qualitative Evaluations
• Qualitative methods can be equally as effective
in evaluating programs and policies as are
quantitative methods.
• The most effective evaluation research often
uses both quantitative and qualitative methods.
Questions?