Download Continuous - rxsped596

Session 5: Causal-comparative & Correlational
Research
& Survey Methods

Today! NIH CITI training modules
◦

Moved back to August 5th : Annotated Bibliography
◦
◦

Make sure you do BASIC/REFRESHER course
Each individual reviews 3 research articles regarding their topic
See Example:
http://rxsped596.pbworks.com/w/file/fetch/54804527/Example%20Annotated%20Bibliogra
phy.pdf
Today! : Conceptual Framework
◦
◦
◦
◦
Group submits short summary of literature and presents a conceptual framework for
theories that drive their proposal. Group submits short summary of literature and presents a
conceptual framework for theories that drive their proposal.
Group composes a one page double-spaced document---can be used within your final
proposal
Use your articles from your Annotated Bibliography to form a framework for your research
proposal
Explain to the reader why should the intervention or the questions you are asking make
sense (based on the literature)

Cancelled: Review of Research Article

August 12th : Written Research Proposal (group assignment)

August 14th: Presentation of Research Proposal (group assignment)
Using Single Subject Research to
Establish “Evidence-based Practices”

A “practice” may be considered “evidence-based”
when:
 The practice is operationally defined, and implemented




with fidelity.
The outcomes associated with the practice are
operationally defined.
The context in which the practice in use is operationally
defined
Results from the single subject studies used to assess the
practice demonstrate experimental control.
The effects are replicated across 5 single subject studies
conducted in at least 3 locations, and with at least 20
different participants.
Dependent and independent variables

Dependent variable (DV) – the behavior
(measure) that you are analyzing



You want to produce change (variability) in the
dependent variable
Studies may have multiple DVs
Independent variable (IV) – the variable (event,
intervention, condition) that is of experimental
interest and that the researcher manipulates in an
experimental research design

Studies may have multiple IVs
Phase A
Phase B
Phase A
Immediacy of
Effect
Level
Phase B
Variability
Trend
Overlap
Research Question???
In SSD, a Functional
Relationship/Experimental Control
has occurred when

There are 3 demonstrations of an effect at 3
points in time.



Effect could be: change in trend or level
Also want to see immediacy of effect
Good research has at least 5 data points in
each phase to establish a consistent pattern in
the data.
Defining Features of
Multiple Baseline Designs
A multiple baseline design involves three or
more AB interventions (series) with phase
changes staggered across at least three points
in time.
 Key Features


Series are independent of each other
 People,
places, materials, behaviors/skills
The same IV is applied in each series
 Staggered implementation of IV

BL
Lollipop for R+
Treatment
6
100
80
60
Percentage of Correct Responding
40
20
Vivian
0
Lollipop for R+
100
80
60
40
20
Tammy
0
Lollipop for R+
100
80
60
40
20
Dr. Cathy
0
10
20
30
40
Sessions
50
60
70
Defining features of withdrawal and
reversal designs

Sequential phases of data collection involving the
implementation and withdrawal of an independent
variable(s)



within each phase, multiple data points are collected to establish
a representative pattern of behavior
phase change should occur only after stability of behavior within
the phase is established
traditionally, the first phase is Baseline, followed by
implementation of the IV (Intervention)

this is not required, however, as you may begin a study with an
intervention phase
4B
Baseline
FCT
Baseline
FCT
6
Total SIB per minute
5
4
3
2
1
0
1
5
10
15
Sessions
20
25
30
35
Alternating Treatment Designs
 Alternating Treatment Designs employ
rapid phase reversals across 2 or more
conditions to assess sensitivity of change in
the dependent variable to change in
condition.
Student 1
Hypothesis: Escape Math Work
Percent Intervals with Occurrence of Problem Behavior
100%
90%
80%
70%
2. Is Esc
different
than Attn?
Control Condition
Escape Condition
60%
Attention Condition
50%
IOA
1. Is Esc
different than
Control?
40%
30%
20%
10%
0%
1
2
3
Sessions
4
5
15
Descriptive Statistics
Who is in your data?
sample
population
Inferential Statistics
What your sample says
about the population
sample
population
Mean, Median, Mode,
standard deviation, variance
Prepared by M. Hara ([email protected])
Tests of significance
(t-, F-Tests)

Central Tendency
◦ Mean- average
◦ Median- midpoint in distribution of scores
◦ Mode- most frequent score

Variability
◦ Range- total extension of the data (e.g., 1-10)
◦ Standard Deviation- sum of deviations from the
mean squared. How well the mean summarizes the
data.
◦ Variance- standard deviation squared. Used in
sophisticated analyses
Inferential Statistics
What your sample says
about the population
sample
population
Prepared by M. Hara ([email protected])

T tests- used when have two groups to compare.
◦ Independent samples t- if groups are independent
 Different people in each group
◦ Dependent samples t-: if two sets of scores are available for the same
people
 Matched groups


ANOVA (analysis of variance)- when you have more than 2
groups to compare OR more than one independent variable
(reports an F-statistic, which is basically a t-value squared)
ANCOVA (analysis of covariance)- ANOVA that allows for control
of the influence of an IV (e.g., characteristics of people) that may
vary between your groups before treatment is introduced.
◦ Post-hoc method for matching groups on variables such as age, prior
education, SES, or a measure of performance

Statistical analyses to determine whether a
difference is statistically significant (probability
for result to occur by chance).

Yes or No answer

Alpha level (p=)
◦ An established probability level which serves as the
criterion to determine whether to accept or reject the
null hypothesis
◦ Common levels in education
 .01
 .05
 .10
Objectives 4.1 & 6.1
Addressing “WHAT” questions?
Depth of
Information
Representative, Generalizability
Details, Depth, and Variability
Quantitative Data
• Survey
Prepared by M. Hara ([email protected])
• Large Scale Assessments
Variable Type
Example
Nominal
• Gender
• Yes/No
Prepared by M. Hara ([email protected])
Male
(0)
Female
(1)
No
(0)
Yes
(1)
24
Check All that Apply
Q. Which of the following applications have you used with
your students? (Please check ALL that apply)
MS Word
iMovie
MS Excel
iDVD
MS PowerPoint
iTunes
SPSS
iWeb
Prepared by M. Hara ([email protected])
25
Variable Type
Example
Nominal
• Gender
• Yes/No
Ordinal
(0)
Female
(1)
No
(0)
Yes
(1)
• Likert-scale
Strongly
Disagree
(1)
Prepared by M. Hara ([email protected])
Male
Disagree
(2)
Agree
(3)
Strongly
Agree
(4)
26
Likert-scale
Strongly
Somewhat Somewhat
Disagree
Agree
Disagree
Disagree Agree
Strongly
Agree
Q1. Overall, I have a good Parentteacher Relationship.
Prepared by M. Hara ([email protected])
27
Likert-scale
Strongly
Somewhat Somewhat
Disagree
Agree
Disagree
Disagree Agree
Q1. Overall, I have a good Parentteacher Relationship.
Strongly
Agree
1 2 3 4 5 6
NOTE: Distinguishable
Prepared by M. Hara ([email protected])
28
Multiple Categories
Ethnicity
African American
Asian
Caucasian
Hispanic
Other ____________
Education Completed
High School
Some College
BA/BS
Master’s
Doctoral
Decline to state
Prepared by M. Hara ([email protected])
29
Multiple Categories
Ethnicity
NOTE: Mutually Exclusive, Exhaustive,
and Distinguishable
African American
1
Asian
2
Caucasian
3
Hispanic
4
Other ____________ 5
Education Completed
High School
1
Some College
2
BA/BS
3
Master’s
4
Doctoral
5
Decline to state
Prepared by M. Hara ([email protected])
888
30
Variable Type
Example
Nominal
• Gender
• Yes/No
Ordinal
(0)
Female
(1)
No
(0)
Yes
(1)
• Likert-scale
Strongly
Disagree
(1)
Scale
Male
Disagree
(2)
Agree
(3)
Strongly
Agree
(4)
• Age, Annual Income, Test-score
(Interval/Ratio)
31
Numbers with Different Meaning
Example
Nominal
• Gender
Numerical/
Continuous
Categorical
Variable Type
• Yes/No
Ordinal
(0)
Female
(1)
No
(0)
Yes
(1)
• Likert-scale
Strongly
Disagree
(1)
Scale
Male
Disagree
(2)
Agree
(3)
Strongly
Agree
(4)
• Age, Annual Income, Test-score
(Interval/Ratio)
Prepared by M. Hara ([email protected])
32
Numbers with Different Meaning
Example
Nominal
• Gender
Numerical/
Continuous
Categorical
Variable Type
• Yes/No
Ordinal
(0)
Female
(1)
No
(0)
Yes
(1)
• Likert-scale
Strongly
Disagree
(1)
Scale
Male
Disagree
(2)
Agree
(3)
Strongly
Agree
(4)
• Age, Annual Income, Test-score
(Interval/Ratio)
Prepared by M. Hara ([email protected])
33
Numbers with Different Meaning
Example
Nominal
• Gender
Numerical/
Continuous
Categorical
Variable Type
• Yes/No
Ordinal
(0)
Female
(1)
No
(0)
Yes
(1)
• Likert-scale
Strongly
Disagree
(1)
Scale
Male
Disagree
(2)
Agree
(3)
Strongly
Agree
(4)
• Age, Annual Income, Test-score
(Interval/Ratio)
Prepared by M. Hara ([email protected])
34
Variable Types and Analysis
Dependent
Variable
(a.k.a., Outcome)
Prepared by M. Hara ([email protected])
Is there
an association?
Independent
Variable
(a.k.a., Predictor)
35
Variable Types and Analysis
Dependent
Variable
(a.k.a., Outcome)
Is there
an association?
Independent
Variable
(a.k.a., Predictor)
Where differences
culminate
Prepared by M. Hara ([email protected])
36
Variable Types and Analysis
Dependent
Variable
(a.k.a., Outcome)
Where differences
culminate
Prepared by M. Hara ([email protected])
Is there
an association?
Independent
Variable
(a.k.a., Predictor,
Intervention)
Contributing
Factors
37
Variable Types and Analysis
Dependent
Variable
Is there
an association?
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
38
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Chi-square test
Or χ²-test
Categorical
Numerical/
Continuous
Prepared by M. Hara ([email protected])
Contingency Tables
(a.k.a. Cross-tabs)
Categorical
Numerical/
Continuous
39
Variable Types and Analysis
Dependent
Variable
Categorical
Annual
Salary
Prepared by M. Hara ([email protected])
Independent
Variable
Contingency Tables
(a.k.a. Cross-tabs)
Categorical
Gender
40
Variable Types and Analysis
Dependent
Variable
Categorical
Independent
Variable
Contingency Tables
(a.k.a. Cross-tabs)
Categorical
Gender
Male
Female
12%
4%
26K - 35K
18%
6%
36K - 45K
24%
11%
46K - 55K
36%
39%
56K - 65K
8%
28%
66K and up
2%
12%
Annual Salary
25K or below
Prepared by M. Hara ([email protected])
41
Your Research Topic???
Dependent
Variable
Categorical
Independent
Variable
Contingency Tables
(a.k.a. Cross-tabs)
Categorical
Annual
Salary
Gender
Your DV?
Your IV?
Prepared by M. Hara ([email protected])
42
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
43
Variable Types and Analysis
Dependent
Variable
t-test or F-test
Numerical/
Continuous
Analysis of Variance
(a.k.a. ANOVA)
0.45
Males
0.45
0.05
0
0
-0
-0
-0
-0
-1
-1
-1
-1
-1
-2
-2
-2
-2
-2
-3
Prepared by M. Hara ([email protected])
Categorical
Females
SAT9 Math Score
.2
0.
0
0.
2
0.
4
-30
.06
-20
.8
-21
.60
-21
.42
-21
.24
-21
.06
-1
.8
-12
.60
-12
.42
-12
.24
-12
.06
-02
.8
-03
.60
-0
.4
-0
.2
0.
0
0.
2
0.
4
0.
6
0.
8
1.
0
1.
2
1.
4
1.
6
1.
8
2.
0
2.
2
2.
4
2.
6
2.
8
3.
0
0.05
.4
0.1
.6
0.1
.8
0.15
.0
0.15
.2
0.2
.4
0.2
.6
0.25
.8
0.25
.0
0.3
.2
0.3
.4
0.35
.6
0.35
.8
0.4
.0
0.4
Independent
Variable
44
Your Research Topic??
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
45
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Prepared by M. Hara ([email protected])
Regression
Numerical/
Continuous
46
Variable Types and Analysis
Dependent
Variable
Numerical/
Continuous
Independent
Variable
Regression
Numerical/
Continuous
SAT9 Math Score
Household Income
Prepared by M. Hara ([email protected])
47
Your Research Topic??
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Prepared by M. Hara ([email protected])
Regression
Numerical/
Continuous
48
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
49
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Logistic Regression
Categorical
Numerical/
Continuous
Pass
High school
Exit Exam
Probability of
passing h.s. exam
based on SAT-9
score
Fail
SAT 9 Math
Prepared by M. Hara ([email protected])
50
Your Research Topic??
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
51
Numbers with Different Meaning
Example
Nominal
• Gender
Numerical/
Continuous
Categorical
Variable Type
• Yes/No
Ordinal
(0)
Female
(1)
No
(0)
Yes
(1)
• Likert-scale
Strongly
Disagree
(1)
Scale
Male
Disagree
(2)
Agree
(3)
Strongly
Agree
(4)
• Age, Annual Income, Test-score
(Interval/Ratio)
Prepared by M. Hara ([email protected])
52
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Chi-square test
Or χ²-test
Categorical
Numerical/
Continuous
Prepared by M. Hara ([email protected])
Contingency Tables
(a.k.a. Cross-tabs)
Categorical
Numerical/
Continuous
53
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
54
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Prepared by M. Hara ([email protected])
Regression
Numerical/
Continuous
55
Variable Types and Analysis
Dependent
Variable
Independent
Variable
Categorical
Categorical
Numerical/
Continuous
Numerical/
Continuous
Prepared by M. Hara ([email protected])
56

Statistics that function to describe the
strength and direction of a relationship
between two or more variables
◦ Simple correlation coefficient (r=)
◦ Coefficient Determination- (r-squared). Amount of
variance that is accounted for by the explanatory
(independent or predictor) variable in the response
variable (criterion variable).
◦ Multiple regression- to indicate the mount of
variance that all of the predictor variables explain.

Way of quantifying the difference between two
groups.

Not just was there an effect, but the magnitude
of the effect.

Many ways to calculate

ES= [Mean of experimental group] – [Mean of control
group]/Standard Deviation

R-squared, Cohens-D

Standard deviation is how well the mean summarizes the data

Investigators attempt to determine the cause of

Describes conditions that already exist (a.k.a.
ex post facto).


differences that already exist between or
among groups of individuals.
The group difference variable is either a
variable that cannot be manipulated or one that
might have been manipulated but for one
reason or another, has not been.
Studies in medicine and sociology are causalcomparative in nature, as are studies of
differences between men and women.
Similarities and Differences Between CausalComparative and
Correlational Research
• Similarities
– Ex Post Facto research
– Attempt to explain
phenomena of interest
– Seek to identify variables
that are worthy of later
exploration through
experimental research
– Neither permits the
manipulation of variables
– Attempt to explore
causation
• Differences
– Causal studies compare
two or more groups of
subjects
– Causal studies involve at
least one categorical
variable
– Causal studies often
compare averages or use
tables instead of
scatterplots and correlation
coefficients
Similarities and Differences Between CausalComparative and Experimental Research
• Similarities
– Require at least one
categorical variable
– Both compare group
performances to determine
relationships
– Both compare separate
groups of subjects
• Differences
– In experimental research, the
independent variable is
manipulated
– Causal studies are likely to
provide much weaker
evidence for causation
– In experimental studies,
researchers can assign
subjects to treatment groups
– The researcher has greater
flexibility in formulating the
structure of the design in
experimental research

Problem Formulation

Sample

Instrumentation

Design
 The first step is to identify and define the particular phenomena
of interest and consider possible causes
 Selection of the sample of individuals to be studied by carefully
identifying the characteristics of select groups
 There are no limits on the types of instruments that are used in
Causal-comparative studies
 The basic design involves selecting two or more groups that
differ on a particular variable of interest and comparing them on
another variable(s) without manipulation
The Basic Causal-Comparative Designs
Independent
variable
Dependent
variable
I
C
(Group possesses
characteristic)
O
(Measurement)
II
–C
(Group does
not possess
characteristic)
O
(Measurement)
I
C1
(Group possesses
characteristic 1)
O
(Measurement)
II
C2
(Group possesses
characteristic 2)
O
(Measurement)
Group
(a)
(b)

Subject Characteristics
 The possibility exists that the groups are not equivalent
on one or more important variables
 One way to control for an extraneous variable is to match
subjects from the comparison groups on that variable
 Creating or finding homogeneous subgroups would be
another way to control for an extraneous variable
 The third way to control for an extraneous variable is to
use the technique of statistical matching
Other Threats
•
•
•
•
•
Loss of subjects
Location
Instrumentation
History
Maturation
• Data collector bias
• Attitude

In a Causal-Comparative Study, the first step is to
construct frequency charts & graphs.

Means and SD are usually calculated if the variables
involved are quantitative.

The most commonly used inference test is a t-test
for differences between means.

ANCOVAs are useful for these types of studies.

Results should always be interpreted with caution
since they do not prove cause and effect.


PSU uses a free web-based survey software
called Qualtrics (available for student use)
http://oit.pdx.edu/node/908



Is there another way of collecting the
information?
Is this method the most efficient and cost
effective?
Will it provide you with the information you
want in a valid manner?






Have specific goals for the survey
Consider alternatives to using a survey to
collect information.
Select samples that well represent the
population studied.
Use designs that balance costs with errors.
Take great care in matching wording to
concepts being measured and the population
being studied
Pretest questionnaires and procedures to
identify problems prior to the survey




Construct quality checks for each stage of the
survey
Maximize cooperation or response rates
within the limits of ethical treatment.
Carefully develop and fulfill pledges of
confidentiality given to respondents.
Disclose all methods of the survey to permit
evaluation and replication
Step
Step
Step
Step
Step
Step
Step
1234567-
Determine Purpose
Identify a Sampling Plan & Mode
Design survey instrument
Test survey instrument
Send out a letter of transmittal
Deliver the survey
Analyze data from survey



State specific objectives
Consider the types of information needed
Use your purpose to guide each of your other
steps and check back to this purpose often!
◦ To make sure you are answering research questions




Simple descriptive- one-shot to describe
moment in time
Cross-sectional- several groups at one point in
time (e.g., 1st, 3rd, 5th grade students)
Longitudinal- cohort at different points (e.g.,
year 1, 2…after leaving school
Strengths and limitations of each??

Identify respondents (based on population you want
to sample)
◦ Think external validity…way you will generalize your
results.

probability or nonprobability methods
◦ Probability- each person has a probability of being
surveyed.
 random sampling, systematic sampling, and stratified sampling
◦ Nonprobability methods
 convenience sampling, judgment sampling, quota sampling,
and snowball sampling

Based on what know about population decide how
collect data:
◦ Email, Web-based surveys, mail, telephone, personal
interviews, etc.



Random sampling- Each member of the population has an
equal and known chance of being selected
Systematic sampling- called an Nth name selection
technique
Stratified sampling- (1) Identify a subset of the population
that share at least one common characteristic (males &
females; managers & non-managers) and (2) their actual
representation in the population, then (3) random
sampling is used to select a sufficient number of subjects
from each stratum.

Convenience sampling- selected because they are convenient.




often used during preliminary research efforts to get a gross estimate of
the results, without incurring the cost or time required to select a random
sample.
Judgment sampling- sample based on judgment. For example, draw
the entire sample from one "representative" city, even though the
population includes all cities. When using this method, the
researcher must be confident that the chosen sample is truly
representative of the entire population.
Quota sampling- like stratified sampling: (1) identify like
characteristics and percentage of population, (2) convenience or
judgment to select participants to represent characteristics and meet
quota to represent population
Snowball sampling- Snowball sampling relies on referrals from
initial subjects to generate additional subjects.










Review the literature
There may be a survey that matches what you need (then
use and cite OR modify and cite)
Determine question format
◦ Open-ended, close-ended, likert scale (0-5; never, sometimes,
always)
Avoid sensitive questions
Be very clear
Short items are better
Avoid negative wording OR use positive wording
Avoid asking about more than one idea…stay away from
AND…make another question to ask for additional
information
Avoid jargon/big words
Emphasize …. (underline) critical words



Test instrument on yourself and others (not
part of your sample!)
Pilot test to similar population.
Use feedback to improve administration of
survey


Cover letter for mail surveys or as
introductory letter/email for a phone, web,
email, or personal interview survey.
Increases participation and limits incomplete
surveys





Deliver the survey according to timeline
Web-based or email surveys may have a start
date and date for completion by.
Send follow up emails/letter/calls
Monitor data as they come in to check for
errors----sooner than later.
Send thank you emails/letters/calls

Follow up with non-respondents so you have
some ideas why there was no response
◦ You can then compare respondents vs nonrespondents





Decide how to handle missing data
Complete descriptive statistics: frequency,
percentages, mean, median, etc.
Look for interesting patterns in the data
Do sub-group analyses if possible
Display data in tables & graphs


Get together with your research group
Based on your overall topic and possible
research questions...outline survey methods
that you might give to participants or other
stakeholders for the purpose of your
research.