Download SS - Piazza

EXPERIMENTAL METHODS IN
PSYCHOLINGUISTIC RESEARCH
SUMMER SEMESTER 2015
RESEARCH CYCLE
Course content
STEPS IN
EXPERIMENTAL DESIGN
Define the research hypothesis and make it testable
•  IV and DV?
Choose between within- and between-subject designs
•  Same/different groups of participants in different conditions
Control extraneous variables
•  Randomly assign participants to conditions
•  Balance conditions in terms of linguistic variables
INTERNAL AND
EXTERNAL VALIDITY
Internal validity
The extent to which the observed changes can be attributed to the
manipulation and not to other possible causes
External validity
The extent to which the results of the experiment can be
generalized to people/situations/stimuli different from those in the
experiment
POPULATION VERSUS
SAMPLE
SAMPLING IN
PSYCHOLINGUISTICS
Non-probability sampling (convenience/volunteer samples)
•  Treated as random samples (importance of replication)
In psycholinguistic experiments we sample from:
1)  Speakers (analyses by Subjects)
2)  Language (analyses by Items)
• 
e.g., “Transformation” stories / specific nouns used in the
stories
INFERENTIAL STATISTICS
Probability distribution
Specifies the likelihood (probability, proportion) of the different
values of a random variable
Known probability distributions:
•  Normal
•  z
•  t
•  F
•  χ2
T DISTRIBUTION
LOGIC OF A T-TEST
Participant
N-Tknowledgeable
N-Tnaive
1
312
325
2
365
356
3
200
224
4
324
388
5
356
412
6
326
378
7
279
299
…
…
…
20
323
340
x
320
350
s
48
55
0
30
Is area under the curve which starts at 30 < α ?
STATISTICAL TESTS
Statistical tests compare two general types of variability in
the data:
a) Systematic variance (something you can explain)
the result of some identifiable factor (either the variable of interest
or some factor that you’ve failed to control adequately)
•  e.g., mean difference of 30 ms in the two conditions
b) Error variance (something you cannot explain)
nonsystematic variability due to individual differences within and
between groups and any number of random, unpredictable effects
• 
e.g., variability of individual scores around the respective
means within the two conditions
POSSIBLE OUTCOMES
Classic inferential statistical analyses yield two results:
Reject H0 => the difference is not very likely if H0 is true
(hence it is probably not true) so you believe that an effect
truly happened in your study and that the results can be
generalized !
•  You find a significant result
Fail to reject H0 => the difference is quite likely even due to
random variability so you should not conclude that the
results can be generalized
•  You find a null result
CHOOSING STATISTICAL TEST(S)
What kind of statistical test should be used (e.g., t-test,
ANOVA, χ2) depends on:
•  The type of data / DV (e.g., categorical vs. continuous)
•  The assumptions of a test (e.g., underlying distributions)
and whether your data is likely to meet those assumptions
•  Number of IVs
•  Whether the design is between- or within-subjects
ONE-WAY ANOVA
Tests whether means of obtained scores in more than two
groups deviate significantly from each other.!
!
Appropriate when (assumptions):!
-  Individual scores are independent"
-  Scores in the groups/categories are independent"
-  Means of the groups/categories come from the normally
distributed sampling distribution of the mean (aka Normality)"
-  Scores in the groups/categories come from the populations
where scores have equal variability (aka Homogeneity of
Variances / Homoscedasticity assumption)"
!
ONE-WAY ANOVA
dfnumerator = 3, dfdenominator=100
dfnumerator = 3, dfdenominator=10
dfnumerator = 3, dfdenominator=5
df numerator / treatment = k −1
df deno min ator / error = n total − k
k = number of categories /groups
n total = total number of scores
€
When ANOVA assumptions are met, F distribution allows to obtain probability
for observed or higher F value. !
!
The shape of F distribution (and probability) will depend on the number of
groups/categories and total number of scores.!
ONE-WAY ANOVA
dfnumerator = 10, dfdenominator=100
dfnumerator = 5, dfdenominator=100
df numerator / treatment = k −1
df deno min ator / error = n total − k
dfnumerator = 3, dfdenominator=100
k = number of categories /groups
n total = total number of scores
€
When ANOVA assumptions are met, F distribution allows to obtain probability
for observed or higher F value. !
!
The shape of F distribution (and probability) will depend on the number of
groups/categories and total number of scores.!
ONE-WAY ANOVA
SSTotal = SSTreatment + SSError
∑n
F=
2
*
(x
−
x
)
in a group
group
G
k −1
∑ (x i − x group )2
n total − k
Systematic variance
(something you can explain)
SSTreatment / Numerator
dfTreatment / Numerator
Error variance
(something you cannot explain)
€
SSError / Re sidual / Deno min ator
df Error / Re sidual / Deno min ator
k = number of categories / groups
ntotal = total number of scores
€
xG = grand mean
nin a group = number of scores in each group
xgroup = mean of the group the score comes from
xi = individual score
ONE-WAY ANOVA
HYPOTHETICAL DATA
Item
NonTransfor
med
Transformed
1
312
325
2
365
356
3
200
224
4
324
388
5
356
412
6
326
378
7
279
299
…
…
…
12
323
340
x
315
365
s
48
55
Procedure:!
!
Step 1: state the null and alternative
hypotheses"
"
Step 2: calculate test statistic (F),
associated p value, and effect size"
"
Step 3: if more than 2 groups, conduct
follow-up pairwise comparisons with p
values corrected for multiple
comparisons (e.g., Bonferroni)"
"
Step 4: make a conclusion / describe
the results"
ONE-WAY ANOVA
HYPOTHETICAL DATA
Step 1: state the null and alternative hypotheses:"
H0: the reading times are equal across different noun
types"
"(µN-T = µT)"
HA: the reading times are different"
ONE-WAY ANOVA
HYPOTHETICAL DATA
Step 2: calculate test statistic (F) and associated p value"
"
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
wid = ItemIndex, # Item index column
!
! between = .(NounType), # IV
!
type = 3, # Type of Sum of Squares (SS)!
!
! detailed = TRUE) # Provides detailed output!
!
ANOVAmodel$ANOVA!
Effect
DFn
(Intercept) 1
NounType
1
DFd SSn
SSd
F
p
22 2693151.2 45992.4 1288.24 <0.001
22
25872.0 45992.4
12.38 0.002
p<.05
*
*
ges!
0.98!
0.36!
ONE-WAY ANOVA
HYPOTHETICAL DATA
Step 2: calculate effect size (e.g., η2 – eta-squared)"
!
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
2
wid = Id, # Subject index column/variable!
Treatment
between = .(NounType), # IV
!
Totaltype = 3, # Type of Sum of Squares (SS)!
!
! detailed = TRUE) # Provides detailed output!
!
ANOVAmodel$ANOVA!
SS
η =
SS
Effect
DFn
(Intercept) 1
NounType
1
25872.0
=
= 0.36
25872.0 + 45992.4
DFd SSn
SSd
F
p
22 2693151.2 45992.4 1288.24 <0.001
22
25872.0 45992.4
12.38 0.002
p<.05
*
*
ges!
0.98!
0.36!
ONE-WAY ANOVA
HYPOTHETICAL DATA
Step 4: make a conclusion / describe the results"
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
ezStats(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
wid = ItemIndex, # Item index column!
between = .(NounType)) # IV(s)
!
NounType N
Mean
SD
FLSD!
NT
12
315
48
38.7!
T
12
365
55
38.7!
REPEATED MEASURES ANOVA
Tests whether means of obtained scores in more than
two groups deviate significantly from each other. !
!!
Appropriate when (assumptions):!
-  Scores in the groups/categories are matched:"
-  e.g., come from the same individual (same person tested more
than 2 times)"
-  Sets of related scores are independent of each other. "
-  All difference scores (difference scores between each
group) have the same variability (variances) (aka Sphericity
assumption). "
-  All means of difference scores come from the normally
distributed sampling distributions of the mean (Normality)."
!
REPEATED MEASURES ANOVA
SSTotal = SSTreatment + SSError + SSSubject
∑n
in a group
F=
* (xgroup − xG )
2
k −1
2
(x
−
x
−
(x
−
x
))
∑ i group subject group
ntotal − k − (nsubjects −1)
MSerror
€
Systematic variance
(something you can explain)
SSTreatment / Numerator
dfTreatment / Numerator
Error variance
(something you cannot explain)
SSError / Re sidual / Deno min ator
df Error / Re sidual / Deno min ator
k = number of categories / groups
ntotal = total number of scores
xG = grand mean
€
n
in a group
= number of scores in each group
xgroup = mean of the group the score comes from
xi = individual score
REPEATED MEASURES ANOVA
HYPOTHETICAL DATA
Participant
N-Tknowledgeable
N-Tnaive
1
312
325
2
365
356
3
200
224
4
324
388
5
356
412
6
326
378
7
279
299
…
…
…
16
323
340
x
320
350
s
43
38
Procedure:!
!
Step 1: state the null and alternative
hypotheses"
"
Step 2: calculate test statistic (F),
associated p value, and effect size"
"
Step 3: if more than 2 groups, conduct
follow-up pairwise comparisons with p
values corrected for multiple
comparisons (e.g., Bonferroni)"
"
Step 4: make a conclusion / describe
the results"
REPEATED MEASURES ANOVA
HYPOTHETICAL DATA
Step 1: state the null and alternative hypotheses:"
H0: the reading times are equal for the different types of
passages"
"(µN-T knowledgeable = µN-T naive)"
HA: the reading times are different"
REPEATED MEASURES ANOVA
HYPOTHETICAL DATA
Step 2: calculate test statistic (F) and associated p value"
"
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
wid = Id, # Subject index column/variable!
within = .(Knowledge), # IV
!
type = 3, # Type of Sum of Squares (SS)!
!
! detailed = TRUE) # Provides detailed output!
!
ANOVAmodel$ANOVA!
Effect
DFn
(Intercept) 1
Knowledge
1
DFd SSn
SSd
F
15 3373454.4 13966.7 3623.03
15
28815.5 34664.3
12.47
p
<0.001
0.003
p<.05
*
*
ges!
0.99!
0.37!
REPEATED MEASURES ANOVA
HYPOTHETICAL DATA
Step 2: calculate effect size (e.g., η2p – partial-eta-squared)"
!
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
2
wid = Id, # Subject index column/variable!
Treatment
P
between = .(NounType), # IV
!
type = Error
3, # Type of Sum of Squares (SS)!
Treatment
!
! detailed = TRUE) # Provides detailed output!
!
ANOVAmodel$ANOVA!
η =
SS
SS
Effect
DFn
(Intercept) 1
Knowledge
1
+ SS
28815.5
=
= 0.45
28815.5 + 34664.3
DFd SSn
SSd
F
15 3373454.4 13966.7 3623.03
15
28815.5 34664.3
12.47
p
<0.001
0.003
p<.05
*
*
ges!
0.99!
0.37!
REPEATED MEASURES ANOVA
HYPOTHETICAL DATA
Step 4: make a conclusion / describe the results"
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
ezStats(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
wid = Id, # Subject index column / variable!
within = .(Knowledge)) # IV(s)
!
Knowledge N
Mean
SD
FLSD!
NTK
16
320
43
28!
NTN
16
350
38
28!
MIXED-FACTORIAL ANOVA
Tests whether:!
•  means of scores in two or more groups based on IV1 (aka Factor 1) deviate
significantly from each other - main effect of IV1 (H01)!
•  means of scores in two or more groups based on IV2 (aka Factor 2) deviate
significantly from each other - main effect of IV2 (H02)!
•  there is an interaction between IV1 and IV2 (H03) / whether the effect of IV1
depends on the level of IV2 / whether the effect of IV2 depends on the level
of IV1"
Appropriate when (assumptions):!
-  Scores in different levels of one of the IVs/factors (aka within-subject factor) are
related"
-  Scores in different levels of one of the IVs/factors (aka between-subjects factor)
are independent"
-  Sphericity and Normality of the difference scores holds for the within-subjects
factor"
- Homogeneity of Variance and Normality holds for the between-subjects factor"
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Item
NT-K
NT-N
T-K
T-N
1
312
315
325
324
2
365
385
356
356
3
200
300
315
326
4
324
324
385
279
5
356
498
300
412
6
326
326
324
378
7
279
279
299
301
…
…
…
…
…
12
323
380
310
330
x
320
415
340
310
s
43
50
38
35
MIXED-FACTORIAL ANOVA
Knowledge"
K"
T"
cellT-K
N"
cellT-N
σ1,1"
Homogeneity of Variance:!
xT"
Noun!
Type"
σ1,2"
σ1,1 = σ1,2"
H01: µT= µNT!
NT"
cellNT-K
cellNT-N
x K"
x N"
xNT"
H02: µK= µN!
H03: µT-K - µNT-K = µT-N - µNT-N / µT-K - µT-N = µNT-K - µNT-N!
!
Sphericity: when there are only two "
levels of within-subjects factor, there is "
only one set of difference scores"
MIXED-FACTORIAL ANOVA
SSTotal = SSIV1 + SSError −Between + SSIV 2 + SSIV1*IV 2 + SSError−Within
Systematic variance
(something you can explain)
∑ nin each IV1
F1 =
∑ (x
i − x IV1
level * (x IV1
level
− xG ) 2
kIV1 −1
level − (x Subject − x IV1
SSIV1
df IV1
2
))
level
n Subjects − k IV1
€
MSError-Between
€
Error variance
(something you
cannot explain)
SSError−Between
df Error −Between
MIXED-FACTORIAL ANOVA
SSTotal = SSIV1 + SSError −Between + SSIV 2 + SSIV1*IV 2 + SSError−Within
∑n
F2 =
n total
in each IV 2 level
* (x IV 2
level
SSIV 2
df IV 2
− xG ) 2
k IV 2 −1
SSError−Within
− (n Subjects −1) − (kIV 2 −1) − (k IV1 −1) * (k IV 2 −1) −1
SSError−Within = SSTotal − SSIV1 − SSError−Between
df Error −Within
€
− SSIV 2 − SSIV1*IV 2
€
∑n
F3 =
n total
cell
* (x cell − x IV1
level
− x IV 2
level
)
(kIV1 −1) * (k IV 2 −1)
SSError−Within
− (n Subjects −1) − (kIV 2 −1) − (k IV1 −1) * (k IV 2 −1) −1
MSError-Within
€
SSIV1*IV 2
df IV1*IV 2
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Procedure:!
!
Step 1: state the null hypotheses"
"
Step 2: calculate test statistics (F), associated p values, and effect sizes"
"
Step 3: If main effects (for IV’s with more than 2 categoties/levels/groups) are
significant, conduct follow up pairwise comparisons (control for Family Wise
Type I error)"
"
Step 4: If interaction is significant, conduct follow-up simple effect tests and/or
pairwise comparisons (control for Family Wise Type I error)"
!
Step 5: make a conclusion / describe the results"
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 2: state the null hypotheses"
"
Main effect of Noun Type:!
Reading times of transformed and non-transformed nouns are equal in a
population"
H01: µT= µNT!
!
Main effect of Knowledge State:!
Reading times of nouns when they are uttered by naïve and knowledgeable story
characters are equal in a population!
H02: µK= µN!
!
Interaction between Noun Type and Knowledge State:!
•  Reading times of nouns when they are uttered by naïve and knowledgeable
story characters do not change differently for transformed and non-transformed
nouns in a population !
•  Reading times of transformed and non-transformed nouns do not change
differently when they are uttered by naïve and knowledgeable story characters
in a population.!
H03: µT-K - µNT-K = µT-N - µNT-N / µT-K - µT-N = µNT-K - µNT-N!
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 2: calculate test statistic (F) and associated p value"
"
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
dv=ReadingTime, # DV!
wid = ItemIndex, !
within = .(Knowledge), # within-subjects IVs
between = .(NounType), # between-subjects IVs!
!
! type = 3, # Type of Sum of Squares (SS)!
!
! detailed = TRUE) # Provides detailed output!
!
ANOVAmodel$ANOVA!
Effect
DFn
(Intercept)
1
NounType
1
Knowledge
1
NounType:Knowledge 1
DFd
SSn
SSd
F
p
p<.05 ges!
22 5904100 29776 4362.18 <0.001
*
0.99!
22
23527 29776
17.38 <0.001
*
0.27 !
22
5660 34169
3.64 0.069
0.08!
22
71431 34169
45.99 <0.001
*
0.53!
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 2: calculate effect size (e.g., η2p – partial-eta-squared)"
!
SS=NounType
23527
2
ANOVAmodel
ezANOVA(data=ex.d,
# data =frame/set!
ηP =
=
0.44
dv=ReadingTime,
DV!
SSNounType + SSError−Between
23527 +# 29776
wid = Id, # Subject index column/variable!
SSKnowledge
5660
between = .(NounType),
# IV
!
2
ηP =
=
=
0.14
= 3, # Type of Sum of Squares (SS)!
SSKnowledge + SStype
5660 + 34169
!
! Error−Within
detailed = TRUE) # Provides detailed output!
!2
SSNounType:Knowledge
71431
η!P =
=
= 0.68
71431+ 34169
! SSNounType:Knowledge + SSError−Within
ANOVAmodel$ANOVA!
Effect
DFn
(Intercept)
1
NounType
1
Knowledge
1
NounType:Knowledge 1
DFd
SSn
SSd
F
p
p<.05 ges!
22 5904100 29776 4362.18 <0.001
*
0.99!
22
23527 29776
17.38 <0.001
*
0.27 !
22
5660 34169
3.64 0.069
0.08!
22
71431 34169
45.99 <0.001
*
0.53!
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 3: If interaction is significant, conduct follow-up simple
effect tests and/or pairwise comparisons (control for Family
Wise Type I error)"
440
!
ezPlot(data=ex.d, !
dv=ReadingTime, !
400
wid = ItemIndex, !
between = .(NounType),!
within = .(Knowledge), !360
x=Knowledge,!
split=NounType)!
320
!
!
280
Mean
IV1
NT
T
K
N
Knowledge
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 3: If interaction is significant, conduct follow-up simple
effect tests and/or pairwise comparisons (control for Family
Wise Type I error)"
!
t.test(ReadingTime~NounType, !
!data = subset(ex.d, Knowledge == “K”), !
!paired = F, !
!alternative = ”two.sided", !
440
!var.equal = T,!
n.s
!mu = 0)!
400
!!
!Two Sample t-test!
360
!
320
t = -0.3024, df = 22, p-value = 0.7652!
Mean
IV1
NT
T
!
!
!
280
K
N
Knowledge
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 3: If interaction is significant, conduct follow-up simple
effect tests and/or pairwise comparisons (control for Family
Wise Type I error)"
!
Mean
t.test(ReadingTime~NounType, !
!data = subset(ex.d, Knowledge == “N”), !
!paired = F, !
!alternative = ”two.sided", !
440
!var.equal = T,!
!mu = 0)!
400
!!
!Two Sample t-test!
360
*
!
320
t = 6.4936, df = 22, p-value = 1.564e-06!
!
280
K
N
!
Knowledge
IV1
NT
T
MIXED-FACTORIAL ANOVA
HYPOTHETICAL DATA
Step 4: make a conclusion / describe the results"
ANOVAmodel = ezANOVA(data=ex.d, # data frame/set!
ezStats(data=ex.d, !
dv=ReadingTime, !
wid = ItemIndex, !
within = .(Knowledge),!
between = .(NounType))
!
NounType Knowledge N
Mean
SD FLSD!
NT
K
12
320
43 33.36648!
NT
N
12
415
50 33.36648!
T
K
12
340
38 33.36648!
T
N
12
310
35 33.36648!