Download Confidence Intervals

Confidence Intervals
Studies to Review
• Loftus and Masson(1994)
• Masson and Loftus(2003)
• Jarmasz and Hollands(2009)
Using Confidence Intervals in
within-subject designs
Loftus and Masson(1994)
Intro
• Bayesian Technique
• Null Hypothesis and Significance Testing
• Competing Hypotheses
• Using Confidence Intervals
Confidence Intervals
• “How well does an observed pattern of sample means represent the
underlying pattern of population means?”
• Confidence intervals have been argued to replace formal stat analysis
• Hypothesis testing is designed to address a restricted, convoluted, and usually
uninteresting question
• A finding of statistical significance implies that the experiment has enough
statistical power
Confidence Intervals
• Confidence intervals can provide
• An initial estimate and intuitive assessment
• The best estimate of the underlying pattern of population means
• The degree to which the observed pattern should be taken seriously
Confidence Intervals for Within Subject
Design
• Current textbooks say that CIs are used for between subject designs
• A CI used for between subject design has two useful properties
• Determined by the same error term as an ANOVA
• A confidence interval around a sample mean and one around the difference
between 2 means are related by a factor of sq(2^3)
• Confidence in patterns of means can be judged on the basis of
confidence intervals plotted around the individual means
A Hypothetical Experiment
• Measures the effects of study time on free recall
• Participants are presented a list of 20 words
• 1, 2, or 5s per word
• Interested in the relationship between study time and words recalled
Between-Subject Data
• Analyzing data as if it were between subjects
• N=30, 10 subjects per group in 3 groups
• Each group participates in 1 of the 3 time conditions with recall
recorded
Between Subject Confidence Intervals
Within-Subject Data
• Same study ran with 10 subjects
• Each subject ran through each condition
Creating a Confidence Interval
• Ex. Creating a confidence interval for the 1s condition
• Researchers would need to create a CI that has the same inter-subject
variability as the between-subject error variance
• This would cause the CI to give a different impression than the within-subject
ANOVA
• This error occurs because inter-subject variance is irrelevant to a within
subject ANOVA, but decides the size of a confidence interval
• Generally speaking the ANOVA and confidence interval are based on
different error terms, causing conflicting information
A Within Subject Confidence Interval
• Ignore the inter-subject variance
• Normalize participant scores by subtracting from the original score a
subject-deviation score
• Mi(rightmost column table 2) – the grand mean M = 12.73(bottom right table
2)
A Within Subject Confidence Interval
• Each subject pattern and condition means remain unchanged
• Each subject now has the same grand mean
• Graphing the data now gives a visual without subject variability
• 2 sources of variability left
• Condition Variance
• Interaction Variance
Conclusion
• Between and within subject confidence intervals both provide:
• Information consistent with an ANOVA
• A clear picture of underlying patterns between means
• A picture of statistical power
• Each experiment has its own challenges
• Specific hypothesis to be evaluated
• Assumptions being met or violated
• Sources of variance that vary in importance
• “No one set of algorithmic rules can appropriately cover all
situations” or “it depends”
Using Confidence Intervals for
Graphically Based Data
Interpretation
Masson and Loftus(2003)
Intro
• Null hypothesis significance testing(NHST) has been heavily debated
• Goal to enhance alternative to NHST for data interpretation
• Prior research has tried to address between/within subject design
and confidence intervals
• This paper considers how to apply a graphical approach to mixed
designs
• “A rule of thumb: plotted means whose confidence intervals overlap
by no more than about half the distance of one side of an interval can
be deemed to differ under NHST”
An Example from prior research
• Similar explanation to Loftus & Masson 1994
• Example of using the Stroop task
• Emphasize that confidence intervals for within-subject research only
support inferences about patterns of means across conditions, not
inferences about population means
Assumptions
• Between subject – homogeneity of variance
• Within-subject – sphericity assumption
• An ANOVA would calculate an e value to correct degrees of freedom if
violated
• Researchers suggest calculating individual confidence intervals if e < .75
however a negative variance can result
• To avoid this, a researcher could construct confidence intervals on single
degree of freedom contrasts
Example of individual CIs
• Table 2 data produces an e of <.6
and violates sphericity
Multifactor Designs
Jarmasz and Hollands(2009)
Multifactor Designs
• Major considerations when trying to graphically present multifactor
designs
• How to illustrate main effects
• How to illustrate interactions
• How to handle violations of homogeneity and sphericity
• If violations occur, transform data to reduce heterogeneity of variance
Designs with 2 levels
• Factorial designs question which mean squared should be used for
confidence intervals
• A regular between subject design has just MSwithin or 1 error term
• This gives a researcher a single confidence interval
• A violation of homogeneity could allow separate intervals
Contrasts
• “Researchers consider an effect produced by a linear combination of
means where that combination is defined by a set of weights applied
to the means.”
• These weights must sum to 0
• Contrast effect
• A difference between 2 means
• A main effect
• An interaction effect
• Computing a confidence interval for this
Graphing Contrasts
• After calculating weights and
creating confidence intervals,
graphs would be plotted
normally
• If a confidence interval does not
include 0, it is thought to be a
significant interaction
• E.g. Encoding interaction with
condition A2
Designs with 2 levels – Within Subjects
• Unlike a between subject study, within subject design includes
multiple MS error terms
• One way to calculate a confidence interval would be to pool your MS
error terms and divide them by the sum of degrees of freedom
Designs with 2 levels – Within Subjects
• What if you should not pool all error terms?
• Pool error terms within a factor of 2 and compute an individual CI for others
• The individual CI only applies to the effect it is associated with
Mixed Designs
• One between subject and one within subject factor
• This means pooling MS error terms would not be possible
• Separate confidence intervals must be constructed for each
Designs with 3 or more Factors
• First plot means with Cis based on pooled MS error estimates then
plot normally
Confidence Intervals in
Repeated-Measures Designs:
The Number of Observations
Principle
Intro
• CIs are a well established statistical method
• They draw inferences well in between subject design
• Using CIs in repeated measure design has been proposed and
accepted
• The size of a CI is affected by the number of observation that create
the mean
• Loftus & Masson papers do not describe calculating n well
• Most think this is total participants
How to Calculate n
• N or number of observations
• Number of participants
• Multiplied by the product of the number of levels of all factors
• Divided by the number of levels for the effect of interest
New CI Formula
• MSrxs = MSE for R
• L = product of the levels of all
RM factors in the analysis
• r is the number of levels for R
Special Scenarios
• IM Factors in Mixed IM-RM Designs
• Mixed IM-RM Interactions
• Repeated Measures Main Effect
• CI(Direction)
Ex. IM Factors in Mixed IM-RM Designs
• Multiple IM factors have no multiplying effect on the MSE
• There is only a since MSE(Mean Square within or MSw)
• Adding RM factors inflate the MSw for any IM factor
• This changes the formula used to calculate a CI
IM-RM Confidence Interval Formula
• Mi is the mean for the relevant
level of Factor A
• ni is the number of participants
in each level of Factor A
• L is the product of the number
of levels of all RM factors
Generalizing the Number of Observations
Principle
• Applying the principle involves 2 steps
• First – Determine how many participants contribute to the effect
• Total participants for pure RM effects
• Participants per condition for IM effects
• Participants for IM-RM interactions
• Multiply the number of participants by the product of the levels of
the remaining RM factors
• In-Article examples