Download Slides

Outline
• National Assessment of Educational
Progress (NAEP)
• Multivariate Design Problem
• Implications for analysis
• Example with similar structure in
Biostatistics
NAEP
• On-going surveys at national and state
levels
• 4th, 8th, and 12th grade students and their
teachers
• math, reading, writing
• background demographic and educational
environment questions
Excellent web site
• http://www.nces.ed.gov/nationsreportcard/
NAEP Mathematics
• Mathematics
– 5 domains/sub-scales/traits/latent proficiencies
– Algebra
– Geometry
• Several hundred potential test questions
NAEP Objectives and
Constraints
• Goal is population estimates
• Individual students (and schools) are NOT
rated based on NAEP
• 45 minutes for cognitive questions (items)
• 15 minutes for background
questions/administration
Matrix Sampling (1984)
Algebra
Student
Q1
1
x
2

Q2
Geometry
Q3
...
x
x




Q76 Q77 Q78
x
x
x


...
x


Model for Cognitive Data
• Longitudinal data model
–  A : Student algebra proficiency
– G : Student geometry proficiency
• IRT (Item response theory)
P Correct algebra answer A   logit 1 item  A item  





Design Issue
• Fixed number of items per student
– How many algebra items?
– How many geometry items?
• Obtain (equally) accurate population
estimates of both algebra and geometry
proficiencies
Balanced Designs
• Give each student approximately the same
number of algebra and geometry items
• Up to 5 or 6 sub-domains, so the number of
items per sub-domain is very small
• Extended collections of related items may
make a balanced design infeasible
Split Designs (symmetric)
• Some students assigned only algebra items
• Same number of other students assigned
only geometry items
• Remaining students are assigned equal
number of algebra and geometry items
Optimal Design and Estimation
• The balanced design is optimal
• Maximum likelihood estimation
– The joint MLE for the algebra distribution and
the geometry distribution is the same as the
univariate MLE with the geometry and algebra
proficiencies estimated separately
• Balanced design
– There is no gain from multivariate estimation
– Estimates for individual student proficiencies are much
improved by multivariate estimation
• Split design
– Multivariate estimation is much better than univariate
– Multivariate estimation for the split design approaches
balanced design efficiency as the proficiency
correlation approaches 1
Bivariate outcomes (Jessica
Mancuso)
• Experimental biomarker for stroke patients
– Measurement error
– It can be applied to the infarct and non-infarct
sides of the brain
– Anticipated that the non-infarct side of the
brain will be predictive of the infarct side
– Evaluate an oral compound using the biomarker
Study design
• Placebo/drug in parallel blinded randomized
groups
• Measurements
– Baseline
– On-dosing measurements (longitudinal)
– Measurements on the infarct and non-infarct
sides of the brain (bivariate)
Estimation
• The primary goal is to estimate the
treatment effect on the infarct side of the
brain
• What is the role of the measurements on the
non-infarct side in the primary estimation?
Depends on other information
• If there is no effect (or a known effect) on
the non-infarct side of the brain, the noninfarct data can improve estimation
– Baseline non-infarct measurement may be very
helpful
– If the treatment does not effect the non-infarct
side of the brain, the on-dosing measurement(s)
are like covariates and may improve estimation
Depends on design
• Balanced design
– Both sides of brain measured each time
– No planned or unplanned missing measurements
– On-dosing non-infarct measurements do not contribute
to estimation of the drug effect on the infarct side
(mostly true)
– Lack of contribution despite improvement in estimation
for individual patients
• Split design
– At some on-dosing times, the non-infarct
measurement is available but the infarct is not
available
– The on-dosing non-infarct data may contribute
substantially
Summary
• The use of multiple outcomes to improve
inference is very complex
• The fact that an outcome can be used to
improve the estimation/prediction of
another outcome at the level of an
individual person is not sufficient