Download Harnessing Untapped Potential: Using International and National

Harnessing Untapped Potential:
Using International and National Assessment
Data to Inform the Transition to Next
Generation Assessment Systems
Laura Egan, NAEP Support and Service Center, Westat
Pamela Byrd, Arkansas Department of Education
Mark DeCandia, Kentucky Department of Education
Kate Beattie, Minnesota Department of Education
Council of Chief State School Officers
2014 National Conference on Student Assessment
Outline of the Session
• Overview of the Literature on Policy Uses of
International Assessments (Jason)
• Mapping the Terrain: Assessment Landscape
and Policy Priorities (Pam)
• The Kentucky Case (Mark)
• The Minnesota Case (Kate)
Background
• Ongoing project
▫ Different subprojects in each state
▫ Growing interest in non-state assessments
▫ Shaped by different contexts of each state
• Rapidly changing terrain
International Assessments in the
Academic Literature
• Increasing interest in international assessments
▫ Come from different disciplinary backgrounds
▫ Use different methodologies
▫ Use different assessment data
 TIMSS, PIRLS, PISA, regional assessments
 Special studies (e.g., 1995 TIMSS International
Curriculum Analysis, 1995 and 1999 Video Case
Studies)
▫ Variety of studies
▫ Focus on different (sets of) countries
International Assessments in the
Academic Literature
• Example from the Literature
▫ Growing body of literature linking performance on
international assessments to economic
performance
Relationship between system-level
student achievement and economic
performance
• Assessments do not directly measure
delivery of curriculum or the impact of an
educational system given their design, crossnational nature
• Instead, assessments like TIMSS, PIRLS,
and PISA provide measures of cognitive
skills, which is an indicator of human capital
Human Capital
• Education, most often in the form of schooling,
raises individuals’ productive value by providing
knowledge, skills, values, and a way of analyzing
problems (Becker, 1993)
• Investments in education should therefore produce
financial returns at the individual (e.g., salary) and
societal (e.g., GDP) level
Human Capital
• Often measured in terms of years of schooling, but
this assumes that
▫ 1 year of schooling delivers same increase in
knowledge and skills regardless of education system
▫ Formal schooling is the primary source of education
and variations in the quality of non-school factors
have negligible effect on education outcomes
(Hanushek and Woessman, 2012)
• Relationship between years of schooling and
economic growth is loose, ambiguous
Assessment Data as a Measure of
Cognitive Skills
• International assessments can provide a better
measure of human capital through measurement
of cognitive skills – what kids know and can do
▫ Human capital does not have to be gained through
schooling, as long as effect is the same
• Assessments provide data at a system level, so
use GDP as economic indicator
• Can combine TIMSS, PISA, PIRLS, regional
assessments, prior assessments onto single scale
with common standard deviation
Cognitive skills and economic growth
• Hanushek and Kimko (2000), Hanushek and
Woessman (2009, 2011, 2012) use econometric
modeling to control for a number of variables,
rule out omitted variables, etc., and demonstrate
 “growth rates changing in a manner
consistent with changes in cognitive skills”
(Hanushek and Woessman, 2011, p 296)
• Pattern is seen over time, across regions
Educational performance and economic growth across OECD
Countries (Hanushek & Woessman, 2012)
Educational performance and economic growth across World
Regions. (Hanushek & Woessman, 2012)
Conclusions
• International large-scale assessments measure
something that matters
• Suggest that there is an economic value in
reforms that lead to increased knowledge and
skills as measured by these assessments
• But…policy path is not immediately clear
Policy implications of international
assessment data
• Provide a great deal of policy-relevant
information on the
▫ context in which learning occurs
▫ cross-national patterns in instructional practices
▫ instructional practices and curricular focus of high
performers (and low performers)
• Do not provide policy prescriptions or
demonstrate causal relationships between
aspects of education system and its performance
Policy implications of international
assessment data
• Used to to advocate for change
▫ Strong focus on curricular change - more rigor,
focus, coherency
▫ Findings/advocacy not always supported
empirically
Implications for State Assessment
Policy
• Comparative data on instructional practices,
which can be correlated to performance on
cognitive items, may be useful for
▫ starting conversations
▫ identifying avenues for future research
▫ identifying practices to be explored
• Empirical relationship between assessment scores
and economic growth fosters conversation with
stakeholders outside of education arena
Implications for State Assessment
Policy
• Important to consider
▫ Different assessments measure different constructs
with different purposes
▫ State contexts differ
 System performance
 Policy context