Download Students with better conceptual understanding of physics do

Relationship
between
Physics Understanding
and
Paragraph Coherence
Reva Freedman
November 15, 2012
Outline
•
•
•
•
•
•
What is AIED?
Intelligent tutoring systems for physics
Coherence: an important factor
Database and experiment
Results
Conclusions
What is AIED?
• Artificial Intelligence in Education
• Breakthrough in last decade
– Follows transformation in AI (and NLP) in
the previous decade
• “Science of learning”
– Not just intuition any more
AIED + NLP: A good match
• Natural Language Processing
– NL = what you speak
• Intelligent tutoring systems often use
NL for the same reasons people do
– More expressive than GUI
– More flexible
Teaching/Learning Physics
• Two goals in physics teaching
– Conceptual understanding
– Problem solving
• Does teaching one help the other?
• Same issue in other sciences
ITSPOKE System
• Intelligent tutoring system for physics
– Latest in a series
 Andes/Atlas  ...  APE
(Python/TCP-IP  C++/etc/.COM)
• Experiment
– Students solve a physics problem
– Then write essay about it
– Then rewrite essay until tutor is satisfied
Sample Physics Problem
A person running in a straight line at a
constant velocity throws a pumpkin
straight up. Where does it land and why?
Sample Student Essay
1) While you are carrying the pumpkin, it has
the same horizontal velocity as you do.
2) When you throw it up, there is the force of
your throw which carries it vertically up,
but there is no horizontal force to affect its
horizontal velocity.
3) Eventually gravity will overcome the force
of your throw and the pumpkin will come
back down.
Sample Student Essay
4) During the pumpkin’s entire path up and
back down, there is no horizontal force
acting on it, so it will maintain the same
constant velocity that it had when you
were carrying it.
5) So the pumpkin will land on the ground
right by your feet.
Coherence
• Coherence used to be hard to get hold of
– “Know it when you see it”
– But impossible to measure
• Barzilay and Lapata (2005, 2008)
– New theory based on the new AI
– Coherence is based on the relationship
between the use (e.g., pumpkin) of entities
in successive sentences
Importance of Coherence
• Local coherence is the major cause of
students’ understanding what they read
– McKoon and Ratcliff (1992)
Data Structure: Entity Grid
Sent. #1
Sent. #2
noun 1
s
-
noun 2
s
x
noun 3
-
o
s = subject
x = other role
o = object
- = none
Algorithm (Barzilay/Lapata)
• Assume human texts are more
coherent than permutations of them
• Choose a permutation function
• For every permutation of each essay in
the training set, compare original vs.
permutation
• Induce coherence function that prefers
the original human text most often
A Permutation Function
• Binary discrimination
– Compare original against random
permutation
– Fastest and easiest
– Run multiple times and see who wins
Database
• Data
– Experiment | student | problem | essay
– No. of “wins” for binary test (0 – 20)
– Physics pretest and posttest
• 2219 data points
– Small number but big records
Methods
• Quinlan’s C4.5 algorithm
– Implementation in Weka’s J48
– Numeric fields considered as nominal
values, e.g., four subgroups for posttest
values
Results: Null Hypothesis!
• No relationship between essay
sequence number and coherence
– When students rewrite their essays, they
do not become more coherent
• No relationship between posttest score
and coherence
– Students with better conceptual
understanding of physics do not write
more coherent essays
Current Work
• Using NLP to break down data
• Relationships between linguistic
features of the data and
– Coherence
– Physics knowledge
Wider Interests
• Science of learning without AI
– What makes CS students successful?
• Computational linguistics w/o learning
– Machine translation of lesser studied
languages
• Programming
– Building ITSs
Acknowledgements
• Learning Research and Development
Center
– University of Pittsburgh
• Diane Litman
– Founder and director of ITSPOKE lab
• Kate Forbes-Riley
– ITSPOKE researcher
Coherence: Underlying Ideas
• Important entities are more likely to be
occur in key syntactic positions such as
subject or object
• They are more likely to be introduced
in the main clause
• They are more likely to be referred to
with pronouns in later mentions
Coherence and Entity Use
• Texts about the same entity will appear
more coherent to the reader than texts
with multiple topic switches
• Continuity of topic leads to consistent
patterns of entity use
Permutation Functions
1) Binary discrimination
– Compare original against random
permutation
– Fastest
– Easiest
2) Insertion
– Insert one sentence, given the others
– Quadratic in document length
Permutation Functions
3) Ordering
– Try all permutations
– Exponential in document length
• Implementation
– Brown Coherence Toolkit (Elsner, 2008)
Entity Grid: Assumptions
• Noun phrases with the same head
noun refer to the same entity
– The bank I went to
– The bank up the street
• Salience (importance) is measured by
number of uses
• Assumptions can be changed