Download Z(s) - WordPress.com

+
Autonomously Generating Hints by
Inferring Problem Solving Policies
C Piech, M Sahami, J Huang, L Guibas
L@S 2015, Vancouver, BC, Canada (March 14–18)
+
2
Highlights


Autonomously generating hints for Code.org ‘Hour of Code’

Predict the way expert think student should move forward

Predictions are basis of hints
Introducing a statistic highly predictive of future performance
+
3
Code.org: Hour of Code

20 introductory programming challenges

Students generate program by putting together parts of code

The largest MOOC ever!

In less than 2 years had over 27M learners

has been taught in over 90K classrooms
HOC Example
+

4
Data
Maze World puzzles
PA
PB

Records student work after each run

Builds a raw series of partial solutions
after final submission or give up

Dec. 2013- Mar. 2014

Over 137 partial solution

Age (M=13.2, SD=4.3); Range: [5,98]
http://stanford.edu/~cpiech/codedotorg.html
+

5
What is Problem Solving Policy (PSP)?
For any partial solution s,
policy decides what the next
partial solution s’ student
should take:
A visualization of a problem solving policy for PA
s' = p (s)
•
•
•
Node: unique partial solution
Node 0: correct answer
Edge: show partial solution students should move to
+
6
Normalizing differences between
Partial Solutions

A student could make incremental or large changes

Student series were interpolated that transition between
partial solutions had exactly one program edit
1.
Build a ‘legal move graph’ of all ways users can move between
partial solutions using single edits.
2.
Replace sequences of states with more than one edit with the
most likely path between the states in the ‘legal move graph’
+
7
Gold Standard Data

7 experts labeled more than 200 student programs (PA,PB)

They defined transitions between partial solutions

Multiple correct ways were allowed in case of disagreement
+
8
Methods

Desirable path algorithms

desirability of a partial solution to the ‘average’ student is a
function of historical counts and

experts suggest that learners make progress along the most
desirable path-to-solution
1)
Poisson path
2)
Independent probable path
+
9
Poisson Path

Poisson path from a partial
solution s is:

path from s to a perfect
solution with the smallest
expected time to be
generated by the average
successful student
Z(s): all the paths-to-solution from s
λx : #times partial solution x is seen
in successful student data
π(s): first partial solution in the path γ(s)
+
10
Independent Poisson Path

Markov-zero most likely path-to-solution for a randomly
sampled student in the class
Z(s): all the paths-to-solution from s
λx : #successful student who submitted x
k : #total submission
+

11
Baselines
Markov Decision Problem (Barnes
and Stamper, 2008)

States: set of partial solutions

Actions: legal changes to program

Rewards: 0,100

Transition probability between
partial solutions a and b :
p(ψt+1 = b|ψt = a)

Rivers Policy (2013)

Next best partial solution
N(x): neighbors of x in the legal move graph
λn : popularity of the partial solution
δ(n, g): min edit distance between a
neighbor n and correct answer g
v(n): unit-test score of the neighbor
+
12

Static analysis

uses algorithm to compare sequence of actions in:
partial solution and correct answer


selects the first divergence

recreates the expert labels
Most likely next


for each partial solution x, selects the partial solution that most successful
students transitioned to from x.
Most common path:

finds the most common entire path to a solution in dataset

sets π(x) to be the first partial solution in the path from x to correct answer
+
13

Expected success


for each partial solution x, selects the next partial solution y that
maximizes the probability of a student arriving at the final answer
given that they transition from x to y.
Min time

selects the next partial solution that has minimal expected time
until the student arrives at the solution
+
14

Ability Model
tj: true next step for partial solution j
αi: ability of student i
βj: difficulty of selecting right answer for partial solution j
lij: next step taken by student i from state j
+

15
Evaluation
Accuracy of policy π


overlap between π and the expert map
weighted by λk (#students who submitted partial solution k)
+
16
Accuracy of PSP algorithms
+
17
Effect of Dataset Size on Accuracy

Policy learning process with
different sample sizes

M and SD are from 100 runs
+
18
Path Score: Student Statistics

Path Score of students series T

time an average student would take to complete the series

sum of the inverse of the rate of each partial solution
+
19
Linear Relationship:
-Log(Path Score), Pr.(Success in Next Challenge)
+
20
Take Away

A useful algorithm would be able to predict what an average
successful student would

Partial solution submission counts captures how desirable a
solution was to average student

Transition counts captures what the biased population of
students who had arrived at a given partial solution, not the
average student, would do

To find next best step search for an entire path to solution
instead of only neighboring solutions