Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
+ Autonomously Generating Hints by Inferring Problem Solving Policies C Piech, M Sahami, J Huang, L Guibas L@S 2015, Vancouver, BC, Canada (March 14–18) Slides prepared by: Roya Hosseini + 2 Highlights Autonomously generating hints for Code.org ‘Hour of Code’ Predict the way expert think student should move forward Predictions are basis of hints Introducing a statistic highly predictive of future performance + 3 Code.org: Hour of Code 20 introductory programming challenges Students generate program by putting together parts of code The largest MOOC ever! In less than 2 years had over 27M learners has been taught in over 90K classrooms HOC Example + 4 Data Maze World puzzles PA PB Records student work after each run Builds a raw series of partial solutions after final submission or give up Dec. 2013- Mar. 2014 Over 137 partial solution Age (M=13.2, SD=4.3); Range: [5,98] http://stanford.edu/~cpiech/codedotorg.html + 5 What is Problem Solving Policy (PSP)? For any partial solution s, policy decides what the next partial solution s’ student should take: A visualization of a problem solving policy for PA s' = p (s) • • • Node: unique partial solution Node 0: correct answer Edge: show partial solution students should move to + 6 Normalizing differences between Partial Solutions A student could make incremental or large changes Student series were interpolated that transition between partial solutions had exactly one program edit 1. Build a ‘legal move graph’ of all ways users can move between partial solutions using single edits. 2. Replace sequences of states with more than one edit with the most likely path between the states in the ‘legal move graph’ + 7 Gold Standard Data 7 experts labeled more than 200 student programs (PA,PB) They defined transitions between partial solutions Multiple correct ways were allowed in case of disagreement + 8 Methods Desirable path algorithms desirability of a partial solution to the ‘average’ student is a function of historical counts and experts suggest that learners make progress along the most desirable path-to-solution 1) Poisson path 2) Independent probable path + 9 Poisson Path Poisson path from a partial solution s is: path from s to a perfect solution with the smallest expected time to be generated by the average successful student Z(s): all the paths-to-solution from s λx : #times partial solution x is seen in successful student data π(s): first partial solution in the path γ(s) + 10 Independent Poisson Path Markov-zero most likely path-to-solution for a randomly sampled student in the class Z(s): all the paths-to-solution from s λx : #successful student who submitted x k : #total submission + 11 Baselines Markov Decision Problem (Barnes and Stamper, 2008) States: set of partial solutions Actions: legal changes to program Rewards: 0,100 Transition probability between partial solutions a and b : p(ψt+1 = b|ψt = a) Rivers Policy (2013) Next best partial solution N(x): neighbors of x in the legal move graph λn : popularity of the partial solution δ(n, g): min edit distance between a neighbor n and correct answer g v(n): unit-test score of the neighbor + 12 Static analysis uses algorithm to compare sequence of actions in: partial solution and correct answer selects the first divergence recreates the expert labels Most likely next for each partial solution x, selects the partial solution that most successful students transitioned to from x. Most common path: finds the most common entire path to a solution in dataset sets π(x) to be the first partial solution in the path from x to correct answer + 13 Expected success for each partial solution x, selects the next partial solution y that maximizes the probability of a student arriving at the final answer given that they transition from x to y. Min time selects the next partial solution that has minimal expected time until the student arrives at the solution + 14 Ability Model tj: true next step for partial solution j αi: ability of student i βj: difficulty of selecting right answer for partial solution j lij: next step taken by student i from state j + 15 Evaluation Accuracy of policy π overlap between π and the expert map weighted by λk (#students who submitted partial solution k) + 16 Accuracy of PSP algorithms + 17 Effect of Dataset Size on Accuracy Policy learning process with different sample sizes M and SD are from 100 runs + 18 Path Score: Student Statistics Path Score of students series T time an average student would take to complete the series sum of the inverse of the rate of each partial solution + 19 Linear Relationship: -Log(Path Score), Pr.(Success in Next Challenge) + 20 Take Away A useful algorithm would be able to predict what an average successful student would Partial solution submission counts captures how desirable a solution was to average student Transition counts captures what the biased population of students who had arrived at a given partial solution, not the average student, would do To find next best step search for an entire path to solution instead of only neighboring solutions