Codewebs: Scalable Homework Search for Massive Open Online Programming Courses

Transcription

1 Codewebs: Scalable Homework Search for Massive Open Online Programming Courses Leonidas Guibas Andy Nguyen Chris Piech Jonathan Huang Educause, 11/6/2013

2 Massive Scale Education 4.5 million users 750,000 users 1.2 million users

3 Complex and informative feedback in MOOCs Multiple choice Coding assignments Proofs Essay questions Short Response Easy to automate Limited ability to ask expressive questions or require creativity Hard to grade automatically Long Response Can assign complex assignments and provide complex feedback 3

4 Binary feedback for coding questions Linear Regression submission (Homework 1) for Coursera s ML class Test Inputs function [theta, J_history] = gradientdescent(x, y, theta, alpha, num_iters) %GRADIENTDESCENT Performs gradient descent to learn theta % theta = GRADIENTDESCENT(X, y, theta, alpha, num_iters) updates theta by % taking num_iters gradient steps with learning rate alpha What would a human grader m = length(y); % number of training examples J_history = zeros(num_iters, 1); (TA) do? for iter = 1:num_iters theta = theta-alpha*1/m*(x'*(x*theta-y)); J_history(iter) = computecost(x, y, theta); end Test Outputs Correct / Incorrect? 4

5 Complex, informative feedback function [theta, J_history] = gradientdescent(x, y, theta, alpha, num_iters) m = length(y); J_history = zeros(num_iters, 1); for iter = 1:num_iters hypo = X*theta; newmat = hypo y; trans1 = (X(:,1)); trans1 = trans1 ; newmat1 = trans1 * newmat; temp1 = sum(newmat1); temp1 = (temp1 *alpha)/m; A = [temp1]; theta(1) = theta(1) - A; trans2 = (X(:,2)) ; newmat2 = trans2*newmat; temp2 = sum(newmat2); temp2 = (temp2 *alpha)/m; B = [temp2]; theta(2)= theta(2) - B; J_history(iter) = computecost(x, y, theta); end theta(1) = theta(1) theta(2)= theta(2); Better: theta = theta-(alpha/m) Why?? *X'*(X*theta-y) Correctness Efficiency Style Elegance Good Good Poor Poor 5

6 Technical Challenges sum(x'*(x*theta-y)); sum(((theta *X ) -y) *X); sum(transpose(x*theta-y)*x); Source code similarity Scalability

7 The Data Code Web: Improving Life for Future Tas John, Andy, Chris, Leo

8 Coursera Machine Learning > 1 million submissions

9 Why Care? Code Web: Improving Life for Future Tas John, Andy, Chris, Leo

10 The feedback paradox Fast feedback is key for student success [Dihoff 04] Human feedback is an overwhelming time commitment for teachers [Sadler 06].

11 Cost disease

12 Grand Challenge Automatically provide feedback to students (for programming assignments)

13 Basic Idea Code Web: Improving Life for Future Tas John, Andy, Chris, Leo

14 [Sad Teacher] Input: many ungraded submissions for a programming assignment

15 [Happy Teacher] Output: annotated submissions

16 [Effective [Happy Teacher] Force multiply teacher effort.

17 Build A Homework Search Engine Code Web: Improving Life for Future Tas John, Andy, Chris, Leo

18 Abstract syntax tree representations function A = warmupexercise() A = []; A = eye(5); endfunction ASSIGN IDENT (A) INDEX_EXP ASTs ignore: Whitespace Comments IDENT (eye) ARGUMENT_LIST CONST (5)

19 Abstract syntax tree representations function A = warmupexercise() A = []; A = eye(5); endfunction ASTs ignore: Whitespace Comments

20 Indexing documents by phrases blue sky yellow submarine The bright and blue butterfly hangs on the breeze term/phrase best {1,3} blue {2,4,6} document list What basic queries should code We all something something yellow submarine search engine bright support? {7,8,10,11,12} heat {1,5,13} kernel {2,5,6,9,56} sky {1,2} submarine {2,3,4} woes {10,19,38} yellow {2,4}

21 Find the Feet That Fit Query ASTs that match on context

22 Find the Feet That Fit Query ASTs that match on context

23 Runtime (seconds) Runtime (seconds) Running time for indexing # ASTs indexed Average AST size (# nodes)

24 Subtree frequency Zipf s Law for code phrases starter code elbow Subtree rank

25 Great!

26 So what?

27 Data driven equivalence classes of code Code phrases Web: Improving Life for Future Tas John, Andy, Chris, Leo

28 There is Structure to Code

29 Exponential Solution Space def dosomething() { Part A Part B Part C 500 ways 200 ways 200 ways Without factoring: params = With factoring: params = }

30 Say you have two sub-forests Sub-forest Solution A Solution B

31 Each sub-forest has a complement Solution A Solution B

32 If those complements are equal Solution A Solution B

33 And the programs have the same output Solution A Solution B

34 The sub-forests were analogous in that context Code Block A Code Block B

35 {m} m rows (X) rows (y) size (X, 1) length (y) size (y, 1) length (x (:, 1)) length (X) size (X) (1) {alphaoverm} alpha / {m} 1 / {m} * alpha alpha.* (1 / {m}) alpha./ {m} alpha * inv ({m}) alpha * (1./ {m}) 1 * alpha / {m} alpha * pinv ({m}) alpha * 1./ {m} alpha.* 1 / {m} 1.* alpha./ {m} alpha * (1 / {m}).01 / {m} alpha.* (1./ {m}) alpha * {m} ^ -1 {hypothesis} (X * theta) (theta' * X')' [X] * theta (X * theta (:)) theta(1) + theta (2) * X (:, 2) sum(x.*repmat(theta',{m},1), 2) {residual} (X * theta - y) (theta' * X' - y')' ({hypothesis} - y) ({hypothesis}' - y )' [{hypothesis} - y] sum({hypothesis} - y, 2)

36 % remaining ASTs after canonicalization Reduction in #ASTs via canonicalization equivalence More class equivalence classes 90 means more reduction 80 means fewer ASTs We get better 10 results on the more common 0 ASTs # unique ASTs considered (Ordered by frequency of AST)

37 # submissions covered (out of 40,000) How many submissions can we give feedback to with fixed effort? with 25 ASTs marked with 200 ASTs marked # equivalence classes

38 # submissions covered (out of 40,000) How many submissions can we give feedback to with fixed effort? with 25 ASTs marked with 200 ASTs marked Equivalences + 25 marked No Equivalences marked # equivalence classes

39 Syntactic bug isolation using Codewebs Web: Improving Life for Future Tas John, Andy, Chris, Leo

40 Recall Bug detection evaluation (Difficult to evaluate localization!!) Each point represents a single coding problem in Coursera s ML class regularized backpropagation for neural nets Precision

41 Higher is better F-score Bug detection evaluation with canonicalization without canonicalization # unique ASTs considered

42 Concrete Example Code Web: Improving Life for Future Tas John, Andy, Chris, Leo

43 Case study: The extraneous sum function [theta, J_history] = gradientdescent(x, y, theta, alpha, num_iters) %GRADIENTDESCENT Performs gradient descent to learn theta % theta = GRADIENTDESCENT(X, y, theta, alpha, num_iters) updates theta by % taking num_iters gradient steps with learning rate alpha Syntax based approach: m = length(y); % number of training examples J_history = zeros(num_iters, 1); Attach this message to everyone containing that exact expression for iter = 1:num_iters theta = theta-alpha*1/m*(x'*(x*theta-y)); J_history(iter) (covers = computecost(x, 99 submissions) y, theta); end function [theta, J_history] = gradientdescent(x, y, theta, alpha, num_iters) %GRADIENTDESCENT Performs gradient descent Dear to learn Lisa Simpson, theta consider the % theta = GRADIENTDESCENT(X, y, theta, dimension alpha, num_iters) of the expression: updates theta by % taking num_iters gradient steps with learning rate alpha m = length(y); % number of training examples J_history = zeros(num_iters, 1); Correct X'*(X*theta-y) and what happens after you call sum on it for iter = 1:num_iters theta = theta-alpha*1/m*sum(x'*(x*theta-y)); J_history(iter) = computecost(x, y, theta); end Incorrect

44 The extraneous sum bug takes many forms theta = theta-alpha*1/m*sum(x'*(x*theta-y)); theta = theta-alpha*1/m*sum(((theta *X ) -y) *X); theta = theta-alpha*1/m*sum(transpose(x*theta-y)*x); (Easier) Output based approach: Attach this message to everyone containing that exact expression (covers 1091 submissions)

45 Codewebs approach to feedback theta = theta-alpha*1/m*sum(x'*(x*theta-y)); Step 1: Find equivalent ways of writing buggy expression using Codewebs engine Step 2: Write a thoughtful/meaningful hint or explanation ~47% improvement over Step 3: Propagate just feedback using an message output based to any submission containing feedback equivalent system!! expression 1091 Output based 1208 Codewebs 1604 Combined # submissions covered by single message

46 Big data as a problem: can we give human quality feedback to a million code submissions? Big data as a solution: structure (clustering, subtree equivalence) of the solution space invisible without dense sampling! Where we re going next: Extensions to other languages Applications to new problem domains Data association problem for variables Dynamic analysis Temporal analysis

47 How do solutions evolve modeling the human creative process Feedback on progress, not just on final solution Think beyond computer science, beyond education