Peer Assessment in Massive Online Classes

Transcription

1 Peer Assessment in Massive Online Classes CHINMAY KULKARNI1, KOH PANG WEI1,2, HUY LE2, DANIEL CHIA1,2, KATHRYN PAPADOPOULOS1, JUSTIN CHENG1, DAPHNE KOLLER1,2, SCOTT R. KLEMMER1 1Stanford University; 2Coursera, Inc

2 Overview Experience with using peer and self assessment 1 st MOOC(Massive Open Online Course) to use peer & self assessment Challenges

3 Overview Experiments Giving students feedback Providing fortune-cookie feedback snippets Data driven approach to identify areas for improvement

4 Overview Results 2 nd iteration 42.9% student grades within 5% of staff grade 65% within 10% staff grade Reduced grading error 12.4% to 9.9%

5

6 Importance

7 Design Studio Public visibility of self and peer work gives students nuanced understanding of design Design tradeoffs Iterative practice

8 Design Studio Suggests requirements for successful design Support open ended designs with multiple correct solutions Allow students to learn unspoken criteria of good design Assessment must provide qualitative & formative feedback

9 Challenges Lack of colocation Peer learning Scale Evaluation Open ended design not easy to evaluate automatically

10 Demographics

11 Peer Assessment Scale evaluation & peer learning Introduce students to different perspectives Increase student involvement Enhance classroom discussion

12 Peer Assessment Issues Disparity between peer & staff grades Errors in grading Feedback quality 3 techniques evaluated to address these issues

13 Peer Assessment Calibrated peer review Calibration Student compare staff grade with theirs along with explanation If grades close, students move on to assessment Otherwise continue calibration for up to 5 submissions After 5 move on to assessment regardless of matching Assessment Student assess 5 submissions Immediately after asses their own work

14 Assignments

15 Peer Assessment Self assessment score granted if within 5% of median score Double blind

16 Assessment Accuracy Methods Est ground truth comparison between student & staff Median grade taken from small number of random peers Compare with ground truth staff grade 1000 such samples taken

17 Results: Student v Staff Grades

18 Results: Staff v Staff Much higher correspondence than among students Limitations still present however Aggregating peer grades increases agreement with staff grades

19 Student Reaction Students report giving feedback & self assessment are valuable learning 42% seeing others work biggest benefit 31% learning to communicate ideas 20% students voluntarily did more than required assessments

20 Student Reaction Illusory superiority effect

21 Discussion How can we change the perception of effort so as to reduce illusory superiority effect?

22 Patriotic Grading

23 Improving Peer Assessment Providing Feedback Providing fortune-cookie feedback snippets Data driven approach to identify areas for improvement

24 Providing Feedback Method Between subjects: Feedback No feedback

25 Results 4.4% more samples in the feedback condition obtained a grade within 5% of the staff grade

26 Personalized Feedback on Assignments Combats weakness of rubrics Provide actionable feedback

27 Results Cookies do not encourage more feedback beyond the cookie text Do not improve feedback However Reduce feedback cost Encourages critical thinking Trigger recollection of concept

28 Data Driven Improvements Low rater agreement indicate areas for revision Rubric revisions Parallel sentence structure Splitting up complex rubric items Using less ambiguous words

29 Inflating Grades & Cheating Not widespread

30 Discussion How do you think cheating behavior would change if classes were offered for college credit?

31 Shifting Roles Teacher Coaching TA Mentoring, revisions to material Student Larger impact Classroom Great statistics Staff have no knowledge of student progress

32 Discussion How would you feel if someone was able to get the same degree as you from the same university, online, for free?

33 Cognitive Tutors: Technology bringing learning science to the classroom Kenneth R. Koedinger Albert Corbett

34

35 Individual Tutoring More effective than many-to-one instruction Tutor for every student not economically feasible HW & SW capability can provide this for every student

36 History Computer-Aided Instruction (Eberts, 1997) Implementation of cognitive models (Newell & Simon, 1972) John Anderson & colleagues Combine computer aided instruction with cognitive models Intelligent tutors emerge

37 Cognitive Tutors Express learner cognition in domain of interest Math, programming, Includes Representation of learner strategies Common misconceptions at different levels of mastery Approach effectiveness of human tutor

38 Discussion In what domains would cognitive tutors be less effective? How can we compensate?

39 Cognitive Tutors Aspects of human tutoring: Monitor performance Provide feedback when needed Monitoring learning Provide questions within reach of student ability

40 Cognitive Tutor Algebra

41 Example id=fraction-addition

42 Learning Theory Act-R Like any cognitive architecture, ACT-R aims to define the basic and irreducible cognitive and perceptual operations that enable the human mind. In theory, each task that humans can perform should consist of a series of these discrete operations. wiki/act-r

43 Learning Theory Procedural Knowledge Can only gain by doing something Encoded in production rules, if-then statement Production Rules Thinking patterns Methods for discovering approaches to solutions Non traditional strategies

44

45 Cognitive Model & Performance Monitoring Model Tracing Monitor each problem solving step & provide help when needed Knowledge Tracing Track acquisition of production rules Update estimate of problems student can solve each time they apply rule Bayesian update Adapts pacing

46

47 Cognitive Tutor Design Principles Represent student competence as a production set Provide instruction in a problem-solving context Communicate goal structure in underlying problem solving Promote correct general understanding of problem-solving knowledge Minimize working memory load Provide immediate feedback on errors relative to model of desired performance

48 What should students be learning? What problem-solving activities help? What relevant knowledge to students bring?

49 Meta-Design Principles Design with instructors & classroom in mind Design full course Designs should be based on empirical evidence Initial development Fine grain prob solving activity Overall evaluation

50 Example: Reducing working memory

51 Example: Self Reflection

52 Cognitive Tutor Algebra 15-25% better performance on standardized tests % better problems solving & representation use

53 Classroom Feedback Students have more positive outlook on material More engaged Video game appeal Provide assistance without social stigma Long term achievement boosting confidence

54 Conclusion Cognitive tutors can effectively capture tutoring behavior Buying computer cheaper than hiring teacher