The Power of Peers in Online Education

Transcription

1 The Power of Peers in Online Education Michael Bernstein Stanford Computer Science! with Chinmay Kulkarni, Scott Klemmer, Julia Cambre and others

2 Our vision Peer processes can improve motivation+learning, provide deeper feedback, and do so at scale.

3 thousands of students

4 thousands of singletons

5 the largest, most diverse classroom in the world

6 Kizilcec, Collaborative Learning in Geographically Distributed and In-person Groups MOOCShop 62013

7 Diversity improves learning Highlight students own assumptions, backgrounds and perspectives [Nemeth 86] Shift students from a passive learning mode to active, effortful, conscious thinking [Gurin et al. 02] Academic improvements as well as sociopolitical inclusion [Gurin et al. 02]

8 = In a classroom of global peers, why are we alone together?

9 how might we distill diversity into a unique asset for online education?

10 Could we connect students to a new mini-u.n. each week?

11 Talkabout: video discussions with global peers Under review

12 3,200+ students from 134 countries across 5 courses

13 Discussions are guided, but not monitored Discussion Prompts Live video

14 Sign up for a time

15 Talkabouts are diverse Median six-student discussions have five different countries represented Median pairwise distance between discussants is greater than New York London Number of Discussions Number of countries in discussion

16 Talkabouts are sticky Social psychology MOOC required 30min attendance Median actual attendance: 58 minutes (2x) 250 Number of students :30 1 hr 1:30 2 hr 2:30 Length of discussion attended

17 Structuring effective discussion

18 A sample of student talk: Irrational behavior Life experiences So, in our ladies lunch group, we made a pact that the first one to check her phone pays. Is that an Ulysses contract? Questions Instead of self-control as a character, maybe we should see it as a muscle it gets tired after a while Tips Yeah, it probably is. And I guess you can exercise I feel most in control when I meditate

19 Self-reference in discussion on prejudice! I think this may have been the first time the lady from Saudi Arabia had spoken to a Jew I told her about the prejudice from Christians I experienced growing up in [US state] in the 40's and the effect of segregation on blacks

20 Do diverse discussions improve learning?

21 Study 1: Randomize diversity levels Randomize N=55 students in Coursera s Organizational Analysis class into low-diversity or high-diversity conditions Low-diversity: peers from your World Bank region High-diversity: peers from any region Manipulation check: median #countries in low-diversity condition = 2 median #countries in high-diversity condition = 4

22 Study 1: Randomize diversity levels After discussion, all students answer an open-ended question on course material, scored in consultation with TAs Analyze via an OLS regression controlling for prior performance

23 Students in diverse discussions perform better Students in the high-diversity condition had higher scores (beta=0.41, p<.05)

24 Study 2: Again, at scale! 2,422 students in Coursera s Social Psychology class Quasi-experiment: students are grouped with peers who arrive at the same time Result: natural variation in diversity Measured performance on final exam one week later OLS regression controlling for prior performance

25 Students in diverse discussions still do better! Each additional country in the discussion increases the student s final exam score by 2.4% (β=1.8pts, t(129)=1.78, p<0.05)

26 The mini-un effect Online It was like a mini-un. We had an Australian currently residing in Dubai, an Afghan, a Romanian, an Indian and myself (a Pakistani).! vs. In person We have the similar views and there was no much discussion, it was simple plain talk between friends. There was a lack of tension and active opposite person.

27 beyond being there!!!

28 Peer assessment Classmates assess each other Provide summative evaluation, and constructive criticism

29 Peer assessment used in 100+ classes Human-computer Interaction Design Programming in Python Code Introduction to Philosophy Essays Teaching character Management Child Nutrition Recipes Social Psychology Essays Constitutional law Arguments World Music! Music

30 Open-ended assignments are pedagogically valuable Open ended assignments Require students to generate solutions, not only recognize them Can assess on more realistic tasks Embrace multiple solutions Closed ended questions constrain choices Recognizing a correct solution does not mean you can generate it Veloski et al 1999 Thompson et al 2000

31 Challenges of open-ended assessment Realistic, open-ended assignments require lots of grading time staff grading takes prohibitive labor (400+ hours/week) Though sharing work provides inspiration and encourages discussion, students don t see others work

32 The problem of peer processes non-experts performing expert work Hinds (1999) The Curse of Expertise: The Effects of Expertise and Debiasing Methods on Predictions of Novice Performance

33 Scaffold micro-expertise in novices

34 In-person classes Peer grades correlate well with staff Peers can provide constructive criticism! Does this scale to global online classes? Falchikov and Goldfinch (2000) Sluijsmans et al. (2002) Kulkarni&Klemmer (2012) Tinapple et al (2013)

35 Our peer assessment process staff-graded 1) Practice 2) Assess 5 Peers 3) Self-Assess Similar to CPR but calibrate students, not the algorithm.! Final peer grade is a simple median, not a calibrated, weighted mean. Carlson & Berry(2003)

36 HCI Online Subtitle Text Free; anyone can enroll. Open-ended assignments central to class, 6-7 hours/week! In all, students watched videos, 5,876 students submitted open-ended assignments

37 Assignments step through a human-centered design process Needfinding and low-fi prototypes Implementation plan Functional prototypes User testing and iteration

38

39 staff-graded 1) Practice 2) Assess 5 Peers 3) Self-Assess

40 Median peer grades correlate with staff Simulated percentage %! within 5% 59% within 10% First two runs of HCI class Mean Staff variation 6.7% Median grade minus staff grade (% of total)

41 Grading adequate for pass-fail class Errors are nearly symmetric around staff grades Simulation based on our data: Correctly cert Cert by grading error No cert by grading error

42 Assessing others yields perspective and inspiration seeing how others tackled the assignments sometimes helped me point out things I missed and offered some perspective 114 similar responses Giving me a wider point of view based on the others work 36 similar responses Consistent with in-person effects Chinn (2005); Tinapple et al. (2013)

43 Assessing yourself after peers yields reflection and comparison Was nice to evaluate my own work AFTER evaluat[ing] others because I could compare my work and effort 175 similar responses helped me see problems that i didn't see before the peer evaluation 50 similar responses Consistent with in-person effects Boud (1994)

44 Qualitative feedback improvement-oriented feedback beyond the rubric

45 Some feedback minimal/ superficial Minimal Superficial Better Great idea! I can't read the words in the pics clearly Solution requirement is vague here but I'm excited to see where you take this in the storyboards!

46 The return of the novices-as-experts problem Experts: capture the structure of rubric Peers: Focus on superficial features, even when asked not to fully interactive, page flow is complete make it clearer what people should do next unpolished Try to make UI less coloured.

47 Fortune cookies for qualitative, personalized feedback Peers can recognize errors from a list of patterns, even if they can t articulate them Most errors are variations on a theme +...because Cue Variation Anderson and Bower (1972) Recognition and retrieval processes in free recall.

48 Encouraging Richer Feedback Copy, then paste Make the prototype more interactive so the user test represents a more real-life interaction: The prototype does everything you're testing, but it couldn't hurt to make it more interactive. If the user can't possibly stray from the things you want to test, how do you know that the user can actually use the full application without making mistakes?

49 Fortune cookies feedback actionable and detailed Clarify the concerns, goals, and expectations of the user tests: try to expand the information in the implementation plan! Prototype should relate to the user needs in the storyboard more. I dont see the proposed functionallity from the storyboards here in the prototypes.

50 Our vision Peer processes can improve motivation+learning, provide deeper feedback, and do so at scale.

51 7 habits of creating successful peer processes 1. Scaffold: create micro-expertise 2. Leverage scale and diversity 3. Design learning opportunities, not labor pools: People aren t just computers with better AI 4. Use a (wait-list) control condition 5. Involve instructors: their buy-in determines student enthusiasm 6. Beta-test with community TAs 7.

52 The Power of Peers in Online Education Michael Bernstein Stanford Computer Science! with Chinmay Kulkarni, Scott Klemmer, Julia Cambre and others