Peer Assessment. Measuring & Monitoring Team Performances. Ir. Vincent Brugemann and Robert-Vincent de Koning Bsc. Challenge the future

Transcription

1 Peer Assessment Measuring & Monitoring Team Performances Ir. Vincent Brugemann and Robert-Vincent de Koning Bsc Delft University of Technology Challenge the future

2 Presentation for TBM staff About Peer assessment Peer Feedback/Peer Review Peer Evaluation (PE) How to use peer assessment? Peer Feedback/Peer Review Via Blackboard Peer Evaluation (PE) Scorion 2

3 Context Faculty of Aerospace Engineering students in BSc and MSc In BSc much use is made of project based learning (approx. 35 ECTS) Students work in groups of 8-10 on a variety of projects 3

4 Problem: Assessment There is a large need amongst lecturers for assistance in assessing individual performance in a group working environment Major problem areas: Who did what? What is the size of the individual contribution? What is the quality of that contribution? In what way did a person contribute to the group process? 4

5 What is Peer Assessment? Peer Assessment is an umbrella term for a number of different teaching methods in which peers assess each other s products and/or behaviour. Peer Assessment is always formative (diagnostic). The aim of Peer Assessment is to learn from feedback; there are no summative consequences. In other words, assessments by peers will not result in grades. 5

6 When should Peer Assessment be used? Peer Assessment is a useful way to get students to collaborate intensively on the subject you want them to learn. The students: 1. Learn to assess each other using assessment criteria 2. Analyse the subject matter more thoroughly by applying the criteria 3. Learn to give and receive constructive feedback 4. Learn to take responsibility 5. Appreciate being taken seriously 6

7 Two forms of Peer Assessment We distinguish two different forms of Peer Assessment so that you can choose the appropriate instrument for your specific teaching situation. The forms of Peer Assessment applied at TU Delft are: Peer Feedback/Peer Review Peer Evaluation (PE) 7

8 PEER FEEDBACK/PEER REVIEW 8

9 Peer Feedback/Peer Review Students provide commentary on each other s products. Similar to the Peer Review used in the scientific world to assess scientific articles. In an educational environment it can also be used to describe assessments of products (which may include scientific articles). 9

10 Working with Peer Feedback/ Peer Review If you want your students to assess each other s products you can use the Self and Peer Assessment tool in Blackboard The students submit their products and Blackboard then distributes them among their peers so that they can assess each other s work. 10

11 PEER EVALUATION (PE) 11

12 Peer Evaluation (PE) Peer Evaluation is a form of Peer Assessment in which students provide commentary on each other s behaviour during cooperation (for example in project groups). The PE instrument Scorion has been developed especially for TU Delft. 12

13 Added value of Peer Evaluation PE can be embedded in teaching and function as an early-warning system, helping projects to run more smoothly. The tutor s own impressions of how the project groups are performing can be compared with the findings of the students themselves. PE enables the students to develop social skills while simultaneously acquiring the relevant subject knowledge. 13

14 Solution: Peer & Self evaluations Peer evaluation: Assessing of team members on their contribution to the project and the way that contribution came into being. Self evaluation: Assessing of one s own contribution to the project and the way that contribution came into being. 14

15 Types of peer evaluations Three types of peer evaluations: 1. Ranking (from top to bottom) 2. Dividing (Dividing up a fixed number of points or amount of money amongst team members) 3. Rubrics (Using qualitative criteria to have students assess each other) For project based learning at Aerospace Engineering it was found that option 3 works the best. 15

16 Criteria Op L&R aan de TU Delft worden 5 criteria gebruikt: Job performance Attitude Leadership / Initiative Communication Management of Resources 16

17 Job Performance Attitude Leadership/Initiative Management of Communication Resources ++ Consistently does more than required. Work is of exceptional quality. Positive and professional attitude which favorably influences other company members. Takes initiative to seek out work, concerned with getting the job done. Very involved in the technical project. Uses time effectively in and out of group and works to get others to do the same. All tasks completed on or ahead of schedule. Oral and written skills excellent. Very effective within the group and to reviewers. + Sometimes does more than required. Work is of high quality. A producer. Positive attitude toward project and the team. Readily accepts tasks, sometimes seeks more work. Gets involved in the project. Uses time effectively in and of group. Completes all tasks on time. Usually effective. 0 Performs all assigned tasks. Quality of work is acceptable. Neutral attitude. Gets involved enough to complete tasks. Does his/her share. Wastes some time in group, but works hard when a deadline is near. Most tasks completed on time. Generally gets the point across. Tries to improve in weak areas. - Performs all assigned tasks. Work must be redone or repaired to meet standards. Negative attitude toward project and/or project. Tends to watch others work. Gets involved only when necessary. Volunteers to help when it will look good. Wastes most of group time. Seldom seen doing productive work. Some tasks completed late. Skills ineffective. Makes an effort to improve. -- Performs some assigned tasks. Work must be redone by others to meet standards. Negative attitude which adversely affects other company members or project. Lets others do the work; does the minimum he/she thinks is needed to get by. Does little useful work in group or out; wastes his/her time and others. Work is constantly late. Skills ineffective. Makes little or no effort to improve. 17

18 Criterion: Job Performance Consistently does more than required. Work is of exceptional quality Sometimes does more than required. Work is of high quality. A producer Performs all assigned tasks. Quality of work is acceptable. Performs all assigned tasks. Work must be redone or repaired to meet standards Performs some assigned tasks. Work must be redone by others to meet standards. 18

19 Criterion: Attitude Positive and professional attitude, which favourably influences other team members. Positive attitude toward project and the team Neutral attitude. Negative attitude toward team and/or project. Negative attitude which adversely affects other team members or project. 19

20 Criterion: Initiative Takes initiative to seek out work, concerned with getting the job done. Very involved in the technical project. Readily accepts tasks, sometimes seeks more work. Gets involved in the project. Gets involved enough to complete tasks. Does his/her share. Tends to watch others work. Gets involved only when necessary. Volunteers to help when it will look good. Lets others do the work; does the minimum he/she thinks is needed to get by. 20

21 Criterion: Management of Resources Uses time effectively in and out of group and works to get others to do the same. All tasks completed on or ahead of schedule. Uses time effectively in and of group. Completes all tasks on time. Wastes some time in group, but works hard when a deadline is near. Most tasks completed on time. Wastes most of group time. Seldom seen doing productive work. Some tasks completed late. Does little useful work in group or out; wastes his/her time and others. Work is constantly late. 21

22 Criterion: Communication Oral and written skills excellent. Very effective within the group and to reviewers. Usually effective. Generally gets the point across. Tries to improve in weak areas. Skills ineffective. Makes an effort to improve. Skills ineffective. Makes little or no effort to improve. 22

23 Experiences with Peer evaluations To make optimal use of peer evaluations they should be carried out at least twice per project (Mid way and at the end) Comment functionality is very useful Because the results are not directly converted into a grade the system is accepted by students as fair A good IT structure is a MUST to avoid overburdening lecturers 23

24 Example: Peer Assessment in the Design and Construction project Background: 1 st year L&R Bachelor 14 week project (Sem 2, period 1 and 2) 5 EC 40 groups of 10 Students working together Assisted by a group of 20 student assistants Peer assessment conducted in weeks 3.7 and 4.8 Results: Made available to the tutors Discussed in a meeting with each tutor Are used as input to improve the group process (formative) & assist the grading process 24

25 Examples: Good student Overestimating Underestimating 25

26 Any questions? If you should have any additional questions contact Vincent Brügemann, tel , 26

27 Peer Assessment Implementatie van de tools Delft University of Technology Challenge the future

28 PEER FEEDBACK/PEER REVIEW 28

29 Peer Review in Blackboard Mogelijkheden Via blackboard het verslag laten inleveren Deadline is in te stellen Aantal reviewers is aan te passen Review kan anoniem gedaan worden Feedback kan per vraag gegeven worden Docent kan model-antwoord geven Docent kan criteria en waardering per vraag instellen Connectie met de GradeCenter van Blackboarden 29

30 Peer Review on Blackboard Tekortkoming: Inactieve studenten Geen of weinig feedback op verslagen Minder nakijkwerk voor sommige studenten Minder inzicht wordt vergaard Ongelijkheid Workaround Verwijder inactieve studenten van cursus Moeilijk te managen 30

31 Peer review assignment in BB 31

32 32

33 Peer review assignment maken Per vraag kan ook een model antwoord gemaakt worden. 33

34 Peer review assignment (2) 34

35 Opdracht inleveren 35

36 Opdracht inleveren (2) 36

37 De opdrachten beoordelen Beoordeling van eigen verslag mbv model antwoorden Beoordeling van twee andere verslagen (anoniem) 37

38 De opdracht beoordelen (2) 38

39 Samenvatting van de review De docent kan een samenvatting bemachtigen: Ga naar Course Tools Self and peer assesment View submissions toont de inlevervooruitgang View evaluations toont de verdeling, voortgang en scores. Evaluaties kunnen ook ingezien worden. View results toont de scores en laat een sync toe met Grade Center 39

40 PEER EVALUATION (PE) 40

41 Scorion In gebruik sinds 2011 Aanzienlijke groei sinds 2011 Afgelopen jaar ingezet voor 29 vakken Aantal gebruikers vorig jaar was ~7000 Rubrics gemaakt door LR is de standaard 41

42 Scorion gang van zaken Docent doet aanvraag bij E-Learning Support (ELS) minimaal twee weken van tevoren. Docent dient keuze te maken in: Aantal evaluaties Formulier template Rapport template Docent deelt groepsindeling met ELS liefste groepsindeling op Blackboard Vragen over mogelijkheden? Bel E-Learning Support voor advies! 42

43 Scorion gang van zaken (2) ELS maakt formulieren ELS verzendt de formulieren op afgesproken tijdstip Docent stuurt eventueel herinnering naar studenten Studenten vullen formulieren in (inloggen met NetID) Formulier sluit automatisch op afgesproken datum Docent kan groepsrapporten inzien Student is in staat persoonlijk rapport in te zien 43

44 Examples Closed questions Title Scoring explanation (optional) Scoring All students in the group have to be graded 44

45 Examples Open questions (standard) Open questions gives the teacher the possibility to allow the students to give feedback on their peers publicly or in private (to their teacher). Feedback for every student. Not compulsory in standard form (can be changed) 45

46 Examples Group report (for teacher) The standard group report contains the average peer scores for a student for each question and the score the student gave oneself. The column Delta gives the difference between these two. 46

47 Examples Group report (for teacher) The standard group report also contains the public and private remarks. These can help the teacher to find problems within groups. If there are any. 47

48 Examples Group report (for teacher) It is also possible to plot the individual results of students in so-called Spiderplots. These plots can be added to the report if it is requested by the teacher. The total average score can also be shown in the report if requested. 48

49 Examples Individual report (for student) The student will only see the average scores he/she has been given and will not see the scores of others. The student will also get a personal Spiderplot. Only the public remarks will be shown. 49

50 Further Possibilities Questions New questions can be implemented. These questions can be open or closed. The title, explanation of the scoring and scoring can be altered if desired. Questions do not necessarily have to be about the students, but can also be used to evaluate the teaching assistents. It is up to the teacher which peer results are available to the student. Open questions can be made anonymous. 50

51 Implementing Peer Assessment Do you want to implement Peer Assessment and/or do you have further questions? Please contact E-Learning support ELS, tel , We need to have your application for a Peer Evaluation with Scorion at least two weeks before the evaluation date. If you will be working with PE for the first time, then we will need to have your application at least two weeks prior to the start of the course. 51

52 Questions? If you should have any additional questions contact Vincent Brügemann, tel , For questions regarding the implementation of Peer Assessments, contact E-Learning Support: