EVALUATE AND IMPROVE YOUR TEACHING: METHODS FOR TEACHING ASSESSMENT MEGAN L. MITTELSTADT, PHD DECEMBER 3, 2013 HARVARD MEDICAL SCHOOL

Transcription

1 EVALUATE AND IMPROVE YOUR TEACHING: METHODS FOR TEACHING ASSESSMENT MEGAN L. MITTELSTADT, PHD DECEMBER 3, 2013 HARVARD MEDICAL SCHOOL

2 OBJECTIVES At the conclusion of this session, the learners will be able to: 1. Describe at least two different methods to obtain feedback from learners 2. Describe a quantitative method to analyze teaching 3. Create a strategy for incorporating self-analysis of teaching

3 TEACHING SAMPLE (Exercise During Session: Record your observations of the teacher in this rather famous teaching clip (who is failing to engage his students). What could he incorporate or change to improve his performance? Answers during session: -Pause after questions, wait for student answer -Read non-verbal cues -Modulate tone, etc

4 WHY ANALYZE YOUR TEACHING METHODS? Self-reflection/ improvement Portfolio documentation of improvement Professional development Improve instructional effectiveness Teaching activities fulfill their intended purpose, function, and goal Impact learner evaluations of instructor/course

5 TEACHING ASSESSMENT Triangulation (using information from students, peers, and self) provides comprehensive and composite assessment of teaching effectiveness (Jahangiri, JDE, 2008) Best done as iterative process Both formative (used to develop or improve persons/programs) and summative (used to determine final achievement) assessments Gather feedback on student learning, as well as on the strengths and weaknesses of the course and the instructor. Use feedback to refine the course.

6 TEACHING ASSESSMENT: DATA SOURCES Instructor Self-Assessment Student Perceptions Teaching and Learning Research Student Learning Administrative Perspective Peer or Colleague Review

7 CLASSROOM ASSESSMENT TECHNIQUES (CAT)

8 CLASSROOM ASSESSMENT TECHNIQUES Prepare CATs ahead of time Experiment with different techniques until you find ones particularly meaningful for you Tell students why you are using CATs, so they understand the impact they can have on their learning experience Actively incorporate the feedback from CATs, so students know you take their feedback seriously

9 CLASSROOM ASSESSMENT TECHNIQUES What CATs have you utilized or experienced in your classrooms? Clickers Survey at end of course Final reports (tests, quizzes, midterm) Interactive brainstorming (focus groups) Think-pair-share Non-verbal cues

10 CLASSROOM ASSESSMENT TECHNIQUES Background knowledge probe Minute paper Muddiest point Student-generated test questions Audience response system Think-pair-share Non-verbal behavior of students Focus groups Student interviews Learning outcome measures (exams, etc) Mid-course Survey Exit/Alumni survey

11 BACKGROUND KNOWLEDGE PROBE Set yourself up for success! Collect specific and useful feedback on prior learning Short, simple questionnaire at beginning of course, new unit, or lesson Can also be used as post-assessment Increase participation Helps teachers determine most effective starting point and appropriate level of instruction

12 MINUTE PAPER Pose 1 or 2 questions What was the most important thing you learned about X today? What important question remains unanswered? Students respond with a few phrases on index cards Review responses and respond in next session Can make fun exercise by varying questions

13 MUDDIEST POINT What was the muddiest point in today s (lecture, discussion, demonstration, reading, etc)? Students quickly identify something they did not understand Review responses and respond in next session

14 STUDENT GENERATED TEST QUESTIONS Prior to exam, ask students to submit questions in format of exam course uses Must provide answers also Possibly use some of the questions on exam What do students see as important points (insights into how they study)

15 THINK-PAIR-SHARE Give your students a question or problem to solve Ask them to spend 2-5 minutes thinking alone Discuss their ideas with the student sitting next to them Then share their ideas with the class

16 AUDIENCE RESPONSE SYSTEM Clickers Immediate feedback from students Interactive engagement Just in time teaching JiTT Use for pre-/post- test Attendance, opinions, monitor individual students If results indicate confusion, can be coupled with Think-Pair-Share (Vote again)

17 NON VERBAL CUES Use eye contact as a tool for continuous feedback

18 FOCUS GROUPS Small groups of 7-10 students with experienced moderator Use standardized open-ended questions Help improve the planning and design of the program/ course Can ask follow-up questions Provide a means of evaluating existing programs Carefully planned, but appear spontaneous Potential Problems Less control over group Uncertainty about accuracy since results could be biased by presence of dominate or opinionated member

19 DESIGNING EFFECTIVE QUESTIONNAIRES Make aims clear from the outset To identify student preference for online tutorials or printed worksheets at training sessions or to gauge student reaction to a new interactive library orientation session Questions should be tightly focused to the aim Only ask about aspects of your teaching you are in a position to act on as a result of responses received Keep it short - lots of white space Arial font Use a combination of closed and open questions Which of the two exercise formats used in this session did you prefer? Online tutorials Printed Worksheets No Preference Why (if you specified a preference) did you prefer this format?

20 CAT ARE MUTUALLY BENEFICIAL Active student participation required Content reinforced Students strengthen self-assessment skills Instructors work closely with students to improve their teaching skills and gain new insights Applicable in all teaching settings

21 PEER ASSESSMENT TECHNIQUES

22 PEER ASSESSMENT High validity for judging course goals and objectives, course content, and appropriateness of tests or assignments Formative assessment tends to have more value than summative (Berk 2005, Seldin 1999, Keig and Waggoner 1995) Don t wait for formal faculty peer evaluations Potential for bias

23 EFFECTIVE PEER REVIEW Flag areas of concern, aspects of your teaching upon which you value the opinion of your partner Agree upon the form in which the data/evidence will be collected and presented at the discussion Set aside time to reflect

24 PEER ASSESSMENT METHODS Informal formative classroom observations (by others) Informal formative classroom observations (of others) Evaluation of course materials (syllabi, goals/ objectives, lectures, outlines) Swapping of best practices Teaching awards by institution

25 SELF-ASSESSMENT TECHNIQUES

26 SELF-ASSESSMENT The Goal of self-assessment is to help teachers identify and make decisions about their strengths and weaknesses (Airasian and Gullickson, 1997) Many teachers do not have a systematic approach to help them improve their instructional behaviors (Keller and Duffy, 2005)

27 METHODS OF SELF-ASSESSMENT Reflection/checklist Videotape Audiotape least obtrusive

28 REFLECTION/CHECKLIST What was the purpose of the session? Did I have any concerns about the session beforehand? Which parts of the session went well and why? Which parts of the session did not go well and why? Were the learning outcomes achieved? What have I learned that can help me improve my performance?

29 METHODS OF SELF-ASSESSMENT Low-Inference self-assessment measure (LISAM) Self-assessment without an objective data source is usually inaccurate. (Freiberg and Driscoll 2000)

30 LOW-INFERENCE SELF-ASSESSMENT MEASURE (LISAM) Focuses on verbal interaction between teacher and learners Captures a slice of classroom interaction Audiotape and/or videotape a 20-minute interaction with your learners Focuses on behaviors that are observable and alterable (Anderson and Freidberg 1995)

31 LISAM Six areas of instruction 1. Questioning skills 2. Teacher talk/student talk 3. Identification of motivating set (introduction) and closure 4. Wait time 5. Number of positive statements made by the teacher 6. Number of times the teacher uses student ideas Listen/review the tape and transfer verbal interaction into frequency counts Provides some objective information on your verbal interaction with learners

32 QUESTIONING SKILLS Review verbal interaction to determine the quantity and types of questions used Opinion (evaluate) Comparison (analyze/synthesize information) Short answer (understand/apply) Yes-No (knowledge/recall) Strive for a balance of 60-40% between factual and higherlevel questions Tally types of questions used

33 TEACHER TALK/STUDENT TALK This ratio captures the level of teacher dominance (high levels of teacher talk) Teacher dominance prevents student participation and responsibility for learning Score by tallying who is speaking every three seconds

34 IDENTIFICATION OF MOTIVATING SET AND CLOSURE Introduction (focus) to the lesson Lesson closure (review, summarize) Describe in writing the introduction and closure Opening Body Closing

35 WAIT TIME Amount of time a teacher should wait after asking a question Research supports waiting 3-7 seconds Short wait times encourage short, recall responses or no answer Longer wait times increase: Length of student response Number of unsolicited responses Frequency of student questions Number of responses from less capable students Student-student interactions Incidence of speculative responses Time the interval between your question and any teacher statement prior to a student s response

36 POSITIVE STATEMENTS Identify (count) the number of positive statements (praise) made by the teacher during the lesson Make a list of positive statements and cite pet phrases Identify # of statements made by the teacher Cite # of times student names were incorporated

37 USE OF STUDENT IDEAS Research indicates that when students answers are expanded or extended by the teacher, students recognize their answer has greater importance Validation Identify the # of times student ideas were incorporated into the lesson/discussion

38 LISAM EXAMPLE (First few minutes, then skip to minute 40)

39 REVIEW Describe at least two different methods to obtain feedback from learners Describe a quantitative method to analyze teaching Create a strategy for incorporating self-analysis of teaching A teacher affects eternity; he never can tell where his influence stops. -Henry Adams, The Education of Henry Adams, 1907