Teacher Effectiveness Webinar Series

Transcription

1 Teacher Effectiveness Webinar Series

2 Using Teacher Evaluation to Change Teaching

3 Welcome Elizabeth Greninger, Ph.D. Webinar facilitator, edcount, LLC Courtney A. Bell, Ph.D. Presenter, Understanding Teaching Quality Center, ETS Teacher Effectiveness Webinar Series 3

4 About REL Mid-Atlantic The Regional Educational Laboratory Mid-Atlantic (REL-MA) serves education stakeholders in five states: DE, MD, NJ, PA, and the District of Columbia. RELs respond to education needs of the region by producing rigorous research, providing technical assistance, and disseminating useful and accessible research-based products. TEACHER EFFECTIVENESS WEBINAR SERIES 4

5 About the webinar series The Teacher Effectiveness Webinar Series provides stakeholders in the Mid-Atlantic region with opportunities to discuss multiple topics within the field of teacher effectiveness. One of the goals of this series is to increase participants understanding of issues surrounding teacher effectiveness that will enable them to share, apply, or implement the knowledge in their classroom, school, or district. TEACHER EFFECTIVENESS WEBINAR SERIES 5

6 Poll: About the Audience Who is joining the webinar today? Teacher Effectiveness Webinar Series 6

7 Courtney Bell, Ph.D. Senior research scientist at ETS in the Understanding Teaching Quality Center Former high school science teacher and teacher educator Current studies use mixed-methods to analyze teacher learning, the measurement of teaching, and the validity of measures of teaching quality Serves on technical advisory committees for three Race to the Top states Co-editing the 5 th edition of the AERA s Handbook of Research on Teaching Currently completing two large validity studies of instruments designed to measure teaching and a third study collecting data in Los Angeles about how principals learn to use observation protocols Teacher Effectiveness Webinar Series 7

8 Using Teacher Evaluation to Change Teaching

9 Overview Professional Practice in Policy Context Teaching v. Teacher Quality The Validity Argument Approach A Theory of Score Use An Interpretive Argument for Improving Teaching and Exemplar Analyses Teacher Effectiveness Webinar Series 9

10 Professional Practice in Teacher Evaluation Policy context is characterized by: A mixture of definitions of the problem A mixture of implicit theories of professional improvement A mixture of uses for the scores A mixture of measures Teacher Effectiveness Webinar Series 10

11 Mixture of Problem Definitions Problematic definitions: Teaching practice is mediocre and teacher evaluation is a way to improve it There are some really poor teachers out there and some good ones. We need a reasonable way to sort these out and treat them accordingly. We need stronger public accountability at all levels of the system and teacher evaluation provides accountability at the teacher level. Stakeholders may have different concerns in teacher evaluation, and thus the purpose and design of the evaluations may be varied Teacher Effectiveness Webinar Series 11

12 Mixture of Improvement Strategies and Implied Theories of Motivation Mixture of theories for professional development: People will improve if you teach them People will improve if you have strong incentives and sanctions People will improve if their performances are more public Theories of improvement have implications for all people in the evaluation system, not just teachers Teacher Effectiveness Webinar Series 12

13 Mixture of Score Uses Two main purposes for the use of scores: To make high stakes personnel decisions To improve teaching Teacher Effectiveness Webinar Series 13

14 A Mixture of Measures Several measures have been used in evaluating teachers, including 1 : Observations Student learning objectives Student belief surveys Parent belief surveys Teacher reports of professional activities Etc. Little, Goe, & Bell (2009) Teacher Effectiveness Webinar Series 14

15 Poll: Measurement Methods Which of the following measures have you had experience with in evaluating teaching quality? Observations Student learning objectives Student belief surveys Parent belief surveys Teacher reports of professional activities Other Teacher Effectiveness Webinar Series 15

16 Overview Professional Practice in Policy Context Teaching v. Teacher Quality The Validity Argument Approach A Theory of Score Use An Interpretive Argument for Improving Teaching and Exemplar Analyses Teacher Effectiveness Webinar Series 16

17 Teacher and Teaching Quality Imagine two surgeons. We would like to evaluate them on the quality of their surgical skills using multiple measures. We will use the size of the scar, the rate of infection, the quality of pain management, and patient satisfaction as our measures of the quality of their surgical skills. One is in Miami, Florida, the other in Moshe, Tanzania. Both must remove a benign tumor from a 53 year old man s abdomen. The surgeon in Miami has a #10 blade steel scalpel that is designed for cutting muscle and skin. The surgeon in Moshe has a well sharpened utility knife that is used for a range of surgical purposes. The excision in Miami will occur in a sterile operating room with no external windows, fans and filters to circulate and clean the air, an anesthesiologist, and a surgical nurse. The excision in Moshe will occur in a clean operating room washed with well water and bleach, windows opened a crack to allow the breeze to circulate the stiflingly hot air, no fans or filters, and a nurse borrowed from the pediatrics unit because she was the only available help. Gitomer & Bell, 2013 Teacher Effectiveness Webinar Series 17

18 Teacher and Teaching Quality Teacher Quality assumes 1 : Stable attributes across contexts Fixed characteristics A focus on the characteristics of the teacher Teaching Quality assumes: Potential for systematic variability with context Potential for malleability A focus on the interactions of teaching and learning Gitomer & Bell (2013) Teacher Effectiveness Webinar Series 18

19 The contextual factors, constructs, and measures associated with teaching quality TARGET DOMAIN Contextual Community Curriculum Factors Students & Colleagues School Leadership Resources Teaching Policy Quality TEACHER CONSTRUCTS STUDENT CONSTRUCTS CONSTRUCTS Teacher Knowledge Teacher Practices Teacher Beliefs Student Beliefs Student Practices Student Knowledge MEASURES Content Knowledge for Teaching Tests Knowledge of Teaching Tests Belief Instruments Observation Measures Artifact Measures Graduation Rates Growth Models Value-Added Methods Course Taking Patterns Teacher Portfolios Student Portfolios Bell, Gitomer, McCaffrey, Hamre, Pianta & Qi (2012) Teacher Effectiveness Webinar Series 19 Confidential and Proprietary. Copyright 2010 Educational Testing Service. All rights reserved.

20 Three Reasons to Clarify Constructs and Measures It is important to have a clear definition of our construct and measures to: Be able to make causal inferences To interpret outcomes and make sensible revisions Design quality control, evaluation, and research plans We must be clear about what we can and cannot claim in evaluation Teacher Effectiveness Webinar Series 20

21 Open-Ended Question: Defining Constructs What constructs are being evaluated in your school/district? Teacher Effectiveness Webinar Series 21

22 The contextual factors, constructs, and measures associated with teaching quality TARGET DOMAIN Contextual Community Curriculum Factors Students & Colleagues School Leadership Resources Teaching Policy Quality TEACHER CONSTRUCTS STUDENT CONSTRUCTS CONSTRUCTS Teacher Knowledge Teacher Practices Teacher Beliefs Student Beliefs Student Practices Student Knowledge MEASURES Content Knowledge for Teaching Tests Knowledge of Teaching Tests Belief Instruments Observation Measures Artifact Measures Graduation Rates Growth Models Value-Added Methods Course Taking Patterns Teacher Portfolios Student Portfolios Teacher Effectiveness Webinar Series 22 Confidential and Proprietary. Copyright 2010 Educational Testing Service. All rights reserved.

23 Overview Professional Practice in Policy Context Teaching v. Teacher Quality The Validity Argument Approach A Theory of Score Use An Interpretive Argument for Improving Teaching and Exemplar Analyses Teacher Effectiveness Webinar Series 23

24 How do we think about evaluating professional practices within a teacher evaluation system? Kane s (2006) validity framework suggests 1 : Interpretive argument Describe the chain of inferences and assumptions that connect observed performance to claims Validity argument Gather empirical evidence or logical support that demonstrates the truthfulness of the chain 1 Kane (2006) An Argument Approach to Validation Teacher Effectiveness Webinar Series 24

25 A Simple Example An Interpretive Argument Luke can ride his bike on our sidewalk the same way he rides it on all the sidewalks on the way to school. When Luke falls down he can get up by himself and decide what to do next. Luke can cross the street safely. Luke knows the way to school. Teacher Effectiveness Webinar Series 25

26 A Simple Example A Validity Argument All summer we watch Luke ride his bike on many of the sidewalks in our neighborhood and he is always able to ride. Luke fell twice over the summer. One time he just sat and cried. The other time he got up and kept going. 50% of the time Luke looks both ways. On 3 occasions, Luke was able to guide us to school. Teacher Effectiveness Webinar Series 26

27 Critical Aspects of Validation What does this mean for our evaluation process? Decide what the purpose is of the evaluation Decide what we can claim about a teacher and how we arrive at that that claim People will still disagree on what evidence shows Clarity on the interpretive and validity arguments helps everyone understand how evaluation measures will be used Teacher Effectiveness Webinar Series 27

28 Open-Ended Question: Assumption and Purpose How is teacher evaluation used within your school or district? Teacher Effectiveness Webinar Series 28

29 Overview Professional Practice in Policy Context Teaching v. Teacher Quality The Validity Argument Approach A Theory of Score Use An Interpretive Argument for Improving Teaching and Exemplar Analyses Teacher Effectiveness Webinar Series 29

30 Theory of Use for a Professional Practice Measure Information Professional practice measure creates information about the level of teaching skill demonstrated by each teacher New Insights Information leads to insights about teaching Practice and Learning Insights lead to the acquisition of new understandings and strategies through professional development, utilization of new tools, etc. Changes in Teaching Understandings and strategies are incorporated into teaching practice, making it more effective Changes in Learning Effective teaching practice results in improved student learning Bell (2012) Teacher Effectiveness Webinar Series 30

31 Importance of the Theory of Use Establishing a theory of use for every professional practice measure is important for clarity What information does the measure provide you? How will that information be used and applied to practice? What are the expected changes to which this information will lead? Teacher Effectiveness Webinar Series 31

32 Overview Professional Practice in Policy Context Teaching v. Teacher Quality The Validity Argument Approach A Theory of Score Use An Interpretive Argument for Improving Teaching and Exemplar Analyses Teacher Effectiveness Webinar Series 32

33 Information on Skill Level Claim 1: Professional practice measure creates information about the level of teaching skill demonstrated by each teacher Assumptions: 1. The scoring rule is appropriate. 2. The scoring rule is applied accurately & consistently. 3. The scoring is bias free. 4. The data fit the scoring model. 5. The sample adequately represents the quality of all relevant lessons. 6. The score on all lessons is related to the teaching quality teachers and students are able to enact. 7. There are not systematic errors that undermine the extrapolation to teaching quality. Teacher Effectiveness Webinar Series 33

34 Information on Skill Level - Analyses Assessing the claim Can observers tell the difference between some and most students? High-level questions and mid-level questions? How much do observers agree on their observations? Teacher Effectiveness Webinar Series 34

35 Information on Skill Level - Analyses (cont d) Assessing the claim Do scores vary depending on who conducted the observation? What time of year the observation was done? Which class was being observed? How much does an individual teacher s score change from year to year? How do systematic errors appear in the scores (e.g., the influence of curriculum, co-teaching)? Teacher Effectiveness Webinar Series 35

36 New Insights Claim 2: The information gathered will lead to new insights Assumptions: 8. Stakeholders understand the information the system produces. 9. Insights are appropriately and accurately related to the system information. 10. Insights are actionable. Teacher Effectiveness Webinar Series 36

37 New Insights Analyses Assessing the claim What did evaluators and teachers learn from the evaluation scores? Would a third party agree with the evaluator s feedback? Teacher Effectiveness Webinar Series 37

38 Poll: Experience and Practice Which of the following do you think would be most likely to lead to new insights about teaching? Stakeholders understand the information the system produces Insights are appropriately and accurately related to the system information Insights are actionable Teacher Effectiveness Webinar Series 38

39 Practice & Learning Claim 3: Insights lead to acquisition of new understandings and strategies, through professional development and utilization of new tools Assumptions: 11. The implications associated with teaching performance are appropriate. 12. The properties of the observed scores on the lessons support the implications associated with the judgments of teaching performance. 13. Stakeholders link insights with appropriate tools, learning opportunities, etc. 14. Learning opportunities and resources are appropriate and of sufficient quality to support teacher learning. Teacher Effectiveness Webinar Series 39

40 Practice & Learning - Analyses Assessing the claim When you share performances at certain score levels, what are stakeholders reactions? What is the rate of misclassification error and level of uncertainty? To what degree does a third party agree with the specified areas of improvement identified for a teacher? What is the quality of PD available to teachers? Teacher Effectiveness Webinar Series 40

41 Change in Teaching Claim 4: Teachers will incorporate what they have learned from their professional learning opportunity into their teaching practice, thereby making it more effective Assumptions: 15. Organizational context supports incorporation of new understandings/strategies. 16. Changes in teaching practice substantively align and are attributable to use of resources/professional learning. Teacher Effectiveness Webinar Series 41

42 Change in Teaching - Analyses Assessing the claim How do stakeholders implement their new knowledge? Do teachers feel supported in their implementation of new knowledge? Randomly assign teachers with similar needs to different professional development/resources and study changes in teaching practices. Teacher Effectiveness Webinar Series 42

43 Changes in Learning Claim 5: Changes in teaching will result in changes in learning Assumptions: 17. Measures of student learning are sensitive to changes in teaching practice. 18. Changes in student learning are attributable to changes in teaching practice. Teacher Effectiveness Webinar Series 43

44 Changes in Learning - Analyses Assessing the claim To what degree does changes in teaching result in changes in student learning? Investigate how other measures might change as teaching improves; do those changes reflect what you theorized? Compare student learning on standardized tests with teacher reports of content coverage 1. Polikoff (2010) Teacher Effectiveness Webinar Series 44

45 Open-Ended Question: Changes in Learning Has teaching changed at your school? How has this affected educators understanding of student learning? Teacher Effectiveness Webinar Series 45

46 Don t Have 10 Million Dollars or Infinite Time? Distinguish between quality control (QC) and validation work and prioritize QC is less costly and can be conducted more regularly, while validation studies can be more expensive and more time consuming Both are important, but prioritize Pay special attention to score validity We must focus on establishing early on that scores mean what we intend them to mean Focus on measures with which we are most concerned Teacher Effectiveness Webinar Series 46

47 Conclusion: Two Cautions Treat teaching quality seriously Be mindful of the increasing validity burden moving toward teacher quality Teacher Effectiveness Webinar Series 47

48 Conclusion: Takeaways Measures of teaching quality are often measuring different constructs. Under a validity framework, you need to both implement a teacher evaluation system and determine the quality of evidence it is producing. Evidence for each of the assumptions is likely to have some ambiguity. To ensure teacher evaluation scores lead to improvements in teaching, monitor how well your theory of action is supported with empirical evidence. Teacher Effectiveness Webinar Series 48

49 Key Takeaways 1. What are your key takeaways from today s presentation? 2. As a result of today s webinar, what action steps do you plan to take? Teacher Effectiveness Webinar Series 49

50 Questions & Answers Teacher Effectiveness Webinar Series 50

51 Contact Information Elizabeth Greninger Courtney Bell Teacher Effectiveness Webinar Series 51

52 Upcoming Webinars Teacher Effectiveness Webinar Series Formative Assessment as the Key to Effective Instructional Practice Presenter: Margaret Heritage, Ph.D. (University of California) June 19, 3:00pm EST Ask an Expert Online Chat Series Restructured Teacher Pay Systems Presenter: Dan Goldhaber (University of Washington) May 27, 3:00pm EST Teacher Effectiveness Webinar Series 52

53 Please take a minute to complete our participant satisfaction feedback survey! Teacher Effectiveness Webinar Series 53

54 Thank you for your participation!

55 References Bell, C.A. (2012). Validation of professional practice components of teacher evaluation systems. Paper presented at the Center for Assessment s 14 th annual Reidy Interactional Lecture Series. Bell, C.A., Gitomer, D.H., McCaffrey, D., Hamre, B., & Pianta, R., Qi, Y. (2012). An argument approach to observation protocol validity. Educational Assessment, 17, Gitomer, D. H., & Bell, C. A. (2013). Evaluating teaching and teachers. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology (Vol. 3, pp ). Washington, DC: American Psychological Association. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational Measurement (4 th edition), pp Westport, CT: American Council on Education/Praeger Publishers. Little, O., Goe, L., & Bell, C. (2009). A practical guide to evaluating teacher effectiveness. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from Polikoff, M.S. (2010). Instructional Sensitivity as a Psychometric Property of Assessments. Educational Measurement: Issues and Practice, 29, Teacher Effectiveness Webinar Series 55