Module 2: Evaluation Essentials

Transcription

1 Do, Measure, Improve Module 2: Evaluation Essentials Module 2: Evaluation Essentials Types of research o Basic new knowledge or evaluation? Framing evaluation research NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 2 1

2 Types of Research Different purposestypically lead to different ways of conceptualizing problems, different designs, different types of data gathering, and different ways of publicizing and disseminatingfindings. (Patton, 2002, p. 222) NatGuild2014 Kamella Tate MFA, EdD 3 Types of Research (variation on Patton, 2002, pp ) 1. Basic/Applied New Knowledge 2. Evaluation a. Formative Summative NatGuild2014 Kamella Tate MFA, EdD 4 2

3 Types of Research (Patton, 2002, pp ) Basic New Knowledge Research To contribute to fundamental knowledge and theoretical understanding about basic human and other natural processes. We have reached a milestone in our understanding of nature... Likely to shed light on other mysteries of our universe. NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 5 Types of Research (Patton, 2002, pp ) Applied New Knowledge Research To produce (generalizable)new knowledge focused on answering practical, real-world questionsand provide relativelyimmediate solutions. Why don t students like school? NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 6 3

4 Types of Research (Patton, 2002, pp ) Formative Evaluation Ongoingimprovement of products, processes, strategies, change interventions, institutions, personnel, policies, etc.; to form or shapethe thing being studied (Patton, 2002, p. 220). NatGuild2014 Kamella Tate MFA, EdD 7 Types of Research (Patton, 2002, pp ) Summative Evaluation To determine the effectiveness of products, processes, strategies, change interventions, institutions, personnel, policies, etc.; to render judgments that inform decisions. NatGuild2014 Kamella Tate MFA, EdD 8 4

5 Distinguishing Between Basic/Applied and Evaluation Research NatGuild2014 Kamella Tate MFA, EdD 9 Framing Evaluation Research Assessment and evaluation basics Characteristics of evaluations NatGuild2014 Kamella Tate MFA, EdD 10 5

6 What we mean by... Assessment Assessment is the organized and ongoing processof collecting and analyzing data and information so as to describe activities, practices, progress, and other dimensions of performance. What s happening? NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 11 What we mean by... Evaluation Evaluations are systematic investigationsthat involve synthesizing and integrating assessment data and then using this information to make inferences and judgments about: the merit(i.e., quality, excellence) the worth(i.e., value, cost-effectiveness) and/or the significance(i.e., importance, impact) of a project, program, or organization. Is it working/not? Why is it working/not? NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 12 6

7 So what? Documentation: What s happening? What s our story? Planning and improvement: Evidence-based quality management. Effectiveness: What s working? What snot working? Advocacy: Find the case. Make the case. Evaluation is accountability-in-action. NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 13 A Note On Advocacy In the same way that an (effective) grant proposal or application is an artifactof a (effective) program/project design process... (Effective) advocacy is an artifactof (effective) assessment and evaluation practices. Systematic. Organized. Purposeful. NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 14 7

8 A Planning & Evaluation Flowchart NatGuild2014 Kamella Tate MFA, EdD 15 Learn More Basically Speaking Part 1: Basically Speaking Part 2: NatGuild2014 Kamella Tate MFA, EdD 16 8

9 An Evaluation Continuum INFORMAL EVALUATION May be more involved with: Day-to-day operations Improvisational, unstructured Action-oriented, problemsolving Staff, clients, participants FORMAL EVALUATION May be more involved with: Systems, policies, models Structured activities, workplans Organizational development Multiple stakeholder groups within the community NatGuild2014 Kamella Tate MFA, EdD 17 Questions and Strategies INFORMAL QUESTIONS What's happening? How are things going? Did you get X done? What do we need to do to get X done? How did they (patrons, students, visitors) like it? Where are we on the budget? How are the reviews? How many people came? FORMAL STRATEGIES Developing scaled surveys and doing statistical analyses Seeking input from a variety of stakeholders Using an outside evaluator Conducting interviews and holding focus groups Measuring impact; cause-effect Publishing white papers NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 18 9

10 Characteristics of Evaluations Evaluation Purpose Audience & Setting Methods & Data Improve practice (formative) Determine value or worth (summative) Users and stakeholders Primarily local (program, organization, community) Quantitative Qualitative Multi/Mixed Generalizability Utility Unimportant Critical BASIC NEW KNOWLEDGE RESEARCH NatGuild2014 Kamella Tate MFA, EdD 19 Evaluation Purposes Formative = Improvement What are a program s strengthsand weaknesses? What are some of the threats and opportunities facing an institution? What s happening?how are things going? o Terms used:recommendations, promising, growth, quality, identify needs, enhancement, suggestions, learning organization, meet challenges, overcome barriers, advance, potential, etc. NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 20 10

11 Evaluation Purposes Summative = Effectiveness Did the program accomplishits purpose? Did the efforts taken ensure the institution achievedits goals? Did it work? o Terms used:conclusions, impact, influence, effect/ effectiveness, results, changes, outcomes, outputs, consequences, benefits, end products, achievements, bottom line, etc. NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 21 Quick Quiz Which evaluation question is formative? 1. As indicated by scores on a skills test, how effective was ScienceGrrrl s From Needs to Outcomes Logic Modeling" seminar? 2. How can ScienceGrrrl enhance its From Needs to Outcomes Logic Modeling" seminar? NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 22 11

12 An Audience for Evaluation Intended Users Program Participants & Personnel [Clients, Customers, Staff, Managers] Organizational Leaders [Directors, Boards, Decision-makers, etc. Funders Stakeholders Community-at-Large Funders, Philanthropists Field, Profession, Sector Civic and Political Leaders Neighbors Researchers Society Etc. NatGuild2014 Kamella Tate MFA, EdD 23 Methods & Data It was a great step in sciencewhen men became convinced that in order to understand the nature of things, they must begin by asking not whether a thing is good or bad, noxious or beneficial, And how much is there of it? but of what kind it is? Qualityand Quantitywere then first recognized as the primary features to be observed in scientific inquiry. James Clerk Maxwell Address to the Mathematical and Physical Sections of the British Association, Liverpool, 15 Sep 1870 The Scientific Papers of James Clerk Maxwell(1890 edition, reprint 2003), Vol. 2, 217 NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 24 12

13 What kind? Qualitative (data, methods, designs) Qualitative data is evidence largely in the form of words andtext quotes, narratives, and stories, as well as descriptions of images, environments, behaviors, etc. NatGuild2014 Kamella Tate MFA, EdD 25 How much? Quantitative(data, methods, designs) Quantitative data is evidence in the form of numbers numerical indices that allow for counting, ranking, or scaled measurement. N = 404; 48.4%; st, 2 nd, 3 rd Strongly Disagree = 1... Strongly Agree = 4 NatGuild2014 Kamella Tate MFA, EdD ktate@musiccenter.org 26 13

14 Quick Quiz QUANT or QUAL? 1. You count how many times people ask questions. 2. Vivid, well-written descriptions are what you find most useful. 3. Zip code NatGuild2014 Kamella Tate MFA, EdD 27 Collecting Two Types of Data Primary methods are in boldface italics Oral Interviews Observations Written or Oral Surveys Collecting Quantitative Data Diagnostic classifications and ratings Records of quantities Frequency counts and rates Ratings, opinion polls, surveys, tests, performance metrics, scaled questionnaires, etc. Collecting Qualitative Data Un-or Semistructured interviews Focus groups Observational fieldwork Extant documents and artifacts Open-ended written or oral questionnaires NatGuild2014 Kamella Tate MFA, EdD 28 14

15 Myth #1 Quantitative methods are more scientific and/or more rigorous than qualitative methods. Myth #2 Qualitative methods are easier than quantitative methods. NatGuild2014 Kamella Tate MFA, EdD 29 Where there are numbers, we also need words. Where there are words, we also need numbers. NatGuild2014 Kamella Tate MFA, EdD 30 15