Community-Based Prevention Marketing Step 7: Monitor and Evaluate Full Transcript

Transcription

1 Community-Based Prevention Marketing Step 7: Monitor and Evaluate Full Transcript Introduction In this step, titled, "Monitor and Evaluate," the coalition will learn about the importance of monitoring and evaluating the chosen policy initiative. The university partner should take the lead in developing a comprehensive plan for monitoring policy implementation and evaluating its impact. Step 7 will help the coalition understand concepts and questions they should consider when developing this plan. For monitoring and evaluation, information is collected about barriers to successful implementation, unanticipated consequences of the policy, and the policy s impact. This will help your coalition assess the policy s potential benefits and enable them to share findings with other coalitions so that they may consider promoting the same policy in their communities. Objectives By the end of this module, you will be able to: help the coalition understand the importance of monitoring and evaluation, and consider the questions to ask when developing a monitoring and evaluation plan in collaboration with a university partner. Together with the marketing plan and budget developed in step 6, the monitoring and evaluation plan will help the coalition monitor its progress, and use the information gained to decide if the policy initiative needs to be adjusted to better meet its goals.

2 Monitoring The first major objective of step 7 is to help the coalition understand the importance of monitoring the implementation of your chosen policy. Since consequences of a policy are never fully known in advance, it is essential to monitor a policy after it has been implemented. The information collected can be used to assess the impact of the policy on beneficiaries, stakeholders, and decision makers. Monitoring is primarily concerned with facts. It answers the questions How did a policy outcome occur? and Why did a policy outcome occur? In addition to understanding the concept of monitoring, we must also answer the question: Why do we monitor? For the answer to this question, click the tab on the left. Also, to understand the practical applications of monitoring, we will explore a case study. Our working example for this study is the implementation of the USDA's nutritional standards for the national school breakfast and lunch programs. To access this case study, click the "Team Nutrition" pin. Monitoring: Why Do We Monitor? Monitoring allows us to assess whether we are doing the right things and how well we are doing them. It can also help us identify where we can improve policy implementation. Monitoring the implementation process can also answer questions like, Does the policy initiative respond to stakeholder interests?, Is the policy initiative sustainable in the community?, and Does the policy initiative reinforce the mission of our coalition? Finally, monitoring the policy can provide evidence to support further funding of these types of activities in the community, and offer the chance to share lessons learned during the process with other coalitions. Click the movie keychain to view Robert J. McDermott, Fellow of the Florida Prevention Research Center at the University of South Florida's College of Public Health, as he outlines the benefits of monitoring. Case Study Beginning July 1, 2012, the US Department of Agriculture established new nutritional standards for the school systems that aligned with dietary guidelines for the American public. Once implemented, monitoring efforts on several fronts helped to assess progress and measure effectiveness in the schools. Click each section below to learn more about how the implementation of this nutritional standard was monitored through four major functions: compliance, auditing, accounting, and explanation.

3 Compliance: Individual state agencies were responsible for monitoring the schools to ensure compliance with dietary specifications. Auditing: As a result of the updated standards, additional resources were made available to school food programs to cover the added costs of meeting the new meal requirements. School officials audited their capacity for meeting these standards. Accounting: A study of 330 Minnesota school districts revealed that healthier meals required additional labor costs. By using an accounting approach, the school districts were able to measure the need to hire additional staff. Explanation: Monitoring the outcome of the new dietary requirements in schools allowed school officials to explain how this change in policy improved healthy eating habits. Evaluation In addition to monitoring, evaluation is another important process to consider when planning your policy initiative. Whereas monitoring looks forward, focusing on the implementation of a policy, evaluation looks back to assess what has happened so far. Evaluation allows us to know whether we have accomplished our goals and made an impact. While evaluation is often considered the final phase of the policy initiative, it plays a role throughout the process. Though often not considered when developing a policy initiative, evaluation can be a very useful tool. The university partner should focus on answering one key question during evaluation: Has our policy initiative made a difference? To complete our understanding of evaluation, we must answer four other questions: Why do we evaluate?, What do we evaluate?, How do we evaluate?, and finally, Who should evaluate? Click the tabs on the left to explore these questions in more detail. Click the photo

4 of the facilitator for an important note about the coalition's unique role during the process of evaluation. Finally, click the movie keychain to listen in as Robert J. McDermott describes the strengths of the coalition-university partnership during this step. Facilitator Note Before we continue our discussion of evaluation, it s important to note that evaluation can feel like a daunting task to those unfamiliar with it. Let s be clear what the coalition s role in evaluation is: providing community knowledge and guidance to help experienced evaluators develop and implement an evaluation plan. It is not the coalition s job to conduct an evaluation! Your primary emphasis during implementation of your initiative is monitoring the progress of the work you re doing. Again, the collective knowledge of the coalition is essential for evaluators to be successful on your behalf. Why Do We Evaluate? There are several good reasons to evaluate your policy. First, collecting data on the impact of a specific policy can improve the initiative s credibility for stakeholders and the wider community. Second, answering evaluation questions can help stakeholders track progress, and determine if the policy has been implemented as planned; this is known as implementation fidelity. Third, evaluation activities allow you to make mid-course revisions when necessary, thus continually improving the implementation process. Finally, conducting an evaluation of your policy can provide evidence that the work is meeting stated objectives. What Do We Evaluate? Deciding what to evaluate involves consideration of several factors. Generally, a policy initiative promotes a change in behavior, knowledge, or beliefs; improved awareness of a problem; or a change in the environment. Depending on the focus of your initiative, you may want to measure things like: changes in participation and utilization, modifications to the environment or policies that support change,

5 and changes in community infrastructure. The evaluation worksheet and example provided in the resources area will help you understand the types of questions to ask and draft your own questions for your specific policy initiative. For examples of the types of questions you might ask, click the "Questions" paper on the right. Questions In general, you may want to ask the following types of questions: What actions have the coalition and other stakeholders taken to effect policy change? How do we define success? For our coalition? For each of our audience groups? Finally, perhaps the most important question to ask is: Given our timeframe and available resources, what information can we gather? How Do We Evaluate? Once you have chosen the questions you want to answer, the coalition then decides how to obtain the information needed. This can be done in several ways. First, the coalition should look for information that already exists. Many local, state, and federal agencies publish reports, directories, and survey data that can be very helpful. Using existing data can save the coalition time, money, and other valuable resources. However, some evaluation questions can only be answered by collecting information specific to your policy, in your local community. There are many ways to collect new data. Coalition members and university partners can observe the behavior of priority audiences, documenting any changes. You might also interview audience groups identified in step 4, or use focus groups or questionnaires to survey them. The Data Collection worksheet provided in the resources area will help your coalition decide what type of data to collect. This is one of the activities for the Step 7 meeting and will be discussed in greater detail in a moment. Click the video keychain to listen in as Robert J. McDermott discusses the advantage of data in evaluating outcomes.

6 Who Should Evaluate? Given the complexity of evaluation design, it is critical to work closely with university partners who have experience designing evaluation plans and collecting data. This is also a good time to draw on coalition members with knowledge of evaluation. Other potential contributors include community members, students, and consultants. As you begin to formulate the monitoring and evaluation plan, it will be more clear who will need to assist in this process. Before the Coalition Meeting Several tasks need to be completed in preparation for the coalition meeting. These tasks include selecting an evaluation lead, reviewing documents, and providing an introduction to coalition members regarding the evaluation process. Click the timeline dot labeled "Select" to get started. Select First, select an evaluation lead. This is likely to be a university partner or someone else with experience designing and implementing evaluation plans. Click the Review dot to continue. Review The facilitator, along with the evaluation lead, will begin by reviewing the CDC Evaluation Handbook provided in the resources area. This handbook discusses many of the same concepts outlined in this module in greater detail, and provides examples of key evaluation questions. It is also important to review the worksheets that the coalition will complete during the step 7 meeting. These include the monitoring worksheet, the evaluation worksheet and the evaluation worksheet example, and the data collection worksheet. These worksheets can serve as guides for planning the meeting s activities. Click the timeline dot labeled Provide to continue. Provide Finally, the facilitator should make copies of the document titled, What is Program Evaluation?, located in the resources area, and make those copies available to coalition members. This provides a very brief introduction to evaluation for those unfamiliar with the process, and it should help prepare the coalition for completing the worksheets. Click any of the timeline dots below to review these tasks.

7 During the Coalition Meeting The tasks to be accomplished during the coalition meeting include reviewing the concepts, completing the worksheets, and designating a subcommittee. Click the timeline dot labeled Review to begin. Review First, the facilitator and evaluation lead will review the key concepts of monitoring and evaluation with the coalition. This presentation, along with the What is Program Evaluation? handout, are useful resources. If more detailed information is needed, reference the CDC Evaluation Handbook, which should be reviewed by the facilitator and evaluation lead before the meeting. Click the Complete dot to continue. Complete The role of the coalition is to help guide the evaluation lead in developing the types of questions needed to determine whether your initiative is effective. This is done by using the evaluation and monitoring worksheets provided. The monitoring worksheet provides much of the key information to help the coalition work through designing a plan for monitoring the policy initiative. There are two evaluation worksheets to help the coalition draft a plan. The first is a completed worksheet showing an example from a policy to promote physical activity in schoolchildren. This will guide the coalition in completing their own evaluation worksheet. This worksheet, once completed, will help the coalition work through the data collection worksheet. The data collection worksheet is designed to help the coalition think through how the evaluation lead will collect the information. Once the data collection worksheet is complete, the facilitator and evaluation lead will assign roles and tasks to complete the data collection process. Click the timeline dot labeled Designate to continue. Designate Finally, designate a monitoring and evaluation subcommittee to track progress in these areas and be responsible for keeping the data. Click any of the timeline dots below to review these tasks.

8 After the Coalition Meeting After the meeting, the facilitator and evaluation lead will work with data collectors to ensure tasks are completed. This is when a monitoring and evaluation subcommittee can begin coordinating these activities. It s important to remember that monitoring and evaluation are important activities throughout the entire advocacy process. Depending on the specific goals of the coalition, data may need to be collected before, during, and after policy implementation. These are issues that depend entirely on the coalition s unique needs and goals, and the evaluation lead can help the coalition answer these questions. The evaluation lead and subcommittee should schedule time to review monitoring and evaluation activities and track progress. Again, these meetings should depend on the needs and goals of the coalition. Note to Facilitator At this point, you should be ready to explain why your coalition needs to monitor and evaluate its progress, and how the evaluation lead will work with the coalition to complete these tasks. Remember, this is not a task for your coalition to take on independently. The coalition s emphasis is on monitoring the progress of the policy initiative. Your evaluation task is to provide guidance from a community perspective to the evaluation lead who provides the technical expertise. The information provided in Step 7, along with the worksheets and handouts in the resources area, will help the coalition coordinate with an evaluation lead to monitor policy implementation and evaluate its impact. Moving to the Next Step Congratulations on completing this learning module! You are now ready to move to Step 8, titled, "Are We Following the Plan?"