Safe and Healthy Minnesota Students Planning and Evaluation Toolkit

Transcription

1 Safe and Healthy Minnesota Students Planning and Evaluation Toolkit AUGUST 2008

2 Safe and Healthy Minnesota Students Planning and Evaluation Toolkit Tips, techniques, and resources for planning and evaluating alcohol, tobacco, other drug and violence (ATOD-V) prevention programs Prepared by: Cheryl Holm-Hansen Amy Leite Julie Atella Wilder Research 451 Lexington Parkway North Saint Paul, Minnesota Development of the Safe and Healthy Minnesota Students Planning and Evaluation Toolkit was supported by the Minnesota Department of Education through funds from a Safe and Drug-Free Schools and Communities National Programs grant, CFDA R. This toolkit may be reproduced for private, non-commercial and education purposes. All reproductions require an acknowledgement of the source and authors of the work. No resale use may be without written permission. Introduction to program evaluation Wilder Research, August 2008

3 Table of CONTENTS 1 Introduction to program evaluation... 1 Why should I consider conducting a program evaluation?... 1 Things to think about before beginning an evaluation... 2 The evaluation process Identifying the information needs of stakeholders... 5 Identifying potential stakeholders... 5 Prioritizing stakeholder groups... 6 Determining stakeholder needs... 7 Involving stakeholders in the evaluation process... 8 Some things to consider when evaluating youth programs Needs assessment What is needs assessment?...11 Why should community assessments be conducted?...12 When should community assessments be conducted?...13 How are we going to do this?...14 Using community assessment results Program selection Effective atod-v interventions and practices...17 Practical considerations for program selection...19 Where to find evidence-based programs and practices Evaluation readiness Resources for evaluation capacity...24 External evaluators...26 Finding external evaluators...29 How much will it cost?...29 How long will it take?...31 Evaluability assessment...33 Evaluation as a priority...34 For more information...35 Introduction to program evaluation Wilder Research, August 2008

4 6 Program theory and logic models Sample outline for a program theory...38 Here s how to start...40 How does a theory differ from a logic model?...40 How to build a logic model Understanding evaluation issues Types of evaluations...48 Return on investment...51 Writing evaluation questions...51 Tips for prioritizing your evaluation questions...53 Setting evaluation standards Protecting human rights Key ethical issues related to program evaluation...57 Federal privacy laws...60 Key ethical issues related to evaluator s roles...62 Addressing ethical challenges Data collection approaches Data collection approaches...65 Evaluating youth programs...66 Cultural considerations...66 Collecting new information...67 Creating your data collection plan...69 Time and budget considerations Reviewing existing data collection tools Characteristics of good evaluation tools...73 Locating existing material...75 Questions to ask when reviewing existing materials Writing and developing data collection tools When to consider using an interview...78 Types of interviews...78 The interview process...79 Conducting focus groups...82 Writing good surveys...83 Developing your survey...84 Other things to consider...89 Introduction to program evaluation Wilder Research, August 2008

5 12 Use of existing data Using primary versus secondary data...93 Using the portal...93 Other resources Carrying out the evaluation Involving stakeholders as the evaluation is in process...97 Common pitfalls to avoid related to budget and staff time Organizing and analyzing information Analyzing quantitative data...99 Analyzing qualitative data Interpreting your results and drawing conclusions Sharing evaluation results with stakeholders Select the right communications strategy Research Findings Implications Recommendations Cultural considerations for presenting results Writing the report Consider creative strategies for disseminating results Invite stakeholders to review and discuss the results Using evaluation results Evaluation results for program improvement Develop an action plan Tips for using evaluation results for program improvement Evaluation results for policy development and advocacy Evaluating prevention program Program theory Evaluation design Pilot programs Introduction to program evaluation Wilder Research, August 2008

6 Appendix Stakeholders and priorities [Form 1] Prioritizing needs assessment findings [Form 2] Research skills checklist [Form 3] Organization capacity worksheet [Form 4] Questions to cover in the interview [Form 5] Strategies for promoting capacity building [Form 6] Budget template [Form 7] Logic model example 1 [Form 8] Logic model example 2 [Form 9] Program theory development [Form 10] Logic model template [Form 11] Determining project goals [Form 12] How to build a logic model [Form 13] Logic model checklist [Form 14] Creating evaluation questions [Form 15] Determining evaluation needs [Form 16] EXAMPLE: Student assent form [Form 17] EXAMPLE: Passive parent consent form [Form 18] EXAMPLE: Active parental consent form [Form 19] Selecting the best data collection method [Form 20] Which data collection approach for us [Form 21] Which data collection approach for us [Form 22] Focus group checklist [Form 23] Focus group protocol [Form 24] Budget check-in [Form 25] Work plan [Form 26] Who, what, where [Form 27] Research implications [Form 28] Action plan [Form 29] References Stakeholder engagement Needs assessments Program selection Evaluation readiness Program theories and logic models Understanding and prioritizing evaluation questions Protecting human subjects Data collection approaches Reviewing existing data collection tools Writing and developing data collection tools Use of existing data Carrying out the evaluation Organizing and analyzing information Sharing evaluation results with stakeholders Using evaluation results Challenges of evaluating prevention programs Introduction to program evaluation Wilder Research, August 2008

7 USING THE SAFE AND HEALTHY MINNESOTA STUDENTS PLANNING AND EVALUATION TOOLKIT This toolkit, developed by Wilder Research for the Minnesota Department of Education Safe and Healthy Minnesota Students program, is intended to increase the planning and evaluation capacity of K-12 Minnesota schools and their community partners as they implement alcohol, tobacco, other drugs and violence (ATOD-V) prevention programs. The design ensures that schools and their partners have access to planning and evaluation guidelines that are useful, relevant, understandable, and driven by effective practices. Evaluation topics are separated into sections, such as engaging stakeholders and using evaluation results. Each section provides an overview of the topic, real-life examples to illustrate ideas and concepts, and practical, self-guided worksheets that schools and programs can use to help guide the design and use of an evaluation. In addition to worksheets, the toolkit is designed to support itself in that each section links to others in the toolkit. Evaluation topics often overlap with one another, and it is common to revisit early evaluation topics as an evaluation proceeds. Each section also provides links to external web-based resources where programs can go for more information about specific evaluation topics. Introduction to program evaluation Wilder Research, August 2008

8 INTRODUCTION TO PROGRAM EVALUATION 1 Does it work? This is the question most often asked of any alcohol, tobacco, other drug and violence (ATOD-V) prevention program. Program evaluation, the systematic collection of information about the activities, process, and outcomes of programs, is often designed to answer this very question. Evaluation can be used to assess program design and implementation, improve program performance and demonstrate success of a program. A well-thought-out evaluation has some major benefits for your program and the people who use your services. It lets you assess the quality and results of your current services, improve specific things that aren t working as well as you want them to, and share the benefit of your experience with others like policy-makers and professionals. An evaluation does not have to be time-consuming or expensive in order to be useful. It is important, however, to ensure that the time you spend on it is time well spent. In this toolkit, we will focus on strategies for making your evaluation both useful and reasonable in terms of dollars and time. WHY SHOULD I CONSIDER CONDUCTING A PROGRAM EVALUATION? Program evaluation is often required by your funders. You may also have a board of directors or others who need to understand the program s results and challenges. But the most important benefit of a good evaluation is to help you monitor service delivery, assess the needs of participants or the community, and identify ways to improve your services or program. Additionally, evaluation results can provide a rationale for funding requests, by showing the strengths of a program or identifying a need for improved services. Information that is shared with others can be used to build other types of support as well, including recruiting staff or volunteers, engaging potential collaborators, or attracting participants. Program evaluation helps advance the understanding of effective service. There is increasing evidence out there about effective and ineffective ways of providing services, including ATOD-V prevention services. Evaluation is critical in this learning process. Introduction to program evaluation 1 Wilder Research, August 2008

9 You can identify helpful approaches, learn why some approaches may not work, and test new ways of addressing individual or community concerns. By sharing this information with others, you strengthen the effectiveness of others who are working toward the same goals. People who choose to work in programs like yours are typically motivated by the mission, so it is gratifying for them to see how the work has made a difference. Hearing feedback from participants or seeing evidence of program success can help to maintain the commitment of program staff. At the same time, evaluation can give participants a safe forum for passing along their experiences, allowing you to modify the delivery of services to better meets their needs. THINGS TO THINK ABOUT BEFORE BEGINNING AN EVALUATION Here are a few key considerations before starting an evaluation. These sections of the toolkit will walk you through the process of identifying the needs of your community and target audience, give practical tips for engaging stakeholders in the process, and guide the selection and development of a sound ATOD-V prevention program. Even if you have already conducted a needs assessment or have selected your program, these sections will still provide valuable information and might help to guide your thinking when it comes to evaluation. Know your audience. Before starting a new program or collecting any information, identify the most important customers for your program, such as program staff, current and potential funders, advisory boards, or others. Once you prioritize these groups, think about the specific types of evaluation information that would interest them most. Is there information that could help guide program decision-making? Is there information that you want to be able to share with others outside your organization? Assess the needs. Conducting a needs assessment is a useful way to establish the level of service need of a community, whether that includes students or the larger community. If done well, a quality needs assessment can not only help identify critical issues in a community, but can also identify community resources that are available to address these issues. The results can play a critical role in improving ATOD-V prevention programs and planning for future programs. Select the right program. Results from a needs assessment can help to identify and develop the right program or practice for your target audience and target issue. Often organizations and funders will look to evidence-based interventions and practices when designing and choosing a program. When selecting or developing a program, take into Introduction to program evaluation 2 Wilder Research, August 2008

10 account the capacity and support provided by your organization, adapt the program to the needs of your target population, and monitor implementation. THE EVALUATION PROCESS Without making the process cumbersome, it is important to take the right steps to ensure that you end up with useful information. This toolkit will outline the following steps to a solid evaluation and provide useful and practical tips for evaluation of ATOD-V prevention programs, including worksheets you can use to design, implement, and share your evaluation. Know your resources. It is important to have an accurate perception of the resources available for evaluation before you begin. You ll need to design a process that can be completed within the budget and time available. If no funding is available for evaluation, explore strategies for capitalizing on other resources (such as volunteer assistance). If current program staff members are not familiar with evaluation, consider options for obtaining evaluation assistance. Spell out your program s theory. Most programs are based on certain values and assumptions about how to best prevent ATOD and violence. A very useful step in the evaluation process is to identify and clarify your program s theory. This theory should clearly explain the reasons why you provide each specific service and the ways in which these services are expected to benefit participants or the community A program theory can help focus the evaluation on the most important expected results. Even if you don t conduct an evaluation, spelling out the program s theory can help to build a shared understanding of the program among stakeholders and guide program enhancements. Build a logic model. Logic models sound technical, but they are actually greatly appreciated by program staff and funders as a useful tool to visually display a program s underlying theory. Step back and prioritize. Typically, these first few steps yield many ideas, questions, and issues to explore in the evaluation. Before proceeding, take some time to review all of these and prioritize those that are most important to include in the evaluation plan, given the available resources. Introduction to program evaluation 3 Wilder Research, August 2008

11 Ensure ethical practice and protect participants. From the first planning discussions through the final report, you must consider ethical issues such as: How can we protect the confidentiality of those who provide information? How do we ensure that people willingly choose to participate? How can we minimize risk of any potential harm to participants while maximizing the potential benefit of the evaluation? Decide how to collect information. Once you have selected the key evaluation issues or questions, you can develop a plan for gathering the right information. This might include collecting information directly from participants. Other methods include documenting the services you provide, collecting information from staff or other knowledgeable observers, or monitoring information available outside your organization. All strategies for gathering information have strengths and weaknesses; considering them helps you choose the most appropriate and reasonable options for your situation. Create the tools for gathering information. These materials could include survey or interview questionnaires, forms or information systems used by program staff, or a wide range of other materials. It is important that the tools are easy to use and collect, and that they provide valid and reliable information. Start gathering information. During this phase, you ll need a way to monitor and ensure that all evaluation activities are completed as intended and on schedule. Regardless of whether you are collecting information directly from program participants, other close observers, or by some other method, you need strategies for gathering this feedback appropriately and accurately. Sort out the information. Many programs successfully gather evaluation information, but then struggle with the best ways to organize it and understand it. Your evaluation plan needs good strategies to analyze the information so that you answer your most important evaluation questions. Share the information. What are the best ways to share information with the staff of your program? What other stakeholders should receive the information, and in what format? What needs to happen, who can make it happen, and how can they best gain the information they need? You re not done until you use it. It s easy to feel that your evaluation is complete once the findings have been shared with funders, program staff, or other key people. However, the true value of an evaluation lies in how it is used. Your evaluation plan should include active methods for using the information. This can go beyond enhancing the program s service delivery, to include goals such as public awareness, advocacy, or policy development. Introduction to program evaluation 4 Wilder Research, August 2008

12 IDENTIFYING THE INFORMATION NEEDS OF STAKEHOLDERS 2 An important early stage in selecting a program or designing an evaluation is to identify the most important stakeholders for your program and evaluation. Stakeholders are individuals or groups who have an interest in your program, would be interested in the results of the evaluation, and might have a role in what will be done with the evaluation results. This section will help you identify stakeholders, identify and prioritize their interests in evaluation, and provide practical tips for engaging stakeholders. IDENTIFYING POTENTIAL STAKEHOLDERS A stakeholder is any person or group that has an interest in your program or its evaluation results. Several categories of individuals can often be considered stakeholders, including those directly involved in the operations of your program, those that are served by the organization, or those who will ultimately use and benefit from the evaluation of your program. Typical stakeholder groups include: Program participants Family members of participants Program management or other staff Board of Directors School officials Current or potential funders Collaborators Advocacy groups Policy makers General public Identifying the needs of stakeholders 5 Wilder Research, August 2008

13 Are there other potential stakeholders for your program that are not on this list? What other groups impact or are impacted by your program? PRIORITIZING STAKEHOLDER GROUPS Typically, a program has multiple stakeholders. In fact, you may feel that every group listed is a reasonable stakeholder for your program. Identifying stakeholders as part of an evaluation process can be valuable, as it allows you to consider the types of information that these groups will be most interested in before you proceed with your evaluation. However, it is likely that these groups will be interested in different issues, and you may not be able to adequately meet the information needs of all of your target audiences. Therefore, it is important to carefully prioritize among these groups. Take time to consider the following questions: Are there groups, such as funders or a Board of Directors, to which you have a contractual obligation to provide evaluation information? If so, what are you required to provide? Is there information that you would like to receive from participants, such as descriptions of the benefits of services, recommendations for program enhancements, or clarification of their needs Are there significant decisions facing your program in terms of the nature or amount of services that can be provided? If so, is there any information that would be helpful in making these decisions? Are there groups that can be helpful in ensuring program continuation or expansion? Do you need to solicit funding from new sources to meet your programming goals? What information would be helpful in this process? Are you currently collaborating with other agencies or organizations? Is there information that potential collaborators would want to know about your program? Does your program address an issue that is important to the general public? If so, are you interested in collecting information that will help shape their perceptions of this issue or effective service options? Given your answers to these questions, which stakeholders do you feel are most important as you consider your evaluation needs? If you re unsure, talk to others in your organization, as interesting insights can emerge from a group discussion on the topic. Identifying the needs of stakeholders 6 Wilder Research, August 2008

14 DETERMINING STAKEHOLDER NEEDS Once you have identified your core stakeholders, it is important to identify the specific types of information that they will be interested in providing or receiving. It is common to realize that the expectations or interests of potential stakeholders are unclear. In this event, take the time to clarify their interests prior to proceeding with the evaluation. Your strategies for communicating with your stakeholders should vary depending on the specific group and the nature of your current relationship. Formal opportunities to gather information can be used, such as a brief survey with community residents or a focus group with current or past program participants. However, an informal meeting or just a phone call is often sufficient. Most people will feel comfortable describing the kind of information that would be most useful to them in understanding your program or making necessary decisions. Once you have finished gathering any needed information, make a list [form1] of core stakeholders, and the evaluation questions and issues that have been established through conversations with those stakeholders. As you move forward with your evaluation design, refer to this list to make sure that these issues are addressed in your plan. Stakeholder(s) Issues/concerns/area of interest Priority High Moderate Low Parents Is it easy to get kids signed up for the program? Moderate Program participants Funders School officials Advocacy organizations Does the program keep youth interested and engaged? Is the program effective in reducing youth alcohol use? Does the program align with the school mission of ATOD prevention education? Does the program align with current best practices in youth alcohol prevention? High High High Moderate Identifying the needs of stakeholders 7 Wilder Research, August 2008

15 INVOLVING STAKEHOLDERS IN THE EVALUATION PROCESS In addition to understanding the specific needs and interests of these groups prior to developing your evaluation plan, it may be important to directly involve some stakeholders in the evaluation design process. This may be especially important if there are stakeholders who are not clear about their needs, but who might benefit from participating in the process, or if they will need to approve of your resulting evaluation plan. You might consider involving stakeholders in the development of evaluation questions to ensure that the priorities of different stakeholders are addressed. Involving stakeholders does not necessarily mean they have complete control of the evaluation, nor does it mean that the evaluation must take into account the ideas and points of view of every stakeholder. It likely cannot. Involving stakeholders does, however, help everyone understand the process of prioritizing and logic behind the decisions that are made. Consider these questions: Which stakeholders are MOST important to include the evaluation planning process? Why? What steps can be taken to ensure that the perspectives of these key stakeholders are incorporated into the evaluation design process? This could include providing opportunities for period review or feedback to including them at all stages of the process. These steps will help to ensure that stakeholders will continue to buy-in to the evaluation process, and to help to guide the efforts required to complete an evaluation. SOME THINGS TO CONSIDER WHEN EVALUATING YOUTH PROGRAMS Youth as stakeholders. All too often, adults and communities develop programs for youth, rather than with youth. Programs developed by adults without the involvement of youth may not be as effective as programs where youth have a voice. Young people know how to talk to and relate to other young people. They know best what types of activities and programs are interesting and likely to engage other youth. Attention to this issue is growing, indicating a desire for greater youth involvement at all stages of youth programming. Evaluation of youth-focused programs is no exception. If your program serves youth, it will be critical to get their early input Identifying the needs of stakeholders 8 Wilder Research, August 2008

16 and buy-in to the evaluation process, just like any other group of stakeholders. Youth will bring a different and unique perspective to your program and the evaluation, and that information would work to inform the rest of the process. There are several ways to get youth interested and involved in the evaluation process. You could encourage youth representation if you have an evaluation committee or provide opportunities for youth to take leadership roles in terms of the evaluation process. School or school district approval for evaluation activities. An essential part of any evaluation is to ensure stakeholder buy-in to the process of the evaluation. Stakeholders should understand the goals of an evaluation, the logic behind the evaluation design, and potential implications. One way to increase stakeholder buy-in might be to identify and recruit someone from within the school who can be an evaluation champion, or someone who will work to solve problems and increase understanding of the evaluation process. An evaluation champion can give the process more legitimacy and build buy-in from skeptical staff members. Communication with teachers, principals, and other school administrators. Communication can be challenging, given their busy schedules and other competing interests. Consider having a liaison between the school and the evaluation team, someone such as a retired teacher who is familiar with the school system but not busy because of it. Tips for engaging stakeholders Involve stakeholders early in the process and allow enough time for meaningful involvement Focus on key stakeholders research has found that it is more beneficial to have deep involvement of core stakeholders, rather than marginal involvement of many Provide stakeholders with options about how they would like to provide feedback or participate in the process Use stakeholders to identify and contact other people to bring into the process Do not assume that people in leadership or decision making roles are the only or most important stakeholders Do not attempt to address every stakeholders interests when, in reality, they cannot all be addressed Identifying the needs of stakeholders 9 Wilder Research, August 2008

17 Be certain not to exclude any stakeholder because of gender, ethnicity, or language background Throughout the evaluation, be alert to identifying additional people who should be involved Avoid giving the impression that all questions will be answered in the evaluation Engaging stakeholders early in the process is important and useful for several reasons. Perspectives of different stakeholders can strengthen the evaluation design, and those involved in the evaluation might help to find resources to carry out the evaluation. Additionally, engaging stakeholders in the evaluation will encourage them to support the evaluation, understand what the evaluation means, and increase the likelihood that they will use the findings. Stakeholders who participate in the planning and carrying out of an evaluation will likely gain a new understanding and appreciation for your program. FOR MORE INFORMATION ABOUT STAKEHOLDER ENGAGMENT WRAP UP Engaging stakeholders is generally the first step of engaging in any evaluation process. But stakeholder engagement does not stop there! It is important that stakeholders feel engaged throughout the evaluation process, in order to provide support for putting evaluation results into action. Identifying stakeholders, including program participants, ensures that all points of view are considered in the development of a program or evaluation. Determining and prioritizing stakeholder needs increases the usefulness of evaluation results. Evaluating youth programs presents opportunities for partnering with youth, parents, teachers and other school officials. Identifying the needs of stakeholders 10 Wilder Research, August 2008

18 NEEDS ASSESSMENT 3 A needs assessment is a useful tool to help establish the services needed in your community. Community can mean your actual neighborhood or geographic community, but can also refer to different systems that people are a part of. For example, community can refer to schools, a collection of professionals, or a group dedicated to a particular cause. If done well, a quality needs assessment can not only help you identify critical issues in your community, but can also identify community resources that are available to address these issues. The results can play a critical role in improving ATOD-V prevention programs and planning for future programs. This section will help identify when to conduct a needs assessment, how to identify the right questions, and tips for putting what was learned into action. WHAT IS NEEDS ASSESSMENT? Assessments provide starting information about the current state of the community. This information can be used to create a vision of where the community should be and to develop plans for achieving these goals. Assessments can also be critical in bringing significant issues to the attention of policy makers and the general public. Needs assessments often address the following key areas: Current and future trends (Are things getting better or worse? What is on the horizon? Is the community prepared for these changes?) Resources (What are your community s assets? Do you have access to resources outside the community? Has the community fully mobilized these potential resources?) Weaknesses (What issues or problems are facing the community? What areas are most in need of improvement? Which pose the greatest threat?) Needs assessment 11 Wilder Research, August 2008

19 Opportunities (Are you taking full advantage of current opportunities? What opportunities are likely to occur in the future?) External pressures (What regional, national, or global issues and trends are affecting the community? How are these likely to change in the future? Is the community prepared to deal with these issues?) WHY SHOULD COMMUNITY ASSESSMENTS BE CONDUCTED? Needs assessment is a systematic process of gathering, analyzing, and reporting information about the needs of your community and the capacities or strengths that are currently available to meet those needs. The goal is not to gather information for the sake of gathering information. The potential value of a quality needs assessment cannot be overstated. There are four primary benefits of conducting a needs assessment, including: To create a common understanding of community issues. Needs assessments help to ensure that individuals have a common information base from which to make decisions, rather than being limited by incomplete knowledge. Having a common understanding about community issues is an important part of building consensus, and working together to create solutions. To create an information base for service delivery decisions. Needs assessment allows you to make decisions about programs and services that are grounded in the needs, priorities and resources existing within the community. For example, an assessment can help you make decisions about expanding or modifying services. It can also help you consider alternative strategies for solving a problem or forging networks or alliances to address issues. To get community involvement and support and establish joint ownership of change efforts. Needs assessment can engage community members before decisions need to be made. Allowing residents, community organizations, businesses, schools, and others to be involved in the process may increase their investment in your program and promote effective partnerships. At a minimum, community members may be involved by providing information about their perceptions of a community. There are also benefits to including them in designing the assessment, collecting information, and using the results to guide service or community change. Needs assessment 12 Wilder Research, August 2008

20 To create a baseline picture of the community. An assessment can create a baseline understanding of the current issues or resources facing a community. This baseline information can be used over time to identify changes and emerging issues if your goal is to promote community change. Baseline information will also be useful as you evaluate program effectiveness, which we will discuss at length throughout this toolkit. Steps required of any needs assessment: Determine the goals of the assessment Identify the resources needed to conduct the assessment Review potential community indicators relevant to ATOD-V prevention Locate and access existing community information Develop data collection strategies and materials Use results to develop program enhancement plans Many assessments tend to be problem-oriented, focusing on identifying concerns or issues in the community. Quality community assessment identifies not only emerging concerns, but also available resources and community strengths. Looking at both assets and needs will enhance your ability to harness resources to better meet community needs. WHEN SHOULD COMMUNITY ASSESSMENTS BE CONDUCTED? Identifying needs and assets can be helpful to your organization at almost any point in your initiative. Assessment should be an ongoing process. Regularly updating community information ensures that assessment results are available and relevant when you need to make a decision or take an action. It will also help you to continually strengthen relationships and networks. Fully incorporating assessment information into ongoing program management will greatly enhance the value of your evaluation process. Needs assessment 13 Wilder Research, August 2008

21 While community assessment can be useful in a wide range of situations, it is especially important when: a community problem or issue is not clearly defined. little is known about a community problem or its possible consequences. community perspectives of an issue are unknown. information will help guide program or system improvement. it is important to identify individuals, groups, or other community resources that may be able to take action HOW ARE WE GOING TO DO THIS? When determining the questions you and your community want to address, it will be very helpful to engage stakeholders in the process of designing and prioritizing evaluation questions. This process provides a forum for anyone who has an interest in the assessment findings, and helps ensure that the needs assessment covers all important areas and topics within your community. Prioritizing questions with stakeholders will increase their buyin to the process and promote use of the findings once the assessment in completed. Sample needs assessment questions for ATOD-V prevention What strengths and assets do we currently have in terms of ATOD-V prevention? What is the greatest violence or safety related problem in our community? What is the greatest threat to the health of youth in this community? What kind of violence is occurring in the community? What is causing youth to use/abuse ATOD? Why are youth in this community engaging in violence? What resources are in place to address these problems? What resources are lacking to address these problems? Needs assessment 14 Wilder Research, August 2008

22 After determining what questions to ask, how are you going to get this information? There are four commonly used approaches for data collection for needs assessments. These approaches are detailed in other sections of this toolkit, so this will provide a brief overview. Key informant interview. In this approach, community leaders and decision makers who are both knowledgeable and invested in the community are asked about their priorities and concerns about ATOD-V prevention. This approach provides a rich source of information about your community. If you are addressing needs of youth in the community, be sure to include individuals who work with youth and young Focus groups. Similar to the key informant interviews, this approach benefits from group interaction around a specific issue or topic. Valuable information can be gained by bringing together individuals with similar interests and engaging in a shared conversation about the issues. Community survey. Surveying a sample of community members will give you an idea about the whole community s priorities and interests. Use of existing data. Some existing data source to consider for ATOD-V prevention might be census data, school record data, existing state or county survey data, or other school information. Data available in the Minnesota Department of Education web portal provides a good source of existing data. This is generally a less costly approach to needs assessment, as new data collection is not required. With any data collection approach, you will need to have the capacity to analyze the results and make sense of the findings. Following sections of this toolkit will provide guidelines and tips for identifying and increasing this capacity for yourself or the needs assessment team. USING COMMUNITY ASSESSMENT RESULTS There are many ways to use the findings of a needs assessment, including: determining your program s philosophy and long-range and short-range program objectives. determining the services that are most needed and the program options that will be implemented. determining the populations that will be served by the program, if limitations in the amount of resources make it impossible to serve everyone. Needs assessment 15 Wilder Research, August 2008

23 Now that you have established the needs of your community, it is time to select a program that will best fill the gaps in your community. Start by prioritizing the findings [form 2] from your assessment. How important is addressing the need to your organization? To the community? Is your organization the best one to address this issue? FOR MORE INFORMATION ABOUT NEEDS ASSESSMENT WRAP UP A needs assessment is a useful tool to help establish the level of service need of a community, whether that includes students or the larger community. If done well, a quality needs assessment can not only help identify critical issues in a community, but can also identify community resources that are available to address these issues. Needs assessments can be conducted to help a community identify a problem, or to learn community perspectives on a particular issues, such as youth ATOD use. This information can help guide program work, either through development of new programs or guide improvement of existing programs. The process of conducting a needs assessment is similar to conducting a program evaluation. It is necessary to engage stakeholders, assess goals, collect information and put the results to use. Putting needs assessment results into action is the most important element of any needs assessment. Work with stakeholders to use the findings to develop and improve programs. Needs assessment 16 Wilder Research, August 2008

24 PROGRAM SELECTION 4 After completing a needs assessment, you will have identified needs and assets within your community. You will likely be faced with more options for interventions and practices than you could ever implement. How do you determine which is the right fit for your organization and target population, given the support of stakeholders, organizational capacity, funding and staff? These are just a few of the considerations that go into selecting an intervention or adopting a practice within an organization. This section will discuss both practices and interventions. Practices are a particular way of doing things, and may encompass an entire program or refer to only a specific way of doing part of a program, for example using youth engagement in the program planning process. Interventions are entire programs designed to meet an overall objective, such as an integrated ATOD and violence prevention program adopted throughout a school district or a large public awareness campaign. EFFECTIVE ATOD-V INTERVENTIONS AND PRACTICES Effective ATOD-V programs are often referred to as evidence-based interventions and practices. There are a lot of different terms used to describe evidence-based programs. What is important is that a program has, through evaluation, shown effectiveness in doing what it is designed to do. Evidence-based programs have indicated effectiveness for diverse audiences, demonstrated changes in knowledge, attitude and behavior over time, and are ready to put into practice. There are many interventions and practices across the country concerning ATOD-V prevention. Some are effective, and others are not. Generally, effective interventions and practices have the four following elements in common: Consideration of the interests and the needs of students. Integrated efforts to reinforce the lessons learned in the program and create a more holistic approach to prevention education. Program selection 17 Wilder Research, August 2008

25 Emphasis on healthy youth development and resilience by promoting independence and responsibility Inclusion of youth who are abstaining and those who have already engaged in some degree of alcohol, tobacco or other drug use. Best practices guidelines for ATOD-V prevention were created to help programs design and choose interventions and practices that are likely to be effective. Evidence-based interventions and practices follow these guidelines. Interventions and practices are designed to develop skills among youth that will promote responsible ATOD-V behaviors. Best practice guidelines encourage ATOD and violence prevention programs to contain the following: Positive prevention messages. Being drug and alcohol free is something to strive for and be proud of. Grounded in human growth and development research, by addressing the susceptibility to drugs and alcohol at various stages of normal development and presenting age appropriate information and skill building exercises. Theme of responsibility both for one s self and for others in the community. Youth should learn that they are an important part of the community and that they are responsible for their own behavior which will help to keep the community ATOD-free for youth. Clear messages that drug use is wrong and always involves some risk. Programs should emphasize that trying a drug just once is unsafe. Scientifically accurate and timely information. Outdated facts might damage credibility with the audience and could portray inaccurate information. Information that is culturally relevant and sensitive. Along with guidelines for elements that should be present in an effective ATOD-V prevention program, there are several things to avoid: Opportunities for youth to make excuses for their behavior. Information that could teach youth how to obtain, prepare, or use alcohol, tobacco, or other drugs. Program selection 18 Wilder Research, August 2008

26 Use of recovering addicts or alcoholics as role models. This can romanticize the behavior of the addict and have adverse affects on youth, especially high-risk youth. This is also true of violence prevention. Care should be taken to avoid romanticize or normalize youth violence. Discussions of improper use of drugs. Messages that imply that some drug use is acceptable, or some drugs are acceptable in certain situations, can be confusing to youth. Be clear that youth should abstain from all forms of ATOD. Implications that drug use is an individual s choice. Youth should not be instructed to think of substance use or abuse as a choice they have the power to make for themselves. Scare tactics, such as alcohol-related car crash reenactments or display of damaged body organs due to smoking or other drug use. PRACTICAL CONSIDERATIONS FOR PROGRAM SELECTION Organizational capacity and support As you begin sifting through the potential interventions and practices, the support and capacity of your organization will significantly impact your choice of an intervention or practice. Organizational support can include stakeholder input and support, the capacity of staff to do the work, and support from administrators and others in authority for the selected intervention or practice. In addition to personnel capacity and support, does your program or organization have the maturity and motivation necessary to successfully implement a given program? What about the technical capability and financial resources? Assessment of needs In section 3 of this toolkit you learned of the importance and practicality of assessing the needs of a target population in terms of programming. This process of defining the problem and identifying where to focus efforts and resources is important for both increasing the effectiveness of programs and tracking progress of the chosen program. Program selection 19 Wilder Research, August 2008

27 Select evidence-based programs and practices Once you have identified the capacity and support within your organization and determined the needs of your target populations, it is time to identify the best intervention or practice for your organization. You will likely identify several options, through conversations with others in your field, research of online databases, and ideas from stakeholders. When selecting an intervention or practice, consider the following factors: The intervention or practice has been evaluated and has demonstrated effectiveness of outcomes in settings similar to yours. Programs have been successfully implemented with your intended target population, considering factors such as age, race and ethnicity, socio-economic status, and geographic location. The intervention or practice aligns with identified community needs. There is fit with the capacity and support of your organization, including personnel, physical and financial resources. Along with adaptations of the intervention or practice, there will likely be organizational changes based on the needs of the program. Consider the following questions as you decide on a program: Does this intervention or practice fit with the mission of our organization? How does this practice or intervention reflect the values and practices of our community? How is the intervention or practice different than what is currently being offered in the community? What staff changes (paid or volunteer) will likely need to be made in order for this intervention or practice to be successful? These questions, along with other considerations, will help you and your stakeholders identify and adopt the best intervention or practice for your program. Keep stakeholders engaged throughout the selection process, as they might have different ideas about the fit of the program given the needs of your community. Stakeholder buy-in will be essential as you begin to adopt and implement any intervention or practice. Program selection 20 Wilder Research, August 2008

28 Implement After determining the needs of your target population and choosing the most promising an intervention, it is time to implement your intervention or practice. It is important that implementation includes monitoring of the program, to ensure that the core components of the intervention or practice are delivered with fidelity, or strict adherence to the original design. Additionally, implementation should monitor the satisfaction of participants. If you see implementation beginning to stray from the original design of the program, consider these strategies for realigning the program and program staff: Identify program mentors, or staff members with the ability and interest to provide assistance and mentoring to other staff. Provide program staff with quick access to resources when they need more information or assistance. Offer booster trainings to program staff at strategic intervals, to ensure that the program is being implemented with the most fidelity possible. The successful implementation of a new intervention or practice can lead to the sustainability of the program. Here are some tips for working to make your program sustainable: Adapt Highlight program outcomes that relate to other results you hope to see. For example, if your program is being implemented in a school setting, show how reducing ATOD use and school violence may promote academic achievement. Embed prevention programming into the overall mission of your organization or school strategic plan. Incorporate program activities into other existing activities within your organization or within schools. While often designed to be implemented with fidelity, evidence-based interventions and practices as designed might not meet the needs of your particular target population. Adaptation is not always permitted when implementing an evidence-based intervention or practice. When permitted, adaptation of an intervention or practice is a delicate balance between preserving the integrity and core elements of a program while adapting that same intervention to be meaningful to participants. Program selection 21 Wilder Research, August 2008