Rural Behavioral Health Programs and Promising Practices

Transcription

1 Rural Behavioral Health Programs and Promising Practices June 2011 U. S. Department of Health and Human Services Health Resources and Services Administration Office of Rural Health Policy

2 This document was prepared for the U.S. Department of Health and Human Services (HHS), Health Resources and Services Administration (HRSA), Office of Rural Health Policy (ORHP) under HRSA Contract # HHSH C. This publication lists non-federal resources in order to provide additional information to consumers. The views and content in these resources have not been formally approved by HHS. Listing these resources is not an endorsement by HHS or its components. 2

3 Table of Contents Overview... 4 Introduction... 6 Methodology... 8 Overview of Nominated Programs... 9 Developing a Promising Practice Focused Technical Assistance Conclusions References Appendices Appendix A: Questions included in the Nomination Form Appendix B: Phone Interview Follow-Up Questions Appendix C: Description of Rural Behavioral Health Programs Appendix D: Web site Links

4 Rural Behavioral Health Programs and Promising Practices Overview The U.S. Department of Health and Human Services (HHS), Health Resources and Services Administration (HRSA), Office of Rural Health Policy (ORHP) coordinates activities related to rural health care within the U.S. Department of Health and Human Services. ORHP has department-wide responsibility for analyzing the possible effects of policy decisions on 62 million residents of rural communities. ORHP was created by Section 711 of the Social Security Act to advise the Secretary of Health and Human Services on health issues within rural communities, including the effects of Medicare and Medicaid on rural citizens access to care, the viability of rural hospitals, and the availability of physicians and other health professionals. ORHP administers grant programs designed to build health care capacity at both the local and State levels. In particular, the community-based grant programs within ORHP that draw authority from Section 330A of the Public Health Service Act, such as the Rural Health Care Services Outreach grant program, are mandated to expand access to, coordinate, restrain the cost of, and improve the quality of essential health care services, including preventative and emergency services, through the development of integrated health care delivery systems or networks in the rural areas and regions. As a result of the program s legislative focus, the Outreach grant program provides funding to consortia for the direct provision of health care services as well as for community health service collaboration. Currently there are many challenges to providing mental health and substance abuse (behavioral health) care in rural America. As such, ORHP supported a study in 2008 to examine the barriers to evaluating programs in rural and frontier behavioral health and to disseminate this information among current grantees and future applicants for the Rural Health Care Services Outreach, Rural Health Network Development, and Rural Health Network Development Planning programs that have a focus on behavioral health delivery in rural communities. The intent was to closely examine these programs in a descriptive manner and outline requirements to move these programs/practices to more rigorous scientific validation that would be valuable in demonstrating their effectiveness. This document will allow rural behavioral health programs to learn from the information gathered in this study and the value of collecting and using data to make program improvements, demonstrate effectiveness and importance of services, and find funding for sustainability. Evidence-based Practices (EBPs) are practices that integrate best research evidence with clinical expertise and patient values (Institute of Medicine, 2001). EBPs are increasing in the field of behavioral health: consumers want treatments with proven effectiveness; providers want to increase their knowledge to enable provision of those treatments; research and funding entities look for opportunities to identify new evidence-based treatments; and, in some cases, policymakers are legislating the provision of evidence-based practices (e.g., Oregon Statute , which mandates expenditures for evidence-based practices). Overall this shift toward the use and development of EBPs in behavioral health services is moving in a positive direction, with the end result being the provision of more effective and standardized behavioral health treatments. However, many of these EBPs are tested in urban areas, and are not easily adapted to rural areas without losing key components of the tested practice. Due to the difficulty of developing and evaluating EBPs, the term, promising practice, has been used to refer to behavioral health practices that do not yet have the evidence-base to be considered EBPs, but which appear to be effective based on a less stringent definition of research evidence (e.g., using a comparison group that is not wellmatched to the treatment group) or preliminary and/or simple data evaluation. The term, best practice, is used to refer to a practice which is generally thought to be effective based upon anecdotal evidence, but for which objective data is lacking. Without some degree of data and evaluation to support the effectiveness of a practice, it is difficult to develop an evidence-based practice. 4

5 A practice is defined as a specific procedure that can be generalized to targeted situations. For example, in using a specific depression screening questionnaire, there are standard procedures to follow (e.g., a specific order of questions, a specific method of scoring the responses). This screening questionnaire may be used in a variety of settings, with a variety of patients, but the reliance on a specific procedure makes it a practice. In contrast, a program encompasses multiple practices, activities, and strategies. For example, a Depression Awareness Program would not only incorporate the practice of depression screenings, but would use targeted behavioral health education, outreach, health fairs, and marketing, which are all part in parcel of a program and specific to the target population. A program may be an EBP, promising, or best practice if program variables can be generalized to different populations and/or settings. Although there are a number of interesting and novel behavioral health practices currently being implemented in rural America, there are fewer practices that are evaluated and tested for their effectiveness than in urban areas. Much of the evidence for the effectiveness of these practices comes from subjective experience and observation, rather than data tracking and analysis. This report documents a study to identify innovative rural programs (i.e., sets of practices, activities, and strategies), and uses the results to suggest basic tools for rural organizations to embark upon the steps necessary to move a program that is perceived to be effective into a promising practice. Achieving the status of promising practice is beneficial to programs because it provides evidence of the program s effectiveness to a target population and funders who may invest in the program, and also enables other rural programs to adapt and use the program in their own community. This document will provide an overview of tools such as community building, grant-writing, data collection, and program definition, which may prove useful to organizations that are interested in transitioning their unique and innovative behavioral health programs to promising practices. In the initial stage of this study, the Western Interstate Commission for Higher Education (WICHE) Mental Health Program identified programs in rural behavioral health. WICHE staff spoke with 62 rural behavioral health and substance abuse providers around the country who had developed novel and innovative programs to improve behavioral health services in rural areas. Although most programs collected some form of data, few programs utilized those data to determine their programs effectiveness. Over the course of these interviews, most organizations reported needing additional resources and knowledge in order to evaluate the effectiveness of their programs. Many programs reported a lack of staff time and funding to support program evaluation and/or research activities. Overworked staff members are unable to set aside time for data-related activities, such as the development of data collection procedures, data collection itself, and data analysis. In addition, staff members do not have time to document program-related activities, or apply for funding to support program and evaluation activities. Finding assistance from outside researchers with expertise in research design, data collection, and program evaluation is also a challenge, as most programs do not have the financial resources to support external consultants. Thus, rural organizations wishing to evaluate the effectiveness of a program are in a double bind they often do not have the time to develop a data collection and evaluation procedure, and also do not have financial resources to hire anyone to assist with this process. Even if programs do have the staff time to devote to evaluation activities, they often do not have the expertise to carry out those activities. Providers in general, and rural providers specifically, are often not trained in research design, data collection and analysis, and program evaluation. In addition, providers often do not have knowledge or experience with grant-writing (e.g., where to identify grant opportunities, how to apply for them, and how to write grant applications). Without successful grant-writing opportunities, it becomes nearly impossible to fund research and evaluation efforts. 5

6 A further hurdle in a program s ability to become a promising practice is the lack of understanding as to what constitutes a practice. Ten percent of organizations (7) responded to WICHE s initial call for nominations by sending information about the organization as a whole; nominating a group of providers or a facility rather than a method of providing treatment or in some way ameliorating the problems facing individuals who need behavioral health services in underserved rural areas. Some programs cannot reach the status of promising practice not only because they do not have data, but also because of a lack of understanding as to what constitutes a practice. This document is primarily targeted toward providers, but is meant to apply to a range of individuals: Providers can learn about data collection and analysis, community engagement, and grant-writing. Policy makers can learn what rural organizations need to implement and develop promising practices, and how this information is instructive for legislative changes to support these efforts. Researchers can learn what providers need in the way of research tools to enable effective evaluation of rural practices (e.g., survey design assistance). Consumers can learn what to look for in order to evaluate whether a given practice is effective in their treatment and whether it would be a good fit for their behavioral health concerns. The intent of this document is to provide a starting point for rural stakeholders to collaborate and ensure innovative practices demonstrate effectiveness based on sound science and have the funding necessary to sustain their activities. Introduction Mental illnesses affect up to 1 in 5 individuals in the United States (U.S. Department of Health and Human Services, 1999). When compared to major physical illnesses, such as cardiovascular diseases, cancer, respiratory conditions, and infectious diseases, mental illness ranks second in the calculated burden of disease (i.e., the number of years of life lost to premature death and years lived with the disability; Murray & Lopez, 1996). While often perceived to have minimal influence on an individual s life, mental illnesses are severely disabling. The disability associated with major depression is equivalent to the disability associated with blindness or paraplegia, and the disability associated with active psychoses seen in schizophrenia is equal in burden to quadriplegia (U.S. Department of Health and Human Services, 1999). A recent report by the World Health Organization (2008) indicates mental illnesses are the biggest health burden in North America, largely due to disability which results in a loss of productive years of life. Behavioral Health in Rural America Approximately 20 percent of the United States population is affected by behavioral health issues each year (Kessler et al., 2004), and although the rate of behavioral health problems does not differ substantially between rural and urban areas, the experience of mental illness differs dramatically (Mohatt, Adams, Bradley, Morris, 2005). The prevalence of and entry into care for behavioral health problems is generally comparable in rural and urban populations, but the quality of care that rural patients receive for behavioral health problems may be poorer, particularly for residents in outlying rural areas (Fortney, Rost, & Zhang, 1999; Kessler et al., 2004). Rural Americans with behavioral health disorders are significantly less likely to receive any type of treatment for their behavioral health problems than urban and suburban Americans (Wang, Lane, Olfson, Pincus, Wells, & Kessler, 2005). Individuals living in rural areas are significantly less likely than their urban counterparts to receive specialty behavioral health care (Wang, et. al. 2005) and more likely to receive general medical care only or human services only (e.g., pastoral counseling) (Wang, Demler, Olfson, Pincus, Wells, & Kessler, 2006). Unfortunately, the likelihood of receiving minimally adequate behavioral health care in the general medical sector and human services sector is substantially lower than in the specialty behavioral health sector (Wang, et. al. 2005). 6

7 For the past 40 years, 60 percent of rural America has been underserved for behavioral health needs (New Freedom Commission on Mental Health, 2003), and more than 85 percent of the nation s behavioral health professional shortage areas are located in rural America (Bird, Dempsey, & Hartley, 2001). Individuals living in rural communities are faced with three distinct burdens to adequate behavioral health care: accessibility, availability, and acceptability (Mohatt et al., 2005; New Freedom Commission on Mental Health, 2003). In terms of accessibility, rural residents typically do not know when they need behavioral health care, where they can find that care, and what care options are available to meet their behavioral health needs (New Freedom Commission on Mental Health, 2003). When they do find behavioral health care, it is not uncommon for individuals in rural areas to travel hundreds of miles to access those services. The availability of behavioral health providers in rural areas is too limited to support urban models of service delivery, in which individuals needing behavioral health services have a variety of behavioral health providers from which to choose. In addition to physical barriers to behavioral health care, rural residents also face psychological barriers. Many Americans attach stigma to having or seeking help for behavioral health problems. This stigma is particularly detrimental in rural areas where there is little to no anonymity in seeking behavioral health services (Mohatt et al., 2005). These psychological barriers severely limit the acceptability of behavioral health services in rural areas. Challenges Confronting Rural Evidence-Based Practices The behavioral health care barriers faced by rural Americans make behavioral health care delivery in rural areas substantially different than in urban areas, which is a particular challenge for creating evidence-based practices for and delivering evidence-based practices to rural residents. In recent years, evidence-based practices (EBPs) have emerged as a means to ensure quality behavioral health care among individuals suffering from mental illnesses. EBPs have been extremely successful in treating mental illnesses. However, most EBPs are developed in urban areas and tested on urban residents, with little, if any, thought given to how they might be implemented in rural areas where resources are much scarcer. The drastic differences in behavioral health care availability, accessibility, and acceptability between rural and urban residents make it challenging to conduct many EBPs in rural areas, leading to a strong need to develop ruralspecific EBPs, and/or determine whether and how existing EBPs can be modified to produce similar treatment effects in rural areas as are observed in urban areas. Unfortunately, a majority of rural areas lack the resources needed to develop and sustain rural-specific EBPs. Rural areas are already underserved and underfunded, and most rural behavioral health professionals do not have the time or training to plan and conduct clinical trials, or to analyze and document results from scientific studies. Even with time and expertise available to carry out and document scientific studies of rural-specific practices, rural behavioral health services often do not have the funds to carry out pilot studies necessary for obtaining larger grants. In many rural areas it is therefore a challenge to carry out large-scale clinical trials necessary to test the effectiveness of rural programs. In addition, few agencies provide funding opportunities to individuals outside of academic centers, which once again puts rural areas at a disadvantage for developing new EBPs specific for rural areas, or ensuring EBPs are effectively adapted for rural areas. The fact that rural behavioral health providers do not have many resources to implement EBPs, or to develop new EBPs leaves rural providers facing a real challenge to providing EBPs for rural residents. However, many rural providers have developed programs specific to their areas, which may be quite successful in improving the availability, accessibility, and acceptability of behavioral health services in rural areas. Rural providers are in a better position to understand the unique needs of their communities, and many have adapted their programs and treatments to meet those needs. Unfortunately there is no medium for sharing this knowledge, and it often remains isolated in specific communities, with few resources available for documenting and publicizing the effectiveness of these unique rural programs. 7

8 Goals In response to the need for technical assistance in determining a program s effectiveness, WICHE conducted a study to a) identify rural programs that were attempting to meet a behavioral health need in a rural area, and b) determine the general resources these programs needed to move their programs toward promising practices. This process included a solicitation for nominations via a web-based survey, and follow-up interviews with key individuals from each program. The end result of this process is a snapshot of approximately 60 rural behavioral health programs throughout the country, and a more detailed understanding of what rural programs need to do to become promising practices. Methodology The identification of rural behavioral health programs involved a multi-step process that first solicited nominations for novel and innovative programs, and then collected additional information about the programs through a survey and follow-up interviews. Below, is a discussion of the procedure for soliciting nominations, surveying, and interviewing each of these programs. Reaching Programs/Practices Criteria were not given for the type of programs to nominate for inclusion in this study (e.g., hotline, outreach), other than the criteria that the program have a positive impact on the behavioral health of individuals living in rural areas. Nominations were solicited via announcements distributed to national rural health organizations (the National Association for Rural Mental Health, the National Association for Rural Health, the National Association of State Mental Health Program Directors), national rural behavioral health funding agencies (Substance Abuse and Mental Health Services Administration, Health Resources and Services Administration), as well as organization-specific contacts such as the Rural Assistance Center and the WICHE Mental Health Program. The purpose of these s was to ensure broad distribution of the announcement to rural consumers, providers, and policy makers. The announcement stated the goal of the survey was to identify practices that were successful in meeting a behavioral health need in a rural area. The definition of "meeting a behavioral health need in a rural area" was purposefully vague. Potential respondents were informed meeting a need" could include increasing access to and/or availability of behavioral health services, increasing awareness about behavioral health issues, recruitment and/or retention of behavioral health providers, treatment of a behavioral health problem, and prevention services, among others. The diversity of programs responding to the nomination indicates there are very diverse behavioral health needs in rural areas, and a large number of ways to meet those various needs. The Survey Individuals who nominated programs by responding to the survey answered a series of questions designed to create a comprehensive summary of the program, and determine whether the programs had any data on their effectiveness. The survey is included as Appendix A and included categories of questions such as: contact information, characteristics of the nominated program, characteristics of the targeted population, training of staff, documentation of program-related information, and data collection/research capacity. 8

9 Phone Interviews Phone interviews were conducted with 62 organizations that completed the survey, to collect more information on the nominated programs. Organizations that started but did not complete the survey, or did not respond to requests for phone interviews, were not interviewed. The interviews focused on further understanding the programs, and included additional questions, such as those directed at the sustainability of the program, and its uniqueness. The list of questions used for these phone interviews appears in Appendix B. In addition to these standard questions, phone interviews also included program-specific questions whose goal was to clarify responses given on the survey. Site Visits Eleven programs were selected for site visits by the project team and project officer at the Office of Rural Health Policy. In cases where the program was confined to one location, these site visits included an in-person tour of the program and meetings with essential program staff. In cases where the program was more distributed, a phone call was conducted, including as many staff members as possible. The goal was simply to get a broader picture of the programs by speaking with people not on the initial phone interviews (providers, directors, consumers) and to visit the program sites. Programs were selected for site visits based on geographical distribution, as well as the type of program it represented (e.g., hotline, integrated care clinic, etc.). The goal of the site visit selection process was to gain an in-depth view of a variety of programs throughout all regions of the country. Therefore, a geographically and topically diverse set of programs was selected for the site visits. Overview of Nominated Programs Sixty-nine programs from across the country completed the initial survey. The programs, representing 32 States, were spread across all four Census regions, and all nine Census divisions. Figure 1 shows the locations of these programs (note that some locations nominated more than one program, so there are fewer than 69 locations in the figure). Although there are likely substantially more behavioral health programs in the U.S. specifically designed to serve rural areas, the fact that respondents represented all areas of the country suggests the programs described in this document are somewhat representative of the types of rural behavioral health programs implemented in rural areas. Figure 1: Locations of organizations that completed the initial survey 9

10 Categorization of Nominations After learning about the nominated programs, it became clear they fell into one of eleven categories, based on the type of program and/or population being served. Some programs fell under more than one category, but all were able to be encompassed by the categories below. Programs adapting EBPs for rural areas found ways to carry out existing EBPs in rural areas, despite being unable to replicate programs with full fidelity. Community education and outreach programs educated individuals in the community about behavioral health issues. Court teams aimed to keep individuals with behavioral health issues from returning to the justice system. Crisis services programs provided care in some way for acute behavioral health problems. Hotline programs used telephone services to provide support to rural residents with behavioral health concerns or issues. Integrated care programs blended physical and behavioral health services in some way. Peer support programs used peers to provide basic treatment or support to individuals with mental illnesses. Programs serving special populations targeted interventions to a particular population, such as children or the elderly. Telemental health programs used phone or video conferencing to provide behavioral health treatment. Training programs educated future rural behavioral health providers. Themes of Robust Programs After reviewing the survey responses, talking with programs, and conducting the site visits, a few themes emerged as being indicative of a program s success and ability to serve its rural community. Each of these themes is described, below. These themes are used in the program descriptions in Appendix C to highlight each program s particular strengths, and are also expanded upon in the next section, to describe how rural organizations in general can strengthen their programs in each of these areas in an effort to develop their unique practices into promising practices. The interview and selection process led to insights into how programs can excel in each of these areas, and begin to develop the data collection and evaluation tools needed to become a promising practice. Relevance to Rural A program s relevance reflects its specificity to a rural community. Some programs bring much-needed services to rural areas, but it is unclear whether the programs were designed for or adapted to the rural area. A program successful in its relevance to rural communities develops a program specific for a rural community, or implements an established program in a way that is specifically adapted for a rural community. Impact on Rural A program that increases the availability, accessibility, and/or acceptability of behavioral health services in rural areas has a successful impact on rural communities. A program with high impact on rural is addressing one or more of the barriers to behavioral health services in rural communities. Sustainability and Expansion Capability Sustainability and expansion capability reflects the awareness of and ability to acquire long-term funding for the program. Rural programs are dependent on a sustainable and expandable source of funding for their programs. Some highly promising rural programs cannot continue or expand due to lack of funding. One feature of programs that are successful in meeting a rural need is a reliance on multiple funding sources, and creativity in seeking and obtaining new funding. 10

11 Capacity A program s capacity reflects the extent to which a program s staff has, or is able to obtain, the qualifications necessary to accomplish the goals of the program. Programs that demonstrate success in program capacity have a highly qualified staff, and provide opportunities for staff to gain additional training in topics specific to the program. For example, providing or supporting additional training in elderly behavioral health issues can substantially improve staff qualifications to implement a behavioral health program for elderly individuals in rural areas. Documentation of Program Information Program documentation refers to the internal and external materials programs use to describe and/or advertise their services. Although most organizations have policy and procedure manuals for the organization, a majority of programs lack formal, internal program documentation, such as a procedure manual for the nominated program, to allow for replicating the program in other communities. However, programs that successfully document program information generally have external marketing materials, and have developed effective efforts to communicate with key audiences and potential funders in their respective communities. Effectiveness A program s effectiveness is dependent on the extent the program tracks and uses data for decision-making. Few programs collect data on their effectiveness, know how to begin collecting data, and know how to use data to evaluate the program s effectiveness. Lack of data is a limitation in creating a promising practice. Community Engagement Community engagement reflects the degree to which a program involves multiple stakeholders from the community in its development, execution, and expansion. Programs included in this document tend to be strong on the other themes when they have a high degree of community engagement. Developing a Promising Practice The seven characteristics of successful programs, as defined by this study, provide a foundation by which programs can take additional steps to move toward promising practices. A majority of programs already have strengths in some of these areas. For example, a program may have engaged community members since its inception, and may have excellent staff capacity for the program. Programs should use these seven characteristics to assess where their programs excel and where they might target further development. The initial study of rural practices highlights the fact that rural programs already have strengths in some of these areas, and this section describes, in brief, some of the steps organizations may take to improve their program in each of these seven areas. The topics of effectiveness and sustainability came up repeatedly as areas where programs wanted more support. The Focused Technical Assistance section is a supplementary section at the end of this document devoted to these topics. Relevance/Impact on rural behavioral health One of the first things an organization needs to do to move its rural practices forward is to determine what the relevance is to the rural community it serves. Most organizations interviewed have mission statements for the organization as a whole, but very few have formal mission statements for the programs they nominated. Identifying the impact a practice will have on a rural community is an essential first step in creating and testing a promising practice. There are three steps to improving a practice s relevance to and impact on rural behavioral health: Defining what is rural, defining the practice, and identifying the practice s goals. 11

12 What is rural? The definition of what constitutes a rural practice is not static. Rural can and often does mean different things based on where an individual is located and who is using the term. For example, rural Pennsylvania and rural Alaska are two very different places with highly unique obstacles to providing behavioral health services for individuals living in those communities. The most common definitions of rural come from the Census Bureau and the Office of Management and Budget (OMB), but there are more than a dozen different definitions of rural. Both the Census Bureau and the OMB have slightly different definitions of rural, and the definition used may have an impact on the funding of the practice, or the funding opportunities available to the practice. For example, the Census Bureau defines an urbanized area or urban cluster as a core region with a population density of at least 1,000 per square mile surrounding regions with a population density of at least 500 people per square mile. Rural areas are those outside of urbanized areas or urban clusters. The OMB defines an urban area, or Metropolitan Statistical Area, as a central county with at least one urbanized area, and nearby counties where 25 percent of individuals commute into or out of the core county. Rural areas, or non-metropolitan counties, are those outside of the Metropolitan Statistical Areas. Thus, the difference between the definitions is largely an outcome of the different ways of defining urban areas. For a more thorough discussion of this topic, refer to the Rural Assistance Center (RAC) online resource page regarding the definition of rural and the related tool for determining eligibility based on these definitions of rural : To view the rural geographic eligibility for the Office of Rural Health Policy programs, please refer to: Defining program The definition of a program is an essential starting point to developing a promising practice. A promising practice is not simply a group of providers offering services in an underserved area, nor is it simply bringing a service to a rural area not already present (e.g., adding a substance abuse treatment provider to the organization). When defining a program, for evaluation or funding, it is important to have a program in mind that is meant to improve a specific, measureable outcome on behavioral health in a rural community and has yet to be identified as having a positive effect on the community. For example, increasing the number of individuals screened for depression, increasing the number of graduates from a rural training program, and decreasing the wait time for an initial visit are all specific, measurable outcomes that might result from implementing new programs. Developing new procedures or programs to meet a specific need in a rural area and improving these aforementioned types of outcomes, constitutes a testable, rural program that could move toward a promising practice with a well-designed study. Identifying program goals In defining a program, it is important to have specific goals in mind as to what the program will accomplish in order to meet a rural need. In rural areas, the most commonly discussed barriers to treatment include: accessibility, availability, and acceptability (Mohatt, 2005). Rural residents often have trouble accessing care providers may be hundreds of miles away. The availability of behavioral health providers in rural areas is often too limited to support urban models of service delivery (e.g., evidence-based practices tested in urban areas). In addition, the stigma many Americans attach to having or seeking help for behavioral health problems is particularly detrimental in rural areas where there is little to no anonymity in seeking behavioral health services (Mohatt et al., 2005). For a program to meet a need in a rural area, it will generally address one of these three barriers to treatment. A program that cannot define how it is specifically addressing a rural need in a novel and innovative way will have a difficult time convincing funding agencies to contribute to the development and testing of the program. 12

13 Community Engagement One key feature of programs successful in other areas is the engagement of many different stakeholders within a rural community. Programs that effectively address rural behavioral health needs incorporate a number of different agencies, providers, and often consumers into their programs, whether they are just starting out, or have been serving rural communities for decades. This engagement of communities demonstrates the community understands and is supportive of the need for the program. Furthermore, active engagement demonstrates the program is able to successfully market what it does to the community, thereby increasing awareness of the services provided and how to access them. The key in building more community engagement is to realize others in the community are also interested in building connections. Ensuring the need will be met Bringing a range of organizations and individuals to discussions about a program s goals and impact on the behavioral health of a rural community ensures the program will effectively meet the community s needs. In addition, having a number of different organizations invested in the program will help ensure its longevity and bring additional areas of expertise to the table. Some of the programs identified have monthly or quarterly meetings in which various organizations in the community come together to discuss their work and to find ways to better meet the unique needs of their community. Often, these community discussions can help a program identify sources of funding it would not have found otherwise, make connections to evaluators, and help a program gain the trust and respect it needs to thrive in the rural community. Getting stakeholders involved Many programs with a good deal of community engagement started by simply picking up the phone and/or making an in-person visit to local providers, consumers, policy makers, and/or researchers. Most programs were met with enthusiasm when they took this approach, and set up groups of community members who meet regularly to discuss community behavioral health issues. Programs successful in engaging the community work hard to maintain these connections despite busy schedules, staff turnover, and lack of resources. Programs successfully engaging their communities also provide communities with information about behavioral health in general, and their program specifically. By engaging communities in this way, these programs are able to get valuable knowledge into the community and decrease stigma in the process of educating communities about behavioral health issues. For example, programs may provide their staff members as community educators, who go to area primary care providers (e.g., doctors and nurses), housing communities, church groups, schools, etc. to teach people about behavioral health issues. The programs may offer free behavioral health screenings at hospital health fairs and set up booths at community events. Each program sees these connections as vital to its success, and as observed during the nomination and interview process, these community connections are essential to a program s sustainability and impact in the community. Connecting with researchers and evaluators A few of the identified programs have connections with researchers and/or evaluators at local universities or colleges to assist with data collection and analysis, and/or grant-writing. These connections are fewer than the connections with other stakeholders in the communities; some programs note the cost of using a researcher and/or evaluator to help with evaluation as a major hurdle in connecting with these individuals. Some organizations already have ties to local universities and colleges. For those organizations that do not have existing ties to a higher education institution, a perusal of faculty with interests in Psychology, Medicine, Social Work, or similar area on a campus web site will generally turn up an address or phone number with which to initiate contact. These connections can be local, within the program s community, or outside of State or regional boundaries. As with connecting to other stakeholders in the community, the program needs to take charge of building these connections with researchers to ensure it has the expertise needed to evaluate the impact of the program in the rural community. 13

14 Although it can be a hurdle to find additional funding to support an evaluation when an organization is already underfunded, there are a few ways to address this issue. The first is to include a researcher or evaluator in a grant application. Some individuals may be willing to help with the grant application if they will be a part of the evaluation process. A second option is to send a request to a Psychology, Medicine, or Social Work department, asking whether or not there are any graduate students who may be interested in helping with evaluating and testing the program as a part of their graduate work. Most departments have distribution lists, and it is not uncommon for requests for this type of help to be distributed to faculty and students via . A third way to connect with researchers is at national conferences, such as the annual meeting of the National Rural Health Association (NRHA) or the National Association for Rural Mental Health (NARMH). Given the existence of computers, phones, and the internet, it is not necessary for programs to connect solely with researchers at nearby universities. Some programs interviewed are thousands of miles away from their evaluators. Capacity and Documentation In order to be a promising practice, a program must document its practices so that other organizations can carry out those practices in accordance with the original program and so that the program is able to advertise its services to the community. Closely related to this idea of documentation is the definition of staff roles, and having the appropriate staff capacity to carry out the specified practices. Specifying details It is important to document the program or practice in as much detail and as clearly as possible, so that someone else could replicate the program based on documentation alone. The following questions can be helpful in getting started with this documentation process: Who is the target population? What are the main components of the program? Where does the program take place? When does the program happen? Why is the program needed? How does the program happen (e.g., what are staff members roles)? Marketing program services to communities Once a program appears to have positive effects, it is important to start marketing the program to both the local community, and the larger rural behavioral health community. Local marketing can be done by developing a brochure, handout, or some other marketing tool (e.g., a magnet or pen) to deliver to organizations in the area that may have an interest in the program. This brochure/handout should be succinct, summarize the main details of the program, and provide contact information (such as a phone number, address, or Web site) for people who wish to know more about the program. Once developed, this brochure/handout can be distributed to relevant organizations in the community. For example, a crisis line may distribute a handout describing their services to local hospitals, schools, law enforcement offices, nursing homes, faith communities, and businesses. In addition to marketing the program to organizations within the community, it is also important to market the program to other organizations with interests in rural behavioral health. Sending out notices of the program to organizations with similar interests can spread information about the program. The Rural Assistance Center (RAC) Online provides a number of potential avenues for marketing programs. In addition to specialty conferences (e.g., the annual meeting of the American Psychological Association), conferences such as the annual meetings of the National Rural Health Association (NRHA) and the National Association for Rural Mental Health (NARMH) offer additional venues for marketing programs that appear to be effective. Presenting a poster or sponsoring a booth at one of these conferences can be effective methods to ensure others in the larger rural and behavioral health practice community are aware of the program. 14

15 Defining staff roles It is important to demonstrate how the program staff is qualified to perform the role for which they are assigned. Collecting information on the characteristics of program staff (e.g., their education and experience), how they were recruited and hired, their job descriptions, the training they received to perform their roles, and how performance is tracked via supervision protocols will help define staff roles. By defining staff roles, a program demonstrates the staff capacity to carry out program activities as they were designed. Importance for replication and evaluation In order to ensure replicability of a program, it is essential to have a rigorous documentation of the details of the program. Program documentation serves two purposes: it helps an organization implement a program consistently, and it helps individuals outside the organization replicate the program. Program documentation is also necessary when applying for funding to support the program s goals (or its evaluation). Carefully documenting the program s implementation facilitates the process of writing grants and cuts back on the staff time needed to apply for funding opportunities, since much of the information needed for the grant will be written before grant-writing begins. Effectiveness The effectiveness of a program can only be measured through data collection. Although it is a good sign for a program to receive positive feedback from the individuals served and for program staff to notice a positive impact, these observations are not sufficient for the purposes of defining a promising practice. Developing a data collection/evaluation plan An evaluation plan is a lot like an architect's plans for a house. It is a written document that specifies the evaluation design and details the practices and procedures to use to conduct the evaluation. (U.S. Department of Health and Human Services, 2010). One way to begin developing an evaluation plan is to answer the question: What is important to know or demonstrate about the practice? Then, after answering this question, develop objectives in measurable terms. For example, rather than asking whether a depression screening questionnaire is effective, focus the question to a measurable outcome, such as, Is the depression screening questionnaire leading to more referrals to behavioral health providers? It is important to focus on the original purpose and related goals of the program when developing an evaluation plan to determine whether or not a program is effective. Assessing effectiveness Once a program has an evaluation plan and has collected data, statistical analyses are needed to determine whether or not the program is effective. As a result of limited resources and manpower, rural programs when compared to urban programs do not readily have expertise in statistics or the time to learn new skills in experimental design and analysis. As noted in the preceding section titled, Community Engagement, connecting with students and researchers at local universities is one way to supplement research and evaluation knowledge within a program. Often, a student in a graduate program may be interested in assisting with an evaluation, in order to gain access to data for a thesis or dissertation. Making connections with local researchers expands the range of available knowledge and may provide a means for assessing a program s effectiveness that would not otherwise be available to a rural organization. 15

16 Sustainability/Expansion A program needs to have a long-term plan for funding to be a sustainable program. Some innovative programs identified lacked a long-term plan and had terminated or were close to terminating by the time of the interviews. The most successful programs with regard to sustainability and expansion gave thought to how the program would be sustained before it was implemented; a sustainability plan is a part of the initial development of the program. Sustainability and expansion success is closely related to a program s successful engagement of the community. Often, a program is sustainable because of its ties to other community programs. In some cases, programs are successful in funding the program solely from contributions from other community organizations. Developing a sustainability and expansion plan Having a sustainability plan in place from the beginning of a program s development is essential for the long-term ability of a program to serve its community. Often these plans include procedures such as getting Medicare or Medicaid reimbursement for non-traditional services; getting services at reduced or no cost from other organizations in the community; creatively applying for funding from local, State, and Federal sources; and contracting with other organizations in the community to provide essential services. Based on WICHE s discussions with the nominated rural programs, the most robust sustainability plans form when community stakeholders are involved in the creation and execution of the sustainability plan. Community stakeholders who are involved from the beginning are able to share their own sustainability plans, help identify and connect programs with sources of funding, as well as donate their own resources to the program. Resources from stakeholders can appear in the form of funding and also in the form of donated time, services, skills. Some programs are sustainable because they are able to rely fully on small contributions from other community stakeholders, who perform a critical function for the program. Applying for funding There are myriad funding opportunities to support a program s expenses. Often these funding opportunities come from Federal entities, such as the Health Resources and Services Administration, Office of Rural Health Policy, the Centers for Medicaid and Medicare, and the Substance Abuse and Mental Health Services Administration. Additionally, funding comes from State offices, such as the Division of Behavioral Health, or the State Legislature as well as from hospitals, providers, private health foundations or others within a rural community. Many programs in existence for years report using a mix of funding resources to get the program started. Over time, as the program s goals solidified and its role in the community became more apparent, some programs were able to rely on funding from one or more sources, such as a local hospital or Medicare/Medicaid. Programs that submitted applications and are highly successful in obtaining funding are quite creative in the types of funding agencies and grant opportunities they target. Some organizations report watching out for grant opportunities in many different areas (e.g., housing-related grants), and often join with other community agencies to apply for these diverse opportunities (with great success). Focused Technical Assistance The Focused Technical Assistance section provides more detail regarding the development of three key characteristics, which the rural programs included in the study struggled with, and are necessary to create a promising practice: relevance to rural, effectiveness, and sustainability and expansion capacity. All programs had varying degrees of strength in areas such as community engagement, or staff capacity. However, all programs spoke to a need for more data on their effectiveness, and additional funding to expand the program and make it more sustainable. Therefore, this section focuses on these two areas, as well as relevance to rural (specifically, definition of a program ), which was noted by the initial nomination process as an area for which some programs needed assistance, due to confusion over the type of program that can become a promising practice. 16

17 Relevance to Rural Communities As noted above, a program s relevance to rural is apparent in its ability to affect the availability, acceptability, and/or accessibility of behavioral health care in rural areas. Identifying the relevance to rural communities is the first area to target in order to develop a plan for effectiveness research, as well as for securing program funding. If a program is not clear on what its goals are, it becomes very difficult to know where the program is having a positive effect on rural communities. The program definition offers the best opportunity for explaining a program s relevance to rural communities. In describing a program, focusing on the practices it encompasses (as defined in the Overview) and its utility for rural behavioral health are key features of the program definition. The program definition should be clear, concise, and describe the main components of the program and its impact on rural behavioral health. Appendix C provides program descriptions for a variety of program types identified in this study. These descriptions cover all the components of a rural behavioral health program definition and serve as an example on how to describe the program clearly and concisely. The program description should identify the organization that houses the program. It should include a statement about the type of services provided, and the community served. It should highlight the key practices in the program, the relevance to rural communities, and any data relating to the program s effectiveness. It should also note any connections to other stakeholders that enhance the program s reach or relevance to the community. Additional resources The Substance Abuse and Mental Health Services Administration s (SAMHSA s) National Registry of Evidence-Based Programs and Practices (NREPP) is an additional resource from which to gather additional examples and descriptions of effective behavioral health program descriptions: Effectiveness To achieve the status of a promising practice, programs must collect data and use them to determine the impact of the program. This measurement of effectiveness will demonstrate to providers whether a program is on the right track in solving a particular behavioral health problem in a rural area (i.e., whether or not the program is meeting its goals, as defined by its relevance to rural), demonstrate to consumers whether they should seek out that program over another, demonstrate to policy makers whether they should promote widespread use of the program, and demonstrate to funding agencies whether the program is worth their investment. For these reasons, data are essential to a program s development and viability. Programs must collect data to reach the status of a promising practice, but perhaps more importantly, data are essential for program staff to determine whether or not a program is effective in meeting its goals. The Administration for Children and Families, Office of Planning, Research, and Evaluation has published a guide to program evaluation, written for program managers, that is an exceptionally valuable resource for individuals in need of assistance in program evaluation (2010). This guide was developed and maintained to explain the topic of program evaluation and give guidance on how to conduct a program evaluation and discuss the results. The following sections highlight some of the main points that are complementary to the aforementioned guide for rural behavioral health programs, and expound on the information based on the results of the study and the needs of rural behavioral health programs. The full guide provides substantially more detail on the proper collection and use of data. 17

18 Data collection There are a number of different types of data to collect during the evaluation process depending on the program goals, objectives, outcomes, and the impact the program intends to have. Any information collected can serve as data, even a simple count of the number of clients/patients who receive services on a given day, month, or year. Data may include the number of participating sites in a program, the cost of the program, changes in knowledge and/or behavior, or a program s return on investment. The fact that anything can serve as data is important to emphasize because most organizations are collecting information all the time (who is and how many people are using services, how they are paying for services, the length of treatment, and so on). Implementing a data collection plan does not necessarily mean collecting additional information. It may simply involve taking advantage of data that already exist. As such, it is important to understand the program goals when determining the necessary types of data for evaluation. The Program Manager s Guide to Evaluation provides further emphasis for this point (U. S. Department of Health and Human Services, 2010): when preparing to collect information (or data ) on your program, remember that any information collected can be considered evaluation data. The important thing is to identify what you would like to demonstrate about your program or practice. Once the goals of the program are determined, it is possible to determine how to evaluate those goals. For example, if the goal of a program is to increase the use of depression screening questionnaires in elderly primary care visits, it is necessary to collect information about the characteristics of the individuals screened, the numbers of individuals screened, how they were recruited, barriers encountered in the recruitment process, and factors that facilitated recruitment. If the goal of a program is to increase the number of primary care offices that offer telemental health services, it is important to collect information on the type of services offered in primary care offices, the educational and training opportunities that exist, and the type of technical equipment available in those offices. If the goal of a program is to offer statewide crisis hotline services, it is important to collect information on the locations of individuals who use the hotline, marketing efforts for the hotline, and the number of individuals who are using the service. Once the program s goals are determined, an evaluation plan can be developed to assess whether or not the program is effectively meeting its goals. Types of Data Program effectiveness can be measured in terms of a process or an outcome. Process objectives and measures describe what a program is doing and how it will be accomplished. Process measures may include many aspects of the program such as participant information, planning products, activities, services, and community engagement actions. This information can help a program be accountable by meeting specific objectives (e.g., hosting a certain number of trainings, serving a certain number of people). For example, a process objective may be: By June 2012, 50 percent of rural primary care providers in the community will include a depression screening questionnaire in primary care visits. Process objectives help monitor and manage a program, and they can help keep a program on track and alert staff when things need to be changed regarding activities and strategies. Whereas process evaluation is conducted during the program implementation phase, outcome evaluation is conducted when an objective, project, or program is completed. Outcome measures are used typically within outcome evaluation to identify the short-term and long-term results and effects of particular program activities (e.g., practices). Although outcome measures can be used to evaluate progress attained during the implementation phase, the goal of outcome evaluation is to look at the evidence of the program s effectiveness. Stated another way, outcome measures help answer the question, "What difference did the program make?" An example of an outcome measure may be: By January 2012, there will be a 25 percent decrease in the rate of suicide attempts in the community that implemented the routine depression screenings. 18

19 As an additional example of the difference between process and outcome measures, consider a community that has developed and implemented a suicide prevention program. Process components would be measured by answering the following questions in relation to the initial timeline for the prevention program: 1. How closely does implementation match the plan? 2. What types of deviation from the plan occurred? 3. What led to the deviations? 4. What effect did the deviations have on the intervention and evaluation? 5. How can the deviation(s) be corrected? Or should the deviation be corrected? 6. Who provided services (e.g., which organization, and which staff)? 7. What services were provided (e.g., treatment approach, type of disorder treated, location of treatment, and length of treatment)? 8. To whom were services provided (i.e., who is the target population)? 9. In what context were services provided (e.g., in a hospital system, or in a community)? 10. At what cost were services provided (e.g., the cost of facilities, the cost of personnel, and the cost to the consumer)? Outcome components will further clarify data from the process evaluation using the following questions and methods: 1. What was the effect suicide prevention activities had on service capacity (e.g., increases or decreases in usage), and other system outcomes? 2. What program/contextual factors were associated with outcomes? 3. What individual factors were associated with outcomes? 4. How lasting were the effects? Additional resources Below are additional resources explaining the different types of data collection, as well as tools that may assist in program evaluation. Resources 1. The Program Manager s Guide to Evaluation: 2. Center for Disease Control Evaluation Working Group: 3. The Kellogg Foundation Evaluation Handbook: Kellogg-Foundation-Evaluation-Handbook.aspx 19

20 Tools 1. The Office of Rural Health Policy is currently working to develop a generalizable formula to help rural programs measure the impact they are making in their community. This Economic Impact Analysis Formula should be finalized and available by the end of 2010, and will be posted at 2. The National Institute on Drug Abuse provides Cost Analysis Tools for substance abuse programs that may also be useful in determining a program s cost effectiveness: a. The Drug Abuse Treatment Cost Analysis Program (DATCAP) is a cost data collection instrument and interview guide designed to be used in a variety of health-related settings. The DATCAP helps collect and organize detailed information on resources used in service delivery and their dollar cost. The DATCAP instrument is available at the following Web site: b. The Services Cost Analysis Program (SASCAP) estimates the costs of substance abuse treatment services by collecting information on the resources needed by treatment programs to provide specific services and how these resource needs may differ across treatment services: c. The Treatment Cost Analysis Tool (TCU TCAT) is a self-administered workbook designed for Financial Officers and Directors to allocate, analyze, and estimate treatment costs, as well as to forecast effects of future changes in staffing, client flow, program design, and other resources: - ComTreatmentCosts Sustainability and Expansion Capacity Based on the information gathered during the study, programs commonly had difficulty identifying consistent revenue streams to support their activities. The issue of funding is of course central to a program s ability to sustain itself and to expand. However, in relation to creating a promising practice, grant funding can be important for conducting the research needed to determine a program s effectiveness. This section therefore focuses on how to begin looking for grants and how to apply for grants once opportunities have been identified. This section will highlight some of the places to sign up for grant announcements and touch on some of the steps programs can take to develop effective applications. Where to look for grants/signing up for announcements Grants are generally classified according to the entity that is offering the grant opportunity. Within each entity, there may be a number of different types of grants. However, this section will focus on the general classes of Federal, State, and private Foundation funding opportunities. Federal Grants The primary resource for searching Federal grants is Grants.gov, a Web site listing all available grant opportunities within Federal agencies ( Individuals can subscribe to notifications to receive information about various types of grant opportunities when they become available and search for current opportunities based on keywords, funding organizations, and types of grant mechanisms. Furthermore, Grants.gov offers tips on effective searching of its database and a number of resources to help applicants identify grant opportunities and technical assistance on completing Federal grant application forms. The RAC online Web site is another resource to search for available rural specific Federal grants: State Grants To identify State grants, checking the State Health and Human Services Web site will generally provide information on currently active funding opportunities in given topic areas. State-based health foundations also vary by State, but searching the Web sites for these organizations can often lead to grant opportunities. In particular for rural communities, the State Offices of Rural Health (SORH) are not only a major partner with the ORHP, but the SORHs provide rural communities within their respective States technical assistance as well as information on available State funding opportunities and other resources ( 20