EVALUATION OF THE ONTARIO COMMON ASSESSMENT OF NEED (OCAN) IN ABORIGINAL MENTAL HEALTH PROGRAMS

Transcription

1 EVALUATION OF THE ONTARIO COMMON ASSESSMENT OF NEED (OCAN) IN ABORIGINAL MENTAL HEALTH PROGRAMS PROLOGUE BY THE CMH CAP TEAM As part of gathering learnings about implementing OCAN within the NE LHIN, the project engaged with independent researchers to conduct an evaluation of how best to implement OCAN within the Aboriginal population. The evaluation involved a survey, focus groups and interviews with staff, service providers and clients at James Bay General Hospital Community Mental Health Program, Noojmowin Teg Health Centre and Mamaweswen, the North Shore Tribal Council who were participants in the NE LHIN OCAN implementation pilot. The focus was mainly on learning how well OCAN fits Aboriginal mental health programs, identifying implementation best practices, determining the needed cultural supports and the barriers and challenges experienced by those who did not use OCAN. The following report cites that the concept of a common assessment tool was supported by participating providers. However, a recommendation was made to devise cultural supports to better reflect the Aboriginal individual s journey and community realities in order to improve uptake and implementation of OCAN. CMH CAP is currently in the process of investigating how this could best be accomplished.

2 EVALUATION OF THE ONTARIO COMMON ASSESSMENT OF NEED (OCAN) IN ABORIGINAL MENTAL HEALTH PROGRAMS March 2010 Evaluation Report of the OCAN Implementation in Aboriginal Mental Health Programs in the Ontario North East Local Health Integration Network Submitted to: Canadian Mental Health Association, Ontario Division on behalf of Community Care Information Management Submitted by: Mariette Sutherland, B.Eng. Whitefish River First Nation Birch Island, Ontario, P0P 1A0 and Marion Maar, PhD Faculty of Medicine, Human Sciences Division Northern Ontario School of Medicine, Sudbury, Ontario

3 Table of Contents INTRODUCTION...5 BACKGROUND...6 THE COMMUNITY CARE INFORMATION MANAGEMENT (CCIM) PROGRAM...6 COMMUNITY MENTAL HEALTH AND ADDICTIONS (CMH&A)...7 ONTARIO COMMON ASSESSMENT PROJECT...8 EVALUATION OF THE OCAN PILOT IN MAINSTREAM MENTAL HEALTH PROGRAMS...9 EVALUATION OF THE OCAN PILOT IN ABORIGINAL MENTAL HEALTH PROGRAMS...9 The Evaluation Team...9 PURPOSE OF THE EVALUATION OF THE OCAN IMPLEMENTATION IN ABORIGINAL MENTAL HEALTH PROGRAMS...10 OBJECTIVES OF THE EVALUATION...10 OUR APPROACH TO THE EVALUATION...10 EVALUATION QUESTIONS...13 EVALUATION METHODOLOGY...16 OVERVIEW OF METHODS...16 TIMELINES AND PARTICIPANTS AT EACH PARTICIPATING SITE...17 OVERVIEW OF PRESENTATION OF FINDINGS...18 RESULTS FOR THE JAMES BAY COAST SITES...19 DESCRIPTION OF THE JAMES BAY COAST COMMUNITIES...20 THE SERVICE CONTEXT: THE JAMES BAY COMMUNITY MENTAL HEALTH PROGRAM...21 OVERVIEW OF THE OCAN IMPLEMENTATION AT THE JAMES BAY COAST...22 Providers Understanding of OCAN...22 Perceived Adequacy of OCAN Staff Training and Support...23 Current status of OCAN implementation...23 OCAN S FIT WITHIN THE SERVICE DELIVERY MODEL AT JBCMHS...24 Perceived importance of a standardized assessment tool...24 The realities of implementing OCAN into the daily workflow...24 Fit of the automated OCAN tool...25 Fit of OCAN with Aboriginal client population and presenting concerns...25 Fit of OCAN within Cree Language and Culture...26 RESULTS FOR NORTH CENTRAL ONTARIO REGION SITES...27 DESCRIPTION OF NORTH SHORE AND MANITOULIN COMMUNITIES...27 SERVICE CONTEXT: N'MNINOEYAA ABORIGINAL MENTAL HEALTH SERVICES...28 OVERVIEW OF THE OCAN IMPLEMENTATION AT N'MNINOEYAA...28 Providers Understanding of OCAN...28 Perceived Adequacy of OCAN Staff Training and Support...29 Current status of OCAN implementation...29 OCAN S FIT WITHIN THE SERVICE DELIVERY ON THE NORTH SHORE...29 Perceived importance of a standardized assessment tool...29 The realities of implementation of OCAN into the daily workflow...30 Cross jurisdictional issues...31 Sutherland & Maar - March 2010 Page 2

4 Fit of the automated OCAN tool...31 Fit of OCAN with Aboriginal client population and presenting concerns...32 Fit of OCAN with Aboriginal Culture...33 SERVICE CONTEXT: NOOJMOWIN TEG HEALTH ACCESS CENTRE...34 OVERVIEW OF THE OCAN IMPLEMENTATION AT NOOJMOWIN TEG...34 Providers Understanding of OCAN...34 Perceived Adequacy of OCAN Staff Training and Support...35 Current status of OCAN implementation...35 OCAN S FIT WITHIN THE SERVICE DELIVERY ON MANITOULIN ISLAND...35 Perceived importance of a standardized assessment tool...35 The realities of implementation OCAN into the daily workflow...35 Cross jurisdictional issues...36 Fit of the automated OCAN tool...36 Overall workers stated that piloting the OCAN has lead to valuable networking with the NE LHIN and with other Aboriginal mental health services programs...37 Fit of OCAN with Aboriginal client population and presenting concerns...37 Fit of OCAN with Aboriginal Culture...37 IMPLICATIONS FOR OCAN IN ABORIGINAL MENTAL HEALTH SERVICES...38 IMPLICATIONS FOR OCAN IN ABORIGINAL MENTAL HEALTH SERVICES...38 COMPARISONS AND CONTRASTS REGARDING OCAN EVALUATION FINDINGS IN THE DIFFERENT ABORIGINAL CULTURES AND SERVICE CONTEXT...38 Contrasts:...38 CULTURAL INCONGRUENCE RELATED TO SPECIFIC DOMAINS...39 Domain 1: Accommodation...39 Domain 2: Food...39 Domain 17: Sexual Expression...39 Domain 21: Phone...39 Domain 22: Transport...40 Domain 23: Money...40 General comment on open ended questions...40 Open ended Question on Hopes and Dreams...40 Open ended Question on Spirituality...40 MISSING ELEMENTS...40 Abuse history...40 Experience of discrimination and/or racism...41 Education...41 SUGGESTIONS FOR FUTURE IMPLEMENTATION OF OCAN IN ABORIGINAL HEALTH SERVICES CONTEXT...42 OPPORTUNITIES FOR FUTURE SUPPORT OF OCAN IMPLEMENTATION IN ABORIGINAL MENTAL HEALTH SERVICES Readiness for the Implementation of a Common Assessment Tool Training Client population Reliability, validity and cultural safety of the current version of the OCAN in Aboriginal mental health services Automation...45 IMPORTANT CONTEXT OUTSIDE OF OCAN SYSTEM IMPEDIMENTS Impact on clinical and administrative processes Policy implications...47 Sutherland & Maar - March 2010 Page 3

5 APPENDIX A: FOCUS GROUP AND INTERVIEW QUESTIONS FOR PROVIDERS/STAFF...49 [PROVIDER PERSPECTIVES]...49 [FUTURE IMPLEMENTATION OF THE OCAN]...50 APPENDIX B: SURVEY TOOL FOR PROVIDERS/STAFF...51 APPENDIX C: INTERVIEW QUESTIONS FOR CONSUMERS...55 [CLIENT PERSPECTIVES]...55 [FUTURE IMPLEMENTATION OF THE OCAN]...56 APPENDIX D: LETTERS OF INFORMATION AND CONSENT FOR PARTICIPANTS...57 APPENDIX E: SURVEY RESULTS...61 Sutherland & Maar - March 2010 Page 4

6 OCAN in Aboriginal Mental Health Services A Report on the Pilot Implementation of OCAN in Aboriginal Mental Health Programs in the Ontario North East Local Health Integration Network INTRODUCTION In 2008, the North East Local Health Integrated Network (NE LHIN) became the first LHIN to initiate a LHINwide implementation of an automated common assessment tool, the Ontario Common Assessment of Need (OCAN) to all its Community Mental Health (CMH) Programs. This implementation pilot of OCAN plays a key role in shaping the provincial framework for future use of the OCAN. Reflecting Ontario s diversity in the NE LHIN, the OCAN will see use with Aboriginal, Anglophone and Francophone consumer populations and their service providers. In this document we report on the experience of the Aboriginal mental health services stakeholders with respect to the OCAN implementation in diverse regions of Northern Eastern Ontario. This evaluation of the pilot implementation of the OCAN in Aboriginal mental health programs was conducted during the winter of 2009/2010. The evaluation research was conducted by Mariette Sutherland (Consultant, Whitefish River First Nation) and Dr. Marion Maar (Faculty, Northern Ontario School of Medicine). Over the past 10 years, these researchers have collaborated on numerous mental health research projects within the Aboriginal communities and organizations, including First Nation health service organizations based in Northern Ontario, as well as provincial Aboriginal territorial organizations (PTOs) and provincial and federal government departments. This OCAN evaluation report of Aboriginal mental health programs is detailed in the following sections: Background to the OCAN in the NE LHIN and previous OCAN evaluation research Purpose of this evaluation & evaluation questions Methods and timelines of the evaluation Description and results for Ontario regions in the Far North and North Central area Discussion of results and implications for OCAN implementation in Aboriginal mental health services Recommendations for OCAN implementation in Aboriginal mental health services Appendices of research instruments Sutherland & Maar - March 2010 Page 5

7 BACKGROUND In this section we provide background on (1) Ontario s relevant health information management programs, (2) community mental health programs in Ontario, (3) the Ontario Common Assessment of Need (OCAN) and (4) previous evaluation research of the OCAN implementation in mainstream services. The Community Care Information Management (CCIM) Program 1 Since 2005, CCIM has played a critical role in improving the health care system for providers, residents, and consumers across the Long-Term Care, Community Mental Health & Addictions (CMH&A), Community Care Access Centres (CCACs), Community Support Services sectors and, more recently, Small & Chronic Care Hospitals and Community Health Centres. The implementation of common assessment tools and business systems solutions along with two integration initiatives enables access to standardized information for evidence-based care planning, operational improvements, and funding decisions that directly support the delivery of better health care to clients at home and in our communities. The RAI-MDS 2.0 common assessment tool in Long-term Care Homes and the OCAN tool in Community Mental Health agencies assist consumer-led decision-making at an individual level and make sharing information easy and timely across the sector. The purpose is to implement standard assessment practices that improve business processes and client care management by identifying needs, helping match needs to existing services and identifying service gaps. Also within the CCIM Common Assessment stream, an Integrated Assessment solution is being created, which will allow health care professionals to see community-based assessment data to help inform the treatment planning for people in their care. Plans are being made to begin a pilot in two or three volunteer LHINs this summer, prior to a provincial rollout. Two business systems solutions are being rolled-out in the CMH&A, Community Support Services, and Small & Chronic Care Hospitals sectors. A financial and statistical software solution (MIS) is currently being implemented in organizations to create consistent, reliable internal processes and systems and to meet external Ontario Healthcare Reporting Standards (OHRS) reporting requirements. This solution is also in the initiation stage in the Community Health Centres sector. The human resources and payroll software solution will make it easier for organizations to track the valuable services provided to clients and enable organizations to automate and streamline their payroll and human resources processes for improved efficiency, accountability and reporting. Both systems will make standardized reporting possible at the local level for Health Service Providers, Local Health Integration Networks (LHINs) and the Ministry. The Integrated Data Strategy is an overall initiative that builds on the past and present work being done in the Business Systems and Common Assessment project streams regarding the implementation of standardized data, tools, and processes across the community care sectors. Currently, information is being reported individually by these streams to reflect the sectors clinical, financial and HR information. The Integrated Data Strategy aims to combine this information to more clearly demonstrate the relationship between needs, services, outcomes, and cost. 1 Information provided in the sections (i.e. CCIM, CMH&A and OCAN) are adapted from materials provided by the Ministry of Health and Long Term Care. Sutherland & Maar - March 2010 Page 6

8 All this is built on a foundation delivered and supported by the Solutions Group with its extensive expertise and focus on continuous development and service delivery through its core competency areas Architecture & Integration; Standards; Security, Privacy and Risk Management; and Transition. A key success factor for CCIM is that every project team and initiative has the appropriate level of business and technical expertise to facilitate effective project planning and implementation. Community Mental Health and Addictions (CMH&A) The community mental health sector is a diverse sector providing a range of community mental health services by over 300 not-for-profit organizations of various sizes and in a variety of settings in Ontario. CCIM supports the CMH&A with tools to improve business processes and enhance the management of client care. CCIM is working with sector leaders to pilot and implement three projects that will help achieve this goal. 1. CMH CAP is implementing a streamlined assessment process that will standardize current practices across the province. OCAN is a standardized, consumer-led decision-making tool that allows key information to be electronically gathered in a secure and efficient manner. Since the initial consultation with the CMH sector in December 2007, the project continues to work province-wide with sector stakeholders through information exchanges and consultations, steering committees and working groups to promote best practices across the sector. Most importantly, consumers are playing a critical role in the constant development and improvement of common assessment practices. Lessons learned from implementation pilots as well as research formed the basis for the CCIM CMH CAP Steering Committee s recommendations to the Ministry of Health and Long- Term Care for province-wide implementation. 2. The Human Resources Information Systems CMH&A project will provide a Payroll and HR software solution that is integrated with the current MIS/OHRS-compliant financial solution offered by the CMH&A MIS project. This information is needed by the LHINs, the Ministry and by service providers to have better indicators for evidence-based resource planning. The project aligns with multiple LHIN priorities. 3. The Community Mental Health and Addictions Management Information Systems Project (CMH&A MIS) has standardized reporting within the CMH&A sector, providing integrated long-term financial and statistical information that is accurate and consistent across the mental health spectrum. The project is in Phase 3, which will introduce a financial and statistical management software solution to ensure there are reliable processes and systems in place within organizations for full MIS integration and ongoing reporting. These three complementary CCIM initiatives are building an improved, comprehensive solution for the sector by producing the following benefits: Client-focused, more individualized care delivery by streamlining intake processes and removing duplicate administrative processes Sutherland & Maar - March 2010 Page 7

9 Improved quality of information for planning and decision-making Standardized, consistent and reliable data and technical and business processes Enhanced information aids benchmarking, policy development and sector planning. Ontario Common Assessment Project Sponsored by CCIM and supported by the MOHLTC, the goal of CMH CAP is to streamline the assessment process by standardizing current practices in the form of an automated common assessment tool for use by community mental health providers. The project has finalized the automated common assessment tool, now called Ontario Common Assessment of Need (OCAN). OCAN is comprised of the Camberwell Assessment of Need (CAN-C) and additional data elements. This tool allows key information to be gathered quickly from both people seeking service and providers while ensuring consistent assessment practices across the province. OCAN provides consumers with an active role in their service planning, while supporting community mental health programs to share and re-use consumer-consented information in a way that provides structure and focus for comprehensive assessments. OCAN is designed to enable consumer-led decision making at the individual level of service; reduce repetitive information gathering; standardize, streamline and unify assessments; and provide an aggregate view of the mental health sector to support informed health care planning and decisions. In the fall of 2008, the North East Local Health Integrated Network (NE LHIN) expressed interest in being the first LHIN to implement OCAN to all its Community Mental Health (CMH) Programs. The NE LHIN implementation approach is a LHIN-wide implementation pilot of OCAN, and is playing a key role in shaping the provincial framework for future LHIN-wide use. This pilot implementation offers a landscape for developing a strategy that reflects Ontario s diversity. Throughout this pilot, OCAN will see use with Aboriginal and Francophone consumer populations in the NE LHIN and their service providers. Stage One - piloting of OCAN was completed in 2008 and involved 16 community mental health organizations throughout Ontario. A number of reports summarizing the lessons and outcomes of that work are available for review. The project is currently in Stage Two, testing LHIN-wide implementation of OCAN in community mental health programs in Northeastern Ontario. A number of Aboriginal organizations and/or Aboriginal mental health programs are involved in this pilot implementation. Sutherland & Maar - March 2010 Page 8

10 Evaluation of the OCAN Pilot in mainstream mental health programs In 2008, the Community Mental Health Common Assessment Project CMH CAP commissioned Caislyn Consulting Inc. to conduct an independent evaluation study of the OCAN pilot in sixteen participating mental health programs to examine the following: The impact of the CMHCA approach on people seeking services and community mental health providers Emerging best practices regarding use of OCAN Feedback on the prescribed methods of the OCAN process 2 The comprehensive report documented and described a number of key learnings about how the OCAN is being implemented in 16 selected pilot sites and suggested areas for improvement that could be further explored. Evaluation of the OCAN Pilot in Aboriginal mental health programs The CMH CAP released a request for proposals (RFP) in the fall of 2009 to evaluate the implementation of OCAN in Aboriginal mental health programs in selected Aboriginal mental health programs in the Ontario North East Local Health Integration Network. The Evaluation Team The CMH CAP commissioned the authors of this report (Mariette Sutherland and Marion Maar) to conduct this evaluation. The results of the evaluation and related recommendations are provided in this document. Mariette Sutherland is a health services research consultant from Whitefish River First Nation. Mariette s work is focused on the planning and evaluation of health services for Aboriginal people in Ontario. Dr. Marion Maar is a fulltime faculty member at the Northern Ontario School of Medicine in Sudbury. Marion s research focuses on Aboriginal health issues, including chronic care, mental health and e-health applications Kate Pautler. Common Assessment Evaluation Study Final Report. Caislyn Consulting Inc. Sutherland & Maar - March 2010 Page 9

11 PURPOSE OF THE EVALUATION OF THE OCAN IMPLEMENTATION IN ABORIGINAL MENTAL HEALTH PROGRAMS Objectives of the evaluation The goal of the evaluation was to undertake an evaluation study examining the processes and content of the OCAN as it relates specifically to the Aboriginal populations in the Ontario NE LHIN CMH CAP pilot project. CMH CAP developed the following OCAN evaluation objectives: Analyzing the degree of fit of the OCAN tool for completing assessments by staff and consumers within Aboriginal mental health programs. Using a variety of data collection methods, giving due consideration to the administrative burden on Community Mental Health organizations, clinicians and consumers, as well as the evaluation questions, timeframe, and budget. Determining what best practices were used during the process of completing OCAN. Collecting and using existing and proven or best practice tools. The Selected Proponent shall work with the project to propose, discuss and advise on the merits of validated tools for addressing quality, access, acceptability and cost. Evaluating what cultural supports need to be in place for both staff and consumers in completing the tool and incorporating OCAN processes in Aboriginal mental health programs. Examining barriers and challenges for using OCAN in Aboriginal programs that did not elect to participate in the NE LHIN pilot. Our Approach to the Evaluation Our approach to the evaluation was collaborative with the CMH CAP team. Furthermore, we used a participatory evaluation framework with Aboriginal stakeholders and participants. We utilized a variety of data collection methods, to document the experience of Aboriginal Mental Health organizations, clinicians and consumers. We were guided by the following conceptual framework: Building on what has been done already: The evaluation plan developed for Aboriginal mental health programs was designed to build on the previously completed evaluation of OCAN in mainstream mental health programs conducted by Caislyn Consulting in The framework used in that evaluation incorporated many aspects that are also relevant within the Aboriginal context. Therefore, similar - or at times even identical - evaluation questions were used whenever possible. Sutherland & Maar - March 2010 Page 10

12 This approach allowed for clear comparisons of the implementation of OCAN in mainstream and Aboriginal programs on several levels. Identifying unique aspects of Aboriginal services: While some aspects of the implementation of OCAN apply to Aboriginal programs as well as mainstream programs, there are other aspects that are unique to the Aboriginal service context. We therefore developed a research plan that included a review and examination of differences in the implementation of OCAN in Aboriginal mental health programs that may be a result of the diversity of Aboriginal peoples, cultures, service environments and geography. In addition, we examined potential differences in service provision in First Nations communities, funding and jurisdictional issues, cultural safety in mental health as well as historic issues. Cultural Safety: Cultural awareness, competence, sensitivity and safety have different meanings that their use is not always consistent in the academic literature. The concept of cultural safety originates with the Maori People of New Zealand, and embodies a concept that goes beyond providers learning about cultural differences. It describes an approach to service provision that incorporates provider self-reflection and understanding of power differentials that often exist between Aboriginal clients or patients and health services providers. Therefore it is vital that in culturally safe services, it is the client who defines "safe services 3 4. Of great importance to many Aboriginal people in the North Eastern Ontario is service provider acceptance of beliefs, religions, backgrounds, and history, and a focus on building on the strengths of Aboriginal people 5. Respect for Aboriginal perspectives on research: Aboriginal people often feel that they have been over researched and are thus frequently critical of government or university interests in conducting research within their communities. Research initiated by outsiders may raise concerns, particularly when the topic of research is sensitive such as mental health research. Researchers who work in Aboriginal communities often come in contact with communities and individuals who have participated in research projects and have felt harmed or violated, despite the fact that research protocols received ethics approval from a university or hospital-based research ethics board (REB). Aboriginal people commonly see the lack of collaboration and community focus in research design and implementation as an ethical issue that must be resolved. Our evaluation team was committed to a research approach, informed by and respectful of cultural values of the Aboriginal people and communities who participated in this evaluation. We strived for transparency in the research process and sought feedback on evaluation instruments and methods from each participating organization. For example, we encouraged review and feedback on interview questions for clients from representatives from the Aboriginal community and partners 3 Ramsden I: Cultural safety. N Z Nurs J 1990, 83(11): Varcoe C, McCormick J: Racing around the classroom margins: Race, racism and teaching nursing. In Teaching nursing: Developing a student-centred learning environment. Edited by: Young L, Paterson B. Philadelphia: Lippincott, Williams & Williams; 2006: Maar, M.., B. Erskine, L. McGregor, M. Sutherland, D. Graham, T. Larose, M. Shawande, and T. Gordon. Innovations on a shoestring: A Study of a Collaborative Community-based Aboriginal Mental Health Service Model in Rural Canada. International Journal of Mental Health Systems 3:27. Accessible at Sutherland & Maar - March 2010 Page 11

13 involved in this evaluation prior to implementation. We also respected all advice regarding the recruitment of participants for this project. In particular, issue surrounding the recruitment of clients (consumers) was discussed with providers at each site. Clients perspectives were only included in the evaluation if providers felt it was appropriate for outside evaluators to interview clients. Cultural safety of clients was of utmost importance. Focus groups with clients were found to be inappropriate due to confidentiality issues related to this method in small communities. Interviews with clients were only conducted if approved and supported by local providers. We learned that in this service environment, many mental health services clients simply do no trust outsiders enough to volunteer for an open discussion of their experience with the OCAN. In those cases where client interviews were deemed appropriate, local mental health workers explained the process prior to the interview to their clients; and often participated in the interviews along with their clients as well as participated in follow up debriefing discussions with the evaluation team interviewers. Follow up debriefings were also offered to clients.. Sutherland & Maar - March 2010 Page 12

14 EVALUATION QUESTIONS Generally speaking, a process evaluation examines if a program or process is implemented and running as envisioned when it was designed. This evaluation went beyond the normal parameters of a process evaluation to also elucidate cultural and systems challenges and facilitators associated with OCAN implementation in the Ontario Aboriginal mental health service systems. Therefore the evaluation was designed to answer questions related to OCAN that focus on the fit of the design of the OCAN for Aboriginal services, providers and clients, as well as the relative success of the implementation of the OCAN in Aboriginal programs. In short, the overarching question of the evaluation was How well was the OCAN implemented in Aboriginal populations in the Ontario NE LHIN CMH CAP pilot project? Table 1 provides a detailed overview our evaluation approach including: 1. evaluation questions used in this project; 2. their relationship to evaluation questions posed in the evaluation of OCAN in mainstream services (as conducted by Caislyn Consulting Inc. in 2008); 3. additional questions posed to address the unique circumstances faced by Aboriginal mental health services; and 4. tools used to address each question. Sutherland & Maar - March 2010 Page 13

15 Table 1: Research domains, evaluation questions, methods and tools Research Domain Evaluation question from evaluation of mainstream (non-aboriginal) Services by Caislyn Additional evaluation questions/objectives specific to Aboriginal services and culture Methodology and tools to collect this information Service Context (not covered) What is the service context of the OCAN in Aboriginal programs? What kind of mental health services exist? What are the main mental health concerns observed? Were there any cross jurisdictional issues that affected implementation? Program reports (to be requested from participating projects Key informant interviews (using staff interview schedule in Appendix) Was the OCAN implemented by all workers and as planned? Is integration/lack of integration of federal and provincial services affecting the implementation of OCAN? Were there any other issues that affected implementation? Provider perspectives What did assessors value in the common assessment approach? How did the common assessment compare with existing assessment processes? Did assessors think they were well prepared to implement use of the tool? How was OCAN introduced to clients? Which domains were particularly useful? Which domains - if any - were problematic or culturally inappropriate? Were translation services required? Where there concerns regarding the collection of electronic mental health information? Focus groups using staff focus group guide (see Appendix) Survey (using Survey Monkey tool in Appendix) How satisfied are assessors with the CA tool? Client perspectives Did participants appreciate the self assessment approach? What supports did participants find effective to completing the self-assessment? What features of the CA tool did participants endorse? How satisfied were Which domains were particularly useful? Which domains - if any - were problematic or culturally inappropriate? Were translation services required? Where there concerns regarding the collection of electronic mental health information? Key informant interviews with clients (if seen as appropriate by project staff) Client interview schedule (see Appendix) Sutherland & Maar - March 2010 Page 14

16 Research Domain Evaluation question from evaluation of mainstream (non-aboriginal) Services by Caislyn Additional evaluation questions/objectives specific to Aboriginal services and culture Methodology and tools to collect this information participants with the process? Future of the OCAN process Is technology working in the common assessment process? What programs are best suited for implementation of the common assessment? What changes should be made to the CA tools and tool use? Are additional domains required/desirable for Aboriginal people? Are reports useful to program planning and reporting? Is integration/lack of integration of federal and provincial services affecting the implementation of OCAN? Focus groups, using staff focus group guide (see Appendix) Survey (using Survey Monkey tool in Appendix) Client interview schedule (see Appendix) What training and resources are essential to future implementation of the CA process? Sutherland & Maar - March 2010 Page 15

17 EVALUATION METHODOLOGY This evaluation was a process evaluation of the pilot implementation of OCAN in Aboriginal mental health services. The evaluation was designed to allow for comparisons with the OCAN evaluation in mainstream mental health programs where possible while focusing on the unique aspects of Aboriginal culture and experiences. Overview of Methods We used mixed methods in this evaluation, however our main focus was on the collection of in depth, qualitative data. This was done for the following reasons: 1. Research shows that questionnaires based research has only limited success with Aboriginal communities. 2. Little is known about the fit of OCAN within Aboriginal culture and services, therefore open ended qualitative research allows researchers to explore topics that are brought up by participants, but were not anticipated by researchers. 3. The unique cultural, service and geographic context of the OCAN implementation within Aboriginal mental health services is best captured and understood through the richness of qualitative research data. In summary, we conducted a review of local Aboriginal mental health program documents, interviews, focus groups and a short survey with providers/staff, as well as key informant interviews with clients as follows: 1. Mental health services document review At each site we reviewed service information, regional and local program descriptions and any other reports made available to the evaluation team by local staff, prior to or following the site visits. 2. Focus groups and interviews with service providers/staff Interviews and focus groups were audio recorded and thematically analyzed by the researchers. Summary reports for each site were sent back to participants to allow staff to verify the accuracy of the research teams findings. 3. Short survey with service providers/staff We also used a survey tool using Survey Monkey to collect survey data from providers/staff to obtain quantitative data on key features of the OCAN implementation. This tool was developed to allow direct comparisons to the OCAN evaluation conducted with mainstream service providers. We provided the option of internet based contributions and paper surveys to ensure remote sites had equal opportunity to contribute to all areas of the evaluation. 4. Interviews with clients Interviews with clients were also audio recorded. Clients experiences with the OCAN tool were thematically analyzed. Given that many Aboriginal communities have had poor experiences with research in the past, the researchers implemented a detailed informed consent process ( see Appendix D). All participants provided either written Sutherland & Maar - March 2010 Page 16

18 or verbal consent to participate in the evaluation after reviewing all sections in the letter of information (Appendix D). Timelines and Participants at each Participating Site Data was collected during site visits conducted during February and March A total of 26 providers/staff and 8 clients participated in the evaluation. Details for each site visit are provided in Table 2. Table 2: Timelines and Participants at each Participating Site Site Timelines for site visit Number of staff participants Number of consumer participants Interviews/focus groups survey James Bay Community Mental Health Program 34 Revlon Moosonee, ON P0L 1Y0 PH: FAX: Noojmowin Teg Health Centre Postal Bag 2002, Hwy Hillside Rd., Aundeck Omni Kaning Little Current, Ontario P0P 1K0Phone: (705) North Shore Tribal Council Gloria Daybutch Box 28, Cutler, ON P0P 1B0 (705) Feb 2-5, Feb 10, March 24, April 19, Sutherland & Maar - March 2010 Page 17

19 OVERVIEW OF PRESENTATION OF FINDINGS The evaluation of the implementation of OCAN in Aboriginal mental health services included culturally, geographically and politically diverse Aboriginal regions in Northern Eastern Ontario. It should not come as a surprise that while we found some common themes, we also found distinct differences between sites, mainly based on the degree of isolation, languages spoken by staff and clients and the state of development of mental health services. Lumping these diverse realities into a single results sections would belie the diverse contexts, experiences and service realities of participating Aboriginal mental health services staff and clients. Furthermore we believe that specific information on the influence of geography, culture, language and the Aboriginal mental health service context is valuable information to determine the future of OCAN in Aboriginal mental health services. It is important to note that Aboriginal communities in Canada have experienced profound disruptions to their traditional lifestyles over several generations. Many communities have experienced major epidemics, losses of languages and lifestyles and multi-generational abuse within the residential school systems. A large body of research shows that these historical events have resulted in higher rates of mental illness, family violence, suicides and addictions in Aboriginal communities when compared with the mainstream population 6. The impact of these events on the health and the determinants of health of Aboriginal populations has often been profound, however it has not necessarily affected communities in a uniform manner. We provide related background information relevant to the implementation of OCAN throughout this report. To avoid generalizations related to Aboriginal people and Aboriginal mental health issues, we divided the results section into 2 sections, representing the cultural and geographic areas where we conducted the evaluation. 1. West Bank of the James Bay Coast; 2. North Central Ontario, encompassing Manitoulin Island and the North Shore of Lake Huron; We paid particular attention to describe the geographic, cultural and service context in each of these areas to demonstrate the impact of these factors on the OCAN implementation. Following the results sections we discuss the findings and report on related implications for the implementation of OCAN in Aboriginal mental health services. Our discussion is focused on the following topics related to OCAN and its implementation: 1. Aspects of OCAN that are easily compatible with Aboriginal services, and therefore should be continued; 2. Aspects of OCAN that are not congruent with Aboriginal services and could be modified; 3. Aspects of OCAN that are difficult to reconcile with Aboriginal services and require significant alteration. Finally we list recommendations for the implementation of OCAN in Aboriginal mental health services. 6 For example see: Kirmayer LJ, Guthrie Valaskakis G., (Eds.). Healing Traditions: The Mental Health of Aboriginal Peoples in Canada. Vancouver: UBC Press; Brave Heart MY, De Bruyn LM. The American Indian Holocaust: Healing historical unresolved grief. American Indian and Alaska Native Mental Health Research. 1998;8(2): Sutherland & Maar - March 2010 Page 18

20 RESULTS FOR THE JAMES BAY COAST SITES The James Bay Community Mental Health Program (JBCMHP) on the James Bay Coast in the Far North of Ontario took part in this evaluation of OCAN. This program provides services to Aboriginal people who live on-reserve in five Cree communities along the western coast of James Bay as well as one off-reserve community: 1. Attawapiskat First Nation 2. Kashechewan First Nation 3. Fort Albany First Nation 4. Moose Cree First Nation (Moose Factory) 5. Peawanuck First Nation (Weenusk) 6. Moosonee (off reserve town) Figure 1: Map of Ontario with James Bay Communities 7 7 Atlas of Canada, accessible on the Natural Resources Canada website at Sutherland & Maar - March 2010 Page 19

21 Description of the James Bay Coast Communities There is no road access to the James Bay communities from the more southern towns of Ontario. However there is rail way service to Moosonee and there are air strips in all communities except Moose Factory. A ferry runs from Moose Factory to Moosonee in the summer and a helicopter service operates between these communities during freeze up and break up of the ice. Ice roads connect all the communities except Peawanuck starting in late winter 8. Compared to other inhabited areas in north eastern Ontario the climate is harsh, with long dry winters with temperatures often reaching -40 degrees Celsius and short summers. The communities frequently have to deal with the effects of severe flooding and at times evacuation of their communities during the breakup of the spring ice. Moosonee is located on the Moose River 12 miles south of James Bay. Approximately 3,500 people live in Moosonee with about 85% being Cree. The main language spoken in this town is English with Cree as a second language. The town is not connected to the road system in Ontario, however it is accessible by a 4-5 hour train ride from Cochrane, Ontario, the closest town accessible by train. Scheduled air service is provided by Air Creebec based in Timmins. In addition there are several small charter air companies based in Moosonee 9. During the winter months, ice roads lead to the communities of Attawapiskat, Kashechewan, Fort Albany, Moose Factory. Moose Factory is located near the mouth of the Moose River on an island approximately 13 miles long. The neighbouring community of Moosonee which is located on the mainland approximately 5 km from Moose Factory is accessible by boat in the summer and by ice road in the winter. The total population is about Kashechewan is located on the north side of the Albany River at a distance of about 130km from Moosonee. It is accessible by year-round air service and a winter ice road. The majority of the people speak Cree as a first language and Cree is spoken in most homes. Fort Albany is located on the south side of the Albany River at approximately 128 km from Moosone by air. It is accessible by air year-round air service and a winter ice road. The majority of the people speak Cree as a first language and Cree is spoken in most homes. Attawapiskat is located approximately 260 km north of Moosonee (20 minute flight), near the mouth of the Attawapiskat River. It has an on-reserve population of approximately 1500 people. Air Creebec has regularly scheduled flights to Attawapiskat. In addition there is a winter road connecting the community to Mossonee and more northern communities in the winter and a barge runs during the summer. The majority of the people speak Cree as a first language and Cree is spoken in most homes. Peawanuck is at a distance of about 520 Km from Moosonee and is the most northern community services by JBMHP and has about 300 residents. It is accessible by year-round air service and by barge during the short 8 Weeneebayko General Hospital website, accessible at 9 Moosonee website accessible at 10 Moose Cree website accessible at Sutherland & Maar - March 2010 Page 20

22 time of open water (September). Recently, a winter road to Gillam, Manitoba has been prepared; it is the world s longest winter road 11. The Service Context: the James Bay Community Mental Health Program 12 Alemotaeta, the James Bay Community Mental Health Program, is committed to establishing a healthier and happier environment for the people in each community along the James Bay Coast. The program provides services with a holistic approach, in a manner that recognizes the local cultures, heritages and traditions. Individuals, families and whole communities are offered ongoing support and developmental assistance to honestly face the aspects of their lives which stop them from being fully healthy, physically, emotionally, mentally and spiritually and to develop their strengths in living a more balanced life. The program networks with other community resources and agencies within the local communities and outside area. The program offers front-line services that are always respectful of the culture and language of the people. The service provides continuity of care for those people over the age 16 living in the James Bay area, particularly those struggling with depression, concurrent disorder, court support & diversion, follow-up and after care, grief, addiction to alcohol/drugs, effects of abuse, problems with relationships, low self-esteem, Moderate Mental Illness and Severe Mental Illness. The following service providers are available: Mental Health Counselor (available in all sites): To provide a variety of appropriate client-directed counseling services. Regional Team Leader: To provide counseling and professional support for program staff and community partners. To provide counseling and referral services for individuals, couples and families as appropriate. Regional Clinician Worker: To provide community based counseling services to persons with mental illness which are culturally relative, accessible, and effective in the client s home community as part of the program s clinical team. To provide joint clinical counseling with a Community Counselor in the coastal communities served by the James Bay Community Mental Health Program. Regional Concurrent Disorder Worker: To provide culturally appropriate assessment, crisis management, supportive counseling and referral services in their language of choice in home community for clients with SMI (Serious Mental Health Illness) and/or CD (Concurrent Disorders - a major mental illness in combination with addictions issues and substance misuse). Regional Court Worker/Case Manager: The primary goal of the program is to divert people with mental illness away from or out of the criminal justice system to more appropriate community mental health services. Regional Crisis Intervention/Early Episode Clinician: Provides comprehensive case management and crisis services to clients that are experiencing early psychosis. 11 Peawanuck website accessible at 12 Adapted from the James Bay Community Mental Health Services Program literature, prepared by James Bay Community Mental Health Services Program staff Sutherland & Maar - March 2010 Page 21

23 Addictions Worker: The program provides intake, assessment, referral, community treatment, group intervention, public education and community awareness to adults, youth, family members and Problem Gamblers. The head office is located in Moosonee and regional specialized workers fly into the coastal communities to provide outresearch services; workers also provide sessions to clients over the phone. Due to the geographical barriers, workers see their clients only rarely in person (often 1-3 times per year) and some workers have never met some of their clients in person. Larger coastal communities have, in addition to visiting workers, also a full time worker in the community. Telehealth/telepsychiatry facilities are only accessible in Moosonee and Moosefactory, there is no telemedicine within the coast communities. Internet access is unreliable with slow connections when it is working. Phone access to the communities normally works, however even this connection is at times quite poor. The services are geared to address the broad range of need of the client population, including: family issues, crisis, court referrals for anger management, suicidal ideation, mood disorders, depression, trauma, Post Traumatic Stress Disorder and anxiety disorders. These presenting concerns are in most cases complicated by intergenerational trauma, that may include residential school trauma, physical and sexual abuse and other issues. Substance use is mostly confined to alcohol. Suicidal ideation is a major concern and suicide waves have occurred in some communities in the recent past. The current client case load for JBCMHS is approximately clients, however many more contact this team for brief mental health services and these clients often return for further brief services after a period of time. Common presenting concerns according to staff are: family issues, crisis, court referrals (e.g.,: for anger management) mood disorders, depression, anxiety disorder, trauma, complicated trauma, Post Traumatic Stress Disorder, suicidal ideation is a major issue as well as addictions to alcohol. During our site visit, local staff explained that need for services and travel demands are overwhelming for workers. Overview of the OCAN Implementation at the James Bay Coast In this section we report on successes and challenges of implementing OCAN in the JBCMHS program. Providers Understanding of OCAN Providers explained that their interest in participating in the OCAN pilot was rooted in their desire to be involved in important new initiatives that were implemented elsewhere in the province or region. They generally expressed a deep appreciation of the opportunity to provide feedback on an initiative that could potentially be rolled out in Aboriginal mental health services in the future. The James Bay service area is geographically quite isolated and providers felt that participating in the OCAN pilot might result in better connection and networking with other programs within the LHIN and the province as a whole. Providers clearly understood that the OCAN is a standardized tool that has been piloted in mental health programs in North Eastern Ontario, including some Aboriginal mental health services. They also understood that OCAN has not been used in Aboriginal services previously and that their feedback on the fit of OCAN with Aboriginal people and services was sought as part of the evaluation phase. Sutherland & Maar - March 2010 Page 22

24 Staff generally felt that the evaluation was perhaps scheduled too early to provide definitive feedback on the fit of OCAN within the daily business of local service system, and more importantly the fit with the culture and presenting concerns of Aboriginal clients in the James Bay area. One provider even stated it is too soon to tell if it is working, come back again next year! The research team explored the underlying reasons for this sentiment throughout the two days spent on site. However one theme emerged almost immediately: While providers were initially interested in OCAN because it is a standardized tool and used elsewhere in North Eastern Ontario, they also reported that they increasingly questioned if OCAN information from the James Bay coast clients can be read accurately by service providers elsewhere in the province to reflect the clients story. There were concerns that the OCAN may not accurately portray the needs of the individual client or client population. Essentially, providers were questioning the reliability and validity of the tool within the Aboriginal client population of the James Bay area. We will elaborate more on this sentiment in the following sections. On a more positive note, providers shared the hope that the OCAN would collect the necessary evidence to support program development. Staff generally felt that the JBC mental health program is servicing many clients with extremely high needs and without the necessary range of specialized mental health services and resources. Perceived Adequacy of OCAN Staff Training and Support Staff felt that the OCAN training they received was a good start, however they agreed that it felt rushed and that is was too short. Most importantly the client scenarios used during the training session were not seen as very helpful since they did not match the reality of the Cree client population on the James Bay coast. For example, staff therefore felt they were not sufficiently prepared during the training sessions to implement OCAN in a service environment with clients who are predominantly Cree speakers and culturally quite different from mainstream Canadians. Only some providers are fluent speakers of Cree and translation is frequently required to complete the OCAN. Providers experienced that the OCAN was very challenging to translate into Cree, since many of the concepts were based on Western culture or clinical mental health concepts that are difficult to translate into Cree. Frequently concepts used in OCAN and OCAN training do not exist in the Cree culture or language. Furthermore, some areas of the OCAN ask very direct and intrusive questions of the client, which is incongruent with Cree cultural norms and even violates cultural protocols. Providers felt that training session should include focused discussions on how to address these specific cultural issues, so that workers would not have to individually work this out through trial and error. Providers felt that the OCAN telephone support line was very helpful and allowed for quick access to more general questions related to OCAN use. The OCAN portal on the other hand has caused some difficulties. Providers report intermitted problems with (a) gaining access to the portal, (b) finding information and (c) downloading information. Current status of OCAN implementation Staff explained that most clients will see staff sporadically and often only during a crisis and it is therefore difficult to implement the OCAN with many of their clients. One provider explained: mental health management is a very hard concept to get across to most of our clients, as they are often struggling with many issues and are unable to focus on managing their mental health. After several attempts with various Sutherland & Maar - March 2010 Page 23