Integrated Dual Disorders Treatment Fidelity Scale

Transcription

1 IDDT Fidelity Scale Protocol (Revision of ) Page 1 Integrated Dual Disorders Treatment Fidelity Scale This document is intended to help guide you in administering the Integrated Dual Disorders Treatment (IDDT) Fidelity Scale. In this document you will find the following: 1) Introduction: The introduction gives an IDDT overview and a who/what/how of the scale. There is also a checklist of suggested activities for before, during, and after the fidelity assessment that should lead to the collection of higher quality data, more positive interactions with respondents, and a more efficient data collection process. 2) Protocol: The protocol explains how to rate each item. In particular, it provides: a) A definition and rationale for each fidelity item. These items have been derived from comprehensive, evidence-based literature. b) A list of data sources most appropriate for each fidelity item (e.g., chart review, program leader interview, team meeting observation). When appropriate, a set of probe questions is provided to help you elicit the critical information needed to score the fidelity item. These probe questions were specifically generated to help you collect information from respondents that is free from bias such as social desirability. c) Decision rules will facilitate the correct scoring of each item. As you collect information from various sources, these rules will help you determine the specific rating to give for each item. 3) Cover sheet: This is a record form for background information on the study site. The data are not used in determining fidelity, but to provide important information for classifying programs, such as size and duration of program, type of parent organization, and community characteristics. 4) Checklist for multiple sources: The checklist is to be used to assess if each of the multiple sources provides evidence for the presence of critical ingredients specified in each item. 5) Score sheet: The score sheet provides instructions for scoring, including how to handle missing data, and identifies cut-off scores for full, moderate, and inadequate implementation.

2 IDDT Fidelity Scale Protocol (Revision of ) Page 2 Integrated Dual Disorders Treatment Fidelity Scale: Introduction Substance abuse is a common and devastating disorder among persons with severe mental illness (SMI). Dual disorders (DD), which denotes the co-occurrence of substance use disorder and SMI, occur in about 50% of individuals with SMI (Regier et al., 1990) and is associated with a variety of negative outcomes, including higher rates of relapse, violence, hospitalization, homelessness, and incarceration (Drake et al., 2001). Integrated dual disorder treatment (IDDT) is an evidence-based practice that has been found to be effective in the recovery process for clients with DD. In IDDT, the same clinicians or teams of clinicians, working in one setting, provide mental health and substance abuse interventions in a coordinated fashion. As an evidence-based psychiatric rehabilitation practice, IDDT aims to help the client learn to manage both illnesses so that he/she can pursue meaningful life goals. The critical ingredients of IDDT include assertive outreach, motivational interventions, and a comprehensive, long-term, staged and individualized approach to recovery. Overview of the scale. The IDDT Fidelity Scale contains 13 program-specific items that have been developed to measure the adequacy of implementation of IDDT programs. Each item on the scale is rated on a 5-point rating scale ranging from 1 (Not implemented) to 5 (Fully implemented). The standards used for establishing the anchors for the fully implemented ratings were determined through a variety of expert sources as well as empirical research. What is rated. The scale is rated on current behavior and activities, not planned or intended behavior. For example, in order to get full credit for Item 3 (Access for IDDT Clients to Comprehensive DD Services), it is not enough that the agency is planning future changes in this area. Unit of analysis. The scale is appropriate for organizations that are serving clients with SMI and for assessing adherence to evidence-based practices at the agency/clinic level, rather than at the level of a specific clinician. However, separate ratings may be completed for a specialty team in addition to the agency/clinic level. How the rating is done. The fidelity assessment is done in person at the program site, following a prearranged schedule. The fidelity assessment requires a minimum of 4 hours to complete, although a longer period of assessment will offer more opportunity to collect information and hence should result in a more valid assessment. The data collection procedures include chart review, observation of team meeting or supervision, observation of one or more group or counseling sessions, and semi-structured interviews with the program leader, the medication prescriber(s), the clinicians providing the services, and clients. We recommend that interviews with clinicians be done in a group format (the same applies to interviews with clients). If the program has 5 or fewer DD clinicians, it is desirable to interview all of them. If the program has more than 5 DD clinicians, attempts should be made to interview at least 5 of them. In terms of clients targeted for IDDT, we recommend interviewing 3 clients, ideally individuals who have received IDDT for at least one year.

3 IDDT Fidelity Scale Protocol (Revision of ) Page 3 For some items that require chart review for rating, the fidelity assessment involves the examination of 10 charts of IDDT clients. The ideal is that charts are randomly selected. We suggest that you ask the program s contact person to select 20 charts prior to your site visit, and then randomly select and review 10 of those charts during your visit. Coding of many items requires both understanding on the part of clinicians and application of that understanding. If clinicians generally do not understand the concepts, then score as 1. If they understand parts of the concept and if they apply the understanding consistently, score as 3. To score 5, there needs to be consistent evidence that the concepts are applied consistently for 80% or more of clients, as documented across different sources of evidence. Who does the ratings. Fidelity assessments can be made by both external groups as well as by the organization implementing IDDT. Both types of assessment are recommended. We will focus on fidelity assessments made by independent assessors. Fidelity assessments should be administered by individuals who have experience and training in interviewing and data collection procedures (including chart reviews). In addition, raters need to have an understanding of the nature and critical ingredients of IDDT. We recommend that all fidelity assessments be conducted by at least two raters. Missing data: Missing data can occur for many reasons. One might be a failure on the assessor s part to collect the necessary information. This scale is designed to be fully completed, with no missing data on any items. Consequently, fidelity assessors should not leave any item uncoded because of insufficient information. Rather, the assessors should follow up with phone calls, s, or additional visits to ensure completeness of the assessment. It is critical that raters record detailed notes of responses given by the interviewees. Another reason that data might be missing is that the rating scale does not fit the organization s approach to services to this population. For example, the item of stage-wise treatment is rated on the basis of the percentage of clients receiving stage-wise services. However, if the clinicians in a program do not have an understanding of stage-wise interventions and therefore do not use this framework, then the proper scoring on this item is 1. It is not missing. We anticipate that many new programs will receive low fidelity ratings on many items for which the program has not yet formulated a policy. References Drake, R. E., Essock, S. M., Shaner, A., Carey, K. B., Minkoff, K., Kola, L., Lynde, D., Osher, F. C., Clark, R. E., & Rickards, L. (2001). Implementing dual diagnosis services for clients with severe mental illness. Psychiatric Services, 52(4), Regier, D. A., Farmer, M. E., Rae, D. S., Locke, B. Z., Keith, S. J., Judd, L. L., & Goodwin, F. K. (1990). Comorbidity of mental disorders with alcohol and other drug abuse. Results from the Epidemiologic Catchment Area (ECA) Study. Jama, 264(19),

4 IDDT Fidelity Scale Protocol (Revision of ) Page 4 Before the Fidelity Site Visit: IDDT Fidelity Assessor Checklist Review the sample cover sheet. This sheet is useful for organizing your fidelity assessment, identifying where the specific assessment was completed, along with general descriptive information about the site. You may need to tailor this sheet to your specific needs (e.g., unique data sources, purposes for the fidelity assessment). Create a timeline for the fidelity assessment. Fidelity assessments require careful coordination of efforts and good communication, particularly if there are multiple assessors. For instance, the timeline might include a note to make reminder calls to the program site to confirm interview dates and times. Establish a contact person at the program. You should have one key person who arranges your visit and communicates beforehand the purpose and scope of your assessment. Typically this will be the IDDT program director or coordinator. Exercise common courtesy in scheduling well in advance, respecting the competing time demands on clinicians, etc. Identify program staff with whom you will need to meet during your fidelity visit. Work with the program contact person to arrange a schedule of interviews for the day of your visit with case managers, substance abuse specialists, rehabilitation services providers (i.e., vocational staff, relevant PHP staff), therapists, psychiatrist or medication prescriber, etc. Again, scheduling your fidelity visit well in advance will more likely enable you to meet with all necessary staff members. Establish a shared understanding with the site being assessed. It is essential that the fidelity assessment team communicates to the programs the goals of the fidelity assessment. Assessors should also inform program staff about who will see the report, whether the program site will receive this information, and exactly what information will be provided. The most successful fidelity assessments are those in which there is a shared goal among the assessors and the program site to understand how the program is progressing according to evidence based principles. If administrators or line staff fear that they will lose funding or look bad if they don t score well, then the accuracy of the data may be compromised. Indicate what you will need from respondents during your fidelity visit. In addition to the purpose of the assessment, briefly describe what information you will need, who you will need to speak with, and how long each interview or visit will take to complete. The site visit is likely to go the most smoothly if the contact person could, where available, assemble the following information prior to your site visit: A copy of agency brochure A copy of IDDT Program Mission Statement Roster of IDDT staff (roles, FTEs) A copy of the substance use screening instrument used by the agency A copy of the standardized DD assessment instrument used by the program Total number of clients served by the agency Number of active clients receiving DD services

5 IDDT Fidelity Scale Protocol (Revision of ) Page 5 Number of clients served in the previous year Number of clients who dropped out of the program in the previous year Number of active clients receiving specific DD services (e.g., substance abuse counseling, DD group counseling, family interventions) Number of active clients receiving additional rehabilitation services from the agency Number of active clients who attend a self-help group in the community Weekly schedule for counseling services Clinician training curriculum and schedule List of process and/or outcome variables Quality assurance data Inform that you will need to observe at least one team meeting (or supervision meeting) and at least one group or counseling session during your visit. This is an important factor in determining when you should schedule your assessment visit to the program. Alert your contact person that you will need to sample 10 charts. It is preferable from a time efficiency standpoint that the charts be drawn beforehand, using a random selection procedure. Obviously, a program can falsify the system by hand picking charts and/or updating them right before the visit. If there is a shared understanding that the goal is to better understand how a program is implementing services, this is less likely to occur. During Your Fidelity Site Visit: Tailor terminology used in the interview to the site. For example, if the site uses the term consumer for client, use that term. If case managers are referred to as clinicians, use that terminology. Every agency has specific job titles for particular staff roles. By adopting the local terminology, the assessor will improve communication. During the interview, record all the important names and numbers (e.g., numbers of clinicians, active clients, clients served in the preceding year, etc.) If discrepancies between sources occur, query the program leader to get a better sense of the program s performance in a particular area. The most common discrepancy is likely to occur when the interview with program leader gives a more idealistic picture of the program s functioning than do the chart and observational data. For example, on Item 5 (Outreach), the clinicians may report that they often spend their time working in the community, while the chart review may show that client contact takes place largely in the office. To understand and resolve this discrepancy, the assessor may go back to the clinicians and say something like, Our chart review shows client contact is office-based the majority of the time. Since you had reported you often provided outreach services in the community, we wanted your help to understand the difference. Before you leave, check for missing data.

6 IDDT Fidelity Scale Protocol (Revision of ) Page 6 After Your Fidelity Site Visit: The same day of the site visit, both assessors should independently rate the fidelity scale. Within 24 hours the assessors should then compare their ratings and resolve any disagreements. Come up with a consensus rating for each item. Sometimes assessors have collected different data or have interpreted the response differently during the interview. Within a week of the fidelity assessment (ideally, the next day or two), the fidelity assessors should follow up with contact to the program leader to clarify any item for which there is a lack of consensus. This is also the time to follow up on any missing data. Tally the item scores and determine which level of implementation was achieved (See Score Sheet).

7 IDDT Fidelity Scale Protocol (Revision of ) Page 7 Overview: IDDT Fidelity Scale Protocol: Item Definitions and Scoring The IDDT fidelity assessment evaluates services provided to a targeted group of clients with DD and the clinicians who are responsible for their mental health and substance abuse treatment. The fidelity assessment focuses on whomever the program leader designated as the target population. (The organization may have a much larger number of clients who are candidates for the IDDT, but that is a question of penetration, not fidelity.) At the outset of the fidelity assessment, in fact even before the day of the fidelity visit, the fidelity assessors should make clear which clients are the IDDT clients and which staff are designated as IDDT staff. For a new program that has not yet adopted IDDT, some of the questions will be unclear, because the program is not organized consistently with IDDT. If a program is hard to rate on an item because the philosophical assumptions differ from the premises of the model (e.g., they are not following a stagewise approach to treatment), the site will get a low rating on items related to these concepts, rather than a not applicable rating. 1a. Multidisciplinary Team Definition: All clients targeted for IDDT receive care from a multidisciplinary team. A multi-disciplinary team consists of, in addition to a DD clinician, two or more of the following: a physician, a nurse, a case manager, or providers of ancillary rehabilitation services described in Item 3. Rationale: Although a major focus of treatment is the elimination or reduction of substance abuse, this goal is more effectively met when other domains of functioning in which clients are typically impaired are also addressed. Competent IDDT programs coordinate all elements of treatment and rehabilitation to ensure that everyone is working toward the same goals in a collaborative manner. Sources of Information: a) Program leader interview Thinking about your IDDT clients, who provides their mental health case management? Describe these services. Do these clinicians have team meetings? How often? Who is present? Are nurses, residential staff, employment specialists, and substance abuse counselors involved in joint planning? What about the client s psychiatrist? How much contact do case managers have with other team members in a typical week? b) Clinician interview

8 IDDT Fidelity Scale Protocol (Revision of ) Page 8 Ask similar questions as asked of program leader, regarding clients on their caseload. c) Employment specialist and residential staff interview How often do you attend treatment team meetings with DD clients case mangers? Are you consulted regarding treatment decisions? Do case mangers help with housing/employment? d) Client interview Do you also receive employment [housing, family, illness management, or ACT/ICM] services from this agency? [If yes] Does your DD clinician have contact with your employment specialist [housing specialist, family counselor, case manager] regularly so that they are on the same page in helping you? Were there any other services you wanted, but were not available? Item Response Coding: First determine if the agency s mental health case managers, DD clinicians, and rehabilitation service providers, and other professional staff work together as a team, as manifested by regular contacts and collaborative treatment planning. If this is generally not true, for example, if the substance abuse counselor attends a treatment team meeting less than once every two weeks, then this item should be scored lower. If the treatment approach is mostly parallel or brokered (different clinicians working in different buildings or different parts of the same building but not meeting together on a regular basis), score this as 1. If the treatment approach is a mix between parallel and multidisciplinary (e.g., nurse and substance abuse counselor present at weekly treatment team meetings, but other key rehabilitation staff are not), score as 3. If the organization embraces a multidisciplinary approach, but it is inconsistently applied, then it may be more appropriate to determine the percentage of clients receiving multidisciplinary services, using team rosters as the primary data source, and determining whether the activities are documented in the charts. 1a. Multidisciplinary Team: Case managers, psychiatrist, nurses, residential staff, and vocational specialists work collaboratively on mental health treatment team % - 40% of clients 41% - 60% of clients 61% -79% of clients receive care from a receive care from a receive care from a multidisciplinary multidisciplinary team multidisciplinary team team < 20% of clients receive care from multidisciplinary team (i.e., most care follows a brokered CM or traditional outpatient approach) >80% of clients receive care from a fully multidisciplinary team with a strong emphasis on accessing a broad range of services and excellent communication between all disciplines

9 IDDT Fidelity Scale Protocol (Revision of ) Page 9 1b. Integrated Substance Abuse Specialist Definition: A substance abuse specialist who has at least 2 years of experience works collaboratively with the treatment team. The experience can be in a variety of settings, preferably working with clients with a dual disorder, but any substance abuse treatment experience will qualify for rating this item. Rationale: Having an experienced substance abuse specialist integrated into the treatment team is essential for ensuring a sustained focus on substance use. a) Program leader interview How often does the substance abuse counselor attend team meetings? How often does the substance abuse counselor have contact with the client s CM in a typical week? Is the substance abuse specialist considered a member of the team? How so? Do they carry a caseload? Are they involved in treatment planning for IDDT clients? Do you talk to him/her a lot? b) Clinician interview Ask similar questions as asked of program leader c) Substance abuse specialist interview Do you attend team meetings? How often? What is your role with regard to the CM/Treatment team? (If there s contact with the team, probe for whether a member, supervisor, consulting or any combination.) How many IDDT clients do you see? What is your role for them? (Probe for CM, assessment, treatment planning, groups, individual, etc.) d) Chart Review Check for Substance abuse specialist involvement in treatment planning Check for individual and group sessions conducted by the SA specialist for IDDT clients

10 IDDT Fidelity Scale Protocol (Revision of ) Page 10 1b. Integrated Substance Abuse Specialist: Substance abuse specialist works collaboratively with the treatment team, modeling IDDT skills and training other staff in IDDT No substance abuse specialist connected with agency IDDT clients are referred to a separate substance abuse department within the agency (e.g., referred to drug and alcohol staff) Substance abuse specialist serves as a consultant to treatment team; does not attend meetings; is not involved in treatment planning Substance abuse specialist is assigned to the team, but is not fully integrated; attends some meetings; may be involved in treatment planning but not systematically Substance abuse specialist is a fully integrated member of the treatment team; attends all team meetings; involved in treatment planning for IDDT clients; models IDDT skills and trains other staff in IDDT 2. Stage-Wise Interventions Definition: All interventions (including ancillary rehabilitation services) are consistent with and determined by the client's stage of treatment or recovery. The concept of stages of treatment (or stages of change) include: Engagement: Forming a trusting working alliance/relationship. Motivation: Helping the engaged client develop the motivation to participate in recoveryoriented interventions. Action: Helping the motivated client acquire skills and supports for managing illnesses and pursuing goals. Relapse Prevention: Helping clients in stable remission develop and use strategies for maintaining recovery. Rationale: Research suggests that modifications in maladaptive behavior occur most effectively when stages of treatment is taken into account. Sources of Information: a) Program leader interview What is the treatment model used to treat clients with substance abuse problems? Do you refer clients to AA? What about detox programs? How do you deal with clients who appear unwilling to change? (Probe for whether confrontation is used) Do you see the goal as abstinence? (Probe if this is a short- vs. long-term goal) How does your team view abstinence versus reduction of use? What kind of relapse prevention skills do you teach? Do you teach relapse prevention skills to clients who are actively using drugs/alcohol?

11 IDDT Fidelity Scale Protocol (Revision of ) Page 11 Has the organization ever offered training on stages of treatment [change]? b) Clinician interview Are you familiar with a stage-wise approach to substance use treatment? [if yes] What stages are defined in the approach your program uses? If the clinicians say they do use stage-wise model, ask them to go through caseload and identify the stage each client is in. Try to get an idea of what the clinician is trying to accomplish with each client (i.e., are they trying to get someone in the engagement stage to attend AA/NA or are they building rapport and providing support?). The goal is to identify how many active clients currently fit in each of the four stages. Items 7 and 10 will need these numbers! Note: Labeling of stages is not as critical as intention and actual practice. c) Team meeting/supervision observation Listen for discussion of interventions based on stages of treatment [change]. d) Observation of group or counseling sessions Listen for interventions based on stages of treatment [change]. e) Chart review (especially treatment plan) Examine 10 charts for documentation of stage-wise treatment. Count the number of charts for which treatment matches stage. Item Response Coding: Coding of this item requires both understanding on the part of clinicians and application of that understanding. If clinicians generally do not understand the concepts, then score as 1. If they understand parts of the concept (for example, if they differentiate between engagement and action), and if they apply the understanding consistently (e.g., different goals for clients in these two stages), score as 3. To score 5 on this item, there needs to be consistent evidence that the stage-wise concepts are applied consistently for 80% or more of clients, as documented across different sources of evidence. 2. Stage-Wise Interventions: Treatment consistent with each client s stage of recovery (engagement, motivation, action, relapse prevention) 20% of interventions are consistent with client s stage of recovery 21%- 40% of interventions are consistent 41%- 60% of interventions are consistent 61% - 79% of interventions are consistent >80% of interventions are consistent with client s stage of recovery

12 IDDT Fidelity Scale Protocol (Revision of ) Page Access for IDDT Clients to Comprehensive DD Services Definition: To address a range of needs of clients targeted for IDDT, agency offers the following five ancillary rehabilitation services (for a service to be considered available, it must both exist and be accessible within 2 months of referral by clients targeted for IDDT who need the service): Residential service: Supervised residential services that accept clients targeted for IDDT, including supported housing (i.e., outreach for housing purposes to clients living independently) and residential programs with on-site residential staff. Exclude short-term residential services (i.e., a month or less). Supported employment: Vocational program that stresses competitive employment in integrated community settings and provides ongoing support. IDDT clients who are not abstinent are not excluded. Family psychoeducation: A collaborative relationship between the treatment team and family (or significant others) that includes basic psychoeducation about SMI and its management, social support and empathy, interventions targeted to reducing tension and stress in the family as well as improving functioning in all family members. Illness management and recovery: Systematic provision of necessary knowledge and skills through psychoeducation, behavioral tailoring, coping skills training and a cognitive-behavioral approach, to help clients learn to manage their illness, find their own goals for recovery, and make informed decisions about their treatment. Assertive community treatment (ACT) or intensive case management (ICM): A multidisciplinary team (client-to-clinician ratios of 15:1 or lower) with at least 50% of client contact occurring in the community and 24-hour access. Ancillary services are consistent with IDDT philosophy and stages of treatment/recovery. For example, housing program encompasses approaches for clients who are in engagement and motivation stages of recovery. Rationale: Individuals with DD have a wide range of needs, such as developing a capacity for independent living, obtaining employment or some other meaningful activity, improving the quality of their family and social relationships, and managing anxiety and other negative moods. Competent IDDT programs must be comprehensive because the recovery process occurs longitudinally in the context of making many life changes. Sources of Information: a) Program leader interview Does your agency provide residential [vocational, family psychoeducation, illness management and recovery, or ACT/ICM] services? [If yes] Probe for specifics of each service area, e.g., What kind of residential services? How long is your residential service? What do you mean by supported housing? Please describe the referral process to these services. What is the waiting period for clients targeted for IDDT to obtain these services after the referral is made?

13 IDDT Fidelity Scale Protocol (Revision of ) Page 13 Are clients targeted for IDDT eligible for these services? What are the admission criteria? Probe and listen for exclusion criteria (e.g., The state vocational rehabilitation agency won t let us take clients with DD into VR until they have been sober for 6 months). Request a copy of agency brochure, if available, and look for description of available rehabilitation services. b) Clinician interview Ask similar questions as for program leader. Then follow up by going through caseload and determine which services each IDDT client is currently receiving. Probe for reasons why client is not receiving a relevant service, e.g., supported employment. In order to document access to a service, a minimum requirement is at least one IDDT client must currently be receiving that service. c) Rehabilitation service provider interview Interview rehabilitation service provider, either by phone or in person, to confirm whether they accept clients who have drug/alcohol problem. Probe for the service provider s philosophy regarding DD clients. d) Chart review (especially treatment plan) Look for documentation of referrals made to the 5 services. Item Response Coding: Evaluate the availability of each of the services above. To count as available, the service must be offered by the organization AND clients with IDDT must have genuine access to the service if they need it. In order to document access to a service, a minimum requirement is at least one IDDT client must currently be receiving that service. If a service is not currently being used by any clients or so restricted that IDDT clients rarely receive it, then that service is counted as unavailable. If multiple sources confirm that all 5 services are available to clients targeted for IDDT, the item would be coded as a Access for IDDT Clients to Comprehensive DD Services Residential services Supported employment Family psychoeducation Illness management ACT or ICM Less than 2 services are provided by the service provider that IDDT clients can access 2 services are provided by the service provider and IDDT clients have genuine access to these services 3 services are provided by the service provider and IDDT clients have genuine access to these services 4 services are provided by the service provider and IDDT clients have genuine access to these services All 5 services are provided by the service provider and IDDT clients have genuine access to these services

14 IDDT Fidelity Scale Protocol (Revision of ) Page Time-Unlimited Services Definition: Clients with DD are treated on a long-term basis with intensity modified according to need and degree of recovery. The following services are available on a time-unlimited basis: Substance abuse counseling Residential service Supported employment Family psychoeducation Illness management and recovery ACT or ICM Notes: 1. Score this item for available services only. For example, if the site has residential services and ACT, but not the other services, then evaluate if these two services are time-unlimited or not. If both are time-unlimited, then the site receives full credit for this item, even though the other services are not available (which is rated on preceding item). This item refers to the program policy regarding time limits or graduation program initiated time limits. The next item refers to clients who are hard to engage or who drop out. Rationale: The evidence suggests that both disorders tend to be chronic and severe. A time-unlimited service that meets individual client s needs is believed to be the most effective strategy for this population. Sources of Information: a) Program leader interview Are there any time limits for the provision of DD treatment in your agency? [If yes] How long? How do you determine the duration of support clients receive?

15 IDDT Fidelity Scale Protocol (Revision of ) Page 15 Do you graduate clients from IDDT after they have completed a certain number of sessions or groups? Which of your DD treatment services are given on a time-unlimited basis? Are clients funded for a particular period of time, for example, to receive substance abuse or employment services? b) Clinician interview Ask the same questions as for program leader. Have you had anyone who graduated from IDDT in the last 6 months? [If yes] Please describe the circumstances. c) Employment specialists and residential program case manager interview Inquire whether these services are time-limited. d) Chart review (especially treatment plan) Examine length of time in services and reasons for termination. Item Response Coding: If 80% or more of DD treatment services that an agency does provide are provided on a long-term basis, the item would be coded as a 5. (If an agency does not provide a service at all, then this is coded under Item 3). 4. Time-Unlimited Services Substance abuse counseling Residential services Supported employment Family psychoeducation Illness management 20% of available services are provided on a time-unlimited basis (e.g., clients are closed out of most services after a defined period of time) 21%- 40% of available services are provided on a time-unlimited basis 41%- 60% of available services are provided on a timeunlimited basis 61%- 79% of available services are provided on a timeunlimited basis >80% of available services are provided on a time-unlimited basis with intensity modified according to each client s needs ACT or ICM 5. Outreach Definition: For all IDDT clients, but especially those in the engagement stage, the IDDT program provides assertive outreach, characterized by some combination of meetings and practical assistance (e.g., housing assistance, medical care, crisis management, legal aid, etc.) in their natural living environments as a means of developing trust and a working alliance. Other clients continue to receive outreach as needed.

16 IDDT Fidelity Scale Protocol (Revision of ) Page 16 Rationale: Many clients targeted for IDDT tend to drop out of treatment due to the chaos in their lives, low motivation, cognitive impairment, and hopelessness. Effective IDDT programs use assertive outreach to keep the clients engaged. Sources of Information: a) Program leader interview Do you have a policy about closing out people who don t show up for treatment? Often clients targeted for IDDT drop out of treatment. How do you engage or re-engage such clients? What kind of strategies do you use to develop a working alliance with your clients? How do you engage clients targeted for IDDT that are homeless? How does a client reach you in a time of crisis? Probe further to determine types/frequency of services provided outside the office. b) Clinician interview Ask similar questions as for program leader. Also ask about several clients who were hardest to engage and what the clinicians did. c) Client interview Have you ever received services/support from your DD clinician [employment specialist, housing specialist] outside of the office, e.g., in your home, in the park, or at work? [If yes] How often? Do you feel that he/she would come out to wherever you are to help you when you are in trouble and need help urgently? d) Chart review (especially treatment plan) Examine length of time in services and reasons for termination. Item Response Coding: If program demonstrates consistently well-thought-out strategies and uses street outreach whenever appropriate, code as 5.

17 IDDT Fidelity Scale Protocol (Revision of ) Page Outreach: Program demonstrates consistently well-thought-out strategies and uses outreach to community whenever appropriate: Housing assistance Medical care Crisis management Legal aid Program is passive in recruitment and reengagement; almost never uses outreach mechanisms. Program makes initial attempts to engage but generally focuses efforts on most motivated clients. Program attempts outreach mechanisms only as convenient. Program usually has plan for engagement and uses most of the outreach mechanisms that are available. Program demonstrates consistently wellthought-out strategies and uses outreach whenever appropriate. 6. Motivational Interventions Definition: All interactions with DD clients are based on motivational interviewing that includes: Expressing empathy Developing discrepancy between goals and continued use Avoiding argumentation Rolling with resistance Instilling self-efficacy and hope Rationale: Motivational interviewing involves helping the client identify his/her own goals and to recognize, through a systematic examination of the individual s ambivalence, that not managing one s illnesses interferes with attaining those goals. Research has demonstrated that clients targeted for IDDT who are unmotivated can be readily identified and effectively helped with motivational interventions. Sources of Information: a) Program leader interview Are you familiar with the concept of motivational interviewing [interventions]? [If yes] How do you understand the concept? Could you give us examples of motivational interventions? Has the agency ever offered training on motivational interventions? How do you instill self-confidence and hope in your clients? b) Clinician interview

18 IDDT Fidelity Scale Protocol (Revision of ) Page 18 Ask similar questions as for program leader. Also, go through a review of a couple of clients who might benefit from motivational strategies and query how the clinician would respond. c) Team meeting/supervision observation Listen for discussion of motivational interventions. d) Observation of group or counseling sessions Listen for discussion of motivational interventions. e) Chart review (especially treatment plan) Examining 10 charts, look for documentation of motivational interventions. f) Client interview Do you like the DD clinicians? Do you have a good relationship? Was there a time when it wasn t a good relationship? Do the DD clinicians help to identify your goals Do they help you focus on your goals? Are the DD clinicians good listeners? Do they do a good job in making you feel hopeful, capable, and confident? Do the DD clinicians keep you motivated to cut back/stay clean? How do they keep you motivated? Item Response Coding: Coding of this item requires both understanding on the part of clinicians and application of that understanding. If clinicians generally do not understand the concepts, then score as 1. If they understand parts of the concept, and if they apply the understanding consistently, score as 3. To score 5 on this item, there needs to be consistent evidence that the concepts are applied consistently for 80% or more of clients for whom motivational interventions are indicated, as documented across different sources of evidence.

19 IDDT Fidelity Scale Protocol (Revision of ) Page Motivational Interventions: Clinicians who treat IDDT clients use strategies such as: Express empathy Develop discrepancy between goals and continued use Avoid argumentation Roll with resistance Instill self-efficacy and hope Clinicians providing IDDT treatment do not understand motivational interventions and 20% of interactions with clients are based on motivational approaches Some clinicians providing IDDT treatment understand motivational interventions and 21%- 40% of interactions with clients are based on motivational approaches Most clinicians providing IDDT treatment understand motivational interventions and 41%- 60% of interactions with clients are based on motivational approaches All clinicians providing IDDT treatment understand motivational interventions and 61%- 79% of interactions with clients are based on motivational approaches All clinicians providing IDDT treatment understand motivational interventions and >80% of interactions with clients are based on motivational approaches 7. Substance Abuse Counseling Note: Rating this item requires that the caseload be previously assessed according to stage (in Item 2). If the program has not formally assessed their caseload, then estimate number of clients who are in these stages (after briefly defining). Definition: Clients who are in the action stage or relapse prevention stage receive substance abuse counseling aimed at: Teaching how to manage cues to use and consequences of use Teaching relapse prevention strategies Teaching drug and alcohol refusal skills Problem-solving skills training to avoid high-risk situations Challenging clients beliefs about substance use; and Coping skills and social skills training to deal with symptoms or negative mood states related to substance abuse (e.g., relaxation training, teaching sleep hygiene, cognitive-behavioral therapy for depression or anxiety, coping strategies for hallucinations) The counseling may take different forms and formats, such as individual, group (including 12-Step programs), or family therapy or a combination. Rationale: Once clients are motivated to manage their own illnesses, they need to develop skills and supports to control symptoms and to pursue an abstinent lifestyle. Effective IDDT programs provide some form of counseling that promotes cognitive-behavioral skills at this stage. Sources of Information:

20 IDDT Fidelity Scale Protocol (Revision of ) Page 20 a) Program leader interview Could you tell me about substance abuse counseling offered in your program? Do you offer individual [group, family] substance abuse counseling? How often? Please describe the program philosophy and strategies your clinicians use. Request a copy of the program s substance abuse counseling schedule and curriculum. b) Clinician interview What kind of skills do you teach in the individual [group, family] substance abuse counseling? Probe to confirm if each of the five areas listed above is addressed. Do all clients who are motivated receive some form of substance abuse counseling? [If no] Who do NOT receive substance abuse counseling? Probe if the clinicians take into account clients motivational stage when introducing substance abuse counseling. c) Chart review Look for documentation of motivational stage and substance abuse counseling. d) Observation of group or counseling sessions Listen for discussion of motivational stage. Observe techniques/topics during group and whether they are appropriate for group members stage of treatment. Item Response Coding: Coding of this item requires both understanding on the part of clinicians and application of that understanding. If clinicians generally do not understand the concepts, then score as 1. If they understand parts of the concept, and if they apply the understanding consistently, score as 3. To score 5 on this item, there needs to be consistent evidence that >80% of clients in action stage or relapse prevention stage receive substance abuse counseling, the item would be coded as a 5. Note: Rating this item requires that the caseload be previously assessed according to stage (in Item 2). If the program has not formally assessed their caseload, then estimate the number of clients who are in these stages (after briefly defining).